The Measurement Trap
A salesperson inks a deal for your company. That person drove that revenue gain. However, a whole sequence of events took place before the sale could even take place. Multiple people (engineers, designers, and the like) contributed to this final sale.
The contribution of each is inseparable from the contributions of the others. Removing any one of them changes what the others could have accomplished. To truly measure the salesperson’s contribution we would have to “re-run” the path to the sales event without that specific salesperson in the room and allow the company to re-optimize with that absence. However, we can never observe the alternative universe where one was absent and the rest remained identical.
At this point you might think there has to be a solution out there. Maybe the Shapley value? But this does not solve the problem. You will have a neat formula but the inputs to it (the values of all possible configurations of the group) are completely impossible to measure.
And once we start considering counterfactual worlds then the list of questions we might have about a contribution keeps growing: Could the salesperson have done more? Could a different salesperson have closed a bigger deal? The deal that was signed may contain hidden costs that only surface years later. Should we subtract those from the salesperson’s credit? When?
This is entanglement. Our ability to complete a task critically relies on everyone else in the system doing their thing. Once you realize how entangled we all are you will be able to see it everywhere.
The platform version
Consider YouTube. A premium subscriber watches a video. A pro rata payment is made to the creator based on minutes watched. But that video exists because the creator watched another creator’s video and was inspired to respond. That response video drives traffic back to the original. A third creator references both in a compilation. A viewer discovers all three through a recommendation algorithm that surfaced the compilation. Which creator generated the value?
YouTube’s payment system counts minutes. But the contribution of any single video to the ecosystem is entangled with every other video on the platform. The simple arithmetic masks a complexity that the arithmetic cannot represent.
Now consider platforms that want to incentivize remixing. A hip-hop producer uses a snare drum sample from a sound library. That sample appears on a track that goes viral. What is the value of the snare drum sample? Without it, the producer would have used a different sound. The track would still exist. It might have been better or worse. We cannot know because we cannot have both versions coexist in order to compare them.
However, we have new digital tools. We can measure anything with all of the data we are collecting!
The measurement trap in practice
So, our instinct, faced with this complexity, is to measure harder. Consider the simplest version: marketing attribution. I ran TV ads for soda during the summer. Sales went up. Was it the ads or the season? To find out I am going to use experiments to measure. I split ad exposure by geography: ads in some regions, no ads in others. But is there a comparable city to New York? The business says don’t cut ads in important markets. But important markets are precisely where the measurement matters most. So I run the experiment in less important places and hope the results transfer. But do they?
Now I want to know which ad creative worked best. So I need multiple versions, split across geographies, controlling for seasonality, local preferences, competitive dynamics. Each layer of measurement adds cost and complexity. The measurement apparatus becomes more expensive than the thing being measured. And at the end, I have a point estimate that applies to this creative, in this market, during this period. Next quarter everything changes.
There is a belief that a platform, with all of its data streams, can determine the value of each participant and each contribution. The belief is wrong. Things get worse when you start considering the effect of hidden forces that contribute to the path you are on.
Harry Nyquist and invisible contributions
Harry Nyquist worked at Bell Labs for decades. He made important contributions to communication theory. However, he also performed an invisible function: he was famous for asking the right questions. Researchers who spoke with Nyquist frequently left the conversation with a new direction, a reframed problem, a clarified intuition. Many of Bell Labs’ most important outputs passed through Nyquist’s conversations.
How much did Nyquist contribute to Bell Labs? There is no measurement that could capture it. He asked questions that changed how other people thought. The contributions were real and completely invisible to any attribution system.
Every organization has a Nyquist. Every platform has participants whose contributions are catalytic rather than direct. No central measurement system can identify them because their contributions exist in the interactions between other people’s measurable outputs.
The way out
If contributions are entangled and counterfactual worlds are unobservable, then centralized measurement of individual value is a dead end. The question is how to run a system without relying on these attribution computations. The answer is to give agency to the people who experience the value. If you design a system where these people can express their valuation through costly, credible signals (transferring standing from their own balance, staking resources on a collaborator, committing to a joint activity with escrow), then the measurement problem dissolves.
The real world does not have clean answers because contributions are entangled, contexts shift, and counterfactuals are unobservable. The solution is to design systems where the people inside the game reveal value through their own choices, at their own cost, in their own time.
Ansible Architecture designs incentive systems and interaction protocols for platforms and marketplaces.
