The Optimizers We Build

Humans do not optimize. They do not have computers in their brains that help with decision making. In fact, it's not clear how we do all the amazing stuff we do everyday. However, we do know given time and stakes, humans build systems that actually optimize. Delegated optimization is most of what engineering is.

Economics went the other way. For a century it modeled human choice as if the human were the optimizer, then built an entire estimation apparatus on top of that assumption. I have spent a lot of my life inside that apparatus. Here I describe why the homo oeconomicus framework is hard to justify about humans but perfectly designed to handle the agents the human builds.

The post I could not write

The arc was the following: modern machine learning is pattern recognition, transformer-shaped function approximation tuned to interpolate within the data it has seen. It will fail when we move to environments it has never observed, because it has no model of the mechanism generating the data, only the data. And then structural econometrics provides the solution: structural models estimate primitives that are stable across regimes and therefore robust to the Lucas critique. The key assumption that buys you this magical outcome is the rigid optimizing framework applied to human behavior. The methods I was trained on, vindicated by the limits of the methods everyone is excited about.

Here is the problem. We cannot say that humans optimize anything in particular. So every equilibrium model rests on an assumption we have no way to confirm. Worse, we can almost never run the experiment that would test it, the intervention that pushes behavior to a genuinely new equilibrium and checks whether our estimated primitives predicted the move. Absent that, a structural model is an internally consistent story we cannot falsify. I was going to sell robustness to a critique while quietly relying on an assumption at least as fragile as the one the critique attacks.

What actually changed

However, now decisions are increasingly made by synthetic agents that do in fact optimize. A reinforcement learning agent solving a Markov decision process is, unlike a person, actually computing a policy that maximizes a discounted objective.

The optimizing-agent assumption was always the weak joint in structural economics. When a firm delegates a decision to an RL agent, that assumption stops being an idealization of a human and becomes a literal description of the machine. The human encodes what she wants into a reward and delegates the optimization to something that will actually execute it.

Structural econometrics does not become valid because humans became optimizers. It becomes valid because the decision-makers did. As delegation spreads, the share of economically consequential behavior that is genuinely optimizing grows, and a toolkit built on that assumption finally finds the domain where the assumption is true rather than hoped for.

An old idea, made literal

In the early 1980s Margaret Bray asked whether rational expectations, the assumption that agents know the equilibrium mapping of the economy they inhabit, could be justified rather than assumed. Her answer: suppose agents do not know the mapping but estimate it, running least squares on the data the economy itself generates and acting on the forecast. The summary is that rational expectations is what you get if the agents are econometricians.

We now inhabit this world even more directly. The pricing agent is not metaphorically running an estimation loop, it is actually running one. Bray's econometrician-agent, imagined to rationalize equilibrium, is now deployed in production.

A concrete setting

Imagine I run a marketplace. Third parties sell on it and set their own prices, using whatever technology they like. As the owner, I want to know how participants will respond if I change the rules. This is the classic structural question. The difference is that here it has a real chance of capturing the true nature of the agents, because the agents are, by construction, the kind of thing my methods assume.

Take two firms that delegate pricing to agents. Each firm hands its agent a reward, some payoff it cares about, and lets the agent chase it. The agent sets a price, the market responds, and a single number comes back that the agent treats as its reward. Crucially, the agent does not know the demand function, and I, watching from outside, do not know how that reward is built. It might be profit. It might be profit net of some fee, or revenue weighted by inventory pressure, or something with fixed costs and capacity penalties baked in. I will not assume. All I will use is that the agent optimizes whatever it was handed. Prices are set in alternating moves, which captures the fact that firms' decision epochs need not line up. The agents learn by entropy-regularized Q-learning, a pretty popular choice.

Now what?

The owner can reverse-engineer something useful from the behavior. Watch the bots price against each other and you can measure directly the strategies they settle into. But measuring today's strategies is not enough to answer a design question, because the moment the rules change the bots simply re-run their learning and settle somewhere new. What the owner actually wants is to play that forward offline, to simulate the response before committing to the change.

To do that you need the one thing the observed strategies do not hand you: what the bots are chasing, the preferences behind the behavior. This is what the structural econometric method recovers. Structural estimation draws no line between modeling behavior and measuring it. You write down a model of choice with a handful of hidden parameters and invert them out of the observed data. Applied to people, that model was always a heroic assumption. Applied to these agents it is simply true, because explicit optimization is exactly what they are doing, so the hidden parameters are really there to be found. Recover them and you can simulate the re-learning under any new rule, because the preferences are stable even as the strategies churn.

There is one honest wrinkle, and it shapes everything that can be claimed. Sometimes the data do not pin the hidden parameters down uniquely: more than one setting explains the same behavior equally well. When that happens the answer is not to pick one and hope. It is to sweep, to run the counterfactual at every candidate setting and report the whole spread of outcomes. If the spread is tight, the unknown parameter did not matter and you have a number. If it is wide, you have a range, and the range itself is the honest answer.

The core question, answered honestly

The marketplace owner's question was: can I predict how participants respond when I change the rules? The answer is now specific rather than hopeful. For some changes, yes, exactly, with a number that does not depend on anything I cannot measure. For others, only within a band, and I can tell you the band and its endpoints. For a few, not without an assumption no amount of watching will ever justify. There is no dramatic moment of re-optimization to model, because the agents are always learning and simply drift to a new normal when the rules shift. I hold their preferences, so I can simulate that drift, and re-run the inference whenever I want to track them directly.

Humans still do not optimize. But we are handing the optimizing to things that do, tethered to what we want, and for those things the old assumptions are finally more than a convenient fiction. The apparatus I was trained on was not wrong about optimization. It was just early. It was waiting for optimizers to show up. What it still cannot do is turn behavior into a full set of primitives, because some of them are not in the behavior to be found. The advance is not that this limit disappears. It is that, for the first time, we can say precisely which questions it touches and which it does not, and sweep across the gap to bound the rest.

References

I make a lot of claims in the text. For the curious, here is a slightly more detailed discussion of the literature. The (hidden) claim this whole post rests on, that the reinforcement learning agent's choice rule and the econometrician's estimator are the same object, is not an analogy I am reaching for. The formal statement that Maximum Causal Entropy IRL from machine learning and the dynamic discrete choice estimators from economics (the Nested Fixed Point algorithm, the Conditional Choice Probability method, and the Nested Pseudo-Likelihood algorithm) all belong to a single class of optimization problems, sharing a common objective, policy, and gradient, is due to:

• Groeger, J., with N. Sanghvi, S. Usami, M. Sharma, and K. Kitani (2021). "Inverse Reinforcement Learning with Explicit Policy Estimates." Proceedings of the AAAI Conference on Artificial Intelligence, 35(11), 9472–9480. https://doi.org/10.1609/aaai.v35i11.17141

That paper is the bridge, and it highlights that popular choices for Q-learning algorithms coincide with the standard setup for dynamic discrete choice models. Once the estimators are known to coincide, we can lean on the economics results on identification, that is, whether it is possible to map from observed data back to the hidden parameters. That in turn tells us which counterfactuals (experiments that let the agents re-optimize) can be run under minimal assumptions, and which ones force the investigator to compromise and report only bounds.

The supporting literature it connects sits on both sides of that bridge.

On the economics side, the dynamic discrete choice tradition: Rust (1987) on the nested fixed point and the logit-shock formulation; Hotz and Miller (1993) on inverting choice probabilities; Magnac and Thesmar (2002) on what is and is not identified, and the necessity of normalizing the shock scale; Pesendorfer and Schmidt-Dengler (2008) on asymptotic least squares; and Kalouptsidi, Scott, and Souza-Rodrigues (2021) on which counterfactuals are identified when the payoff is not.

On the machine learning side, the entropy-regularized reinforcement learning tradition: Ziebart (2010) on maximum causal entropy IRL; Haarnoja et al. (2017) on soft Q-learning; and Geist, Scherrer, and Pietquin (2019) on regularized Markov decision processes.

On the learning-as-foundation side, the older idea that agents who estimate their environment can converge to equilibrium: Bray (1982) and Bray and Savin (1986) on least-squares learning of rational expectations, with the convergence machinery in Marcet and Sargent (1989); and, for the pricing environment specifically, Calvano, Calzolari, Denicolò, and Pastorello (2020) on Q-learning agents that learn to set supracompetitive prices.