Economists Turn to Game Theory to Keep AI Agents Honest
AI safety researchers are turning to a branch of economics called mechanism design to keep powerful AI models from lying, cheating, or sandbagging their way past safety tests.
The approach, sometimes called "reverse game theory," starts from a blunt premise: we may never understand what frontier AI models are actually doing internally. So instead of trying to peer into a model's "brain" or train it to follow a moral code, researchers want to design rules and reward structures that make good behavior the rational choice — regardless of what the model is really thinking.
The idea borrows from familiar real-world mechanisms. When one person cuts a cake and the other picks first, the cutter is incentivized to make the pieces even. Ride-hailing surge pricing pushes drivers to work busy hours. Second-price auctions, where the winner pays the runner-up's bid, encourage buyers to bid what they're truly willing to pay. Applied to AI, a recent preprint from MIT economist Stephen Morris and colleagues at Yale and Columbia suggests rewarding a model not just for its own test performance, but for how well its answers predict what a second model will say. Deliberately playing dumb — "sandbagging" to avoid triggering safety mechanisms — becomes a losing bet.
Princeton computer scientist Elad Hazan proposes a different mechanism: have models take turns as "solvers" that tackle problems and "auditors" that check the others' work. Solvers caught cheating are punished, but so are auditors that miss the cheating. Disputes can go to "trial" before a human expert. Hazan acknowledges the risk of collusion — auditors siding with the models they're meant to police — but argues that paying auditors well enough, and auditing the auditors themselves, can keep the system honest. He points to Nobel laureate Gary Becker's 1960s work on the "rational offender": people commit crimes not because they're bad, but because the incentives make it rational.
The approach has limits. Mechanism design assumes agents behave rationally, and AI models are not rational — they're algorithms that can be swayed by subtle, poorly understood factors. "Is it even reasonable to assume that they will respond to incentives?" asks Helen Qu, a research fellow at the Flatiron Institute. Morris concedes the point but notes that economists have long treated humans as rational despite knowing otherwise, because it still helps predict behavior. Qu sees mechanism design as one layer in a "defense in depth" strategy, alongside interpretability, regulation, and training — not a replacement for them. And even if the mechanisms work, they must ultimately fit into the biggest incentive system of all: the market. If a more aligned model is slower or more expensive, will any frontier lab actually release it?