AI forecasters rival top humans, but causation remains a blind spot
Specialized AI forecasting systems now equal or beat the best human superforecasters on standard prediction questions, but they still struggle with the conditional and causal forecasts that matter most for policy and business decisions.
The finding, discussed by the Forecasting Research Institute (FRI), marks a shift in how organizations might buy predictions. Until recently, getting top-tier forecasts meant hiring a team of superforecasters at a cost of thousands of dollars and weeks of waiting, or trawling thin prediction markets. Now startups such as FutureSearch, Mantic and Preseen sell AI forecasters that handle standard questions at or above the best human level, and general-purpose models like ChatGPT and Claude are roughly six months behind — meaning they should match top humans by the end of the year.
The catch is what the machines still cannot do reliably. On conditional questions — "if the Democrats win, what happens to the stock market?" — a test by FRI about a year ago found the AI systems of the time did significantly worse than on ordinary questions, apparently failing to grasp the concept. Experts expect gradual improvement rather than a sudden flip, and they are building a new benchmark to measure where the systems stand in a few months. Truly causal questions, which ask what would happen if one factor changed while everything else stayed fixed, remain harder still; prediction markets cannot answer them at all because there is nothing observable to pay out on.
The most surprising detail is how FRI is testing that gap: using Freeciv, a clone of the strategy game Civilization, where researchers can replay a game, change exactly one variable and observe the full range of outcomes. Forecasting the real world is harder, but the game lets them measure whether AI systems can tell a conditional forecast from a causal one.
AI progress itself remains a blind spot. Human and AI forecasters have underestimated the pace of AI advances every year; earlier this summer they put only a 5% chance on AI solving a Millennium Problem this year. That matters for anyone using these tools to plan around AI, because the same failure may contaminate forecasts about the technology's own trajectory.