Molecular Déjà Vu: Frontier Models Are Retrieving Benchmark Answers, Not Predicting Them
Frontier language models are frequently retrieving published molecular property values verbatim rather than predicting them, invalidating accuracy scores on five of twelve standard regression benchmarks.
An audit of 22 frontier models across 12 molecular property regression benchmarks reveals widespread verbatim retrieval of published numerical values. On five of the tested datasets, more than 50% of the evaluated large language models reproduced exact digits from source literature instead of generating predictions based on molecular structure. This memorization is benchmark-specific; on the remaining seven datasets, retrieval appeared only in isolated cells rather than as a systemic failure. The study distinguishes between models that genuinely predict properties and those acting as lookup tables for training data contaminants.
The depth of retrieval shifts significantly depending on the reasoning level applied during inference. When experiments were run at a higher reasoning level using the same molecules and prompts, the frequency of flagged retrieval instances increased by 89% compared to the lowest reasoning level. This suggests that chain-of-thought or deeper processing steps may trigger specific memory pathways that expose memorized values rather than suppressing them. In tests designed to interrupt retrieval, even the strongest models recognized combinations of transformed SMILES strings paired with original labels, indicating robust coupling between structural representations and stored numeric outputs.
Suppressing these retrieval mechanisms altered the error distribution across the model cohort. While differing uses of verbatim retrieval previously spread prediction errors apart, blocking access to memorized values moved the errors of different models closer together in relative terms. This convergence implies that general predictive capability is not solely determined by the volume of memorized values. The findings necessitate a re-evaluation of current evaluation protocols, as high accuracy on contaminated benchmarks reflects data leakage rather than improved architectural reasoning or chemical understanding.