Real-SWE Puts Frontier Models on Private Enterprise Codebases — Fable 5.1 Tops It at 38.8%
Fable 5.1 achieved a 38.8% resolution rate on Real-SWE, the first benchmark evaluating AI agents on licensed private enterprise codebases rather than public repositories.
WithSpecific released Real-SWE, a benchmark designed to evaluate frontier AI models on private, real-world enterprise codebases licensed from actual companies. Unlike existing benchmarks that rely on public internet data or synthetic tasks, Real-SWE uses production codebases from entities including a Luma/Partiful competitor with over 200,000 users and a consumer fintech platform processing 100,000+ bank statements. The evaluation measures model-and-harness combinations on tasks with direct business consequences, such as fixing billing logic, calculating taxes, and migrating customers across multiple services. These tasks require agents to navigate proprietary systems where 99% of tokens are hidden from standard model training sets.
In the initial results, Fable 5.1 running on the Claude Code harness topped the leaderboard with a 38.8% resolution rate. GPT-6 Astra (Codex CLI) followed at 33.8%, and Gemini 3.8 Flash (Gemini CLI) scored 31.2%. Performance dropped significantly for lower-ranked models, with GPT-5.6 Sol achieving only 16.2%. The difficulty of the benchmark is evident in task-specific breakdowns: six of the ten sampled tasks had overall resolution rates below 15%. While models performed well on "Multi-region sweep" (67.2%) and "API keys & environments" (65.6%), they failed almost universally on "Analytics stream reducer" (0.0%) and "Tax jurisdiction" (3.1%).
The analysis highlights that missed requirements are the most common failure mode, with models struggling to interpret company-specific coding patterns and underspecified instructions. Despite the low success rates, rollout costs varied widely, ranging from $2.50 for Gemini 3.8 Flash to $6.96 for Fable 5.1. Notably, rollout duration did not correlate strongly with success; 71.4% of rollouts under 10 minutes failed, compared to 73.4% of longer attempts. The benchmark utilizes native harnesses reflecting actual engineering workflows, integrating tools like AWS emulators, Kubernetes, Linear MCP, and various databases including PostgreSQL and MongoDB.
The findings suggest that current frontier models are not yet ready to autonomously handle complex, cross-functional engineering work within strict operational constraints. The gap between performance on public datasets and private, context-heavy enterprise code remains substantial, indicating that agents frequently miss requirements or fail to verify assumptions when facing out-of-distribution proprietary systems.