GPT-5.6 Sol arrived with OpenAI claiming stronger frontend engineering and greater token efficiency than GPT-5.5, while Anthropic released Fable 5 at a dramatically higher price than the other models. Grok 4.5 launched in the same period as a comparatively inexpensive, Opus-class model that, in theory, should be extremely capable.
I wanted a way to test those claims through real-world use: one sufficiently complex software problem, dissected across enough dimensions to show where the models excel, where they fail, and where their performance is nearly indistinguishable.
Fable 5 frontend prototype showing global search in its Basecamp implementation
I built Basecamp Bench to measure this. Each model starts from an empty workspace with the same product requirements, design system, sample data, and Basecamp 5 reference material, then builds both a frontend application and a backend implementing all 203 operations in the canonical API contract. Each workspace is set up with AGENTS.md, DESIGN.md, and INIT.md to simulate a new greenfield project and reveal how agents make choices and tradeoffs in their work. Fable 5 won both tracks. Grok 4.5 delivered the strongest speed and cost tradeoff, completing both builds in 37 minutes for $9.30, but with significant polish gaps.
Grok 4.5 frontend prototype showing the sample Basecamp project
Fable 5 leads both tracks because it builds the shared Recording model described in the specification and connects every surface and endpoint to it.
Backend scores tightly because most models register the complete route surface, with invariant enforcement, contract-correct responses, and honest failure handling separating them. Frontend scores vary more as models trade breadth for depth.
Sonnet prioritizes working, persistent workflows, GPT-5.5 prioritizes polish (though not particularly well) while leaving core controls inert, and GPT-5.6 Sol pairs a disciplined backend with a shallow frontend. Grok 4.5 delivers the strongest balance of speed, cost, and functional depth, outperforming Sol on the frontend and nearly matching it on the backend.
For me and my taste, Fable 5 is within a few percentage points of the real Basecamp frontend implementation, with very minor adjustments that would take no more than a day to finish.
The gap between Fable 5 and the other models is striking, especially given how heavily OpenAI1, Anthropic2, and xAI3 tout them for frontend work.
You can inspect each frontend submission in the repository:
Every model leaves different work for an engineer to finish, and Basecamp Bench measures that starting point before review and iteration improve it. Fable produces the deepest product and still requires security and correctness hardening. Grok reaches roughly 85–87% of Fable's score for 11% of the cost, finishing in 37 minutes instead of more than two hours. The practical question is whether Fable's stronger foundation justifies its higher price, or whether a cheaper model paired with deliberate engineering review produces better value. Model choice changes the starting point, working style, and review burden more than the final result a skilled engineer can reach.
I used the new GPT Sites feature to publish the full evaluation report, where you can explore the methodology, dimension-level scores, charts, model-by-model commentary, and specific failure modes. The GitHub repo contains the runner, prompts, rubrics, versioned contracts, reference material, baseline runs, evaluator reports, leaderboards, and provenance manifests needed to inspect or reproduce the benchmark.
I also attempted to test ZAI's GLM 5.2 using the Pi harness, but the model is unable to complete the benchmark as written.4
I will add Gemini 3.5 Pro when it releases in the coming weeks. If the rumors are true and GPT-6 lands later this summer, I will add it to the benchmark as well.
If you want to add another harness or benchmark run, submit a PR or send me a note.
This post was dictated to GPT-5.6 Sol using ChatGPT.
Fable 5, Grok 4.5, and GPT-5.6 Sol were used to create the evaluation code. The report content was dictated to both Fable 5 and GPT-5.6 Sol. Fable 5 created the final report, with Sol correcting its writing mistakes and improving its clarity.
Footnotes
-
Both benchmark tracks ask the agent to produce a single-file implementation, and GLM 5.2 repeatedly broke on that constraint. I manually overrode the harness to allow multiple files and tried having the model produce them in stages, but the agent still could not complete either run. ↩