GPT-5.6 Sol arrived with OpenAI claiming stronger frontend engineering and greater token efficiency than GPT-5.5, while Anthropic released Fable 5 at a dramatically higher price than the other models.12 SpaceXAI launched Grok 4.5 in the same period as a comparatively inexpensive, Opus-class model that, in theory, should be extremely capable.3
I wanted a way to test those claims through real-world use. I chose one sufficiently complex software problem and dissected it across 20 dimensions to see where the models excel, where they fail, and where their performance is nearly indistinguishable.
September 2026 update
Fable 5.1 leads frontend at 8.183, ahead of GPT-6 Astra at 7.607. Astra leads backend at 9.305, ahead of Fable at 8.965.
Astra costs about a quarter as much as Fable 5.1 across both tracks ($25.57 versus $92.96), making the quality-versus-cost comparison substantially more interesting.
This post retains the original findings and adds new results, focusing on GPT-6 Astra and Fable 5.1.
Benchmark setup and results
Fable 5 frontend prototype showing global search in its Basecamp implementation
I built Basecamp Bench to measure this. You can read the full report here. Agents start in the same greenfield workspace with AGENTS.md, DESIGN.md, INIT.md, and Basecamp 5 reference material. They then build a frontend from mocks (screenshots) and design tokens, and a backend implementing the canonical API contract, revealing their engineering choices and tradeoffs.
Grok 4.5 frontend prototype showing the sample Basecamp project
Time and cost include implementation and evaluation across both tracks. Table updated September 5.
Agent separation
Astra’s backend follows the API’s required response formats, reproduces the specified sample data consistently, and preserves changes across restarts. Tests also confirmed that it rejects invalid cross-project card moves without changing state and preserves files through upload and download. Tests exercised all 203 API operations: 192 returned the required success statuses and response formats, while 11 explicitly reported that the feature was not implemented.
July backend benchmark scores across evaluation dimensions
Fable 5.1’s frontend advantage comes from product completeness. Fable 5.1 uses the same underlying records across screens, so creating, editing, commenting, and moving items updates the rest of the app consistently. Its automated test suite passes 3,485 checks. Astra implements substantial frontend workflows too, but omits details such as recent items in search and the Sample Project tab, and simplifies parts of Docs and Chat.
July frontend benchmark scores across evaluation dimensions
Both frontends can fail to start when saved data is malformed. Fable 5.1 can also render unsafe HTML from saved content. Astra’s backend gives new projects eight enabled tools instead of an empty dock and accepts reaction text longer than the API’s 16-character limit.
In the original run, I found Fable 5 visually close to the real Basecamp frontend. Fable 5.1 extends its product and workflow coverage, but it still needs to validate saved data before loading it, keep keyboard focus inside open dialogs, and let users move cards without dragging.
The real Basecamp frontend showing its sample podcast project
Fable 5.1 leads Astra by 0.576 on frontend, while Astra leads by 0.340 on backend. The combined cost is $92.96 for Fable and $25.57 for Astra. The choice depends on how much the frontend advantage matters for the work at hand.
The final 10% of a project requires as much work as the first 90%.4 Each point on Basecamp Bench is harder-won than the last. Moving from eight to nine requires far more depth and complexity than moving from five to six.
Browse each frontend submission in the repository ↓
Run variability
Score variability across five Sonnet 5 and GPT-5.6 Sol runs
I ran Sonnet 5 and GPT-5.6 Sol five additional times on each track to measure run-to-run variation. Stronger runs turned more of the requested product into working, persistent behavior and handled API contracts and validation more carefully, while weaker runs often looked similarly complete at a glance but concealed inert controls, brittle state, and shallow backend behavior.
On frontend work, GPT-5.6 Sol and Sonnet 5 are closer than their reputations suggest. Sol's best run beat Sonnet's worst, while its weakest lost to Sonnet's best. That overlap may explain why reasonable people prefer opposite models. They experienced the top of one model's range and the bottom of the other's.
Full report and other agents
I used the new ChatGPT Sites feature to publish the evaluation report, where you can explore the methodology, dimension-level scores, charts, model-by-model commentary, and specific failure modes.
The GitHub repo contains the runner, prompts, rubrics, versioned contracts, reference material, baseline runs, evaluator reports, leaderboards, and provenance manifests needed to inspect or reproduce the benchmark.
Building the project with GPT-5.6 Sol consumed 682 million tokens and $449.33 in model usage.
I attempted to test ZAI's GLM 5.2 using the Pi harness, but the model is unable to complete the benchmark as written.5 I also attempted to test Gemini Flash 3.5 in AGY, but experienced tool call failures across five runs.6 Later Gemini Flash models, 3.7 and 3.8, successfully completed both benchmark tracks through the Antigravity harness. Their scores are included in the table above. I successfully tested Gemini Pro 3.1, but both submissions were largely unfinished, scoring 3.2 on the frontend and 3.9 on the backend. The model appeared unable to sustain agentic work across the benchmark's long time horizon, placing it in a different class from the other agents.
The updated report includes GPT-6 Astra and Fable 5.1, with dimension-level scores, costs, and specific failure modes.
If you want to add another harness or benchmark run, submit a PR or send me a note.
This post was dictated to GPT-5.6 Sol using ChatGPT. It was subsequently updated by GPT-6 Astra under my direction.
Fable 5, Grok 4.5, and GPT-5.6 Sol were used to create the evaluation code. The report content was dictated to both Fable 5 and GPT-5.6 Sol. Fable 5 created the final report, with Sol correcting its writing mistakes and improving its clarity.
Footnotes
-
“The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time.” — Tom Cargill ↩
-
Both benchmark tracks ask the agent to produce a single-file implementation, and GLM 5.2 repeatedly broke on that constraint. I manually overrode the harness to allow multiple files and tried having the model produce them in stages, but the agent still could not complete either run. ↩
-
AGY lost background-task state in all runs, then misclassified completed work as failed. Missing task IDs broke polling on the backend, while missing task logs caused an invalid tool call and runaway generation on the frontend. ↩