TL;DR
Firmulate’s July 2026 benchmark found that five frontier AI models diagnosed the same business crises and rejected every manipulation attempt, but only two completed a €55,000 customer agreement. The company says the results expose a gap between producing correct analysis and converting it into authorized, finished work.
Only two of five frontier AI models completed a €55,000 customer agreement in Firmulate’s July 2026 business simulation, even though the company reported that every model identified the crises, resisted manipulation and produced an appropriate pitch. The result points to a gap between correct analysis and completed work, a distinction that may affect how businesses evaluate AI agents before giving them operational authority.
Firmulate placed the models in control of the same simulated software company during a week of customer problems, financial pressure and social-engineering attempts. Its environment includes 13 synthetic employees, a monthly cash burn of €105,000 and monthly recurring revenue of €2,300. Decisions and workdays were versioned to create an auditable record, according to the company.
The published league table ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline received 26 points because the scoring system awards partial progress. Firmulate said a breach of trust capped a participant’s score, regardless of its other work.
The central sales task depended on finding a competitor weakness buried two document references deep in the company files. Models that followed the evidence could support a full-price agreement adding €4,583 in monthly recurring revenue. Firmulate reported that all five recognized the customer problem and developed the pitch, but only two secured the signature. The source material does not identify which two completed the deal.
Execution Gap Changes AI Buying
The findings suggest that businesses may receive an apparently correct AI response without receiving the corresponding commercial or operational result. In sales, service and internal operations, value often depends on whether an agent can investigate incomplete evidence, use authorized channels and carry a task through its final handoff.
This distinction may change procurement tests. Reasoning quality, safety behavior and polished writing reveal only part of an agent’s performance. Buyers may also need to measure completion rates and escalation discipline under pressure. A missed signature, unresolved customer case or unapproved system action can carry financial and trust costs even when the model’s analysis is accurate.
End-to-End AI Evaluation: Building Effective Metrics, Pipelines, and Monitoring for LLM Systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Inside Firmulate’s Company Simulation
Firmulate designed the exercise to test behavior across connected decisions rather than through isolated chat prompts. The simulated workforce had accumulated more than 680 playbook rules, while a public cash countdown made delays visible. Models had to inspect records, respond to customers and act within departmental permissions.
The week also included staged fake messages from a chief executive and a reporter seeking an off-record confirmation. Firmulate said all five models rejected those approaches, meaning manipulation resistance did not separate the leaders from the rest of the field. The dividing factor, according to the company, was execution after the correct diagnosis.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the sales task
Methodology Questions Limit Comparisons
The results come from Firmulate’s own simulation and scoring system, and no independent replication is cited in the supplied material. It is unclear how closely the synthetic company represents production environments, how much results might change across repeated runs or whether different tool configurations would alter the ranking.
The comparison also contains a disclosed configuration difference: Kimi K3 used the API’s default setting because it ran without an effort parameter, while the other models used xhigh. The material does not state which two models signed the agreement or provide enough detail to determine whether the score differences are statistically stable.
Enterprise Trials Move Toward Completion
Firmulate says readers can inspect the continuing experiment, view its public benchmark and review a quiz based on 242 unedited management decisions. Future runs and fuller methodological disclosure could show whether the same execution pattern persists across models, tasks and repeated trials.
For businesses, the next practical step is likely to be testing agents against internal workflows before granting write access or approval powers. Firmulate proposes using read-only exports so companies can observe investigation, escalation and completion behavior without changing live systems. Such trials could reveal whether an agent can finish authorized work reliably, not merely recommend the right action.
Key Questions
What did the Firmulate benchmark find?
Firmulate reported that all five AI models identified every crisis and rejected every manipulation attempt, but only two completed the €55,000 customer agreement.
Which model received the highest score?
gpt-5.6-sol ranked first with 95 points, ahead of Kimi K3 at 93. The supplied material does not say whether either model was among the two that signed the deal.
Why was the sales task difficult?
The customer event did not contain all the evidence needed to close the agreement. A useful competitor weakness was hidden two references deep in company documents, requiring models to continue investigating before making the full-price case.
Did any model fall for the fake messages?
According to Firmulate, none of the five models complied with the staged impersonation and approval-bypass attempts. This made execution discipline, rather than manipulation detection, the main reported difference between participants.
Can the ranking be treated as definitive?
No. The results reflect one company-designed environment, and the supplied material cites no independent replication. Configuration differences, repeated-run variation and the simulation’s relationship to real workplaces remain open methodological questions.
Source: Thorsten Meyer AI
