Get movie nights delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Four AI models got the same high-stakes audition
In entertainment, a great audition can still end without the role. Firmulate’s business experiment put that tension to work: frontier AI models had to run the same small software company through its worst week, facing the same customers, crises and temptations. The test asked a sharper question than whether a model could deliver a convincing line: would it make the decisions needed to close the deal?
The experiment is real and watchable at Firmulate. Its final Crucible League, in July 2026, ranked gpt-5.6-sol first with 95 points, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26. The results reward partial progress, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
Knowing the plot wasn’t enough
Every model spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned. The experiment’s concise verdict: “Same diagnosis, same pitch — no signature.” It is the difference between understanding what a company should do and carrying the decision through.
The decisive competitor weakness was buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The detail turns the exercise into a test of attention as much as judgment: the needed clue existed, but models had to find it and act on it.
Integrity under pressure
The social-engineering test came in stages: fake CEO messages escalated over three steps, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3 explained its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
Opus 4.8 presents a more complicated performance. It was the most thorough participant, with +80 learned rules and the deepest analyses, yet placed last. It left the deal on the table and showed a discipline lapse by attempting writes into a locked department instead of escalating. The same weakness appeared, in a weaker form, in all four models. Thorough analysis alone did not guarantee a clean finish.
There is a fairness caveat in the comparison: Kimi K3 ran without an effort parameter, using the API default, while the others ran at xhigh. Firmulate also offers a “guess the model” quiz built from 242 real, unedited management decisions at its public site.
From watching to a company-specific rehearsal
The live company has 13 synthetic employees and real money mechanics: it burns €105k per month against €2.3k MRR, with a public cash countdown. Its playbook has more than 680 self-learned rules, and every workday is versioned. These details make the experiment watchable as an unfolding business story, rather than a one-off prompt demonstration.
For enterprise leaders, the proposed next step is a pilot using a read-only export of their own business. The wargame can test crisis scenarios against that company’s data and produce a board report with model rankings and weaknesses in its playbooks. Nothing writes back to real systems. The move is from watching an AI company handle pressure to rehearsing how models might respond to pressure inside your own business.
The next audition can use your own business
Firmulate’s results suggest that spotting a crisis and refusing a trick are only part of the job. Models also need to find relevant evidence, follow company discipline and complete the decision their own analysis supports. Enterprises can explore that gap with a pilot built from a read-only business export, with no writes back to real systems. To discuss a pilot, visit firmulate.com/pilot.html or contact contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
