The AI Post
Agents & CodingOpen ModelsEnterpriseFundraisingGenerative MediaGovernanceInferenceInfrastructureLegal & SafetySector Impact
Research & Evals · Microsoft · Toloka · ThinkingBox · Claude Opus 5.5 · Kimi-K3 · GPT-6 Astra · Hugging Face

Microsoft benchmark finds agents often claim success but leave databases wrong

Microsoft's ThinkingBox checks what an AI agent actually changed in a database, not what it said. Across 121,680 trials, the authors say 67% of failures were clean tool calls with wrong values.

Microsoft's Copilot Studio team and the data company Toloka published ThinkingBox on 3 October, a benchmark that scores AI agents on the state they leave in a database rather than on their replies. The authors say that across 121,680 trials, 67.24 per cent of failures still called tools cleanly but wrote wrong field values. No outside group has reproduced the numbers.

Continue reading

Snippets
  • Kimmonismus's rumour watchlist says Qwen 4 is being discussed for October or early November, with Alibaba confirming it is in training; the other open‑weight dates are his estimates. Kimmonismus on X
  • The Information reports, citing sources, that Anthropic's stock match of employee charity gifts passed $660 million in the six months to March and could reach billions after an IPO. The Information
  • Reuters reports Elon Musk said he will rename SpaceXAI to SpaceXSI. Reuters on X
  • Bloomberg reports former Anthropic researcher Jacob Coxon will testify Monday at a New York City Council hearing alongside representatives of Google, OpenAI and Meta. Bloomberg
  • Elon Musk says five live football games were streamed at once on a single commercial flight over Starlink, in a post with more than 5,000 likes. Elon Musk on X
  • SemiAnalysis says Alibaba's T‑Head showed the Zhenwu V900 AI chip at Apsara 2026, with 216 GB of memory and 1,200 GB/s of interconnect, due to ship in Q1 2027. SemiAnalysis
  • fal says the Manus team has launched Manus 2.0, which integrates H3 Max, the post‑trained video model fal released with MiniMax in August. fal on X
  • The account bridgemindai says it built NerfBench, a benchmark to test whether Claude Opus 5.5 has been quietly nerfed, and posted a video of the results. bridgemindai on X ▶ Watch
  • SemiconductorsX reports Goldman Sachs now expects the AI memory shortage to last until 2030, forecasting global token use up more than 24 times to about 120 quadrillion a month. SemiconductorsX on X
All snippets
Inside Today