Benchmark scores 76 decision systems, puts TypeSafe's Jev on top

Benchmark Heaven published version 1.4.0 of JevBench on Wednesday, a leaderboard that scores 76 typed decision systems on 534 public and 308 sealed decisions. Such systems take a state and a bounded rubric and return a typed answer rather than free text, according to the benchmark's own description.

The leaderboard puts TypeSafe AI's Jev 1.13.0 first on 63.3, followed by JevK5 v0.2.0 on 62.0, Hopper on 59.4, Winnow-12B Q8 on 55.6 and reflex 4B on 54.0. The score weighs accuracy above chance against calibration, then applies separate gates for speed and cost, according to the benchmark's own methodology page.

The run places two models this paper reported on Monday, when community benchmarks disagreed over which was faster. Convai Innovations' open-source Laya sits 32nd on 30.3. An OpenJev entry, credited to razorback16 rather than to the checkpoints trending on Hugging Face, sits 21st on 36.9.

A third of the decisions are held back. The page says the text and answers of the 308 sealed items stay private and only system-level aggregates appear, and that a penalty applies when a system's public accuracy exceeds its sealed accuracy by more than 25 points. The harness, the public tasks and the scoring rules are on GitHub under an MIT licence.

Commenters on Hacker News, where the Show HN post drew 123 points and 33 comments, questioned what the tasks cover. One, hbrn, wrote that Jev performs on a par with a system built in a couple of days and costs twice as much. Another said the tasks do not cover much of what Jev was likely trained to do.

The author, posting as florianstandhar, said a missing zero-shot baseline, BART-large-MNLI, would be added in the next run, and invited submissions through the repository. The benchmark's own page says its scores describe the tested configurations rather than every application.