General Decision Models:
Benchmarking and Insights Beyond Jev

Feiyu Duan1,2,*, Jiayu Lin1,2,*, Jia Wang1,3,*, Jun Xiang2,*, Jialiang Wu2, Xinnong Zhang1,2, Hanqi Yan4, Siyuan Wang5, Zhongyu Wei1,2,†
1Shanghai Innovation Institute 2Fudan University 3Tongji University 4King's College London 5Chinese University of Hong Kong

* Equal contribution, names are listed in alphabetical order. · † Corresponding author.

Get started

Resources

Jump into the code, download JEVal, or choose an SII-Decision model.

Where generality meets fast decisions

General-purpose models can handle many tasks, but structured choices often need a faster interface. JEVal measures that capability; SII-Decision is an open model family built for it.

General intelligence and decisional intelligence as two dimensions, with general decision models occupying their intersection.
General intelligence and decisional intelligence meet in general decision models.

JEVal benchmark

JEVal puts diverse tasks into a shared decision interface: choose an option, make a binary judgment, or assign a score. Its ten domains test whether a model can move from evidence-based judgments to specialist knowledge, long-context understanding, and agent or tool choices without task-specific retraining.

JEVal benchmark framework: sources, data processing, unified decision tasks, evaluation, domain composition, and input lengths.
11,257 bilingual instances from 36 datasets across 10 application domains.

SII-Decision

The First-Token Decision Readout restricts an open-weight LLM's next-token probabilities to valid option labels, producing a structured decision in one forward pass. Reasoning-to-Readout Self-Distillation then teaches this readout to match the distribution the same model reaches after reasoning, without gold labels or a stronger teacher.

SII-Decision first-token decision readout for choice, binary, and score tasks, followed by reasoning-to-readout self-distillation.
The same readout handles all three JEVal formats; self-distillation transfers post-reasoning decisions to the first token.

Key results

From individual decisions to complete agent workflows and large-scale simulations, these results show where fast decision models work well and where they still struggle.

Decision quality on JEVal

The strongest decision models approach leading thinking LLMs on single-step accuracy. SII-Decision-27B reaches 79.24% accuracy and the lowest expected calibration error (ECE) of the 25 configurations. Across domains, decision models are strongest when the evidence is available in the input; the best generative LLMs retain an advantage in medicine, finance, and law.

Overall results on 10,938 choice and binary questions; higher accuracy and lower errors are better.
ModelAccuracy ↑ECE ↓Brierraw ↓
SII-Decision-27B79.24%0.01490.279
SII-Decision-4B71.92%0.08430.383
Jev78.38%0.05320.287
Qwen3.8-27B · thinking79.49%0.10050.319
GLM-5.3-Flash · thinking80.37%0.07840.316
GPT-6-Luna · non-thinking80.93%0.11590.297
JEVal accuracy against short-input latency for SII-Decision and other decision models.
Accuracy–latency comparison for inputs up to 1K tokens. Open models run on one H100; hosted Jev includes network transport.
Accuracy, calibration, and raw Brier error across JEVal's ten domains for 25 configurations.
Domain-level accuracy and probability quality across all model configurations.

Probability estimates need scrutiny

Jev often selects the most likely option on controlled probability tasks, yet its distribution is too concentrated. On prior items, it assigns the true mode 89.35% probability on average, compared with 35.60% in the known distribution. Ranking an option correctly does not ensure the probabilities are faithful.

Jev's probability concentration compared with known prior and posterior distributions.
Probability concentration on known prior and posterior distributions.

Agent workflows: faster, with lower success

In τ-bench, Jev selects the next tool or response action while the same base LLM generates arguments or a user-facing answer. The median episode becomes shorter, but success falls and the observed P95 time rises. A fast local decision can still lead to a less reliable multi-step workflow.

Overall τ-bench results across Retail and Airline.
AgentSuccess ↑Median time ↓P95 time ↓
Base LLM77.6%267.1 s695.5 s
Jev + LLM66.1%160.6 s816.0 s

Social simulation at scale

On 277,500 SocioBench respondent–question pairs, SII-Decision-27B attains 39.87% accuracy at 8.36 seconds per item, close to thinking DeepSeek-V4-Flash-0731 at 40.41% and 211.65 seconds. In ElectionSim, about 330,000 simulated voters yield correct calls for 47 of 51 states and 12 of 15 battleground states. Vote-share errors are about twice those of the original generative pipeline, and the simulation overestimates the Democratic share.

Actual and simulated 2024 election maps, with four incorrectly called states marked.
SII-Decision-27B preserves most state-level winners but misses four Trump-won states.

BibTeX

@misc{duan2026generaldecisionmodelsbenchmarking,
      title={General Decision Models: Benchmarking and Insights Beyond Jev},
      author={Feiyu Duan and Jiayu Lin and Jia Wang and Jun Xiang and Jialiang Wu and Xinnong Zhang and Hanqi Yan and Siyuan Wang and Zhongyu Wei},
      year={2026},
      eprint={2610.03935},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.03935},
}