Model guide · Dracon Bench
mistral-large-4-1t-a52b
Tested
Mistral Large 4 on NIM sits at 46.0 overall across five suites, rank 21. IFEval-50 80%, IFBench 0%, MMLU-Pro 1/10, GSM8K 60%, JSON 4/5; no SVG brief produced an admissible artifact, so this row has no SVG column and no sixth suite. Several IFEval prompts needed retries after timeouts, but the final not_delivered count is 0. Pilot depth: 10 prompts per text suite and the 5 SVG briefs; every suite prints its own n and not_delivered.
- Rank
- #21
- Overall
- 46.0
- Suites
- 5 of 6
- Ranked
- #21 of 21
When the world heard about it
Timeline
- First reachable for us 2026-11-18 our first probe
- First tested by us 2026-11-18 Dracon Bench
- Officially announced not recorded no source lists it
- First official benchmark not recorded no source lists it
- Reachable before we scored it same day
Track A · script-scored subsets
Scores
| Suite | Score | Sample | Precision |
|---|---|---|---|
| IFEval-50 prompt-strict | 80.0% | 8/10 prompts | 49.0%–94.3% |
| IFEval-50 instruction-strict | 76.5% | 13/17 constraints | 52.7%–90.4% |
| IFBench-50 prompt-strict | 0.0% | 0/10 prompts | 0.0%–27.8% |
| IFBench-50 instruction-strict | 0.0% | 0/11 constraints | 0.0%–25.9% |
| MMLU-Pro-50 | 10.0% | 1/10 served | 1.8%–40.4% |
| JSON briefs conformance | 80.0% | 4/5 served | 37.6%–96.4% |
| GSM8K-50 | 60.0% | 6/10 served | 31.3%–83.2% |
The range beside a script-scored figure is a 95% interval over that figure's own sample: the prompts it followed, the constraints it met, or the items it answered. The instruction-strict figures are measured over CONSTRAINTS, which is why their sample line counts constraints rather than prompts. The judged total is not a proportion, so it has no interval: it carries the measured judge band instead: how far a re-judging of the same artifacts moved them, which is a different kind of number. The method page says what each one covers and what it does not.
Track A · per-type breakdown
What it failed
The instruction types these two suites check, and the checks the model missed. A type is checked once per prompt that carries it, so a type with seven checks was asked for seven times. The counts beside a type are that type's own; the suite's totals are in the scores table above. These are the failures behind the instruction-strict figure, which is the sum of them.
IFEval-50
Missed at least one check in 3 of the 15 instruction types checked.
| Instruction type | Checks | Missed |
|---|---|---|
| startend · quotation | 2 | 2 |
| detectable format · json format | 1 | 1 |
| length constraints · number words | 1 | 1 |
IFBench-50
Missed at least one check in 8 of the 8 verifiers checked.
| Verifier | Checks | Missed |
|---|---|---|
| count · unique word count | 3 | 3 |
| count · numbers | 2 | 2 |
| count · conjunctions | 1 | 1 |
| count · person names | 1 | 1 |
| count · word count range | 1 | 1 |
| format · list | 1 | 1 |
| words · keywords specific position | 1 | 1 |
| words · words position | 1 | 1 |
Provider-reported usage
Thinking cost
What the run cost the model in thinking: provider-reported completion tokens per prompt, the thinking share, and the cost per correct answer. Reasoning-heavy models can exhaust fixed budgets before answering.
| Suite | Cost | Reading |
|---|---|---|
| IFEval-50 | 1,901/prompt | 82% thinking· 1,901/correct |
| GSM8K-50 | 375/prompt | 71% thinking· 563/correct |
Head to head
Compare with
- ernie-5.5-flash-preview
- north-mini-code
- dots-3-note-preview
- gemma-4-31b-it
- lfm-2.5-2.6b
- lfm-3-1.7b
- muse-glimmer-30b
- nemotron-3.5-bolt-e4b
- nemotron-3.5-lightning-30b-a3b
- nemotron-3.5-lightning
- nemotron-3.5-super-vl-120b-a12b
- nemotron-3.5-super-vl
- nemotron-3-nano-omni-30b-a3b-reasoning
- nemotron-3-nano-omni-30b-a3b-reasoning
- nemotron-3-ultra-550b-a55b
- nemotron-3-ultra-550b-a55b
- hy4-flash
- particle-1
- laguna-xs-2.1
- flint-v1.0
Run record
Evidence and notes
- NVIDIA NIM free endpoint (integrate.api.nvidia.com/v1), same key and base as every NIM row
- IFBench 0/10, MMLU-Pro 1/10, GSM8K 60%
- No SVG score: every brief failed to produce an admissible artifact, so the overall covers five suites