TestedFirst seen 2026-11-18 (first probed by us) · First tested 2026-11-18 · NVIDIA NIM free endpoint (integrate.api.nvidia.com/v1)
Gemma 4 31B on NIM is a mid-board lane: 55.8 overall across six suites, rank 16. IFEval-50 90%, MMLU-Pro 5/10, GSM8K 50%, IFBench 20%, JSON 3/5, SVG 97/150 from three admissible artifacts. The OpenRouter :free id for this model was skipped (400 from the provider behind that key); the NIM row is a different serving lane and does not unskip it. Pilot depth: 10 prompts per text suite and the 5 SVG briefs; every suite prints its own n and not_delivered.
Rank
#16
Overall
55.8
Suites
6 of 6
Ranked
#16 of 21
When the world heard about it
Timeline
First reachable for us2026-11-18our first probe
First tested by us2026-11-18Dracon Bench
Officially announced2026-04-02llm-stats
First official benchmark2026-11-19the provider's benchmarks block: a public score existed by then — the first day we saw one
Reachable before we scored itsame day
Announced before it reached the lane we served230 days
A public score existed within1 day
Track A · script-scored subsets
Scores
Subset scores for gemma-4-31b-it.
Suite
Score
Sample
Precision
IFEval-50 prompt-strict
90.0%
9/10 prompts
59.6%–98.2%
IFEval-50 instruction-strict
94.1%
16/17 constraints
73.0%–99.0%
IFBench-50 prompt-strict
20.0%
2/10 prompts
5.7%–51.0%
IFBench-50 instruction-strict
18.2%
2/11 constraints
5.1%–47.7%
MMLU-Pro-50
50.0%
5/10 served
23.7%–76.3%
JSON briefs conformance
60.0%
3/5 served
23.1%–88.2%
GSM8K-50
50.0%
5/10 served
23.7%–76.3%
SVG blind rubric
97/150
blind rubric
judge band ±8.3 pts
The range beside a script-scored figure is a 95% interval over that
figure's own sample: the prompts it followed, the constraints it
met, or the items it answered. The instruction-strict figures are
measured over CONSTRAINTS, which is why their sample line counts
constraints rather than prompts. The judged total is not a
proportion, so it has no interval: it carries the measured judge
band instead: how far a re-judging of the same artifacts moved
them, which is a different kind of number. The method page says
what each one covers and what it does not.
Track A · per-type breakdown
What it failed
The instruction types these two suites check, and the checks the model
missed. A type is checked once per prompt that carries it, so a type
with seven checks was asked for seven times. The counts beside a type
are that type's own; the suite's totals are in the scores table above.
These are the failures behind the instruction-strict figure, which is
the sum of them.
IFEval-50
Missed at least one check in 1 of the 15
instruction types checked.
The instruction types 1 of 15 the
IFEval-50 suite checked that google/gemma-4-31b-it missed, most missed first.
Instruction type
Checks
Missed
detectable content · number placeholders
1
1
IFBench-50
Missed at least one check in 7 of the 8
verifiers checked.
The verifiers 7 of 8 the
IFBench-50 suite checked that google/gemma-4-31b-it missed, most missed first.
Verifier
Checks
Missed
count · unique word count
3
3
count · numbers
2
1
count · person names
1
1
count · word count range
1
1
format · list
1
1
words · keywords specific position
1
1
words · words position
1
1
Provider-reported usage
Thinking cost
What the run cost the model in thinking: provider-reported completion
tokens per prompt, the thinking share, and the cost per correct answer.
Reasoning-heavy models can exhaust fixed budgets before answering.
Thinking cost for gemma-4-31b-it.
Suite
Cost
Reading
IFEval-50
359/prompt
0%
thinking· 359/correct
GSM8K-50
135/prompt
0%
thinking· 189/correct
Track B · blind-judged
Artifacts
Blind-judged by nvidia/nemotron-3.5-super-vl:free: the judge saw
the brief and the rendered image only, never the model name. Briefs
are frozen; expand one to read it.
beach
unjudged
no svg
Brief
Draw a beautiful sunset beach scene: a large sun sitting low over the sea
with a shimmering reflection path on the water, a leaning palm tree
silhouette on the left, two dolphins arcing out of the waves, a small
sailboat on the right, birds in the sky, sandy foreground with a starfish.
Warm oranges and pinks fading to teal. Coherent perspective: distant
objects smaller and hazier than near ones.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
clock
42/50Judge reasoning
Brief fidelity (8/10)
All requested elements are present including 12 numerals, tick marks, 10:10 hands, red second hand at 6, and DRACON text, though the hour hand points exactly at 10 rather than just past it and DRACON sits at center rather than under it.
Construction (9/10)
The clock is cleanly assembled with a circular dial, properly joined hands at the center, and no melted or duplicated parts.
Composition (9/10)
The clock is centered in the 800x600 frame with balanced proportions and clear use of the circular format.
Detail finish (8/10)
Flat vector execution includes minute ticks, a red second hand, and legible numerals with no broken edges or cut-off elements.
Visual appeal (8/10)
The dark rim, light face, and red accent create a coherent, confident, and attractive minimalist design.
Brief
Draw a round wall clock face showing exactly 10:10: a circular dial with
all 12 hour numerals in the correct positions, minute tick marks, an hour
hand pointing just past 10, a minute hand pointing exactly at 2, and a thin
second hand pointing at 6. The word DRACON in small capitals under the
center. Dark rim, light face, flat vector style, centered composition.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
duo
unjudged
no svg
Brief
Draw a comical scene: a dog driving a jet ski at speed, paws on the
handlebars, with a raccoon riding behind holding on, both leaning into a
turn. Sea spray behind the jet ski, sun and clouds in the sky, a distant
shoreline. Both characters must be clearly doing something, not just
posed side by side. Bright cartoon vector style.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
monalisa
18/50Judge reasoning
Brief fidelity (3/10)
Major requested elements are missing, the folded hands, winding river, and true three-quarter pose are absent, capping fidelity despite the dark dress, center-parted hair, and smile being present.
Construction (3/10)
The figure is assembled from simple geometric shapes rather than anatomical forms, with an abstract mask-like face and no visible hands or dress structure.
Composition (5/10)
The bust is centered against a layered landscape with a small bridge on the right, but the figure is cut off at the bottom edge where hands should appear.
Detail finish (3/10)
The artwork uses flat fills with no shading, textures, or fine details, and the bottom edge cuts off the figure abruptly.
Visual appeal (4/10)
The muted green, brown, and smoky blue palette is coherent, but the minimalist execution lacks charm and recognizability as the Mona Lisa.
Brief
Redraw the Mona Lisa as a flat vector portrait, bust composition: a seated
woman in three-quarter view facing the viewer, dark dress with neckline
detail, hands folded at the bottom edge, soft enigmatic smile, hair parted
in the middle. Behind her a hazy landscape with a winding river, a bridge
on the right, and blue mountains. Muted greens, browns, and smoky blues.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
pelican
37/50Judge reasoning
Brief fidelity (7/10)
The pelican is present in side profile facing left with a large beak, throat pouch, long neck, folded wing, flat ground, sky, and sea horizon, but the feet are not clearly webbed as requested.
Construction (8/10)
The bird is assembled from clean geometric shapes, oval body, curved neck, triangular beak, and simple legs, with no melted or duplicated parts.
Composition (8/10)
The subject is well-placed in the frame facing into open negative space, with a clear horizon line and full body visible.
Detail finish (7/10)
The flat vector style is clean with an eye, wing suggestion, and horizon strip, though detail is minimal and feet lack definition.
Visual appeal (7/10)
The limited brown-and-blue palette is coherent and the confident, minimalist linework gives the piece quiet charm.
Brief
Draw a brown pelican standing in side profile, facing left, full body visible:
large beak with throat pouch, long neck, folded wing, webbed feet on flat
ground. Simple sky background with a hint of sea on the horizon. Flat vector
style, clean shapes, limited palette.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.