TestedFirst seen 2026-11-18 (first probed by us) · First tested 2026-11-18 · NVIDIA NIM free endpoint (integrate.api.nvidia.com/v1)
The 550B-class Nemotron Ultra on NIM: IFEval-50 70%, IFBench 50%, MMLU-Pro 3/10, GSM8K 60%, JSON 5/5, and SVG 187/250 from five blind-judged artifacts, the highest SVG total among the NIM additions. Overall 64.1 across six suites, rank 10. It is a separate row from the OpenRouter Ultra lane. Pilot depth: 10 prompts per text suite and the 5 SVG briefs; every suite prints its own n and not_delivered.
Rank
#10
Overall
64.1
Suites
6 of 6
Ranked
#10 of 21
When the world heard about it
Timeline
First reachable for us2026-11-18our first probe
First tested by us2026-11-18Dracon Bench
Officially announced2026-06-04llm-stats
First official benchmark2026-11-19the provider's benchmarks block: a public score existed by then — the first day we saw one
Reachable before we scored itsame day
Announced before it reached the lane we served167 days
A public score existed within1 day
Track A · script-scored subsets
Scores
Subset scores for nemotron-3-ultra-550b-a55b.
Suite
Score
Sample
Precision
IFEval-50 prompt-strict
70.0%
7/10 prompts
39.7%–89.2%
IFEval-50 instruction-strict
52.9%
9/17 constraints
31.0%–73.8%
IFBench-50 prompt-strict
50.0%
5/10 prompts
23.7%–76.3%
IFBench-50 instruction-strict
54.5%
6/11 constraints
28.0%–78.7%
MMLU-Pro-50
30.0%
3/10 served
10.8%–60.3%
JSON briefs conformance
100.0%
5/5 served
56.6%–100.0%
GSM8K-50
60.0%
6/10 served
31.3%–83.2%
SVG blind rubric
187/250
blind rubric
judge band ±8.3 pts
The range beside a script-scored figure is a 95% interval over that
figure's own sample: the prompts it followed, the constraints it
met, or the items it answered. The instruction-strict figures are
measured over CONSTRAINTS, which is why their sample line counts
constraints rather than prompts. The judged total is not a
proportion, so it has no interval: it carries the measured judge
band instead: how far a re-judging of the same artifacts moved
them, which is a different kind of number. The method page says
what each one covers and what it does not.
Track A · per-type breakdown
What it failed
The instruction types these two suites check, and the checks the model
missed. A type is checked once per prompt that carries it, so a type
with seven checks was asked for seven times. The counts beside a type
are that type's own; the suite's totals are in the scores table above.
These are the failures behind the instruction-strict figure, which is
the sum of them.
IFEval-50
Missed at least one check in 7 of the 15
instruction types checked.
The instruction types 7 of 15 the
IFEval-50 suite checked that nvidia/nemotron-3-ultra-550b-a55b missed, most missed first.
Instruction type
Checks
Missed
startend · quotation
2
2
change case · capital word frequency
1
1
change case · english capital
1
1
detectable content · postscript
1
1
keywords · frequency
1
1
length constraints · number words
1
1
punctuation · no comma
2
1
IFBench-50
Missed at least one check in 4 of the 8
verifiers checked.
The verifiers 4 of 8 the
IFBench-50 suite checked that nvidia/nemotron-3-ultra-550b-a55b missed, most missed first.
Verifier
Checks
Missed
count · unique word count
3
2
count · numbers
2
1
format · list
1
1
words · keywords specific position
1
1
Provider-reported usage
Thinking cost
What the run cost the model in thinking: provider-reported completion
tokens per prompt, the thinking share, and the cost per correct answer.
Reasoning-heavy models can exhaust fixed budgets before answering.
Thinking cost for nemotron-3-ultra-550b-a55b.
Suite
Cost
Reading
IFEval-50
1,867/prompt
67%
thinking· 1,867/correct
GSM8K-50
354/prompt
59%
thinking· 590/correct
Track B · blind-judged
Artifacts
Blind-judged by nvidia/nemotron-3.5-super-vl:free: the judge saw
the brief and the rendered image only, never the model name. Briefs
are frozen; expand one to read it.
beach
33/50Judge reasoning
Brief fidelity (4/10)
Most requested elements appear (sun, reflection path, leaning palm, sailboat, birds, starfish, gradient sky), but the two arcing dolphins are missing, a headline element that caps this criterion at 4.
Construction (7/10)
Forms are simple silhouettes and geometric shapes rather than detailed anatomy; nothing looks melted, duplicated, or broken, though the water shapes are ambiguous.
Composition (8/10)
The frame is balanced with the palm anchoring the left, the sun centered low over the sea, the small sailboat on the right, and sand with starfish in the foreground; distant sky reads as hazier than the near shore.
Detail finish (7/10)
Includes a shimmering reflection path, scattered birds, a starfish, and a smooth warm-to-cool gradient, but shading and texture are minimal overall.
Visual appeal (7/10)
The palette is coherent and pleasing, the silhouette linework is confident, and the scene has a calm, charming simplicity.
Brief
Draw a beautiful sunset beach scene: a large sun sitting low over the sea
with a shimmering reflection path on the water, a leaning palm tree
silhouette on the left, two dolphins arcing out of the waves, a small
sailboat on the right, birds in the sky, sandy foreground with a starfish.
Warm oranges and pinks fading to teal. Coherent perspective: distant
objects smaller and hazier than near ones.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
clock
44/50Judge reasoning
Brief fidelity (9/10)
All requested elements are present: 12 ordered numerals, minute ticks, hands indicating roughly 10:10, a thin second hand at 6, and the DRACON wordmark under the center.
Construction (9/10)
The clock face, hands, center hub, and numerals are cleanly assembled flat vector shapes with no melting, duplication, or anatomical errors.
Composition (9/10)
The clock is centered in the frame with balanced scale and clear use of the 800x600 viewbox.
Detail finish (9/10)
The flat vector style executes the dark rim, light face, tick marks, red second hand, and small-cap DRACON text without broken edges or stray shapes.
Visual appeal (8/10)
The coherent cream-and-charcoal palette with a red accent creates a confident, minimalist, and visually pleasing clock illustration.
Brief
Draw a round wall clock face showing exactly 10:10: a circular dial with
all 12 hour numerals in the correct positions, minute tick marks, an hour
hand pointing just past 10, a minute hand pointing exactly at 2, and a thin
second hand pointing at 6. The word DRACON in small capitals under the
center. Dark rim, light face, flat vector style, centered composition.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
duo
39/50Judge reasoning
Brief fidelity (8/10)
All headline elements are present: dog driving with paws on handlebars, raccoon behind holding on, both leaning into a turn, sea spray, sun and clouds, distant shoreline, and bright cartoon vector style, though the jet ski itself is minimally rendered.
Construction (8/10)
Bodies and object parts are assembled from clear geometric shapes rather than melted or duplicated; the dog, raccoon, handlebars, and jet ski platform are anatomically coherent within the simplified cartoon style.
Composition (8/10)
The frame is cleanly divided between sky and water with a horizon shoreline, the duo is centered and readable, and the sun/clouds balance the upper space without crowding the subjects.
Detail finish (7/10)
The image uses flat bright colors with small touches like water lines, spray streaks, and cloud shapes, but lacks shading or texture depth and has a somewhat sparse finish.
Visual appeal (8/10)
The palette is coherent and cheerful, the linework is confident, and the simple vector style has charm, though it leans toward minimalism.
Brief
Draw a comical scene: a dog driving a jet ski at speed, paws on the
handlebars, with a raccoon riding behind holding on, both leaning into a
turn. Sea spray behind the jet ski, sun and clouds in the sky, a distant
shoreline. Both characters must be clearly doing something, not just
posed side by side. Bright cartoon vector style.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
monalisa
31/50Judge reasoning
Brief fidelity (6/10)
The seated woman, dark dress, folded hands, smile, and center-parted hair are present, while the three-quarter view, winding river, and bridge are only abstractly suggested.
Construction (7/10)
The face, hands, dress, and hair are distinct assembled shapes without melting or duplication, though highly simplified.
Composition (7/10)
The bust is centered with hands at the bottom edge and layered background hills and mountains filling the frame effectively.
Detail finish (5/10)
The artwork is flat and clean with minimal shading or texture, lacking the fine detail and finish implied by the brief.
Visual appeal (6/10)
The muted green, brown, and smoky blue palette is coherent, but the crude, icon-like execution limits its charm and visual impact.
Brief
Redraw the Mona Lisa as a flat vector portrait, bust composition: a seated
woman in three-quarter view facing the viewer, dark dress with neckline
detail, hands folded at the bottom edge, soft enigmatic smile, hair parted
in the middle. Behind her a hazy landscape with a winding river, a bridge
on the right, and blue mountains. Muted greens, browns, and smoky blues.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.
pelican
40/50Judge reasoning
Brief fidelity (9/10)
All requested elements are present: brown pelican in left-facing side profile, full body, large beak with throat pouch, long neck, folded wing suggested by feather lines, webbed feet on flat ground, sky and sea horizon, flat vector style with limited palette.
Construction (8/10)
The pelican is built from clean, distinct vector shapes, head, beak, pouch, neck, oval body, stick legs, and webbed feet, assembled without melting or duplication, though the neck-to-body joint is stylized.
Composition (7/10)
The bird occupies the left-center of the frame facing left into open sky, with a clear horizon dividing sky, sea, and ground; the composition is readable but leaves large empty areas and the subject looks toward the frame edge.
Detail finish (8/10)
Finishing touches include a drop shadow, eye highlight, beak detail, wing feather lines, and webbed feet, with no broken shapes or cut-off edges visible.
Visual appeal (8/10)
The limited palette of browns, blues, and tan is coherent, the linework is confident and flat, and the simple stylized pelican has modest charm.
Brief
Draw a brown pelican standing in side profile, facing left, full body visible:
large beak with throat pouch, long neck, folded wing, webbed feet on flat
ground. Simple sky background with a hint of sea on the horizon. Flat vector
style, clean shapes, limited palette.
Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external
references) inside a single ```svg fenced block. Short prose outside the
fence is allowed.