Skip to main content

← Dracon Bench leaderboard

Model guide · Dracon Bench

nemotron-3.5-lightning-30b-a3b

Tested First seen 2026-11-18 (first probed by us) · First tested 2026-11-18 · NVIDIA NIM free endpoint (integrate.api.nvidia.com/v1)

Nemotron 3.5 Lightning 30B-A3B on NIM spends much of its budget on reasoning: MMLU-Pro landed 1/10, IFEval-50 80%, IFBench 60%, GSM8K 70%, JSON 4/5, and SVG 55/150 from three admissible artifacts; two briefs produced nothing scoreable. Overall 56.1 across six suites, rank 15. Disclosure: 14,654 reasoning tokens were spent on IFEval alone. Pilot depth: 10 prompts per text suite and the 5 SVG briefs; every suite prints its own n and not_delivered.

Rank
#15
Overall
56.1
Suites
6 of 6
Ranked
#15 of 21

When the world heard about it

Timeline

  1. First reachable for us 2026-11-18 our first probe
  2. First tested by us 2026-11-18 Dracon Bench
  3. Officially announced 2026-08-11 llm-stats
  4. First official benchmark 2026-11-19 llm-stats: a public score existed by then — the first day we saw one
  • Reachable before we scored it same day
  • Announced before it reached the lane we served 99 days
  • A public score existed within 1 day

Track A · script-scored subsets

Scores

Subset scores for nemotron-3.5-lightning-30b-a3b.
SuiteScoreSamplePrecision
IFEval-50 prompt-strict80.0%8/10 prompts49.0%–94.3%
IFEval-50 instruction-strict70.6%12/17 constraints46.9%–86.7%
IFBench-50 prompt-strict60.0%6/10 prompts31.3%–83.2%
IFBench-50 instruction-strict54.5%6/11 constraints28.0%–78.7%
MMLU-Pro-5010.0%1/10 served1.8%–40.4%
JSON briefs conformance80.0%4/5 served37.6%–96.4%
GSM8K-5070.0%7/10 served39.7%–89.2%
SVG blind rubric55/150blind rubricjudge band ±8.3 pts

The range beside a script-scored figure is a 95% interval over that figure's own sample: the prompts it followed, the constraints it met, or the items it answered. The instruction-strict figures are measured over CONSTRAINTS, which is why their sample line counts constraints rather than prompts. The judged total is not a proportion, so it has no interval: it carries the measured judge band instead: how far a re-judging of the same artifacts moved them, which is a different kind of number. The method page says what each one covers and what it does not.

Track A · per-type breakdown

What it failed

The instruction types these two suites check, and the checks the model missed. A type is checked once per prompt that carries it, so a type with seven checks was asked for seven times. The counts beside a type are that type's own; the suite's totals are in the scores table above. These are the failures behind the instruction-strict figure, which is the sum of them.

IFEval-50

Missed at least one check in 4 of the 15 instruction types checked.

The instruction types 4 of 15 the IFEval-50 suite checked that nvidia/nemotron-3.5-lightning-30b-a3b missed, most missed first.
Instruction typeChecksMissed
startend · quotation22
change case · english capital11
detectable content · postscript11
keywords · frequency11

IFBench-50

Missed at least one check in 4 of the 8 verifiers checked.

The verifiers 4 of 8 the IFBench-50 suite checked that nvidia/nemotron-3.5-lightning-30b-a3b missed, most missed first.
VerifierChecksMissed
count · unique word count32
count · word count range11
format · list11
words · keywords specific position11

Provider-reported usage

Thinking cost

What the run cost the model in thinking: provider-reported completion tokens per prompt, the thinking share, and the cost per correct answer. Reasoning-heavy models can exhaust fixed budgets before answering.

Thinking cost for nemotron-3.5-lightning-30b-a3b.
SuiteCostReading
IFEval-502,025/prompt80% thinking· 2,278/correct
GSM8K-50455/prompt77% thinking· 520/correct

Track B · blind-judged

Artifacts

Blind-judged by nvidia/nemotron-3.5-super-vl:free: the judge saw the brief and the rendered image only, never the model name. Briefs are frozen; expand one to read it.

beach

21/50
Judge reasoning
Brief fidelity (3/10)
The sun, sea, sand, and a crude sailboat appear, but the two dolphins, distinct birds, recognizable palm tree, and starfish are missing or illegible, so headline elements are absent and the score is capped.
Construction (5/10)
All visible objects are simple geometric primitives (circles, triangles, lines) that are cleanly assembled with no melted or duplicated parts.
Composition (4/10)
The frame is divided into sky, sea, and sand with the sun centered and a sailboat on the right, but the palm tree is cut off by the left edge and perspective is essentially flat.
Detail finish (4/10)
The gradient sky and white wave lines provide basic finish, but there are no textures, shading, or legible small touches beyond a tiny yellow starfish blob.
Visual appeal (5/10)
The warm-to-teal palette is coherent and the minimal linework is confident, though the extreme abstraction limits charm.
Brief

Draw a beautiful sunset beach scene: a large sun sitting low over the sea with a shimmering reflection path on the water, a leaning palm tree silhouette on the left, two dolphins arcing out of the waves, a small sailboat on the right, birds in the sky, sandy foreground with a starfish. Warm oranges and pinks fading to teal. Coherent perspective: distant objects smaller and hazier than near ones. Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external references) inside a single ```svg fenced block. Short prose outside the fence is allowed.

clock

unjudged

no svg

Brief

Draw a round wall clock face showing exactly 10:10: a circular dial with all 12 hour numerals in the correct positions, minute tick marks, an hour hand pointing just past 10, a minute hand pointing exactly at 2, and a thin second hand pointing at 6. The word DRACON in small capitals under the center. Dark rim, light face, flat vector style, centered composition. Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external references) inside a single ```svg fenced block. Short prose outside the fence is allowed.

duo

18/50
Judge reasoning
Brief fidelity (3/10)
The sun, clouds, and distant shoreline are present, but the dog, raccoon, jet ski, paws on handlebars, and leaning-into-a-turn poses are not clearly recognizable, so key headline elements are missing.
Construction (3/10)
The figures are reduced to abstract circles and the jet ski to basic geometric blocks, lacking assembled anatomy, faces, paws, or coherent object parts.
Composition (4/10)
The subject is tiny and pushed to the lower-left corner with vast empty sky above, resulting in poor use of the frame and weak perspective.
Detail finish (4/10)
The image uses flat colors with no shading, textures, or small finishing touches, though the markup appears intact with no obvious cut-off edges.
Visual appeal (4/10)
The palette is simple and coherent, but the extreme abstraction and emptiness limit charm and visual impact.
Brief

Draw a comical scene: a dog driving a jet ski at speed, paws on the handlebars, with a raccoon riding behind holding on, both leaning into a turn. Sea spray behind the jet ski, sun and clouds in the sky, a distant shoreline. Both characters must be clearly doing something, not just posed side by side. Bright cartoon vector style. Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external references) inside a single ```svg fenced block. Short prose outside the fence is allowed.

monalisa

16/50
Judge reasoning
Brief fidelity (3/10)
The image includes a face, enigmatic smile line, dark dress shape, and background landscape with river and mountains, but lacks the folded hands, bridge on the right, neckline detail, and clear three-quarter view, so major omissions keep this low.
Construction (2/10)
The figure is assembled from crude geometric primitives, an oval face with no eyes or nose, triangular hair, and a black arch for a dress, with no anatomical assembly of hands, arms, or facial features.
Composition (4/10)
The layout is centered and uses the full frame, but the oversized head relative to the body and flat layering make the perspective and relative sizes feel unbalanced and simplistic.
Detail finish (3/10)
The artwork is flat with no shading or textures, contains only a single smile line as a small touch, and shows no obvious broken markup or cut-off edges, but lacks nearly all requested detail.
Visual appeal (4/10)
The muted blue-green-brown palette is coherent and the minimalist shapes have slight charm, but the extreme abstraction makes it unlikely to be called good as a portrait.
Brief

Redraw the Mona Lisa as a flat vector portrait, bust composition: a seated woman in three-quarter view facing the viewer, dark dress with neckline detail, hands folded at the bottom edge, soft enigmatic smile, hair parted in the middle. Behind her a hazy landscape with a winding river, a bridge on the right, and blue mountains. Muted greens, browns, and smoky blues. Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external references) inside a single ```svg fenced block. Short prose outside the fence is allowed.

pelican

unjudged

no svg

Brief

Draw a brown pelican standing in side profile, facing left, full body visible: large beak with throat pouch, long neck, folded wing, webbed feet on flat ground. Simple sky background with a hint of sea on the horizon. Flat vector style, clean shapes, limited palette. Output: exactly one self-contained SVG (viewBox "0 0 800 600", no external references) inside a single ```svg fenced block. Short prose outside the fence is allowed.

Head to head

Compare with

Run record

Evidence and notes

  • NVIDIA NIM free endpoint (integrate.api.nvidia.com/v1), same key and base as every NIM row
  • Reasoning-heavy: 14,654 reasoning tokens on IFEval alone
  • Only 3 of 5 SVG briefs produced an admissible artifact