Bench method
Dracon Bench estimates how competent a free AI model feels in real use: instruction following, reasoning, and visual construction. Suite 1.0, frozen 2026-11-11. Everything below is versioned; nothing is edited in place.
Track A: established subsets, script-scored
| Subset | Source | Sample | Scorer |
|---|---|---|---|
| IFEval-50 | google/IFEval train (541 rows, Apache-2.0) | 50 rows, seed 20261109, IDs published | Port of the 25 official instruction verifiers; strict prompt-level and instruction-level accuracy |
| GSM8K-50 | openai/grade-school-math test (1319 rows) | 50 rows, seed 20261109, IDs published | Last-number exact match against the #### answer |
Every model answers at temperature 0 with one fixed system prompt. Empty, refused, and truncated responses count as failures. Subset scores are always labelled as subsets, never as official full-dataset scores. Three documented approximations in the IFEval port: sentence splitting (no NLTK punkt in JS), script-based language detection (no langdetect), and word tokenising. Each is pinned by unit tests.
Track B: frozen art briefs, blind-judged
Five frozen briefs (pelican, monalisa, duo, clock, beach) ask for a single self-contained SVG. Subjects are iconic on purpose: the visitor brings their own frame of reference. The judge sees the brief and the rendered image only, never the model name, and runs at temperature 0 against a published 5-criterion rubric (brief fidelity, construction, composition, detail and finish, visual appeal), 10 points each. Judge for v1.0: nvidia/nemotron-3.5-super-vl:free. Re-judging the same artifact moves about 5 points, so close totals are ties and every artifact is stored next to its score for audit.
Efficiency
Completion and reasoning token counts come from provider-reported usage on every record. The efficiency table shows tokens per prompt, the thinking share, and tokens per correct answer. It measures what the run cost the model, not billable price (all lanes here are free).
Calibration
Flint IFEval-50 prompt-strict (93.3%) lands within 2 points of the vendor-published full IFEval figure (95.1%), which validates the harness on a 50-prompt sample. Nemotron GSM8K-50 (78.0% zero-shot) sits below the vendor eight-shot chain-of-thought figure (88.1%), lower as expected for zero-shot. Dots IFEval/GSM8K and Flint GSM8K have no published reference found and stand as novel numbers.
Integrity
Bad runs are re-run and logged, never silently dropped: the bench notes on the leaderboard record dead lanes, provider refusals, cooldowns, and harness fixes with dates. Every result pins model id, endpoint, and timestamp because free lanes get re-routed. A rubric or judge change re-scores every artifact and bumps the bench version.
Version history
- 1.0 (2026-11-10): IFEval-50, GSM8K-50, five SVG briefs, blind rubric judging. Inputs sealed with SHA manifest.