From task file to decision.

A task.json file describes everything: the prompt variants you want to compare, the quality checks every answer must pass, and a handful of sample inputs with real variable values.

This example tests product-description generation in two variants — a short prompt and one that also names the price. Five checks guard quality: two length limits, two banned phrases, and a judge model that scores each answer on a rubric.

{
  "name": "Judge-Test: Produktbeschreibung",
  "options": { "temperature": 0, "num_predict": 200 },

  "variants": [
    {
      "name": "kurz",
      "system": "Du schreibst prägnante Produktbeschreibungen für einen Online-Shop. Antworte auf Deutsch.",
      "user": "Schreibe eine Produktbeschreibung für: {{produkt}}\n\nAnforderungen:\n- 2–4 Sätze\n- Erwähne den Hauptvorteil\n- Keine Superlative wie 'das beste' oder 'einzigartig'"
    },
    {
      "name": "mit-preis",
      "system": "Du schreibst prägnante Produktbeschreibungen für einen Online-Shop. Antworte auf Deutsch.",
      "user": "Schreibe eine Produktbeschreibung für: {{produkt}} (Preis: {{preis}})\n\nAnforderungen:\n- 2–4 Sätze\n- Erwähne den Preis und den Hauptvorteil\n- Keine Superlative"
    }
  ],

  "checks": [
    { "type": "minChars", "value": 80,    "name": "mindestens 80 Zeichen" },
    { "type": "maxChars", "value": 600,   "name": "maximal 600 Zeichen" },
    { "type": "notContains", "value": "einzigartig", "name": "kein 'einzigartig'" },
    { "type": "notContains", "value": "das beste",   "name": "kein 'das beste'" },
    {
      "type": "judge",
      "model": "qwen3:30b-instruct",
      "rubric": "Ist die Beschreibung auf Deutsch verfasst? Ist sie sachlich und vermeidet sie Superlative? Wird ein konkreter Vorteil des Produkts genannt? Ist die Länge angemessen (2-4 Sätze)?",
      "minScore": 0.7,
      "name": "Qualitätsprüfung (Judge)"
    }
  ],

  "samples": [
    { "vars": { "produkt": "Mechanische Tastatur mit Cherry MX Brown Switches", "preis": "89 €"  } },
    { "vars": { "produkt": "Stehpult-Aufsatz höhenverstellbar 60–120 cm",       "preis": "149 €" } },
    { "vars": { "produkt": "USB-C Dockingstation mit 4K HDMI, 100W PD",         "preis": "79 €"  } }
  ]
}

variants

Each variant is one version of your prompt. openmodelbench runs all samples through all variants and measures them separately, so you can compare e.g. a short prompt against a detailed one.

checks

minChars / maxChars and notContains are instant rule checks. judge sends each answer to a second model that scores it against your rubric — 0.0 to 1.0. Only answers above minScore count as passing.

samples

Real inputs with concrete variable values. The tool renders {{produkt}} and {{preis}} into the prompt at run time. Use enough samples to catch edge cases — one sample is noise.

Point the CLI at your task file, name the model you want to test, and tell it your production target — how many jobs need to finish within what time window.

openmodelbench run task.json \
  --model qwen2.5:3b \
  --batch-jobs 1000 \
  --window-hours 8

--batch-jobs & --window-hours

These two flags turn raw throughput into a yes/no answer. Here: 1 000 jobs in 8 hours = 125 jobs/h required. The report immediately flags any variant that falls short.

Ollama must be running

openmodelbench talks to Ollama on localhost:11434. Pull the model first (ollama pull qwen2.5:3b) and the CLI handles the rest.

The run took about 4 minutes on an AMD EPYC 7543P with 8 threads. Output is a self-contained HTML report saved next to your task file.

The report opens in any browser — no server needed. It shows the throughput of each variant, whether the quality gate passed, how long the full batch would take, and the individual model answers with their judge scores. You can rate answers Good / Bad; ratings are stored locally in the browser.