Open source · Ollama · CPU-first

Does your LLM batch need a GPU?

Compare local models on your own task. Find out what your CPU server can finish in the time you have.

$ npx openmodelbench run \
    --task task.json \
    --models "qwen2.5:3b,gemma3:4b" \
    --batch-jobs 5000 --window-hours 8

Your task. Your server. Your time window.

Three steps to a decision.

Define your task

Describe it once: prompt variants, sample inputs and simple checks.

Compare models

Every model runs the same inputs. Speed comes from Ollama's own timers.

Check your window

Measured capacity, with a safety margin, against the batch you need to finish.

A decision, not just a benchmark.

Real run — AMD EPYC 7543P, 8 threads, 31 GiB RAM · Task: product description generation · 1,000 jobs / 8 h window · needs 125 jobs/h

Model	Prompt variant	Jobs/h	Quality	Verdict
qwen2.5:3b	short	121	100%	Too slow
qwen2.5:3b	with price	104	67%	Quality gate failed

Required: 125 jobs/h · neither variant fits — this batch needs faster hardware or a smaller model.

See the task file, command and full report →

Checks verify rules.
You judge answers.

Automated checks catch format errors, missing fields and obviously wrong answers.

Quality of a translation, summary or classification needs your eyes. The report lists all answers side by side with Good/Bad buttons.

Short benchmarks are optimistic. Use three or more rounds, keep the safety margin and run a longer test before you commit.

Run it on the machine that will do the batch.

Node.js 18+ and a running Ollama server
No package dependencies
Native Ollama API only in this version
Output: report.html, report.md, results.json

# create an example task
npx openmodelbench init task.json

# pull the models to compare
ollama pull qwen2.5:3b
ollama pull gemma3:4b

# run the comparison
npx openmodelbench run --task task.json \
  --models "qwen2.5:3b,gemma3:4b" \
  --concurrency 1,2 --rounds 3 \
  --batch-jobs 5000 --window-hours 8