An LLM comparison is useful when another person can inspect the same task, outputs, scoring rules, and failure cases. This guide builds that record around a reproducible prompt set, a transparent score matrix, and separately tracked time and cost. The worked numbers are illustrative: they demonstrate a method for your own evaluation, not a ranking from completed paid-model trials.
Comparing ChatGPT, Claude, Gemini, or another model is often presented as a race with one winner. That framing breaks as soon as the task changes. A model that finds current sources quickly may be awkward for a controlled rewrite; a model that explains code well may need more manual fact checking. The useful unit of comparison is therefore a verified workflow: a task, an output, the evidence behind it, and the time required to trust it.
Key Takeaways
- There is no permanent “best” model. A useful comparison fixes the task set, model version, date, and scoring rubric.
- Run the same six prompts and keep raw outputs, source links, failures, and review minutes.
- Separate answer quality from evidence quality, privacy, latency, and total cost.
- Report worked numbers as a worksheet or spot test unless the dataset and protocol support a broader claim.
Why a leaderboard is not an experiment
Model names and subscription limits change. A screenshot of one impressive answer does not say which version ran, what system instructions were active, whether tools or browsing were enabled, or how many failed attempts were discarded. Official evaluation guidance emphasizes task-specific success criteria, repeatable cases, and calibration against human judgment. OpenAI evaluation best practices and Anthropic evaluation guidance provide useful starting points. That principle applies across providers: define the job before choosing the model.
A run needs a passport, not just a score
Task ID · exact prompt · context snapshot · expected behavior · run date
Model/product version · tool access · controls · limits · retry policy
Every raw response · tool errors · citations · latency · charges
Rubric version · score rationale · verifier · minutes · rejected cases
Freeze the comparison before running it
Write a small test card for every task. Record the model identifier, date, temperature or equivalent controls, tool access, context files, and output limit. Use the same prompt and context for each candidate. If a provider cannot expose a setting, record that limitation rather than pretending the runs are identical.
| Task | Prompt held constant | What to measure | Illustrative review time |
|---|---|---|---|
| Current research | Question, date window, source rule | Supported claims and freshness | 8 min verification |
| Code explanation | Language, runtime, input/output | Runnable path and edge cases | 6 min verification |
| Long-context reading | Same document and questions | Recall, citations, omissions | 10 min verification |
| Structured extraction | Same schema and examples | Valid JSON and missing fields | 5 min verification |
| Editorial rewrite | Audience, tone, word budget | Instruction fit and edit distance | 4 min verification |
| Fact check | Claim list and cutoff date | Primary-source support | 9 min verification |
The review times in this table are synthetic, not observations. The six task types above are deliberately mixed. Research tests source handling; code tests executable reasoning; long context tests recall and omission; extraction tests structure; rewriting tests editorial control; and fact checking tests whether uncertainty is visible. Six examples are a smoke test, not evidence of a stable model ranking. Expand each important task type with representative, edge, and failure cases, and keep a held-out set separate from prompt tuning.
A scoring rubric that exposes the trade-offs
Score each response before looking at the model name when possible. A 0–2 scale is easy to audit: “2” means the stated pass condition is met, not that the prose feels impressive. Keep the raw answer and one sentence of justification beside every score.
Keep the six raw rows visible
| Task | Fit | Source | Edges | Style |
|---|---|---|---|---|
| Research | 2 | 1 | 1 | 2 |
| Code | 2 | 2 | 2 | 2 |
| Long context | 1 | 1 | 1 | 2 |
| Extraction | 2 | 2 | 2 | 2 |
| Rewrite | 2 | 2 | 1 | 2 |
| Fact check | 2 | 1 | 1 | 1 |
0 = fail · 1 = partial · 2 = complete. Fit = instruction fit; Source = source traceability; Edges = edge cases; Style = writing control. Six scores per column; means: 1.83, 1.50, 1.33, 1.83. No provider was tested to produce these values.
| Dimension | 0 — fail | 1 — partial | 2 — complete | Evidence to save |
|---|---|---|---|---|
| Instruction fit | Required format or constraint is missing. | The main task works but needs edits. | All explicit constraints are met. | Prompt and raw answer |
| Source traceability | No usable source. | Indirect or incomplete link. | The required source or supplied input supports the claim. | URL or input location and checked statement |
| Reasoning / edge cases | Example breaks or hides uncertainty. | Works on the happy path. | Limits and failure cases are explicit. | Test input and result |
| Writing control | Tone or structure is unusable. | A human must heavily rewrite it. | Audience, length, and format fit. | Before/after edit notes |
Illustrative results: what the numbers do and do not say
The matrix uses invented integer scores to demonstrate a review sheet. No provider was run to produce them. The column totals are 11, 9, 8, and 11; dividing each by six gives 1.83, 1.50, 1.33, and 1.83 after rounding. The earlier chart included averages that could not be obtained from six integer 0–2 scores at the displayed precision. Keeping the raw rows visible makes this kind of mistake detectable.
The first response is only the beginning of the work
| Task | Illustrative review time | What the reviewer checks |
|---|---|---|
| Research | 8 minutes | Open primary sources and check claims. |
| Code | 6 minutes | Execute the stated example and edge cases. |
| Long context | 10 minutes | Locate supporting passages and omissions. |
| Extraction | 5 minutes | Validate fields, types, and missing values. |
| Rewrite | 4 minutes | Check constraints and semantic preservation. |
| Fact check | 9 minutes | Match every claim to dated evidence. |
Total: 42 minutes. No actual model run was timed for this worksheet.
Verification time is part of workflow cost. In this hypothetical worksheet, fact checking takes 9 minutes and long-context review takes 10 minutes, even when the first answer arrives quickly. A low subscription price can still produce a high cost per publishable result if editors must repair citations, rerun code, or rewrite the structure.
Source traceability need not mean an external link for every task. In a rewrite, it can mean that the revised passage preserves the supplied facts. If a dimension genuinely does not apply, record N/A and exclude it from that dimension’s denominator; do not silently score it as zero or full credit. The displayed example assumes all four dimensions apply to each task.
Two test cards you can actually run
A list of task categories is not a reproducible dataset. The cards below provide small synthetic inputs and expected checks. They are starter cases, not a complete evaluation suite.
CASE extract-01
Input: Item A: 3 units at USD 4.50 each. Shipping is not stated.
Task: Return JSON only: item, quantity, unit_price_usd,
line_total_usd, shipping_usd. Use null for missing data.
Expected: {"item":"A","quantity":3,"unit_price_usd":4.5,
"line_total_usd":13.5,"shipping_usd":null}
Fail: invented shipping, wrong arithmetic, extra prose, invalid JSON.
CASE rewrite-01
Input: The trial includes 12 fictional records. Two lack dates.
The report excludes those two from the dated analysis.
Task: Rewrite in two plain-English sentences, under 40 words.
Preserve every quantity and the exclusion rule. Add no claims.
Checks: two sentences; <40 words; 12 total; 2 undated;
exclusions preserved; no invented performance conclusion.Fix the scoring rule before viewing candidate outputs. For the extraction case, compare parsed field values and types, not merely string formatting. Add variants with zero quantity, a missing price, conflicting amounts, and an untrusted instruction embedded in the input. Keep those variants distinct so a failure can be traced to a particular requirement.
Make the test executable and auditable
A spreadsheet is enough to start. The following pseudocode shows the control points; it is a template, not an implementation or a claim that every provider exposes the same API. Freeze model/tool configurations and a retry budget, retain failed attempts, and have the reviewer verify claims independently rather than ask the candidate to grade itself.
for task in fixed_tasks:
for model in models:
output = run(model, task.prompt, task.context)
score = rubric(output, task.criteria)
evidence = verify_claims(output, task.source_rule)
log(task.id, model.id, score, evidence, minutes_spent)
Do not hide failures. Log rejected outputs, tool errors, missing citations, and retries. A model that needs one retry for a rare edge case may be preferable to a model that returns a polished but unsupported answer. The log is also what lets you repeat the test after a model update. Repeat nondeterministic cases, vary candidate presentation order when comparing responses, and record judge disagreements. Calibrate any model-based grader against human-reviewed examples; a confident automated score is not a ground-truth label.
What this comparison can and cannot prove
- It can show, once actual runs are collected: which candidate fits the selected tasks, how much evidence checking costs, and where a prompt or tool change improves the result.
- It cannot show: a universal intelligence ranking, a permanent price/performance winner, or a guarantee that future model versions behave the same way.
- It should not expose: confidential documents or personal data merely to make a benchmark look realistic.
- It should report: the test date, versions, tool settings, sample size, rubric, failed cases, and any provider-specific limitation.
Convert time to money; keep risk separate
Calls including retries: $3
Tool charges: $1
Review: 42 minutes at $30/hour
$25 total$3 + $1 + (42 ÷ 60 × $30).
First: required privacy and safety conditions.
Next: task-specific quality and evidence.
Then: cost per accepted result and latency.
A serious failure cannot be averaged away.
When to use a comparison service—and when not to
Use a comparison service when it exposes the prompt, model identifier, raw output, scoring method, and source links. It can reduce setup time and make repeated spot tests easier. Treat an opaque score as a lead, not evidence. Build your own small worksheet when the workload contains private context, domain-specific definitions, or a strict publication checklist; those details rarely fit a generic leaderboard.
Editorial Verdict
Stop asking which model is smartest in the abstract. Choose the workflow that produces a verifiable result for the job in front of you, and publish enough of the method that another reader can repeat it. Transparent evidence, failed cases, and review time create more value than a confident ranking.
Frequently Asked Questions
How many prompts should I use?
Six is enough to start checking your process, not enough to establish broad superiority. Add multiple cases for important tasks, repeat variable runs, and include failures and rare but consequential edge cases. Report sample counts rather than inventing statistical confidence.
Should style and factual accuracy share one score?
Keep them separate. Define hard failures before testing, such as invented evidence or unauthorized data disclosure. A polished style or low cost should not compensate for failure of a required safety or correctness condition.
Can I call this worksheet a benchmark result?
Not as published here: its numerical examples are synthetic and no provider run produced them. For an actual benchmark, publish enough of the dataset, configurations, scoring method, raw outcomes, and limitations to support the claim. Protect confidential data while doing so.
Can another LLM judge the answers?
It can assist with a clear rubric, but check agreement with human reviewers and save its reasons. Blind candidate names when feasible, vary order, inspect inconsistent cases, and avoid treating agreement between two models as proof that a factual claim is true.
Is a chat product comparable to an API model?
Only if the claim reflects the actual configurations. Browsing, retrieval, system instructions, memory, file tools, and output limits can change the workflow. If you cannot hold them constant, compare the products as configured and disclose the differences rather than attributing everything to the underlying model.
How should I calculate the total cost?
Add charges in the same currency, include retries once, and convert review minutes using a stated labor rate. Divide by accepted outputs only if acceptance is defined and nonzero. Report latency and risk separately; neither is a dollar amount without an explicit, justified conversion.
Official Sources
- OpenAI evaluation guidance — datasets, graders, and repeatable evaluation cycles.
- Anthropic: Define success criteria and build evaluations — define success before comparing outputs.
- Google Gemini API documentation — model and API behavior must be checked against the current version.
Prices, model availability, and API behavior change. Evaluation-method guidance was rechecked on September 19, 2026. This is a provider-neutral workflow, not a dependency on a particular hosted evaluation platform. Recheck product availability and interfaces before implementation.
Editorial record
Editorial correction — September 19, 2026: Replaced inconsistent averages with a transparent six-row score matrix, explicitly labeled every number as illustrative, corrected the cost units, and added runnable test inputs and an expanded FAQ. Original publication date retained; no paid model benchmark was run.