A fluent answer about Civilization VI does not show that a model has applied the game’s rules. A more useful test gives it an explicit city-state state, asks for legal actions, and checks the result against a validator. The protocol below separates state extraction, rule application, and explanation, with illustrative examples rather than an unsupported DeepSeek-versus-ChatGPT winner.
📌 Key Takeaways
- Mechanical Validation Over Conversational Fluency: Evaluating frontier LLMs on discrete game states requires testing exact schema conformance, mathematical threshold checks, and counterfactual sensitivity rather than qualitative prose.
- Reproducible Suzerain Evaluation: The proposed test checks whether DeepSeek and ChatGPT differ in rule application, tied envoy counts, and legal action ranking under the same stated game conditions. The worksheet illustrates how to evaluate those differences; it does not report measured model results.
- Production Verification Standard: Validate the JSON structure and game rules independently, record the model version and supported sampling settings, and repeat cases to check consistency. A fixed temperature or valid JSON does not guarantee a factual answer or eliminate hallucinations.
What a suzerain means in Civilization VI
In Civilization VI, a city-state records envoys sent by each civilization. The civilization with the most envoys, provided it reaches the required threshold, becomes the city-state's suzerain and receives its unique bonus. The official game manual describes the relationship and the three-envoy threshold; the exact bonus depends on the city-state and the ruleset in use. See the Civilization VI manual before treating a particular bonus as universal.
This is a useful AI test because the correct move is conditional. Sending an envoy can secure a science, culture, faith, trade, or military advantage, but it can also fail to change the suzerain if another player has more envoys. A good assistant must read the whole state, apply the threshold rule, and explain what would change the recommendation.
The question this comparison actually asks
“Which model is smarter?” is too vague to reproduce. A tighter question is:
Given the same city-state state, can DeepSeek and ChatGPT extract the rules, rank legal actions, preserve uncertainty, and change their recommendation when one fact changes?
That framing separates four abilities that are often mixed together:
- State extraction: copying envoys, era, quests, war status, resources, and rival counts without inventing values.
- Rule application: deciding whether the suzerain threshold and tie conditions are actually met.
- Action ranking: comparing “send envoy”, “save envoy”, “complete the quest first”, and “challenge the rival” against the stated goal.
- Calibration: lowering confidence when the game state is incomplete or a DLC rule is unknown.
Why token probability is not the same as game understanding
An autoregressive language model predicts a probability distribution over possible next tokens. Temperature changes the shape of that distribution: lower values concentrate probability around high-scoring tokens, while higher values make lower-scoring alternatives more likely. That mechanism is documented in AWS's inference-parameter guide.
Those token probabilities do not automatically equal probabilities of actions. The word “envoy” may be split into tokens, and the model may spend different numbers of tokens explaining each option. To compare decisions, first constrain the output to a fixed list of action labels, then normalize the returned scores or use a structured verdict. If the provider does not expose comparable log probabilities, report the ranking and confidence rubric instead of inventing numeric probabilities.
A safe output contract also prevents a fluent explanation from hiding a rule error. OpenAI's Structured Outputs documentation describes schema-constrained JSON; the same idea can be implemented with a validator around any provider's response.
A worked state: one fact changes the best move
The following state is deliberately small so another reader can reproduce it by hand. It is an editorial worksheet, not a claim about a live game save.
| Field | Value | Why it matters |
|---|---|---|
| City-state | Science city-state | The suzerain bonus supports a science-focused plan. |
| Your envoys | 3 | You meet the manual's threshold. |
| Rival envoys | 4 | You do not currently control the suzerainty. |
| Available envoys | 2 | Sending one still leaves a reserve for a new quest. |
| Quest | Build a campus | Completing it may award another envoy later. |
| Goal | Science victory, 20 turns | Weights durable science against immediate control. |
Under this state, “send one envoy now” is not automatically the best answer: it changes the count to 4–4 but may still leave a tie rule unresolved. A careful assistant should ask how the ruleset handles a tie, or recommend checking the city-state panel before spending the envoy. If the rival count changes from 4 to 2, the same action becomes much easier to justify. The point of the test is the counterfactual, not a dramatic winner.
Prompt before and after: from vague advice to an auditable decision
Weak prompt: “Should I send envoys to this city-state? Explain the best strategy.”
This invites a generic answer because the model has no fixed state, no legal action list, and no acceptance rule.
Controlled prompt:
Return exactly one JSON object. Do not add keys or markdown.
State: city_state_type=science; my_envoys=3; rival_envoys=4;
available_envoys=2; quest=build_campus; goal=science_victory;
turn_horizon=20; ruleset=record_if_unknown.
Actions: [send_one_envoy, save_envoy, complete_quest_first, inspect_city_state_panel].
For each action, record legal, expected_benefit, main_risk, confidence_0_to_1.
Choose one action only if the state is sufficient; otherwise choose inspect_city_state_panel.
Include a counterfactual: rival_envoys=2.The evaluator can then check whether the model preserved the numbers, selected one of the allowed labels, and changed its recommendation under the counterfactual. This is much more informative than judging whether the prose “sounds intelligent.”
How to compare DeepSeek and ChatGPT without a hype test
DeepSeek's
R1 release note
describes a reasoning model and the deepseek-reasoner API model.
ChatGPT is a product surface that can route among changing models, so a fair
API comparison must record the exact model identifier and date rather than
treating “ChatGPT” as a fixed engine.
| Control | Required setting | Why it prevents a false winner |
|---|---|---|
| Same state | One frozen JSON fixture per case | Neither model receives easier numbers. |
| Same actions | Four labels, identical spelling | Free-form prose cannot hide an omitted option. |
| Same task | State extraction, rule check, ranking, counterfactual | Measures reasoning steps separately. |
| Sampling | Temperature 0 for a baseline; three seeds for variance | Distinguishes deterministic errors from sampling noise. |
| Validation | JSON schema plus a rule-based checker | Rejects malformed or illegal actions automatically. |
| Reporting | Model ID, date, latency, tokens, failures, accepted cases | Makes the result auditable after models change. |
A small, reproducible evaluation protocol
Create 20 hand-checked states: ten where sending an envoy is clearly justified, five where saving is safer, and five where the correct response is to inspect missing information. Run each state with both providers and three fixed seeds: 120 primary requests. Keep the raw JSON, the returned model ID, the finish status, latency, and billed usage. Do not convert a refusal, timeout, malformed JSON, or wrong action into a silent retry; record it as a failure and count any retry separately.
| Metric | Definition | Acceptance rule |
|---|---|---|
| State accuracy | Correct fields ÷ fields supplied | At least 95% on the frozen fixture. |
| Rule accuracy | Correct threshold/tie interpretation ÷ cases | No invented bonus or missing condition. |
| Action validity | Allowed action labels ÷ responses | 100% after schema validation. |
| Counterfactual sensitivity | Recommendations changed when the decisive fact changed | Pass on every paired case. |
| Calibration | Confidence bucket vs observed correctness | Report counts; do not claim calibration from five examples. |
These thresholds are editorial acceptance rules, not published model scores. A real result belongs in the article only after the raw fixture and evaluator are available for inspection.
What this method can and cannot establish
- It can show whether a model copies the supplied state, applies a documented threshold, respects a closed action set, and reacts to a counterfactual.
- It cannot prove that a model understands every Civilization VI system, plays a complete campaign, or generalizes to a modded ruleset.
- It cannot turn a worksheet probability into a calibrated forecast unless many comparable cases and proper scoring rules are used.
- It cannot compare subscription prices with API costs; record the dated API rate card and the actual usage export for any cost claim.
TechTrendInsight verdict
DeepSeek and ChatGPT should be judged on the same frozen game states, the same legal actions, and the same validator. A fluent answer that misses the envoy count is a failure; a cautious “inspect the panel first” can be the strongest answer when a decisive fact is absent.
My conclusion is therefore conditional: use the model that produces the lowest cost per accepted, rule-consistent decision for your fixtures. Do not call one model the winner because it writes a longer explanation or because a single prompt feels persuasive.
FAQ
Does the article claim that DeepSeek or ChatGPT can play Civilization VI autonomously?
No. The experiment is a decision-audit worksheet. It tests state reading and action ranking; it does not connect an LLM to the game client or authorize moves.
Why show probabilities if they are illustrative?
They make the decision tree explicit. Unless comparable log probabilities or repeated outcome frequencies are available, the numbers must remain labeled as worksheet values.
What should I record when a model changes its answer?
Keep the original state, the changed field, the exact model ID, sampling settings, raw JSON, validator result, and timestamp. That record is more valuable than a screenshot of a polished explanation.
Official and primary sources
- Civilization VI official manual — city-state envoys and suzerain threshold.
- DeepSeek-R1 release note — official model and API naming.
- OpenAI Structured Outputs — schema-constrained responses.
- Inference parameters and temperature — probability-distribution behavior.
Original publication: February 7, 2025. I updated this article on September 14, 2026 to document the LLM-focused test and preserve a reproducible evaluation record.
Editorial record
Editorial update — September 14, 2026: I revisited this comparison from an LLM perspective and rebuilt it around a concrete question: can a language model represent a Civilization VI city-state state, apply the suzerain rules, rank legal actions, and explain uncertainty?
I added a reproducible prompt, a constrained JSON output contract, an illustrative probability tree, and an evaluation worksheet. The examples show what can be checked mechanically—state extraction, rule application, counterfactual changes, and confidence—without treating a fluent answer as proof of game understanding. Any benchmark result should be published only with the raw requests, model identifiers, and validator output.