AI Technical Articles
Google MKA Explained: Testing AI on Visual Work
The most useful thing about Google DeepMind’s new Multimodal Knowledge Agent benchmark is not its leaderboard. It is the harder question behind the test: Can an AI inspect the files that contain the evidence, reason across them, and produce the finished artifact you asked for?
That distinction matters when the source material includes architectural drawings, maps, charts, scanned documents, spreadsheets, schematics, or other files where converting everything to text can discard relevant information.
Google DeepMind developed MKA with Artificial Analysis and announced it on November 10, 2026. The benchmark contains 200 private tasks across 10 industry verticals, created with domain experts representing 83 occupations. A separate set of 10 public tasks lets outsiders inspect examples of the benchmark format. Tasks require work across visually complex source files. The technical paper reports that 176 of the 200 tasks (88%) require a standalone file; the other 24 can be answered in the final text response. Outputs include spreadsheets, presentations, reports, and marked-up drawings. Google DeepMind describes the benchmark and its design.
For buyers evaluating AI for document-heavy work, that suggests a more useful question than which model ranks first: Does the evaluation resemble the complete job you intend to delegate?
What MKA actually tests
Three terms help explain the difference.
Multimodal AI processes information beyond plain text. In MKA, visual processing matters because relevant evidence can be encoded in geometry, layout, markings, charts, drawings, and relationships among elements on a page.
An AI agent does more than answer one isolated prompt. Artificial Analysis runs MKA models in its open-source Stirrup agent harness, which provides a Linux sandbox, code execution, internet access, web search and fetching, and an image-viewing tool for models that accept image input. Each task permits up to 250 agent turns. Artificial Analysis documents the evaluation environment.
A benchmark is a standardized evaluation designed to make results comparable under defined conditions. Here, the agent receives a workspace containing source files and a specified output requirement. Most tasks require a file rather than only a conversational answer. The technical paper lists formats including .xlsx, .pptx, .pdf, .docx, .html, or .png files.
That last requirement is important. Knowing an answer and delivering usable work are different capabilities.
MKA grades submissions against weighted expert-written criteria. Its Weighted Satisfaction Rate, or WSR, is the weighted share of criteria a submission satisfies. Criteria are divided into Regression checks for baseline competence and harder Climb checks intended to distinguish stronger systems. The headline leaderboard uses Climb WSR. The evaluator explains these score definitions.
For a small hypothetical example, suppose a task has four checks with weights 4, 3, 2 and 1. Passing the first and third earns 6 of 10 available weight units, or 60% WSR. That is not 60% of tasks completed. MKA calculates the relevant weighted score for each task and then averages across tasks, so a headline score also hides which requirements failed.
Google’s announcement describes roughly 70 yes-or-no checks per task, weighted 75% on factual accuracy and 25% on visual design. The automated grader can open and render submitted files and inspect details such as spreadsheet formulas.
The technical paper’s 50-task grader study reports 96.2% agreement with domain experts on criterion judgments. The study focused on high-weight checks and a balanced mix of passes and failures.
That supports agreement within the authors’ study. It does not establish the same reliability for low-weight formatting checks or an organization’s own documents and quality-control requirements.
The architectural example shows why text extraction is not enough
The public drawing-coordination example illustrates the central problem.
An architectural agent must work across floor plans, building sections, elevations, and a review standard, then produce an exception-focused PowerPoint audit. Findings have to distinguish between a mismatch and something that cannot be determined, with evidence from the relevant drawings.
Consider what happens if text extraction successfully recovers a window-width label. That alone may not establish where the window sits or whether its representation is consistent across a plan, section, and elevation. Those relationships can depend on visual structure rather than the extracted text alone.
The deliverable matters too. An agent that identifies a correct dimension but fails to organize the exceptions and their supporting drawing evidence has not completed the same job.
The point is not that every evaluated model necessarily makes those mistakes. It is that MKA’s task design can penalize failures that a text-only question-and-answer benchmark might never expose.
The public examples are available through the MKA dataset on Hugging Face.
Does MKA really make vision matter?
Google’s technical report includes a useful ablation experiment, meaning researchers deliberately removed one capability to measure its effect.
When direct image viewing was removed from Gemini 4 Carbon, its MKA Climb WSR fell by about 5.9 percentage points. The image-disabled agent also used 1.28 times as many turns and 1.40 times as many output tokens. The same experiment found no statistically meaningful change in the model’s overall GDPval score. See Table 5 of the technical report.
The appropriate conclusion is narrow: MKA was more sensitive to direct visual inspection than GDPval for this model under this experiment.
That is meaningful evidence that MKA makes vision consequential. It is not evidence that every workplace process needs direct image reasoning, nor does it show that visual reasoning alone is sufficient to complete these tasks.
A leaderboard score is not an automation percentage
As checked on November 11, 2026, Artificial Analysis reports rounded Climb WSR scores of 57% for Claude Fable 5.2, 54% for Gemini 4 Carbon, and 53% for Claude Opus 5.5. The listed configurations are Max with Default Fallback for the Claude models and High for Carbon. The paper reports that these top three are statistically indistinguishable on Climb WSR under its task-level uncertainty analysis, so small rank differences deserve caution.
Those numbers do not mean that 57% of a profession can now be automated.
Climb WSR is the mean weighted share of the harder rubric criteria satisfied across MKA tasks under the benchmark’s specified agent environment. Models receive tools, a sandbox, and as many as 250 turns. A company’s actual environment may provide different software, permissions, templates, source material, or review procedures.
Cost figures need the same caution. Artificial Analysis’s launch results report average model-call costs of $31.97 per MKA task for Claude Fable 5.2 and $6.29 for Gemini 4 Carbon. Its cost measure excludes judge calls; it also does not cover a company’s integration, supervision or rework. These are Artificial Analysis’s evaluation costs, not the separate cost figures in Google’s paper.
Performance also varies by field. Artificial Analysis reports that no single evaluated model leads all 10 industry verticals.
There is another reason not to treat the ranking as definitive. Google DeepMind developed the benchmark with Artificial Analysis, while Artificial Analysis says that it conducts the model evaluations independently. The scored 200-task set is private to reduce the risk of benchmark tasks leaking into training data or being found through search. Only the separate 10-task showcase set is publicly inspectable.
That protects the evaluation from one kind of contamination, but it also means outside researchers cannot independently audit every scored task.
How to apply the idea to your own AI evaluation
MKA’s design suggests a practical testing method even if you never reproduce its scoring system.
Start by inspecting one of the public MKA examples and its source files and rubric. The useful idea is the connection between source evidence, an observable requirement, and the finished artifact.
Then choose one representative internal workflow with a definite output. For example:
- assemble a monthly spreadsheet from several reports;
- audit drawings against a defined standard;
- produce a presentation from a fixed collection of source documents.
Before testing an agent, define checks in two categories:
- Source fidelity: Did it locate, reconcile, and preserve the evidence that matters?
- Deliverable usability: Is the spreadsheet, presentation, report, or other artifact structured and formatted so that someone can actually use it?
Run competing systems with the same source files, instructions, tool access, and limits. Otherwise, differences in the setup can be mistaken for differences in model capability.
Finally, have someone qualified in the underlying workflow recheck high-consequence requirements. A strong automated benchmark result is evidence about performance under benchmark conditions, not approval for a production process.
This validation method is an editorial recommendation inspired by MKA’s structure, not a certification procedure specified by Google or Artificial Analysis.
The better question for document-heavy AI
For drawings, scans and mixed document collections, “Can this model understand PDFs?” is too vague. Ask whether it can preserve the important evidence across your actual files, use the tools it will really have, and produce a usable artifact against requirements you can inspect. MKA helps make that test concrete; its leaderboard cannot replace your own test.