Code Lab
NVIDIA SkillEvaluator Explained: Do Agent Skills Actually Improve Results?
CODE LAB · PROJECT SPOTLIGHT
NVIDIA SkillEvaluator Python · Apache 2.0
Check skill quality, loading compatibility and measured impact with separate evaluation tiers.
Project wordmark · NVIDIA/SkillEvaluator · creator-supplied artwork.
NVIDIA SkillEvaluator asks two different questions about an agent skill: is its package well formed, and does using it improve task results? The first can be checked without a model run. The second needs a controlled comparison with the skill enabled and disabled.
That distinction matters when a skill looks polished but has never been tested against a baseline. The public Python framework separates static validation, semantic overlap checks and live agent evaluation. We inspected the repository, its report examples and the comparison source at the pinned commit; we did not install the evaluator or run an agent benchmark.
Start with the question you need answered
| Check | What it tells you | What it does not establish |
|---|---|---|
| Static quality | Whether instructions and supporting files satisfy configured quality checks. | A high score does not prove task accuracy or safe execution. |
| Harness compatibility | Whether a versioned agent profile is expected to discover and load the skill. | A loading check does not reproduce a full agent task. |
| Semantic overlap | Where guidance duplicates or conflicts with other guidance. | Similarity alone does not establish that a skill is redundant in every workflow. |
| Live evaluation | How runs compare on the chosen cases, metrics and environment. | One dataset is not a universal performance guarantee. |
The repository calls its support level experimental and community supported, with no enterprise-support entitlement or SLA. Treat the evaluator version, skill revision and dataset as part of the result, not incidental setup details.
The three tiers have different requirements
Tier 1 covers validation, including quality, schema, harness compatibility and configured security or code-integrity checks. Some checks use external scanners. The README warns that a scanner that cannot run is incomplete and can return a nonzero exit status; --no-llm does not supply missing scanner dependencies.
Tier 2 examines duplication using embeddings and chat. A catalog enables comparisons between skills. Check whether that comparison actually ran before interpreting a report as a review of your whole collection.
Tier 3 runs live evaluation with provider access, a supported agent and a sandbox, Docker by default. It can consume model or managed-sandbox resources. A readiness check and a successful inference run are separate observations.
A small starting point without a model key
The documented CLI supports Python 3.12–3.14. For a skill folder you own, the README gives this static-quality route:
pip install skillevaluator
skillevaluator quality-check ./my-skillHere ./my-skill is a placeholder for the folder being reviewed. This command checks static quality and reports findings and a score. The README says publication evidence remains incomplete after this check alone. It does not run the security and live-impact work needed for a broader conclusion.
For a broader static pipeline, the repository documents skillevaluator validate ./my-skill --tiers 1. Review the scanner setup ↗ first. Without a tiers restriction, validate runs all three tiers, so a short command can imply a much larger setup and resource commitment.
The official documentation ↗ also covers CLI, CI and agent integration. NVIDIA Build access described in the README is for eligible prototyping under its terms and limits; it is not a promise that every provider, account or production run is free. We did not enroll, generate a key or execute those workflows.
Read the creator’s report example carefully
The repository publishes before/after benchmark-card assets. They illustrate a change in report presentation: the later example makes the baseline, skill-assisted result and percentage-point difference explicit. These files are not our measurements and do not show that an arbitrary skill improved after editing.


The example’s values are specific to its demo dataset and metadata. Use the layout as a reading aid: find the baseline, task count, attempts, models and environment before interpreting the verdict. A green badge without its evidence is not enough.
Inside the source: compare shared cases for each metric
The comparison code records a common case basis for each metric. This excerpt then calculates the arm means and preserves the number of cases and scored trials:
For each arm, the list includes a value only if the trial’s case belongs to shared[metric] and the metric has a score. It averages those scored trial values and records both the shared-case count and each arm’s trial count. A metric with no shared cases receives an explicit exclusion reason.
Earlier code builds that shared set after considering missing or dropped cases, unavailable metric values and differing judge models. That is more informative than subtracting two averages whose populations may differ. Notice that this excerpt averages trials, not equally weighted case means; differing scored-trial counts remain relevant when interpreting a result.
Actual source: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. Apache License 2.0. Pinned commit affe1acc3d0a24877c5d508d96ac69663a7fd92a; original source text retained, line numbers and syntax coloring added. License ↗ · NOTICE ↗
Design a comparison that can change your decision
Choose tasks that represent the skill’s intended use. Include cases where the skill should activate and cases where it should stay out of the way. Define expected outputs and failure conditions before looking at generated answers, then review generated evaluation cases rather than assuming that a synthetic dataset is complete.
Keep the model, agent version, tools and sandbox conditions comparable between the arms. Retain traces showing whether the skill was loaded, the decisions made and the final artifact. Correctness, activation behavior, cost and execution time answer different questions; an improvement in one does not automatically imply improvement in the others.
The README says its Tier 3 autopilot can create a skill-grounded starter with limited coverage. That is a useful starting artifact, not a substitute for a reviewed task distribution. For repeat runs, also inspect excluded cases and the basis of each metric instead of focusing only on the headline lift.
Keep reports tied to a revision
A useful report names the skill revision, evaluator release, agent/model configuration, dataset and scoring policy. Refresh it when those change. Preserve incomplete checks rather than presenting skipped scanners or an unexecuted agent run as a pass.
Explore the pinned README ↗, full comparison implementation ↗ and releases ↗. The repository is Apache 2.0 and includes third-party software notices; hosted model access and sandbox costs are separate from the source license.
Source attribution and license
SkillEvaluator Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. This product includes software developed by NVIDIA CORPORATION & AFFILIATES. Third-party dependency notices are listed in THIRD_PARTY_NOTICES.md.
Reviewed November 18, 2026. Discovery: Mindung’s November 16 GitHub roundup ↗. We read its auto-generated transcript and pinned repository links, then independently inspected primary source. Creator report examples are distinguished from our source review. No evaluator installation, scanner result or live benchmark is claimed.
