Agent Skills Need Tests, Not Vibes

An Agent Skill is more than a prompt: it can carry tool-calling behavior, scripts, and support artifacts such as references and assets. Publishing one without eval cases is shipping that composite behavior without tests. skill-lens turns those expected behaviors into a report and an exit code your pipeline can gate on.

You write a skill. You try it, the answer looks right, and you publish it.

A few weeks later, someone edits the SKILL.md, reads a new sample answer, and ships that too. Maybe it is better. Maybe it now calls a different tool first, or stops checking the one thing you cared about. Nobody can tell from the diff, because the diff is prose.

I kept seeing that gap. The tooling around agent skills got great at packaging and discovering them, but not at asking the boring question a production artifact should have to answer: what evidence do you have that this still works?

That is why I built skill-lens. It is a small CI-first tool for testing agent skills before they ship.

1. A skill is an interface, not only a prompt

When you publish a skill, you are making promises about behavior, not only about wording. A skill can mix instructions, tool-calling behavior, bundled scripts, and support artifacts such as references and assets. A change in any of those layers can change how the whole skill behaves.

A simple refund skill might say, for example, that it must look up the order before doing anything, and that it must never refund an order outside the return window. That is a contract.

But an agent can honor that contract only if something checks two layers: what it says and what it does. A polished prompt is not enough. If the agent answers in polished prose while calling the wrong tool, the test has to catch that.

Skill-lens treats those promises as eval cases. You describe a task, the mock tools available to the agent, the calls it should and should not make, and the output you expect. Then the tool runs the case and gives you a pass/fail verdict.

2. Start with one skill and one boundary

The mock tools execute nothing. They record the call and return the response you wrote, so the test sees the agent’s decision without touching real systems.

You do not need a full test strategy on day one. Start with the behavior that would actually hurt if it regressed.

For a refund skill, the first question is not "does it explain itself nicely?" It is "does it refuse the refund it should refuse?"

The eval file can look like this:

cases:
  - name: refuses a refund outside the return window
    task: I want a refund for order 1234
    tools:
      - name: lookup_order
        description: Look up an order by its id
        parameters:
          order_id: string
        returns: '{"id": "1234", "status": "delivered", "days_since_delivery": 45}'
      - name: issue_refund
        description: Issue a refund for an order
        parameters:
          order_id: string
        returns: '{"ok": true}'
    trajectory:
      called: [lookup_order]
      forbidden: [issue_refund]
    assertions:
      - kind: contains
        value: "1234"

The mock tools execute nothing. They record the call and return the response you wrote, so a test can inspect the agent’s decision without side effects.

From there, add the cases that matter:

• the happy path, when the skill should do the thing
• the policy edge, when it should refuse
• a negative trigger, when an unrelated task should not activate the skill
• a budget ceiling, so the fix does not get slower or more expensive forever

3. Make it a release path

The first run can be free. Skill-lens has an offline runner, so you can validate the suite and see the report before spending anything.

skill-lens init ./skills/refund
skill-lens run ./skills/refund

That gives you a scaffold with empty fields the tool refuses to run until you fill them in. It is deliberately not a green checkmark before you have written anything.

When the suite is real, run it against one or more real agents. Then move it into CI, so every pull request that touches the skill runs the cases, and the build fails if the behavior regresses. That is the difference between "I tested it once" and "this is a maintained artifact."

4. Test what matters, not what is easy

It is tempting to start with output assertions because they are simple. But a skill that drives tools usually fails in its choices, not its vocabulary.

So test the tools the agent called, the order it called them in, and whether the skill was used at all. Output assertions are still useful, but they are the lower bound of what you can check.

That is also why I kept the tool small. Skill-lens is not another workflow engine. It is one layer of discipline around the thing we already publish.

Further reading

If you want to go deeper, the sources I found most useful are:

• skill-lens documentation — project docs for the CLI, runners, and CI setup used in this post.
• Testing Agent Skills Systematically with Evals — OpenAI’s practical guide to scoring and improving skills over time.
• Skill authoring best practices — Anthropic’s guidance on designing and testing skills.
• Test Agent Skills — Promptfoo’s comparison-oriented testing guide.

Conclusion

Agent skills are becoming real infrastructure. Real infrastructure does not get changed by vibes; it gets changed by tests.

If you publish a skill, give it one boundary. Write it as a case. Then let the tool run it, and let CI keep it honest. That is all skill-lens does, and that is enough to make publishing feel like releasing.