For the past couple of weeks I've been living inside AgentEye, our enterprise platform for understanding how AI agents behave in production. It gives teams visibility into what their agents are actually doing — failures, latency, quality metrics, execution history — all through a CLI called agenteye.

At least for me, and I suspect for most people, a pattern showed up fast. Most command-line tools ship with some level of instructional material, and they usually do a decent job of telling you which command does what. Where they fall short is describing how a command behaves in the hands of an experienced user: the edge cases, the assumptions baked in behind it, and the small variations that quietly change the outcome.

To close that gap we wrote a skill file for Claude. Instead of re-documenting every command, it captures the operational knowledge that experienced users pick up over time — recommended workflows, hidden assumptions, common pitfalls, and behaviors that aren't obvious from the CLI docs alone.

Naturally, that made me curious. So I ran the same tasks past Claude twice: once with the skill file loaded, once without.

The outcome wasn't what I expected. The skill file didn't make Claude "smarter" in any meaningful sense. It changed how Claude interacted with the CLI. Claude stopped making the mistakes that come from missing operational context, even though the underlying reasoning stayed exactly the same.

Not every mistake looks like a failure

Working through this, I landed on two completely different ways for an agent to be wrong.

The first kind is easy to catch. AgentEye expects global flags to come before the command. agenteye --json events is correct; agenteye events --json throws an instant usage error. This one is easy because the CLI tells you: the command fails loudly and explains itself.

The second kind is much worse.

This time the command succeeds. The output is completely valid. Nothing crashes. And yet the answer is wrong or misleading.

The cleanest example is fetching errors from the last 24 hours. Run agenteye errors --since 24h --all and it returns 50 rows — then reports next_cursor: null, which reads as "you've reached the end of the results." Except there are hundreds more rows that were never returned. Nothing in the output hints at this, and nothing explains the behavior.

The same shape shows up in commands that change agent state, like creating keys or disabling users. Run them interactively and the CLI asks for confirmation. Run them in non-interactive mode — Claude's default — and that confirmation is silently skipped.

This is exactly the class of behavior you'd never expect from reading --help, but that matters enormously the moment a tool is being driven by an AI instead of a human.

Without the skill file there's an even more basic failure mode: Claude doesn't know agenteye exists at all. Ask it "did anything break today?" and it reaches for git log and GitHub Actions history — confidently answering a different question than the one I asked.

What the skill file actually changes

Once the skill file is loaded, Claude understands that following the conventions established by power users beats guessing. It sets global flags in the right place, turns on JSON output by default, and leans on exit codes instead of parsing human-readable error messages.

It's easiest to see in practice. Ask "Did anything break today?":

Without the skill file, Claude runs agenteye errors --since 24h --all and immediately hits the default 50-row limit — quietly missing most of the data.
With the skill file, Claude runs agenteye errors --since 24h --aggregate, which summarizes the entire window without getting truncated, and only pulls detailed rows when they're actually needed.

Same question, same model — but now Claude goes straight to live agent data instead of reconstructing an answer from the wrong source.

API keys are another good example. Say a CI pipeline exists and a key looks stale. A reasonable-looking recovery is to just regenerate it. But the CLI can't tell Claude whether that key is actively in use by another system, and regenerating it could break a live integration. The skill file encodes that: key replacements can't happen without prior approval, so Claude stops and asks a human first instead of quietly taking a destructive action.

There's a second, subtler benefit: the skill file lets Claude reach for capabilities I wouldn't think to use myself. As a developer, to analyze agent behavior I'd normally have to learn the CLI deeply enough to know which commands exist and when to use them. The skill file front-loads that context. It knows agenteye query schema and agenteye query run --sql are there for deeper analysis, so Claude can escalate to them when a question warrants it — without me having to discover them first.

Acknowledging incidents works the same way. Rather than blindly acking an incident, the recommended flow is: open the list of active incidents, select the relevant one, examine the details, validate with a human, then act. That interactive validation step disappears in an automated run, so the skill file adds it back explicitly as a safety checkpoint.

Final thoughts

The biggest takeaway from this experiment is that documentation and operational knowledge are two different things.

The CLI documentation tells you which commands exist. The skill file tells you the best way to use them. Pagination limits, skipped confirmation prompts, hidden SQL, the real cost of regenerating an API key — none of that lives in --help. It usually surfaces only after someone has already hit the consequence and written it down.

That's precisely what makes skill files valuable. And because the CLI keeps evolving, the operational knowledge captured in the skill file has to evolve with it — keeping the skill file current is every bit as necessary as keeping the docs current.

Documentation tells you what a command does. The skill file tells you what an experienced user knows.