This follows Running Claude, Codex & Grok as One Team on a Locked-Down Enterprise Project (Day 0).

Agent skill directories keep growing, and it is easy to believe that dropping in a planning skill, a TDD skill, or a wiki skill will make a coding agent noticeably better. I am a backend developer with four years of experience in SI work, mostly on commerce-style services. I wanted a number, not a vibe.

So I ran the same fixed tasks twice: once with skills forbidden, once with the agent required to follow a single skill document. I registered the pass/fail rule before looking at outcomes.

The short version: on 12 November 2026 (Asia/Seoul) I completed ten comparisons. None met the pre-registered adopt bar. A couple came close. Several burned a lot more tokens for a tiny quality bump, or none at all. This post is the write-up.

1. Method

Each comparison is one published agent skill.

• Arm A could not read any skill document.
• Arm B had to read and follow that skill's document.
• The task prompt was identical except for a run tag.
• Every run was a fresh session of the same model (Grok, grok-4.7-build) in headless mode. I did not steer the agent mid-run.

When several skills shared a task, I ran A once and compared every B against that same baseline. That keeps the prompt and the day constant, but it also means one lucky or unlucky A is reused. I treat that as a limitation, not a feature.

I measured the same things every time:

• Non-cache tokens: input minus cache reads, plus output. This is the cost number used in the verdict. I also logged total tokens including cache, but I did not judge on them.
• Active time: time the model was actually working, idle excluded.
• Quality (1–5): the mean of accuracy, completeness, actionability, concision, and scope discipline. Before grading I stripped skill names and run tags and shuffled the labels. A separate Grok session, seeing the work for the first time, did the scoring.
• Review rejects: defects a reviewer would send back, tagged as actual error, spec misunderstanding, missing context, or out-of-scope.
• Tests: implementation task only — 13 hidden tests the agent never saw.
• Contamination and compliance: if A had read a skill, that pair would be voided. I also checked that B actually opened the skill. All ten pairs were clean, and every B read its skill.
• Repo guard: each run was checked so the experiment did not modify the work repositories. All guards passed.

There were four tasks. I kept client identifiers and production data out of the prompts, and I describe the tasks only in general terms here.

1. Plan. Write one implementation plan for wiring a partial loyalty-points recovery action into an e-commerce admin screen. Repositories are read-only.
2. Analysis. In a Spring/MyBatis e-commerce backend, trace how that recovery API should flow and write a short analysis with evidence. Read-only again.
3. Implementation. Implement a small Node.js module from a written spec — the verdict function used by this lab itself. Scoring is backed by those 13 hidden tests.
4. Wiki. Ingest one source document into a local markdown wiki copy and answer one question, citing only that source. This is not a test of "wiki versus no wiki." Both arms ingested into a wiki. The only difference was whether a wiki skill was loaded.

2. Verdict rules (written first)

I did not want to invent a story after seeing the table, so the rule was fixed in code before the runs.

• Adopt (써야 함): quality stays within −0.3 of A, tests pass, rejects do not go up, and non-cache tokens or active time drop by at least 20%. Alternatively, quality rises by 1 or more while tokens rise by no more than 30%.
• Drop (빼기): quality falls by 0.5 or more, or tokens rise by 50% or more while the quality gain is under 0.3.
• Hold (보류): everything in between. A nicer-looking answer that costs too much still lands here.

I also decided in advance that n=1 is only directional. I will not treat a result as settled until the same task has been repeated at least three times. Every figure below is n=1, indicative.

3. Results

All runs are from 12 November 2026. Token counts use the rounded thousands from the result cards. Percentages are the recorded values computed from the raw counts.

Non-cache token change versus the no-skill baseline (left is a cut, right is an increase):

test-driven-development      −10.2%  ██
incremental-implementation    +4.5%  █
graphify                     +14.2%  ███
planning-and-task-breakdown  +48.8%  ██████████
improve (plan)               +68.6%  █████████████
improve (impl)               +77.1%  ███████████████
karpathy-llm-wiki            +89.1%  ██████████████████
grill-me                    +116.7%  ███████████████████████
tdd                         +125.2%  █████████████████████████
writing-plans               +166.1%  █████████████████████████████████

On the implementation task, the hidden tests were 13/13 for A and for every B. On the wiki task, citation accuracy was 2/2 for A and 3/3 for B (both 1.0). Total tokens including cache rose even more on the B arms; those totals were recorded, not used for the verdict.

One summing rule is worth flagging. The improve implementation comparison adds a planning session that used the skill and a follow-up implementation session that did not. The other implementation skills are single sessions.

4. How I read the table

Adopt 0, hold 5, drop 5. After a drop verdict I uninstalled writing-plans, grill-me (and the companion grilling skill), tdd, and karpathy-llm-wiki on the same day. improve split: drop on the implementation task, hold on the planning task.

The closest call was test-driven-development from obra/superpowers. Quality went 4.8→5, rejects 2→0, and non-cache tokens fell 10.2%. Active time, though, rose 31.3%, so it missed the 20% cost cut, and a +0.2 quality bump is far from the +1 alternative. Hence hold. A different skill also named tdd, from mattpocock/skills, reached the same quality of 5 but at +125.2% tokens, which is a drop. The label on the skill is not the result.

The four planning skills either ticked quality up slightly (+0.2 to +0.4) or down (grill-me, −0.4). Tokens rose between 48.8% and 166.1%, and active time roughly doubled. Rejects did not fall (1→1) and in one case rose (1→3). A generic planning skill did not make this review cheaper.

graphify looks harmless at +0.2 quality and no rejects, but tokens were +14.2% and time +85.5%, so it stays on hold. The wiki skill kept quality at 5 vs 5 and nearly doubled tokens (+89.1%) and time (+103.8%), so it was dropped. Again: this is the cost of loading the skill on top of a wiki workflow, not a verdict on keeping a wiki.

5. What was actually useful, and when a skill is worth it

I cannot point to a skill and say "this is the one you need," because nothing in this sample cleared the adopt bar. The only candidate I would rerun with a larger n is test-driven-development.

The more useful artifact was the lab itself: freeze the task, change only whether a skill is loaded, and look at tokens, time, rejects, and blinded quality together. Writing the drop rule in code before the first run mattered. Hidden tests on the implementation task also mattered. They caught that every arm could pass the suite while still disagreeing on the edges of the spec.

I did not measure the next idea, so I will not pretend it is a result. On the planning task, the no-skill arm and several planning-skill arms stumbled on the same project-specific review points. A short checklist of known reject reasons in the prompt may be a cheaper fix than installing a generic planning skill. That comparison is still to do.

Until a skill beats the rule below, I will not leave it on by default just because it is popular or well named:

• keep quality within 0.3 of the baseline, keep tests green, do not add rejects, and cut tokens or time by 20% or more, or
• raise quality by a full point without growing tokens more than 30%.

A +0.2 quality bump does not pay for a doubled token bill.

6. Limits

The numbers only make sense with these limits attached.

• Every comparison is n=1. Until a task is repeated at least three times, this is indicative only.
• Grading was blinded, but it was a single grader session, and it used the same model family as the runner. Blinding removes labels, not shared taste.
• A was reused across skills on the same task, so one baseline affects several pairs.
• I only ran Grok. The same skill may behave differently in another coding agent.
• grill-me is designed to ask a human questions. In headless mode I assumed the recommended answers were approved. An interactive run could look different.
• The improve implementation figure sums two sessions.
• Four task types are not a career. I would not generalize this to longer work, other domains, or runs where a person interrupts mid-task.
• I generalized the work tasks and left identifying names, paths, and business rules out of this post.

The next measurements are straightforward: repeat the closest hold with a larger n, and A/B a reject-reason checklist against "just add another skill." Until then, a new skill is a hypothesis, not a default.

Every figure in this post comes from the 12 November 2026 lab records. I did not add estimates.

Korean version: https://po-p.tistory.com/40

That comes with constraints most AI-coding blog posts don't have:

No direct DB access. I work from exported files, screens, and what the client allows. Legacy code everywhere. A lot of the existing code was copied from other projects and is not "the right answer". Multiple repos. A backend, a new Next.js frontend (read-only for me), and a legacy service I migrate from. Over the past months I've been using three agents together: Claude Code as the implementer, Codex for review and verification, and Grok to challenge facts and designs. This post is Day 0 of a 2-week experiment to find out whether my setup actually helps, or just feels like it helps.

The problem: my instructions grew, and so did the cost My backend repo had a detailed CLAUDE.md: DB-first workflow, known pitfalls, verified facts, agent roles. It works. But when I measured my agent logs for the last two weeks, I found this:

Claude Code (3 working days) Total tokens ~208M Share that was cache reads ~98% Tool calls 760 Search calls (grep/glob/rg) ~23% of tool calls 98% cache reads means most of the cost is the agent re-reading long context every turn, not doing new work. Nearly a quarter of tool calls were searches, which suggests agents were rediscovering things that were already written down.

Why this setup, and why run the experiment

Starting point: Matt Pocock's "Home Repo" pattern The home-folder layout isn't my idea. I took it from Matt Pocock's "Home Repo" pattern on X: keep skills, rule files, and a wiki in one home repo, run the agents from there, and list the other repos' locations in AGENTS.md so the agents can work across several repos at once.

My project is split into three repos: a backend, a new frontend, and a legacy service. On top of that, I work in an SI client environment with no direct DB access. That's why the pattern looked like a good fit.

But does it actually help? A popular post doesn't prove the pattern works. Three reasons to test it:

Different environment. The original assumes agents can freely read and edit everything. Mine has read-only repos, no DB access, and legacy code that can't be trusted as ground truth. The cost is measurable. On day 0, cache reads were about 98% of tokens. Putting the rules in one home folder and keeping them short should lower that, and I can compare before and after. Separating "feels better" from "is better." I'm tracking tokens, the share of search calls, and logged agent mistakes instead of going on gut feeling. In the results post on Nov 25, I'll answer with numbers whether this was the right call. If it didn't help, I'll say so and explain what I removed and why.

References Matt Pocock on X: The Home Repo The setup I'm testing

One "home" folder above all repos All repos live under one workspace/ folder. A single root AGENTS.md holds only cross-repo rules:

Which repo is writable (backend) and which are read-only (frontend, legacy) Where to run each agent. Codex only reads AGENTS.md from the git root down, so it has to start in workspace/. Claude Code also reads parent folders. Who does what (implementer ≠ verifier) A legacy → new-service mapping table, where every row carries its evidence file Repo-specific rules stay in the repo. The same rule is written in exactly one place.

"Short request → fixed procedure" I don't want to type "design first, then ask me" every time. So the root file maps request types to procedures:

"Design a points API" → search existing code → check real DB columns → write docs/-design.md → wait for my approval → implement → build + .http request check → review by a different agent.

A mistake log, not a trend log Every time an agent edits the wrong repo, claims a column exists when it doesn't, re-investigates a documented fact, or reports "done" without verification, it gets one line in agent-misses.md.

Instruction changes are only allowed if they cite a line in that log or a measurement. A new trend alone is not a reason. The root file has a one-page limit: add one rule, remove or merge one.

Measuring it A small Node script reads local Claude Code and Codex session logs and outputs daily tokens, cache reads, tool calls, and search calls per session to a CSV. Twice a week, a review compares the numbers with the baseline above.

What I'll report in 2 weeks Tokens per session and per tool call: before vs. after Search-call ratio Number of logged agent mistakes, and whether repeated ones disappear What I removed from the instructions, and what broke when I did My honest hypothesis: fewer, shorter rules in the right place beat a long, careful instruction file. But I've measured only 3 working days so far, so I could easily be wrong. That's the point of the experiment.

If you work in a constrained enterprise environment with AI agents, I'd love to hear how you handle it.

Part 1 of a series. Results on Nov 25.