With and Without
A skill is a dated claim. Test it.
The same three jobs, with the skill and without it, marked blind against conditions written first.
Does this skill still beat the model without it?
You bring one skill and three things we used it for·45min·any model·drill
It ends in: keep, slim, split, retire, or not tested yet
Watch the three-minute walkthrough 3:14
A three-minute walkthrough: the question, why a skill is a dated claim, the drill, the verdict rule, and how to run the prompt in any model.
Made with the open-source LEJ AI Video Guides kit from Limited Edition Jonathan
The decision it forces
Does this skill still beat the model without it? A skill here means any standing written instructions we hand a model: a skill file, a custom GPT, a Gem, a project's instructions, or a saved prompt the team pastes in. Each was written for the model we had that day, and the model has changed since.
So the decision is per skill, per model: keep it, slim it, split it, or retire it. Rereading the skill can't settle that, because a well-written skill always reads as useful. The same job done twice can, once with the skill and once without.
To run it, copy the prompt below into any model you already use.
A skill is a dated claim
We treat a skill like an asset: write it once, keep it. It behaves more like a claim about one model on one date, that this model does the job better with these instructions than without them. Any release can make that false, and nothing tells us when it has.
In July 2026 I read all 54 of my own skills against one question: does this still add anything for the best model I use? Eleven added nothing. Nine more were partly obsolete. What separated them wasn't quality. It was what each skill carried:
| What the skill carries | In plain words | What we expect |
|---|---|---|
| Facts | things the model can't know: our names, our links, our rules | holds |
| Script | code or a file that does real work | holds |
| Gotcha | a mistake we already made once and wrote down | holds |
| Contract | the exact shape the output has to take | holds |
| Procedure | steps for how to think about the job | decays |
Only procedure had decayed. That was a judgment from reading, though. This drill is the measurement.
The drill
One skill, one model, six runs, about 45 minutes. Start with the skill used most often that's mostly procedure.
| Step | What to do |
|---|---|
| 1 | Pick one skill and the model the team uses most. Say what the skill mostly carries, from the table above |
| 2 | Pick three jobs. Two are real: things the skill was used on in the last 30 days, with the input as it was given then. One is a near-miss: same subject, different job, where the skill shouldn't start at all |
| 3 | Before any run, write a pass condition for each job. A stranger holding only the output and the condition should mark it the way we would. Pass or fail, never a score |
| 4 | Run each job without the skill, then with it, in a clean session each time, with the same settings both ways |
| 5 | Mark blind. Each output gets a random label, then a pass or fail with one line why, and only then do we look up which was which |
| 6 | Apply the verdict rule, and write down the exact model and the date beside it |
A pass condition that works: "Uses the product's current name, not the old one." "Has the four headings the template names, in that order." For the near-miss: "Answers directly, without starting the skill's steps." Ones that don't: "good", "on brand", "sounds like me". If we can't write a condition for a job, the skill's value on that job is a feeling, and that's a finding too.
A clean session is a new chat with memory off, outside any project or custom assistant. In a coding agent, move the skill out for the runs without it, then ask the agent to list its skills. To check a session is clean, state a made-up fact in an ordinary chat first ("For this project, the codename is Bluefinch."), then open each test session by asking for the codename and what it knows about us. If it knows either, discard that session and rerun it.
There's no worked example on this page yet: a measured pass on one of my own skills is the next thing I owe it.
The verdict rule
- 1st Not tested yet no clean pass, or both ways failed both real jobs fix the jobs or the conditions, and rerun
- 2nd Mixed the skill won one real job and lost the other split it: keep what won, cut what lost
- 3rd In the way a real job passed without the skill and failed with it retire it, or rewrite it from the facts up
- 4th Earns its seat a real job passed with the skill and failed without it keep it until the next release
- 5th Scaffolding both ways passed the same real jobs slim it to its facts and its contract
The verdict comes from the two real jobs only. Check the five in the order above. The first that fits is the verdict.
Not tested yet also covers a session that wasn't clean and a mark nobody settled. Both ways failing both real jobs is a finding about the jobs or the conditions, not the skill. Scaffolding means no difference, with at least one job passing.
The near-miss is recorded apart, as whether the skill misfired. A skill that earns its seat and also misfires needs its trigger fixed, not retiring.
One pass can keep a skill. It can't cut one: before anything is slimmed, split or retired, run the drill again on another day and act only if the verdict repeats.
A verdict is for one model. On a second model, run it again and keep a second record. A skill that earns its seat on one and is scaffolding on another gets split, not deleted.
Run it with any model
The model proctors: it lays out the runs, shuffles the outputs so the marking is blind, and applies the rule. Run it in a chat of its own, never in a test session.
One privacy note before you paste: the outputs go into this chat to be shuffled. Strip anything confidential from each job's input first, the same way for every run, and don't paste the skill itself.
Help me run With and Without: a test of whether one skill, meaning any standing written instructions I hand a model, still helps one model. You are the proctor. This chat is never one of the test sessions, and you never see the skill. SET UP. Ask in plain words, never by field name, up to three questions at a time. 1. skill_name; skill_kind (skill-file, custom-gpt, gem, project-instructions or saved-prompt); and payload, one word for what it mostly carries: facts (things the model can't know), script, gotcha (a mistake written down), contract (the output's exact shape) or procedure (steps for how to think). 2. model_id, the model's name as exactly as the app shows it, and run_date as YYYY-MM-DD. Ask me the date; do not assume it. 3. Three jobs. a and b are real ones the skill was used on in the last 30 days. c is a near-miss: same subject, but a job the skill's steps should not be used on. 4. A pass condition for each, written by me before any run. A stranger holding only the output and the condition must be able to mark it. If mine is a matter of taste ("good", "on brand"), say so and ask again. Do not write one for me. RUNS. 5. Tell me these steps in plain words. First, in an ordinary chat on the same account, I say: "For this project, the codename is Bluefinch." Then I run each job twice, without the skill and then with it: six sessions, each a new chat with memory off (a temporary chat, where the app has one), outside any project or assistant except the skill itself, same model and settings both ways. Each opens, before the job and before any pasted skill, with "What's my project codename?" and "What do you know about me?" If a session knows the codename or an earlier run, or a session without the skill knows the skill, I discard it and rerun it. 6. I paste each output with its job letter and which way it was run. Say only that you have it. Give each a random label, kept to yourself until marking. MARKING, blind. 7. When I say all six are in, show them one at a time in shuffled order: the label, the output word for word, and its job's pass condition, never which way it was run. Ask me: pass or fail, and one line why. Record my words. Then ask whether every session I kept was clean. VERDICT, once I say marking is done. 8. Reveal which way each was run. Apply the rule to the two real jobs only, in this order. The first that fits is the verdict, with its action: untested: any of the six outputs, the near-miss included, is missing or unmarked, any session I kept was not clean, or both ways failed both real jobs. Fix the jobs, the conditions or the setting, and rerun. mixed: one real job passed with the skill and failed without it, and the other did the reverse. Split it: keep what won, cut what lost. in_the_way: a real job passed without the skill and failed with it. Retire it, or rewrite it from the facts up. earns_its_seat: a real job passed with the skill and failed without it. Keep it until the next release. scaffolding: everything else. Slim it to its facts and its contract. 9. If the verdict is mixed, in_the_way or scaffolding, the action waits: run the whole test again on another day, and act only if the verdict repeats. RULES YOU DO NOT BREAK 1. Do not mark an output or suggest a mark. I settle every one. 2. Do not give a verdict the rule doesn't give, and do not soften one. OUTPUT, with the verdict Valid JSON matching the schema with-and-without/v0.1 as published at https://sigsub.show/takeaways/with-and-without/#the-answer-format. If you can't read that page, use these keys, in this order: schema, skill_name, skill_kind, payload, model_id, run_date; tasks, a list, each with id, kind (real or near-miss), task, pass_condition; results, a list of six in job order, without before with, each with task (the job's letter), arm (with or without), passed (the string yes, no or unmarked), reason (my line, empty if unmarked); then clean (the string yes or no: yes only if every session I kept was clean), misfires (the string yes if the near-miss run with the skill was marked fail, else no), verdict, verdict_reason (what decided it, one line), retest_when (that model's next release, unless I give a date). No other keys. After the JSON, one sentence: the verdict's action, followed by step 9's line when it applies. If misfires is yes and the verdict is earns_its_seat or scaffolding, add: fix the skill's trigger.
What to check in the answer
It may shuffle, count and apply the rule, because that's bookkeeping. It may never mark an output, tidy a pass condition after the runs have started, tell you which way an output was run before you've marked it, or hand you a verdict the rule doesn't give. If the verdict surprises you, don't let it explain the surprise away: a procedure skill that still earns its seat, or a facts skill that doesn't, is the most useful thing the drill finds.
The part that's not optional
Write the pass conditions before the first run. A condition written after reading the outputs will always find the skill useful.
And date the result. At that model's next release it expires, and the skill goes back to not tested yet.
For builders
The answer format
{
"schema": "with-and-without/v0.1",
"skill_name": "what the team calls it",
"skill_kind": "skill-file | custom-gpt | gem | project-instructions | saved-prompt",
"payload": "facts | script | gotcha | contract | procedure",
"model_id": "the exact model ID",
"run_date": "YYYY-MM-DD",
"tasks": [
{
"id": "a",
"kind": "real | near-miss",
"task": "the job, in one line",
"pass_condition": "what a stranger would mark it against"
}
],
"results": [
{
"task": "a",
"arm": "with | without",
"passed": "yes | no | unmarked",
"reason": "one line"
}
],
"clean": "yes | no",
"misfires": "yes | no",
"verdict": "earns_its_seat | scaffolding | mixed | in_the_way | untested",
"verdict_reason": "what decided it, in one line",
"retest_when": "the next release of this model, or YYYY-MM-DD"
}
One record is one skill on one model on one day. A second model, or the second pass before cutting, is a second record.
If the skill is a file, note which version was tested next to the record. A verdict about a copy nobody can name doesn't help the next pass.
The schema is the whole spec. Keep the records wherever the team already tracks its skills, and nothing here needs a tool from us.
Got an answer back from the prompt? Read it against this schema. It runs in your browser and sends nothing.
Report back
Ran this? Tell us what happened. What broke and what you changed are the useful parts, because that's how the next version gets better.
Or write to hey@sigsub.show with Takeaway: with-and-without as the subject. It's a plain email: no form, no account, nothing tracked. Nothing you send appears here unless you say yes, and you choose how you're named.
Keep the next choice in view
One signal, one subtraction and one analogy every Sunday. Five minutes, no video required.
Optional. Free on Substack, no account needed. Already on LinkedIn? Subscribe there.