Start here · Lesson 1.3
Measuring AI Taste Training: Metrics That Matter
It is easy to track activity in AI taste training: how many rules you saved, how many reviews you did. None of that tells you whether the output improved. This post gives you a few numbers that point at outcomes, and a weekly way to read them. No benchmarks are quoted, since yours will depend on your work. The method being measured is from Lesson 1.3: Teach AI your taste.
Track four numbers each week: repeat notes, hand edits, rule hits and review time. Fewer repeats is the best sign.
Activity versus outcome
Activity metrics feel good and mean little.
- Number of rules saved
- Number of reviews done
- Length of the rule file
A long file is not better. The Claude Code docs say specific, concise instructions are followed more consistently, and note that very large instruction files cause problems.
Outcome metrics ask whether the work is getting closer to what you want.
The four numbers
- Repeat notesHow many notes in this week's review repeat a note from an earlier week. The most important number.
- Hand editsHow many pieces you changed by hand before publishing.
- Rule hitsOf your saved rules, how many were followed in this week's pieces.
- Review timeHow long the review took, start to finish.
Repeat notes: the core signal
Taste training exists to stop you repeating yourself. If you gave the same note last week and give it again this week, the rule did not work, or was never saved.
Count carefully. Keep last week's notes next to this week's and mark matches. Falling repeats suggest rules are held. Flat repeats point to vague wording or a rule that was not stored.
Hand edits
Count the pieces you edited before publishing, not the number of edits. Over several weeks you expect this to fall. If it stays flat while repeats fall, your new notes may be about different issues, which is fine.
Rule hits
Take each rule and check it against the seven new pieces. Mark yes or no.
A rule that is ignored needs work. The docs treat memory files as context, not enforced configuration, so a miss is a normal thing to find. Make the rule shorter, add a concrete example, and look for a clash with another rule.
Review time
Track the minutes. The lesson reviews a week of posts in about five minutes once you know the method. Your time may be longer at first. If review time keeps rising, the cause is usually vague notes or a bloated rule file.
A simple weekly table
Keep a table like this in a spreadsheet. The numbers below are placeholders, not results.
| Week | Repeat notes | Hand edits | Rule hits | Review time |
|---|---|---|---|---|
| 1 | n/a | [count] | n/a | [minutes] |
| 2 | [count] | [count] | [x of y] | [minutes] |
| 3 | [count] | [count] | [x of y] | [minutes] |
Week one has no repeats because there is nothing to repeat yet.
How to read the table
- Repeats down, hits up. Rules are doing their job.
- Repeats flat, hits low. Rules are vague or ignored. Rewrite them.
- Hits high, edits still high. You may need new rules for new problems, or a rule for your voice.
- Review time up. Trim the rule file and tighten your notes.
Metrics that can mislead
- Likes and views. These depend on many things beyond your rules, so do not use them to judge the training alone.
- Number of rules. More is not better.
- How the output feels today. Mood changes. Counts do not.
If you want to link rules to business results, do it carefully and over a long stretch, and do not assume cause.
A worked reading, with invented figures
To show how to read the table, here is an imaginary set of counts for one small shop. These numbers are made up and not a benchmark.
In week two the owner finds that three of the eight notes repeat from week one. In week three, one of seven repeats. Rule hits go from four of six to five of six. Review time stays steady.
The reading: repeats are falling and hits are rising, so the rules are mostly held. The one miss needs a shorter, more concrete rule. Nothing here says how the audience reacted, because that was not measured.
Setting up the tracking in ten minutes
- Make a sheetOne tab, five columns, one row per week.
- Save last week's notesKeep them in the same file so you can compare.
- Mark repeatsNext to this week's notes, tag any that match an old one.
- Check rulesList each rule and mark followed or not for the new batch.
- Write one lineRecord what you will change before the next review.
Pitfalls when counting
- Counting similar notes as different. "No outlines" and "remove the white border" are the same note. Group them.
- Counting only the notes you remember. Use the written file.
- Changing what counts mid way. Define your terms once.
- Reading too much into one week. Small counts swing a lot, so wait for a trend.
Metrics are a way to ask better questions about your rules, not a score to defend.
Questions people ask
What is a good number of repeat notes?
There is no universal figure. Compare your own weeks and aim for a downward trend.
Should I track likes?
Treat them as context only. Too many other factors affect them.
How long before the numbers mean anything?
A few weeks of data gives you a trend. One week gives you a baseline.
Where to go next
Get your baseline with the full week review in Teach AI your taste, Lesson 1.3. For a worked picture of a first week, see A week of AI taste training: an illustrative example. For the routine itself, read The TEACH framework for AI taste training.