Skip to main content

Ideas behind Shaka's PR descriptions

An AI agent wrote this reference: Claude Code running Claude Opus 5.5, on 2026-10-31 (UTC). The agent read the public sources and this repository's PRs and recounted the numbers. Codex and a fresh Claude session reviewed it before publication. Idea 6 was updated on 2026-11-04 (UTC), after Shaka added a readback.

A Shaka pull request (PR) description has two jobs. It shows what the PR is trying to do, and it shows that the verification is adequate for that goal (#421 plan). This page records where the ideas for those jobs came from, what Shaka built, and how each held up. For how Shaka works with those sources today, read the analysis.

Before and after. #543, merged 2026-10-15, shows a summary, a checkpoint status, and a free-form table of areas and results; nothing states the problem, what was left out, or what went untested. #974, merged 2026-10-29, shows a recommendation with links, a summary, an Intent block with Why, Task, Root cause, Safe because, Review effort 2 of 5 with its floor, and Scope, then Verification with a Shaka-written validation and review line and a Setup gaps line saying 1 of 2 configured reviewers completed. Its checks and Not verified line are folded under Evidence and limitations. A note says #984 later found the Setup gaps line wrong: both reviewers ran, and the agent passed one result. Tags name Tan as the source for Intent, and Osmani as the source for the review floor and validation line, by the orchestrating session's account.

Data and limits​

  • Sources: this repository's public PRs and issues, Tan's posts, pstack at c742dd6, and Osmani's factory at 8af1165 and agent-skills at eb07f19, plus two Osmani posts named below.

  • Attribution: the #421 plan cites Tan's work directly. No public Shaka record cites Osmani's. The agent session that orchestrated this work reports reading factory, agent-skills, and both posts on 2026-10-18. It then cited factory in its brief for #740 and #757. That account is first-hand and can't be checked on GitHub.

  • Count: PRs merged from 2026-10-22 through 2026-10-29 UTC, run on 2026-10-30 at 03:00 UTC, after the window closed: gh pr list --repo shakacode/shaka --state merged --search "merged:2026-10-22..2026-10-29" --limit 300 --json number,author,mergedAt,body.

  • Matching: each description was matched by its text:

    • an ## Intent heading, or ## Change and review, its name from #860 until #888;
    • a **Validation and review:** line;
    • Floor N: in the Review effort line;
    • a **Setup gaps:** line naming configured reviewers.
  • Labels: this page uses the labels from the window. Since then, Review effort became Human review (#1024), which the checkpoint's low, medium, or high Risk rating then replaced (#1081). Adequate with accepted gaps became Adequate, with gaps the reviewer judged acceptable (#1021).

  • Restated checks: "11 of 12" is #740's own count of the PRs merged after #553. "32 of 101" comes from the window above, where 101 descriptions list check bullets under Verification. A description counts when one of those bullets matches this case-insensitive expression:

    (bin\/validate|\bvalidate\b[^\n]{0,40}(pass|green|exit 0)|evidence run --command validate|tests?,? 0 failures|RuboCop (clean|passed)|local review[^\n]{0,30}(clean|0 findings|no findings))

    The 32 are #973, #962, #959, #958, #945, #944, #939, #931, #930, #922, #916, #915, #907, #901, #898, #897, #895, #893, #891, #888, #881, #880, #879, #875, #860, #853, #844, #833, #826, #823, #792, and #742. Read by eye, a few matches also prove something about the change.

  • Not in the count: 5 of the 115 have no Intent block: 4 Dependabot PRs and one that Shaka did not publish. The count has no comparison group from another repository, and descriptions are read as they are now.

  • Not measured: whether the fields are true, and whether maintainers decide faster.

Measures from the same 115 PRs​

  • Checkpoint verdicts: across 144 published verification verdicts on 98 PRs, 138 said adequate with accepted gaps, 5 said adequate, and 1 said inadequate (#901, later revised). The final verdicts were 94 adequate with accepted gaps and 4 adequate. Across 176 published checkpoints, the reviewer said Proceed 167 times, Simplify/reframe 5 times, and Do not merge 4 times.
  • Review effort and time to merge: measured from the first awaiting-merge-approval label. Medians were 22 minutes at effort 2, 92 at 3, 66 at 4, and 142 at 5. Of the 44 effort-5 PRs that got that label, 17 merged within 30 minutes. Reading a PR leaves no trace, so this can't show how closely anyone read it.
  • What sets effort 5: 47 PRs rated 5. The docs floor set 45 and the Shaka settings floor set 2. Of the 45, 5 changed only docs and 40 also changed code. 25 of the 40 changed fewer than 20 lines under docs/.
  • People's comments: 36 GitHub comments on 13 of the 110 PRs with Intent. A comment counts when it has no agent identity line or Shaka marker and comes from no bot. Chat feedback does not appear.
  • Cost: across the 109 PRs that report a spend, the median was $15.14, at API prices. Docs-only PRs (5) had a median of $9.22 and code-only PRs (61) $10.54. PRs changing both (40, one reporting no spend) had a median of $21.87, and they are larger. Per 100 changed lines, the mixed and code-only groups cost about the same, $10.79 and $9.68. The other 4 changed only internal/.
  • Visible Verification text: the median was 75 words for PRs merged 10-22 and 10-23 and 11 words for PRs merged 10-29, not counting link targets.
  • Where Not verified appears, anywhere in the description, by merge date. A PR counts once, at its first such line. Across the 110 PRs, 45 were visible, 52 folded, and 13 absent. #860 merged on 10-25.
MergedVisibleFoldedAbsent
10-222802
10-23801
10-25320
10-26053
10-272100
10-28181
10-293276

Stacked bars of Not verified placement by merge date. 10-22: 28 visible. 10-23: 8 visible. 10-24: no merges. 10-25, when #860 merged: 3 visible, 2 folded. 10-26: 5 folded. 10-27: 2 visible, 10 folded. 10-28: 1 visible, 8 folded. 10-29: 3 visible, 27 folded.

Lauren Tan: proof is the limiting step​

Loops You Can Trust argues that verification constrains an agent loop the way the slowest step constrains a production line. pstack is Tan's public skill set in cursor/plugins.

1. Proof should show the work is right, intended, and quick to review​

  • Idea: give agents a way to verify their own work, then have them produce an artifact. It should show the work is correct, matches the intent, and is easier to review than the whole PR (post, 2026-10-11).
  • In Shaka: the two jobs, in the #421 plan four days later. #553 added Intent and Verification to every description.
  • Result: 110 of 115 PRs carry both. Before #553, only a summary was required.
  • Limit: Shaka checks that the fields exist. Whether a Why line is true is left to the agent, the reviewers, and the checkpoint.

2. A PR body is a short briefing for someone who has the diff​

  • Idea: the opening-a-pr playbook gives a one-minute briefing with an always-present Scope. Each Verification bullet is a real run path and its outcome. Other detail goes to linked artifacts.
  • In Shaka: Intent holds Why, Safe because, and Scope, plus optional lines such as Task, Root cause, and Leaves out. Its review line gave way to a Risk and review section after Intent (#1081). Verification holds up to three checks. Audit detail is collapsed.
  • Result: the visible text now names the problem, the scope, and the safety fact. On #543, it named none of them.
  • Limit: bodies stayed long. #974 has about 15,900 characters, and about 2,000 are visible. That is near the same share as #543, at about 1,500 of 10,500.
  • Left out: the playbook's Tradeoffs and Blast Radius sections. Shaka keeps alternatives in the code walkthrough and asks the checkpoint about blast radius.

3. An artifact beats an explanation, and any stage can stop the line​

  • Idea: in Loops You Can Trust, an artifact gives a reviewer something to inspect, where an explanation can sound right and be wrong. Failing to reproduce a bug, or judging a fix too risky, is a useful outcome that keeps bad work from moving on.
  • In Shaka: each check must cite a link or a code span (#553). #742 has the checkpoint judge whether the proof is enough. Under Auto, merge refuses a head judged inadequate or unjudgeable until a person confirms it.
  • Result: #743 tested the checkpoint's question about blast radius. With it, every run on the two code PRs said whether a revert undoes the change. Without it, none did. 2 of 5 runs with it missed a real gap that all 5 runs without it found. Evaluation article.
  • Limit: Shaka can tell that a link or command is cited, not that it ran. This repository still runs Ask, so the Auto stop has not fired here. The evaluation used one model and one scorer, who also wrote the criteria.

4. Find the one fact a change is safe because of​

  • Idea: the blast-radius skill looks for the single fact a change's safety depends on, then proves it by running real code. It ranks evidence from an unsupported claim up to a reproduction in the running app.
  • In Shaka: the Intent block's required Safe because line. #576 also tells agents to reproduce through the real caller before editing, and to check the fix's safety assumption.
  • Result: in #576's comparison, the revised guidance caught an empty-chunk crash the old guidance missed. Both agents reproduced before editing, so testing earlier was a tie.
  • Limit: Safe because is a sentence, and Shaka does not rank its evidence. The comparison was one pair per wording.

5. Fix the environment instead of correcting the agent​

  • Idea: when agents repeat a mistake, change the repository so the mistake can't happen (post). The correct skill tries architecture first, then types, lint, and tests, with docs last.
  • In Shaka: three rules moved from prose into code.
    • #553 refuses a description with a missing field.
    • #740 writes the validation and review line from recorded results.
    • #757 refused a review effort below a floor computed from the changed paths, until #1081 retired the rating.
  • Result: 11 of 12 PRs had spent two of three checks on restated passes. Now 32 of 101 spend one, mostly to add test totals, and none spends two. #553 had asked only for what ran and its outcome. Ratings of 1 on workflow or Ruby changes (#690, #694, #704) could no longer publish.
  • Limit: the windows differ, and nobody ran a controlled comparison. Shaka's rule table, which records what enforces each workflow rule, predates this post.

6. Restate the goal before working​

  • Idea: /in-your-own-words has the agent say what it thinks the goal and problem are before it starts. Osmani's Agentic Verification recommends the same habit.
  • In Shaka: #742 left out this plan-time restatement. #1044 added it on 2026-11-04. Before coding, the agent reads back the outcome, scope, success criteria, and assumptions. Since #1065, the PR description shows that readback as Goal and Done when under the Task line.
  • Limit: clear tasks go ahead without waiting for an answer, so the readback catches a misread goal only when someone reads it in time. #1017 proposed waiting up to five minutes.

Addy Osmani: what proof a human receives​

factory is a reference setup for running agents on a GitHub repository. Humans keep every merge and every load-bearing change. agent-skills is a general skill pack.

The Code Nobody Reads (2026-09-28) argues that line-by-line reading is ending for much code, so checks must replace the trust that reading gave. Agentic Verification (2026-10-14) names verification as the bottleneck and credits Tan on giving agents a way to drive the real system.

By the orchestrating session's account, ideas 7 and 8 led to #740 and #757. Idea 9 predates this work.

7. Report the gate result in a form the agent can't soften​

  • Idea: factory's implement skill makes the agent quote the gate script's final FACTORY_GATES: line verbatim, and forbids restating it. A misconfigured gate or a skipped required gate doesn't count as green. Its verifier reruns the gates and treats a mismatch as a finding.
  • In Shaka: by that account, this rule led to #740. The helper writes the line from results bound to the head, and no agent types it. Any exit code other than 0 shows as failed, and missing results show as UNKNOWN. Hosted validate reruns the suite on GitHub.
  • Result: of 110 lines, 96 report recorded results and 14 report UNKNOWN. In 4 of those, the installed Shaka updated during the run. In another 4, heavy machine load broke timing tests. The other 6 were not traced.
  • Limit: #923 added a line for missing reviewers, and it counted only the result files the agent passed. #984 traced all 8 such lines: 6 were false, and on 2 the agent never ran the second reviewer. When it can read the review ledger, description now refuses results from one ledger batch that leave out a reviewer. A result rebuilt with review check, as on #974, still gets through.

8. Send human attention to load-bearing paths​

  • Idea: factory's charter lists paths that always get the deepest gates and a human read, and are never auto-merged. Its PR template has a Human read required section. The Code Nobody Reads sizes human attention to blast radius: a lighter pass for small changes to less sensitive code, and careful review for core paths.
  • In Shaka: by that account, these led to #757, which computed a review floor from the changed paths. Agents could rate higher, never lower. Tan's correct ladder pointed the same way. Floors came from the default branch, so a PR couldn't lower its own. factory has the same rule against an agent rewriting its own constraints. #812 later let Auto merge only PRs at or below a chosen floor. That came from #780 and the study below, not from factory. #1081 then replaced the floors with the checkpoint's risk rating, which the repository's risk areas can raise but not lower.
  • Result: a study of 309 PRs supports paths over size. A person gave feedback on 65, and 64 of those changed docs or agent instructions. Within the size limits, 0 of 73 PRs that left those files and Shaka's settings alone drew feedback.
    • Ratings here were 2 in 9 PRs, 3 in 30, 4 in 24, and 5 in 47, across the 110 PRs with Intent.
    • This repository's docs floor set 45 of the 47 at 5, by the maintainer's choice.
    • #827 replayed 60 PRs at floor 2, and 14 would have merged without a person.
  • Limit: a path floor missed one risky line inside a low-floor file. #922 added Closer reading for that. 4 PRs with an Intent block name no floor, because they changed only design records under internal/.

9. The writer doesn't grade the work​

  • Idea: factory's contract says "The writer does not grade the work." A fresh verifier reads the diff cold. agent-skills' doubt-driven-development asks a fresh-context reviewer to disprove a claim. The Code Nobody Reads adds that a second copy of the same model shares the first one's blind spots. Tan's skills also use a second model family.
  • In Shaka: this predates the reading. Shaka prefers a reviewer from another provider, who reads each commit before push. The Verification line names who reviewed the head.
  • Result: Codex reported 37 of the 102 defects that local review found, defects Claude did not report (review process study).
  • Limit: #984 found 2 PRs where the agent ran one reviewer, because the helper's note named only the first. That note now names them all.

Not in Shaka​

  • Fixed verification table: factory's verifier reports fixed rows. The #421 plan proposed a similar table, and #743 recommended deferring it.
  • Rejecting when uncertain: factory's verifier leans toward rejecting. Shaka's checkpoint can answer cannot judge, which stops Auto but not Ask.
  • Unattended edits to existing tests: factory bans them. Shaka has no such rule.
  • Agents never merge: factory reserves every merge for a human. Shaka's Auto can merge within trusted limits.
  • Proving a new test fails: factory's prove-test.sh removes the fix and confirms the new test fails. Shaka asks agents to observe the failure, and no script checks it.
  • A verifier kept from the writer's account: factory's verifier ignores any narrative it receives. Shaka's reviewers receive the task's stated goal.
  • Stop rules: factory halts a run when gates fail twice on one item. In Shaka, two repair rounds without convergence prompt a reassessment, and a cap limits reviewed commits.

Where Tan and Osmani agree and differ​

Both name verification as the limit on agent work. For a repeated mistake, Tan's correct skill and Osmani's Agentic Verification both put the fix in the repository: a check, a hook, or shared config. Both check the writer with a fresh reviewer, ideally another model, and both recommend restating the goal before coding, which #1044 added to Shaka on 2026-11-04.

They differ on instruction length. Tan expects skills to compress into short principles: two pstack principle skills run 209 and 324 words, while three agent-skills skills run 2,474 to 3,334 words. The #421 plan chose the short form.

The Code Nobody Reads also tells authors to say in the PR description what they did not review. #553's Not verified line predates Shaka's reading of that post.

Other dead ends​

  • Lost review evidence. Shaka already told agents to keep review ledgers out of temporary directories, and warned when one sat there (#479). On #742 and #743, a restart still wiped a ledger kept in the session scratchpad. #976 tracks durable usage records.
  • The installed Shaka moved mid-task. Recorded evidence bound to the installed revision, and an update during a task superseded it. #812 reran review and validation. Since #1022, an update supersedes evidence only when it changes the code that runs or checks it. #960, #961, #973, and #922 published UNKNOWN results.
  • Machine load. On #868, #881, #927, and #942, local validation hit timing failures. Hosted validation was the gate.
  • PRs that change the renderer can't show themselves. The installed, older Shaka rendered the descriptions of #740, #757, and #810.
  • Noise from the new fields. A new user found the evidence the most useful part of shakacode/shakaperf#243. Routine passes and a Not verified line about work nobody asked for got in the way (#777). #810 folds routine passes. The PR record reference has the current rules.

Open items​

  • #421 stays open. An evaluation of how implementing agents verify is not done. The plan-time restatement (idea 6) arrived in #1044 after this window.
  • #780 closed with #1081: the checkpoint's risk rating, raised by the repository's high- and medium-risk areas, now sets what Auto may merge. This repository still runs Ask.
  • #976: usage records that survive model switches, restarts, and host handoffs.

Check the same fields on your repository​

Ask your agent:

$shaka For PRs merged in this repository last week, count how many
descriptions have an Intent block, Shaka's validation and review line, and a
risk rating (Human review or Review effort on older PRs). List the ones that don't.

In Claude Code, start the prompt with /shaka instead of $shaka.

The ten PRs​

These ten built the two jobs and tested the prompt sentence. Later PRs cited above refined the result. Merge dates are UTC.

MergedPRWhat changed
10-16#547The packaged checkpoint prompt asks whether testing fits what a failure would reach and whether a revert undoes it.
10-16#549This repository's checkpoint prompt asks the same question.
10-16#553Every description gets Intent and Verification.
10-20#743A 30-run evaluation of the #547/#549 sentence.
10-20#759While decisions are open, the walkthrough waits.
10-21#740Shaka writes the validation and review line.
10-21#757Review effort gets a floor computed from the changed paths.
10-21#810Routine passes fold, from field report #777.
10-21#812Auto merges only PRs at or below a chosen floor.
10-22#742The checkpoint judges whether the proof is enough.