From Playbook to Pipeline: Actually Enforcing AI Code Checks

The everyday problem: Knowing which checks to run on AI-generated code is one thing (that's the companion playbook, Verifying AI-Generated Code Before You Push). Actually wiring those checks into your workflow so they run automatically — and getting past the dozen small things that break the first time — is another. This guide is the practical, hands-on half: how to set the checks up locally, how to make them run on every commit, and how to enforce them for a whole team. It's the "from playbook to pipeline" path.

Honesty up front: every command here was run and verified on Windows (with Next.js / Python / TypeScript / React projects). The approach is the same on macOS and Linux, but a few install steps differ — those are noted, and where something is platform- or version-specific it's flagged so you can adapt and fix-forward rather than assume it works identically. Tool versions move fast; pin them (this guide shows where that matters, because unpinned versions caused two of the snags below).

There are three levels of enforcement, each stronger than the last: run the checks manually, run them automatically on every commit (a local git hook), and make them mandatory for merging (server-side CI + branch protection). We'll build up through all three.

Level 0: Run a check manually (and the install gotcha)

Start by proving one check works by hand. We'll use Gitleaks (secret scanning) because it's fully local and the most universally useful.

The first gotcha — installing it. Many guides say scoop install gitleaks. But Scoop is a Windows package manager that isn't installed by default — so you may get 'scoop' is not recognized as an internal or external command. That's not an error in your setup; you just don't have Scoop. You have options:

# Easiest on Windows — download the binary directly, no package manager:
# 1. Go to https://github.com/gitleaks/gitleaks/releases
# 2. Download gitleaks_x.x.x_windows_x64.zip and unzip gitleaks.exe
# 3. Put gitleaks.exe on your PATH, or drop it in the folder you're scanning

# macOS:
brew install gitleaks

# Or, if you DO have Scoop / Chocolatey:
scoop install gitleaks
choco install gitleaks

Run it (note: current Gitleaks uses dir / git subcommands — older guides saying gitleaks detect are out of date):

gitleaks dir . --verbose          # scan files on disk
gitleaks git . --verbose          # scan git history

The noise gotcha. A manual dir scan reads everything, including dependency folders — so you'll get false positives from library code like .venv/Lib/site-packages/... or node_modules/ (e.g. a line of library source that merely mentions "PrivateKey"). Those aren't your secrets. The fix is a config file that tells Gitleaks to skip those paths — covered in the kit's .gitleaks.toml. The key insight: a finding inside a dependency folder is almost always noise; a finding in your code is what matters.

Level 1: Run checks automatically on every commit (pre-commit hook)

Manual runs depend on memory. A pre-commit hook runs the checks automatically every time you git commit, and blocks the commit if something's wrong. The standard tool is the pre-commit framework.

pip install pre-commit             # the framework
# put a .pre-commit-config.yaml in your repo root (see the kit), then:
pre-commit install                 # wires the hooks into git for this repo

The enforcement kit includes a ready-to-use .pre-commit-config.yaml, .gitleaks.toml, CI workflow, and PR checklist template. Download the enforcement kit →

Gotcha 1 — install doesn't scan anything. Running pre-commit install only wires up the hook; it prints "pre-commit installed at .git\hooks\pre-commit" and stops. It does not scan your files at that moment. So if you planted a test secret expecting to see it caught — nothing happens, because no scan ran. The checks only run when you actually git commit, or when you run pre-commit run --all-files manually.

Gotcha 2 — the first run needs internet. The framework downloads each hook's repo the first time. If you're offline or behind a proxy you'll see Could not resolve host: github.com. That's a network issue, not a config one — get connected and re-run; after the first download the hooks are cached and work offline.

Gotcha 3 — it only scans STAGED files. This is the big one. The hook scans what you're committing (staged files), not everything on disk. You'll even see [INFO] Stashing unstaged files in the output — it literally sets aside anything not staged. So a secret in an unstaged file, an untracked file, or a gitignored file (like .env) won't be caught by the hook. To test it, you must git add the file first, then commit.

Gotcha 4 — "fixer" hooks report "Failed" when they change a file. Hooks like end-of-file-fixer and trailing-whitespace don't just check — they edit the file (e.g. adding a missing newline). When they modify something they report "Failed" so you notice and re-stage. That's expected, not a real failure — run again and it passes.

Gotcha 5 — it works from the VS Code commit button too. The hook is wired into git itself, so it runs whether you commit from the terminal or the VS Code Source Control UI (or any git GUI). If a commit silently "doesn't go through" in VS Code, the hook probably blocked it — check the output panel. (Both can bypass with a force/no-verify option — which is why Level 2 exists.)

A real detection nuance we hit: a hardcoded key in a comment may not be flagged, while the same key in actual code is. Gitleaks scores findings by context, so test detection with the secret in real code, not a commented-out line — and remember a clean result means "no recognizable pattern found," not "definitely nothing here."

What to put in the local hook (and what to leave out). Keep the local hook fast — the rule of thumb is under ~5 seconds, because a slow hook gets disabled with git commit --no-verify. So only fast, local checks belong here: Gitleaks (secrets), private-key detection, large-file and merge-conflict checks. The slower or network-dependent checks (dependency audits, full static analysis) go in CI instead.

Level 2: Make checks mandatory to merge (CI + branch protection)

The local hook is great feedback, but it's bypassable (--no-verify) and each developer has to install it. Real enforcement runs the checks on the server, where they can't be skipped. This is a GitHub Actions workflow (in .github/workflows/) that runs on every push and pull request.

The CI workflow runs the full set, including the checks deliberately kept out of the local hook: secret scanning (full history), hallucinated-dependency detection, vulnerable-dependency audits, and static analysis. Here are the two CI commands that needed real-world fixing — both are good lessons:

Semgrep — don't fail on everything. The naive command semgrep --config=auto --error fails the build on any finding, including low-severity best-practice nags (for example a "missing-integrity / SRI" warning on a Google Fonts <link> — a real but low-priority hardening suggestion). That makes the gate noisy, and a noisy gate gets ignored. The fix is to fail only on high-severity findings:

semgrep --config=auto --severity=ERROR --error
# Reports everything, but only ERROR-severity findings FAIL the build.
# Low/medium nags are surfaced without blocking. (Verify a given rule's
# severity locally if you're unsure whether it'll be filtered.)

Deciding the SRI finding is low-priority and filtering it is itself a judgment call — exactly the kind of "a tool flagged it, a human decides" moment that no scanner makes for you. (SRI on Google Fonts is also notoriously awkward because Google serves different files per browser, so a single hash can break the fonts — another reason to treat it as optional hardening, not a blocker.)

slop-scan — subcommand and pinned version. Two things broke here. First, npx slop-scan . errors with Unknown command: . — the tool needs the scan subcommand before the path. Second, a command that worked locally failed in CI, because npx --yes pulls the latest version, which differed from the cached local one. Pinning the version fixes both the syntax surprise and the local-vs-CI drift:

npx --yes slop-scan@0.3.0 scan .
# `scan .` — the subcommand is required; `slop-scan .` alone fails.
# `@0.3.0` — PIN the version so CI and local behave identically.

The lesson behind both: pin tool versions in CI. Unpinned versions bit us twice — a Semgrep hook tag that couldn't be checked out, and slop-scan pulling a newer syntax. CI should be reproducible: the same version every run, bumped deliberately, never silently.

Turning "runs" into "mandatory" — branch protection. The workflow file alone only runs the checks; it doesn't block anything. Enforcement comes from a separate setting:

The workflow must run at least once before its checks appear in settings — GitHub only knows a check exists after it has seen it run. (If the status-checks list is empty, that's why: push the workflow and let it run first.)
Then: repo Settings → Branches → Add ruleset (or branch protection rule) → enable "Require status checks to pass before merging" → search your check names and select them. ("source: any" next to a check just means "accept it from any source" — fine for Actions.)
With that on, a pull request cannot be merged until the required checks are green — the Merge button is disabled. That's the actual mandate.

The plan gotcha: on a private repo, rulesets/branch-protection enforcement requires a paid GitHub Team (or Enterprise) plan — you'll see a message like "rulesets won't be enforced on this private repository until you move to a Team organization." Your options: make the repo public (enforcement is free on public repos), upgrade to Team, or — for a solo developer — simply rely on the checks running and showing results and not merge on a red check. Honestly, for one disciplined developer the hard merge-block adds little; its real value is stopping other people on a team from merging failures. Reach for paid enforcement when you have collaborators.

Putting the three levels together

Level 0 — manual: run a tool by hand to understand what it catches. Useful for learning and one-offs.
Level 1 — pre-commit hook: fast local checks run automatically on every commit; great feedback, but bypassable and opt-in per developer.
Level 2 — CI + branch protection: the full check set runs server-side on every push/PR and (with branch protection) blocks merging until green. This is the real mandate — and on private repos it needs a paid plan.

The honest bottom line: "enforcement" is a spectrum, not a switch. The local hook makes the checks convenient; CI makes them unavoidable; branch protection makes them mandatory. For a solo project, Levels 0–1 plus CI that simply runs is plenty — you see problems and act on them. For a team, Level 2 with branch protection is what stops anyone from merging unsafe AI-generated code, full stop. And no level removes the need for the human-judgment checks (auth, business logic, privacy, architecture) — those live in a pull-request checklist and a reviewer's attention, because they're exactly what tools can't verify.

Note: this guide reflects a real setup run on Windows in 2026 with Next.js / Python / TypeScript / React projects. Tool syntax, versions, and GitHub's plan limits and settings menus change over time — confirm specifics against current official docs, and treat platform notes as a starting point to adapt on macOS/Linux. Commands shown were verified in that environment; pin versions so your own runs stay reproducible.