From Manual Testing to Automated Validation
Manual rule testing: a human runs test prompts, reads the AI output, and judges whether the rules are followed. This works for: initial rule creation (5 prompts, 15 minutes) and occasional validation (quarterly reviews). It does not scale for: continuous integration (every rule change should be validated), regression detection (catching when a new rule breaks an existing pattern), or large rule sets (50+ rules need systematic, not ad-hoc, testing).
Automated validation: a script that runs test prompts through the AI API, parses the output for expected patterns (keywords, code structures, naming conventions), and reports pass/fail per prompt. The script: runs in CI after every rule file change. If a test fails: the CI blocks the deployment with a specific error ('Rule regression: the error handling prompt no longer generates the Result pattern'). The developer: fixes the rule and re-submits.
The automation spectrum: Level 1 — keyword checking (does the output contain 'Result.ok' and not contain 'try-catch'?). Level 2 — structure checking (does the output have a function with specific return type and error handling?). Level 3 — full AI evaluation (a second AI evaluates whether the output follows the rules). Most teams: start at Level 1 (simple, fast, catches 80% of regressions) and add Level 2-3 as the rule set matures. AI rule: 'Level 1 keyword checking: 30 minutes to implement, catches most regressions. Start here.'
Step 1: Design the Automated Test Suite
Each test case: a prompt, an expected pattern, and optional negative patterns (what should NOT appear). Format: { prompt: 'Create a function that fetches a user by ID and handles errors', expected: ['Result.ok', 'Result.err', 'async function'], notExpected: ['try {', 'catch ('], ruleTested: 'error-handling-result-pattern' }. The test: runs the prompt through the AI API, checks the output for expected strings, and reports pass (all expected present, no notExpected present) or fail (with specifics about which check failed).
Test coverage: one test per major rule category. Minimum: error handling (1 test), naming conventions (1 test), testing patterns (1 test), security rules (1 test), and framework patterns (1 test). Total: 5 tests. Each test: exercises 3-5 rules in its category (the prompt is designed to trigger multiple rules). The 5-test suite: validates the majority of the rule file with minimal execution time (5 API calls × 3-5 seconds each = 15-25 seconds total).
Test maintenance: when a rule changes, update the corresponding test's expected patterns. When a rule is added: add a test (or extend an existing test to cover the new rule). When a rule is removed: update the test to no longer expect the removed pattern. The test suite: evolves alongside the rules. AI rule: 'The test suite mirrors the rules. Every rule change: check if a test needs updating. A test suite that does not match the rules: produces false positives or misses real regressions.'
The entire validation suite: 5 API calls. Each call: ~5 seconds (prompt + generation + response). Total: 25 seconds. The CI pipeline: adds 25 seconds to the build time when the rule file changes. The value: catches rule regressions that would otherwise reach the team and cause inconsistent AI output for days or weeks. 25 seconds of CI time: prevents hours of debugging. The ROI: impossible to argue against.
Step 2: Implement the Validation Script
The script (validate-rules.test.ts or validate-rules.js): reads the test suite definition (a JSON file or inline definitions), calls the AI API with each prompt (using the Anthropic API, OpenAI API, or the tool's CLI), captures the AI's response, checks for expected and notExpected patterns in the response, and reports results (pass/fail per test, with details on failures). Total script: 50-100 lines. Dependencies: the AI provider's SDK (already in the project for development use).
API considerations: the validation script calls the AI API programmatically. Cost: 5 prompts × ~1,000 tokens each = ~5,000 tokens per run (~$0.01-0.05 depending on the model). Speed: 15-25 seconds total. The cost and speed: negligible for CI runs. For frequent runs (every PR): use a smaller, faster model for validation (Haiku or GPT-4o-mini) that is sufficient for checking pattern compliance without needing the full model's capabilities.
Handling AI variability: AI output varies slightly between runs (different wording, different variable names, different code structure — same pattern). The validation: should check for patterns (keywords, structural elements), not exact string matches. Expected: 'Result.ok' (a keyword that appears regardless of the surrounding code). Not expected: the exact function body (which varies between runs). Pattern-based checking: robust against AI variability. Exact matching: brittle and fails on valid output. AI rule: 'Check for patterns and keywords, not exact output. The AI generates different code each run. The patterns: consistent. The exact text: variable.'
Test expectation: 'The output should be exactly: async function getUser(id: string): Promise<Result<User, AppError>> {...}'. This test: fails on every run (the AI generates slightly different code each time — different variable names, different formatting, different code structure). Better: expected: ['Result<', 'AppError', 'async function']. This: passes consistently because the AI always uses these keywords regardless of the surrounding code. Pattern matching: robust. Exact matching: brittle.
Step 3: CI Integration and Regression Detection
CI workflow: add the validation script as a CI step that runs whenever the rule file changes. In GitHub Actions: on: push: paths: ['CLAUDE.md', '.cursorrules']. The step: runs the validation script. If all tests pass: CI continues. If any test fails: CI blocks with the failure details. The developer: sees exactly which rule regression occurred and which test caught it.
Regression detection: the most valuable aspect of automated validation. Scenario: a developer adds a new rule that conflicts with an existing error handling rule. The error handling test: fails ('expected Result.ok but found try-catch'). Without automated validation: the conflict goes undetected until a developer notices inconsistent AI output days or weeks later. With automated validation: caught in CI within minutes of the rule change. The fix: resolve the conflict before the rule reaches the team.
Baseline snapshots: for Level 2+ validation, capture a baseline snapshot of AI output for each prompt (the output when the rules are known to be correct). Future runs: compare against the baseline. Significant deviations: flagged for review (even if the keyword checks pass). This catches: subtle changes in AI behavior that keyword checking misses (the output has the right keywords but the structure changed). AI rule: 'Baseline snapshots: the advanced technique. Keyword checking: sufficient for most teams. Add baselines when the team needs to detect subtle behavioral changes, not just keyword-level regressions.'
A developer adds a new rule: 'Use try-catch for all error handling in middleware.' The existing error handling test: checks for 'Result.ok' in the output. The new rule conflicts: the AI now generates try-catch instead of Result.ok for some prompts. The test: fails in CI. The failure message: 'Expected Result.ok in error handling output. Found try-catch instead. The new middleware rule may conflict with the existing error handling rule.' Without automation: the conflict reaches the team. Developers are confused for days. With automation: caught in CI. Fixed before deployment.
Automated Validation Summary
Summary of automatically validating AI rules.
- Manual testing: works for initial creation and quarterly reviews. Does not scale for CI or regression detection
- Automated: test prompt + expected patterns + notExpected patterns. 5 tests minimum, 15-25 seconds total
- Level 1: keyword checking (contains 'Result.ok', not contains 'try {'). 80% of regressions caught
- Level 2: structure checking (function shape, return type, error handling pattern). More precise
- Level 3: AI evaluation (second AI judges whether output follows rules). Most comprehensive
- Script: 50-100 lines. AI API calls. Pattern-based matching (not exact string matching)
- CI: runs on rule file changes. Blocks deployment on regression. Developer sees exact failure
- Cost: ~$0.01-0.05 per run. Speed: 15-25 seconds. Negligible overhead for high-value protection