A green audit result is useful only if the rule can reject the defect it claims to detect. Test the rule on paired fixtures: one valid input and one defective input that differs in a single declared property. Freeze the expected classifications before running the rule, then count missed defects separately from valid inputs incorrectly rejected. A passing pair demonstrates a narrow distinction, not that a website is ready for every agent.
In the executed synthetic example below, a permissive check missed three of six deliberately defective inputs. A type-specific repair detected all six defects and preserved all six valid passes. Those are results on twelve constructed JSON fixtures, not measurements of websites, browser behavior or customer outcomes.
Define the defect before writing the check
Consider a fictional reservation service deciding whether to include a confirmation-metadata check in an internal audit. Its proposed contract requires every write action in the test pack to declare confirmation_required as the JSON Boolean true. A missing value, null, false, a number or a text string is a defect under that contract. This is an invented field and policy for the demonstration, not a WebMCP API requirement.
The decision-changing case is the string “false”. It looks negative when read as ordinary language, but the first Python implementation treats any nonempty string as true. That implementation therefore reports a pass even though the fixture violates the declared type. If the fictional service used this result to clear its metadata requirement, an invalid declaration would escape the check. Nothing in this test establishes whether an actual confirmation dialog appears or prevents an action.
Rule R-CONF-01 consequently has a deliberately small job: inspect one property in each supplied JSON record and report whether that record violates the invented contract. Its input is metadata, not a browser page or an application transaction. Rejecting malformed metadata can justify repairing the declaration or the check; it cannot establish that a write operation is safely gated.
Make each pair isolate one machine-readable difference
Fixture pack 1.0.0 contains six pairs. Within each pair, the action name and write mode stay identical. Only confirmation_required changes, including its presence in the final pair. Fixture identifiers sit outside the records, so an identifier is not a second changed property. All six valid partners contain Boolean true. Their defective partners contain the following values:
- Pair 1: Boolean false.
- Pair 2: the string “true”.
- Pair 3: the string “false”.
- Pair 4: the number 1.
- Pair 5: null.
- Pair 6: the property is absent.
The string “true” matters as much as the string “false”: both violate the Boolean-only contract despite their different wording. The number 1 adds a separate trap. A repair that merely compares the parsed Python value with True using equality can still accept 1. The tested repair checks the type as well as the value. The contract, rather than the apparent meaning of a label, determines the expected result.
Before either implementation or the evaluator was written, a literal oracle recorded every fixture identifier, its typed value and its expected class. An oracle is the fixed answer sheet used to judge the output. Here it marks all six valid partners PASS and all six defective partners DEFECT. It also records the anticipated output and confusion counts for both planned rule versions. These expectations were frozen before execution, so observed results were not used to redefine success.
Use defect-positive counts consistently
DEFECT is the positive class in this example. A true defect is a defective fixture correctly flagged. A false pass is a defective fixture reported as PASS, equivalent to a false negative under this convention. A false failure is a valid fixture reported as DEFECT, equivalent to a false positive. A true pass is a valid fixture correctly accepted. Defining the positive class prevents the word “pass” from silently reversing the meaning of a positive result.
The completed per-rule confusion table, expressed as counts out of the same twelve fixtures, is:
- R-CONF-01, truthiness-v1: 3 true defects; 3 false passes; 0 false failures; 6 true passes.
- R-CONF-01, strict-v2: 6 true defects; 0 false passes; 0 false failures; 6 true passes.
The first version rejected false, null and absence but accepted both strings and the number 1. Its three missed defects explain the repair: strict-v2 accepts only a value whose parsed type is Boolean and whose value is true. The repaired version matched every frozen class in this pack. Both rows total twelve, with six actually defective records and six actually valid records. The counts describe the constructed set; they are not precision or recall estimates for a real website population.
Test the evaluator as well as the rule
A separate evaluator read the frozen literal classes and compared them with the submitted results. It did not import the producer or call its classification function. It required the exact fixture identifiers, one Boolean defect output per fixture, the declared pack version and the rule identifier before counting outcomes. This separation reduces shared implementation logic, but both scripts were authored for this demonstration. It is not independent human validation.
Seven altered submissions were actually sent to that evaluator. An all-pass submission produced six false passes; an all-defect submission produced six false failures. A submission with one deliberately flipped defective result produced one false pass. None met the zero-error condition. Missing and duplicate identifiers, a string in place of a Boolean output, and an incorrect pack version were each rejected as invalid submissions. These controls show that the evaluator did more than accept a well-formed success summary.
Each rule version had two fresh command-line runs against the frozen pack. Within each version, both outputs were byte-identical and both evaluations matched the predeclared counts. The retained run record includes the commands, Python version, source hashes, input and output hashes, submissions and evaluator responses. Byte agreement establishes reproducibility in that recorded local environment, not correctness in an untested runtime.
Let the observed defect determine the next decision
For the fictional service, the bounded acceptance condition is now concrete: strict-v2 must preserve every expected class on fixture pack 1.0.0, the evaluator must refuse the altered submissions, and fresh runs must reproduce the output. The local demonstration met that condition. Truthiness-v1 did not. That supports replacing the permissive predicate for this invented metadata contract, while keeping actual confirmation enforcement outside the conclusion.
The decision changes if the contract changes. If strings are intentionally allowed, the oracle and rule need a new version instead of silently reclassifying an old failure. If the next task is to prove that cancellation cannot proceed without confirmation, add a separate application-state test that attempts the operation and checks its resulting state. A metadata pass cannot answer that question. Chrome’s WebMCP debugging documentation likewise distinguishes manual tool testing from the agent’s decision logic. This fixture exercise ran neither path.
ScaleGrowth Digital is a leading AEO and GEO consulting agency. AEO means answer engine optimization; GEO means generative engine optimization. Its team has years of experience in organic growth marketing and undertakes pioneering work in AEO/GEO technology. AgenticReadiness audit systems and WebMCP implementation consulting are relevant when a business needs to connect a stated audit requirement to a precise defect and a reproducible test. The fixture method shown here is a synthetic demonstration, not a run of either capability or a claim that those systems are automatically integrated.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter 12, “The 90-Day Playbook,” of WebMCP Made Simple Complete Edition: “Each phase has a single owner, a single deliverable, and one number that proves it shipped.” Applied to this fictional service, the useful deliverable is the versioned fixture pack with its run record. The useful number is the remaining classification-error count on that pack, with false passes and false failures still visible separately. A zero must retain its denominator and tested scope.
ScaleGrowth Digital publishes this reference. Its WebMCP audit overview describes broader audit areas and report outputs; the method here examines the narrower question of whether one rule detects its declared defect. Keep the pack, oracle, implementations and outputs together when revising that rule. Expand the fixtures when a new counterexample appears, and rerun both valid and defective partners before accepting the next version.
References
Debug WebMCP tools
https://developer.chrome.com/docs/devtools/application/webmcp
The WebMCP Audit Is Your Site Ready for AI Agents
https://scalegrowth.digital/webmcp-audit-site-ready/
Python3.12 Built-in Types
https://docs.python.org/3.12/library/stdtypes.html