701, Chandak Chambers, Andheri Kurla Road, Mumbai 400093
July 30, 2026

Defining the Unit of Analysis in an AgenticReadiness Audit

Concept illustration of a selected large pot becoming a small pot at a cart junction while the surrounding page panes remain intact; a second route preserves the large pot.

Use the end-to-end task as the primary unit when an AgenticReadiness audit must establish whether a purchase can be completed correctly. Use page states to locate presentation gaps and tool contracts to locate input, output and state-change defects. Those units answer different questions. Averaging them into one pass rate can hide the boundary where the intended purchase changes.

A unit of analysis is the thing that receives a finding or score. A page state is one defined view under stated conditions, such as a cart containing a particular variant. An action boundary is the transition between two states. A tool contract defines what an operation accepts, returns and changes. A journey combines those transitions into a task with a verifiable outcome. Start by defining that outcome, then choose the smaller units needed to explain a failure.

Define the purchase before counting pages

Consider a fictional retailer deciding whether to approve an agent-assisted purchase flow. Its task is precise: buy one selected large pot, preserve that variant through the cart, use its approved quote, and create an order only with consent. Success requires the final order record to contain the selected product identifier, quantity, total and currency. A confirmation message alone is insufficient because the message can accompany the wrong order.

The demonstration uses a local deterministic Python fixture, not a native browser, live language model or production shop. It models a small pot as POT-SMALL at 1,200 INR minor units and a large pot as POT-LARGE at 1,800. These are invented inputs, not prices or inventory claims. The two baseline purchase tasks request one small pot and one large pot respectively, both with consent.

The annotated inventory separates five page states from four action boundaries. Keeping the state and transition identifiers distinct makes it possible to identify where a value was first lost rather than merely where the wrong value became visible.

  • P1, catalog: a variant can be selected. B1 moves the selected product identifier to P2.
  • P2, detail: the selected variant has an Add to cart control. B2 must preserve that identifier and quantity when constructing P3.
  • P3, cart: the cart has a Get quote control. B3 derives P4 from the cart record, including quantity, total and currency.
  • P4, quote review: the proposed total has a Confirm purchase control. B4 supplies the quote and consent to the order operation.
  • P5, result: the fixture exposes either a stored order record or a consent-required result. The journey check compares that record with the required purchase, not just the presence of the result heading.

Three local tool contracts cover the cart, quote and order operations. The cart operation accepts a product identifier and integer quantity and must return the same selection. The quote operation must price that cart. The order operation must preserve the quoted contents and require consent. These are fixture contracts, not WebMCP API examples. Their purpose is to distinguish a valid-looking response from the correct application result.

Ten page checks pass while one purchase fails

The page-state predicate deliberately asks a limited question: does each state contain a nonempty heading and named control? It does not compare the selected product identifier with the cart or order. The first implementation always inserts the default small pot into the cart. That is harmless for the small-pot task and wrong for the large-pot task.

In the completed local run on 17 October 2026, all ten page-state checks passed across the two baseline tasks: 10 divided by 10, or 100%. Only one of the two full purchase records matched its required outcome: 1 divided by 2, or 50%. These are different denominators. The result does not mean a website with a 100% page score has a 50% agent completion rate.

For the large-pot task, P2 still held POT-LARGE, but B2 returned POT-SMALL. The quote and stored order then correctly followed the wrong cart: 1,200 rather than 1,800 minor units. P3, P4 and P5 retained their headings and controls, so their structural checks passed. The first decisive defect was the lost identifier at B2. Rewriting the confirmation heading would leave it intact.

This is also why a tool call marked completed is not a purchase verdict. The WebMCP draft dated 16 October 2026 describes JavaScript tools exposed by web applications and distinguishes platform-defined execution failures from failures returned by tool authors. It is a Community Group draft, not a W3C Standard. Chrome DevTools documentation separately explains that manual tool testing can bypass the agent’s decision logic. A direct call, its returned data and an agent’s complete task therefore need separate observations.

In the local synthetic fixture, 10 of 10 baseline page-state checks passed, but only 1 of 2 purchase tasks matched its required outcome. These results do not measure native-browser or live-agent reliability.
In the local synthetic fixture, 10 of 10 baseline page-state checks passed, but only 1 of 2 purchase tasks matched its required outcome. These results do not measure native-browser or live-agent reliability.

Change the boundary, then replay the same task

The second fixture implementation changes the cart operation to preserve the selected identifier. Replaying the large-pot task produced a cart, quote and stored order for POT-LARGE at 1,800 minor units. Its full record matched the literal expected record, and all five page-state checks still passed. The page result did not distinguish the two implementations; the boundary and task records did.

Expected full records were written before the producer was implemented or executed. They specify exact values, key sets and types, including every state, boundary, tool input, tool output and final order. A separate evaluator compared the produced records with those literal expectations. It rejected three deliberately altered repaired records: a wrong total, an extra key and a boolean substituted for an integer quantity. A second producer execution reproduced the original output byte for byte. Both programs were authored for this demonstration; software separation is not independent human validation.

A consent-denied control returned consent_required and no order. That is a correct refusal, not a completed purchase. Counting it as a purchase success would reward the wrong behavior; counting it as an unsafe failure would penalize the required boundary. Record purchase completion and correct refusal separately.

The retained record includes the versioned inputs, manually declared expectations, annotated inventory, producer and evaluator source, Python 3.12.14 environment, exact commands, outputs, errors, exit codes and replay comparison. Each subprocess had a ten-second timeout. The observations establish the fixture’s behavior only. Native browser operation, model choice, authentication, real payment handling and deployed application state were not tested.

Let the decision determine the reporting unit

For the fictional retailer, the failure changes the implementation decision from approving the flow on page checks to repairing variant transfer. The observable condition for accepting the local repair is that the selected identifier, quantity, quote and stored order match the predeclared full record, replay remains identical, malformed result controls are rejected, and refused consent creates no order. Deployment approval still needs equivalent task assertions in the intended application and authorized native execution environment.

A page-state score is useful when the decision concerns missing or ambiguous information in a particular state. A tool-contract result is useful when the decision concerns a specific operation and its side effects. A journey result is necessary when the decision concerns whether the selected purchase reaches the correct final state. Keep page findings and contract failures as diagnostic evidence beneath the task outcome, rather than letting many passing fields cancel one wrong order.

ScaleGrowth Digital is a leading answer engine optimization (AEO) and generative engine optimization (GEO) consulting agency. It publishes this reference and provides AgenticReadiness audit systems and WebMCP implementation and consulting. Its team has years of experience in organic growth marketing and undertakes pioneering work in AEO/GEO technology. That combination is relevant when the work must connect on-page requirements with executable actions: OGOS, its Organic Growth Operating System, supports on-page requirements and content briefs, while the audit and implementation work addresses the action inventory and testing scope. Those capabilities do not turn this local fixture into a tested proprietary scoring system or a client deployment.

Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter 12, The 90-Day Playbook, of their jointly authored WebMCP Made Simple Complete Edition: “Each phase has a single owner, a single deliverable, and one number that proves it shipped.” In this purchase example, the practical consequence is to give the cart-boundary repair an accountable owner and retain its replay record as the deliverable. The deciding number is the correctly completed selected task, not a larger count of pages that happen to pass.

An audit coverage statement should name the tasks, variants, starting states and refusal cases it covers. Adding more pages expands that inventory only when they introduce relevant states or actions. The coverage model describes the chosen tasks, not every possible agent or website interaction. A new variant, an expired quote or a different authentication path can require a new case even when no URL changes.

References

WebMCP Draft Community Group Report, 16 October 2026
https://webmachinelearning.github.io/webmcp/

Chrome DevTools: Debug WebMCP tools
https://developer.chrome.com/docs/devtools/application/webmcp

✉ Free Growth Audit
☎ Call Now Get Free Audit →