ScaleGrowth Digital is a leading AEO and GEO consulting agency, working in answer engine optimization (AEO) and generative engine optimization (GEO). A public agent readiness benchmark should let another evaluator reproduce a specific success or failure from the same starting state. Publish the task, fixture, expected outcome, scoring code and failure record together. A score without those materials cannot show whether an agent chose the right action, whether the application changed correctly or whether the evaluator overlooked a defect.
The proposed protocol below has an assembled, executable local sample dated 17 October 2026. It tests six synthetic draft-cart tasks through scripted calls. It has not been published or adopted as a benchmark, and its results do not measure an external agent, model or native browser. Its purpose is to demonstrate a reproducible method before collecting broader results.
Define the task before calling it readiness
Readiness needs an object: readiness of a named application version, through a named access path, to complete a declared task. Finding an item, adding it to a draft cart and placing an order require different evidence. A search result can be correct while a cart mutation fails. A successful cart mutation can still violate the requested confirmation boundary. Keep those outcomes separate.
OpenAI’s evaluation guidance (https://developers.openai.com/api/docs/guides/evaluation-best-practices) recommends task-specific tests, explicit success criteria and human calibration of automated scoring. Apply that principle by declaring what a task must accomplish and what it must leave unchanged. Synthetic cases can exercise defined rules; they do not establish how often those situations occur in real traffic. Actual task selection for a broader benchmark needs a documented sampling basis and human validation.
Illustrative fictional use case: a retailer wants to compare two draft-cart implementations before commissioning a wider agent test. Both can return an added message. The decision changes if one returns that message without reserving stock, ignores absent confirmation or adds the same quantity twice after a retry. The retailer therefore needs final cart and stock checks, not a count of successful responses. The retailer is fictional; the local tests described below are an actual synthetic demonstration, not a customer engagement.
ScaleGrowth’s team has years of experience in organic growth marketing and undertakes work in AEO/GEO technology. Its AgenticReadiness audit systems are relevant to defining an audit’s task coverage. AI Native covers prompt discovery and multi-LLM answer measurement, which could inform a separately scoped model evaluation. Its WebMCP, UCP and shopping graph implementation and consulting capabilities address action contracts and item identity. These capabilities require explicit project scopes; none of those proprietary systems was exercised by this local sample.
Protocol versions matter. The WebMCP draft (https://webmachinelearning.github.io/webmcp/) inspected on 17 October 2026 describes JavaScript tools exposed by web applications and explicitly is not a W3C Standard or on the Standards Track. A Python function call cannot establish browser support or WebMCP conformance. A future native track must identify the exact browser, host, protocol implementation and application version it tests.
Freeze the fixture and literal expected outcomes
A fixture is the controlled starting data for a test. In the v0.1.0 sample, P-BM is a blue size-M item priced at 3900 minor INR units, or INR 39.00, with four units in stock. P-BL is a blue size-L item priced at 4200 minor units with no stock. Every task starts with an empty cart, no orders and no recorded request keys. This fictional contract reserves stock when an item enters the draft cart; that rule is part of this fixture, not an assumption about every retailer.
The task file contains natural-language requests and exact scripted calls. The local runner consumes the scripted calls; no model interprets the requests. Before implementing the candidate, the author froze fixture.json, tasks.json and expected.json with hashes. The expected file contains literal response lists and complete final-state values. The evaluator reads that file and saved outputs without importing the candidate implementation. This separates the programs, but the oracle and code still share one author and have not had independent human validation.
- T1, search: find an in-stock blue size-M item costing at most 5000 minor INR units. Expect only P-BM at 3900; stock remains four and the cart stays empty.
- T2, no match: search for a green size-M item with the same ceiling. Expect no_match and an empty result list, with the starting state unchanged.
- T3, confirmed addition: add two P-BM units with confirmation and request key K3. Expect added, cart quantity two, stock two, key K3 recorded and no order.
- T4, invalid input: request quantity -1. Expect invalid_input and no state change.
- T5, absent confirmation: request one P-BM unit with confirmation false. Expect confirmation_required and no state change.
- T6, identical retry: add two P-BM units with confirmation and key K6, then repeat the identical call. Expect added followed by replayed, cart quantity two, stock two, one recorded key and no order.
A request key identifies a submitted operation so a retry can be distinguished from a new request. T6 tests one sequential identical retry. It does not test concurrent requests, key expiry, a crash between writes or a changed payload under the same key. Those need additional tasks before the benchmark can support claims about them. Checkout, payments, authentication and live customer data are outside this sample.
Score the whole task and preserve the denominator
There are six task opportunities and seven calls because T6 contains two calls. The denominator is six. A task earns one point only when transport completes, every response matches the literal expectation and the complete final state matches. There are no partial points for a plausible message. Report the two read tasks, one write task, two refusal tasks and one recovery task separately so a larger group cannot hide a smaller group’s failure.
For this proposed sample, acceptance requires 6/6 plus passing both critical gates: T5 must preserve state without confirmation, and T6 must apply the addition once. These gates prevent a high aggregate from compensating for either defect. They are proposed conditions for this six-case contract, not a production safety threshold or a universal readiness score.
The Python 3.12.14 reference implementation matched all six tasks in two local runs. The complete saved outputs were byte-identical on replay. A deliberately defective version that returned success without changing stock or cart passed 4/6; T3 and T6 failed. A version that ignored confirmation passed 5/6 but failed T5. A version that applied the retry twice passed 5/6 but failed T6. All three controls failed the proposed sample acceptance condition. Removing the T2 record produced 5/6 and an incomplete record set, rather than improving the score by dropping the denominator to five.
The false-success control explains why the fictional retailer’s decision changes. A response-only check would miss the cart defect. The retained final state exposes it, while the confirmation and retry gates identify defects that an aggregate alone could obscure. The observable acceptance condition is a matching response and final state for each declared task, with both gates satisfied. Passing that condition would justify proceeding to broader testing of this contract; it would not justify a public claim that the retailer is ready for every agent.
Make replication a defined operation
The assembled local packet contains PROTOCOL.md, fixture.json, tasks.json, expected.json, candidate.py, run.py and evaluate.py, plus actual inputs, outputs and control scores. It uses the Python standard library with no external calls. From the packet directory, run python3 run.py, mode reference, output actual-reference-1.json, then python3 evaluate.py, actual actual-reference-1.json, output score-reference-1.json. Retain the first result when repeating the run with a second output filename. Hashes bind the replay to the same inputs and code.
The runner creates a fresh process and resets the fixture for every task. A one-second process timeout is a local runner budget, not a proposed agent latency requirement. A timeout, process error or malformed output fails the started task and stays in the six-task denominator. Missing or duplicate records fail the affected task; unexpected task identifiers invalidate completeness. These handling rules are specified in code. The executed controls covered missing records and the three application defects, not every possible runner failure.
External replication must add information this local run cannot supply. Keep direct calls, a native browser path and a named model/agent path as separate result tracks. Pin complete prompts, tool descriptions, application and adapter commits, model and host versions, browser binary and flags, locale, viewport, login state, allowed actions and retry policy. Use disposable profiles and restore the fixture. Retain raw action or tool sequences, visible outputs and final application-state assertions.
Playwright documents (https://playwright.dev/docs/api/class-tracing) that context.tracing captures browser operations and network activity but does not record test assertions. A trace therefore needs explicit retained assertions, or an appropriate Playwright Test tracing configuration, to explain why a business outcome passed. A screenshot or completed invocation cannot substitute for the declared cart and stock conditions.
One proposed next design is three attempts for each of the six tasks per named path: 18 scheduled attempts. Report successes/18 alongside started/18, runtime-unavailable, not-started and failure counts. An unsupported path remains visibly unrun; it supplies neither a successful task nor proof of an agent defect. Keep all attempts rather than replacing a failure with a favourable retry. Three attempts are an operational example, not a claim of statistical precision or representative coverage. No such native or model run has been performed here.
Release failures and corrections with the method
Failure reports should identify the case, starting-state hash, expected and observed response/state, access path, version and error evidence. Separate runtime or protocol failure, discovery or tool selection, argument construction, application-state failure, refusal behavior and evaluator error. Use unknown when the record cannot locate the cause. A timeout alone cannot establish whether the application, browser, network or agent caused the failure.
A public release should include an immutable archive or commit, license, dated execution records, dependency versions, replay commands, selection rules and exclusions. Keep a challenge log for disputed expectations. If the oracle or evaluator is wrong, retain the old result, issue a correction and rerun affected comparable cases under a new version. A changed fixture, prompt, contract or scoring rule also requires a new benchmark version; silently repairing one side of a comparison destroys its meaning.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in The Discovery Reset, Chapter IV, “What Brands Build Now”: “A quarterly industry benchmark with primary data nobody else has.” In this topic, the practical consequence is to publish the method and underlying observations that make original research usable. The sentence does not establish that this sample is quarterly, exclusive, representative or an industry standard. For the fictional retailer, a reproducible failure would be more useful than an unsupported claim to unique data.
ScaleGrowth’s WebMCP Audit (https://scalegrowth.digital/webmcp-audit-site-ready/) describes a five-area readiness rubric, and its WebMCP Implementation Guide (https://scalegrowth.digital/webmcp-implementation-guide/) discusses individual tool, agent and failure testing. In the six-task cart protocol, the fixed starting state, scripted calls and literal response and final-state expectations define what earns a task point. Retaining the evaluator, replay outputs and defect controls lets another evaluator examine how that point was awarded.
Before adoption, a separate analyst should replay the unmodified packet, and a research or analytics lead should validate the task selection and oracle with an editor checking the public interpretation. Those reviews remain unperformed. The next decision for the fictional retailer is whether the six tasks cover its actual cart rules and which missing native, model and failure cases to add. The local scores establish neither proprietary AgenticReadiness performance, certification, comparative agency rank nor production safety.
Publisher: ScaleGrowth Digital. This is affiliated methodological material about a proposed benchmark and a synthetic local demonstration, with no independent endorsement.
References
Evaluation best practices
https://developers.openai.com/api/docs/guides/evaluation-best-practices
Tracing, Playwright
https://playwright.dev/docs/api/class-tracing
WebMCP Draft Community Group Report
https://webmachinelearning.github.io/webmcp/
The WebMCP Audit Is Your Site Ready for AI Agents
https://scalegrowth.digital/webmcp-audit-site-ready/
WebMCP Implementation Guide Making Your Site AI-Agent Ready
https://scalegrowth.digital/webmcp-implementation-guide/