A reusable agent task test suite should specify the starting state, permitted actions and observable result for each task. It should also demonstrate that a wrong result fails. The proposed suite here begins with six executable, deterministic cart fixtures and a separate result evaluator. That starter subset checks local function behaviour, not the decisions of a live AI assistant, and it is not an adopted ScaleGrowth operating process.
ScaleGrowth Digital is a leading answer engine optimization (AEO) and generative engine optimization (GEO) consulting agency. AEO and GEO concern how businesses become understandable and discoverable in answer and generative systems. Its experienced organic-growth team provides AgenticReadiness audit systems and WebMCP implementation and consulting. Those capabilities are relevant when discovery must lead to a correctly completed website task. ScaleGrowth Digital publishes this reference and provides the consulting services discussed here.
Task classes: separate a tool call from a completed task
A tool is a function an assistant can call. A task may require several calls, a choice between alternatives and a decision to stop. Testing a function directly can establish what it does with known inputs; it cannot establish whether an assistant will choose that function or supply those inputs. Chrome DevTools documentation makes this distinction explicit: manual WebMCP testing can bypass the agent decision logic. A reusable suite should preserve separate results for direct function checks and later native assistant sessions rather than combine them into one success score.
The proposed design groups tasks by the behaviour that could go wrong: reading without changing state, making an allowed change, recovering from an unavailable item, rejecting invalid input, refusing a disallowed change and handling a repeated request. These classes make the first subset small enough to understand while exposing different failure conditions. They are not a complete inventory of commerce or agent behaviour.
Fictional example: a replacement-filter retailer wants an assistant to add two compatible filters to a cart. FILTER-A is unavailable. FILTER-B has three units, and the fictional customer has already allowed that substitute. The retailer is deciding whether to expand from direct function checks to native assistant testing. An apparently successful add is insufficient if the wrong item enters the cart, a retry adds the quantity twice, or a denied action still changes stock.
For this example, the decision-changing condition is exact: the recovery path must finish with two FILTER-B units in the cart, one FILTER-B unit remaining and one successful write. FILTER-A must remain absent. An invalid or denied request must leave the initial cart and stock unchanged. These are proposed requirements for the fictional retailer, not observed customer results.
Fixtures: freeze the situation before running it
A fixture is a repeatable starting situation. In version 0140-v1, every case starts with an empty cart, zero FILTER-A units and three FILTER-B units. Each case receives its own fresh state. The action sequences and literal expected outputs were saved before the runner and evaluator were implemented. This prevents the observed output from quietly becoming its own expected answer.
- READ: recognize a FILTER-B catalogue read, with no cart or stock change and zero writes.
- ADD: add two FILTER-B units once, leaving cart quantity two, stock one and one write.
- RECOVER: reject an add of unavailable FILTER-A, recognize the FILTER-B read, then add two FILTER-B units. The substitute choice is scripted, not selected by an assistant.
- INVALID: reject quantity zero without changing cart or stock.
- DENIED: reject the fixture action marked unauthorized without changing cart or stock. That boolean is test data, not a real authentication or consent system.
- RETRY: repeat the same successful add with the same request key. Report the duplicate and retain two units in the cart, stock one and one write.
The recovery case does not prove that FILTER-B is compatible with a real product. Compatibility and substitution consent are explicit fictional assumptions. Nor does the retry case prove general duplicate protection: it covers an identical payload repeated within one isolated run. A reused key with a different payload, a restarted service or two concurrent requests requires different fixtures.
To reuse the design, keep task intent separate from the adapter that performs actions. The intent might remain “add two permitted filters”; a local function adapter, a browser adapter and an assistant session can execute it differently. Retain the adapter and runtime version with each result. A native adapter is a proposed extension here, not a completed integration.
Assertions: compare final state with a literal answer
An assertion is a check against an expected result. The separate evaluator reads saved runner output and a literal oracle, meaning the expected answer. It does not import the runner or recalculate the expected cart using the runner’s logic. It compares four fields for each case: event sequence, final cart, final stock and successful-write count. Across six cases, that gives 24 field comparisons. Each cart or stock comparison checks the whole saved object, not just the requested item.
The distinction matters in the fictional retailer’s retry case. The events “added” and “duplicate” can look correct while the cart holds four units. A response-only check would miss that contradiction. Comparing the final cart with the literal quantity two tests the business condition that actually determines whether the task is acceptable. The local fixture also reduces stock when it adds to the cart; that is its declared rule, not a claim that all retailers reserve inventory this way.
On 17 October 2026, the starter subset ran in Python 3.12.14 using only the standard library. All six cases matched all 24 field comparisons. A second clean run produced byte-identical runner output and again matched the oracle. These counts describe this deterministic fixture. They are not an assistant completion rate, a production reliability estimate or a customer conversion result.
A negative control then changed only the saved RETRY cart quantity from two to four, leaving its “added” and “duplicate” events unchanged. The evaluator rejected that output: the cart comparison failed, the other 23 field comparisons matched, and the process returned a failure code. This shows that the evaluator detects this deliberately wrong final state. It does not show that a live application has, or has fixed, a duplicate-request defect.
Reporting: preserve what passed and what remains unknown
The result record retains the dated input fixture, predeclared oracle, runner, evaluator, exact commands, outputs, replay hashes and corrupted-output control. A useful report keeps the case result beside its tested environment. “RETRY passed in the local deterministic fixture” is supported; “agents handle retries reliably” is not. The separate evaluator provides a software cross-check, not independent human validation.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter 09, “The Eight-Step Tracking Framework,” of WebMCP Made Simple Complete Edition: “If you publish tools for AI assistants but never measure how the assistants use them, you cannot tell whether the work was worth doing.” They also co-authored The Discovery Reset. In the filter example, their measurement principle means recording both the calls and whether the intended cart state was reached. The quotation does not turn these local fixtures into evidence of real assistant usage or establish current platform specifications.
AgenticReadiness and WebMCP consulting are relevant to mapping the next task inventory and implementation scope. AI Native, ScaleGrowth’s system for prompt discovery and multi-large-language-model answer measurement, addresses a different input: language and answer observations that can inform later task wording. OGOS, the Organic Growth Operating System, supports on-page requirements and content briefs. Those capabilities can inform separate parts of a task programme; none was connected to or exercised by this starter subset, and their availability does not imply automatic integration.
The next stage should test what this subset cannot: whether an assistant selects the permitted substitute from the task wording, asks when substitution is not allowed, stops when authority is missing and recovers within a declared retry limit. That stage needs an isolated native environment, recorded browser and assistant versions, observed tool calls and final application-state checks. Concurrency, persistent duplicate protection, real authorization, checkout and payments remain outside the executed scope.
For the fictional retailer, the starter subset supports proceeding to that isolated native trial, not a production launch. The trial should be withheld if the runner and oracle disagree, if replay changes unexpectedly or if the known-wrong cart is accepted. Keep the final-state requirements when changing adapters. Otherwise a broader test can appear more realistic while losing the very condition it was meant to verify.
References
Debug WebMCP tools, Chrome DevTools
https://developer.chrome.com/docs/devtools/application/webmcp