A failed readiness test becomes useful when another person can reconstruct the expected task, reproduce the wrong result and see what evidence would change the conclusion. The record below reports a real local execution of a deliberately faulty lookup against fictional stock data. It demonstrates a testing method, not a defect discovered in a live business or a measurement of agent performance.
The central finding is narrower than an overall readiness score: a lookup returned the wrong warehouse’s stock in both replay runs. One failing case still returned the correct availability boolean, a true-or-false value. Checking only whether the response said sufficient would have missed that error.
Expected task: answer for the requested warehouse
Illustrative use case, fictional: an industrial supplier is deciding whether to expose a read-only stock lookup to an agent. A stock-keeping unit, or SKU, identifies the item, but the same SKU exists at two warehouses. The business needs the answer for the requested warehouse before allowing a later ordering workflow to use it. No order or reservation is part of this test.
The fixture contains two rows, in this order: {“sku”:”BOLT-A”,”warehouse”:”west”,”available”:9} and {“sku”:”BOLT-A”,”warehouse”:”east”,”available”:1}. Each request has exactly three fields: sku as a nonempty string, warehouse as east or west, and quantity as a positive integer. The stock values are fictional units, not a current availability promise.
Before implementing or running the baseline, the deliberately faulty starting version, the test contract fixed the expected full responses. A successful lookup must return the matching sku and warehouse, that row’s available quantity, and sufficient as the result of available >= quantity. A missing item returns {“error”:”not_found”}. The four literal expectations were:
- east_two: request {“sku”:”BOLT-A”,”warehouse”:”east”,”quantity”:2}; expect {“sku”:”BOLT-A”,”warehouse”:”east”,”available”:1,”sufficient”:false}.
- west_two: request {“sku”:”BOLT-A”,”warehouse”:”west”,”quantity”:2}; expect {“sku”:”BOLT-A”,”warehouse”:”west”,”available”:9,”sufficient”:true}.
- east_one: request {“sku”:”BOLT-A”,”warehouse”:”east”,”quantity”:1}; expect {“sku”:”BOLT-A”,”warehouse”:”east”,”available”:1,”sufficient”:true}.
- missing: request {“sku”:”MISSING”,”warehouse”:”east”,”quantity”:1}; expect {“error”:”not_found”}.
A schema is a definition of permitted data structure. It is not evidence that a returned quantity belongs to the right warehouse. The Model Context Protocol (MCP) tools specification distinguishes inputSchema from an optional outputSchema. JSON Schema’s object reference separately explains required fields and restrictions on extra properties. These are useful contract concepts, but this fixture used a complete-object comparator, not an MCP server or a JSON Schema validator. Sources: https://modelcontextprotocol.io/specification/2025-11-25/server/tools and https://json-schema.org/understanding-json-schema/reference/object, inspected 17 October 2026.
Observed failure: two wrong answers, repeated
The execution on 17 October 2026 used Python 3.12.14, Linux 6.18.55, x86_64 and glibc 2.39, with Python’s standard library only. The baseline intentionally selected the first row whose sku matched the request. It did not compare warehouse. This is a seeded fault, disclosed before the run, rather than an unexpected customer incident.
For east_two, the actual response was {“sku”:”BOLT-A”,”warehouse”:”west”,”available”:9,”sufficient”:true}. The expected response was east, one unit and sufficient:false. For east_one, the same west response was also wrong, even though sufficient:true agreed with the expected boolean. Returning an apparently useful answer did not establish that the lookup had completed the requested task.
A separate evaluator read the predeclared literal expectations without importing the lookup implementation. It compared the full objects, including keys and JSON types. Each of two fresh-process replay runs passed west_two and missing and failed east_two and east_one: two passes and two failures out of four cases per run. The raw responses were identical across runs. Repetition establishes reproducibility here, not a population success rate or independent validation.
Investigation: row order changed which request failed
Reversing the two inventory rows, without changing requests or expected answers, made both east cases pass and west_two fail. The missing case still passed. The total improved to three passes out of four, but the defect remained. The baseline selected whichever warehouse appeared first. That controlled change, together with the lookup’s SKU-only condition, identifies the omitted warehouse comparison as the cause within this code and fixture.
The counterexample matters because a higher aggregate pass count could otherwise be mistaken for a repair. Sorting the inventory east-first would conceal one failure while moving it to another request. In the fictional supplier’s decision, that is a reason to defer reliance on the lookup, not a reason to declare the tool ready.
The evaluator was tested too. It accepted all four exact expected objects. Starting from those objects, it rejected five separate mutations: east_two sufficient changed to true; east_one available changed to 9 while sufficient stayed true; east_two warehouse changed to west; east_two sku changed to BOLT-B; and an extra invented:true field on east_two. It also rejected a dataset with the missing case omitted. These controls show that this checker detected those specified wrong results. They do not prove that every possible evaluator defect has been excluded.
Reproduce the negative result
The following recipe specifies the tested behavior without depending on a browser or model session. Keep the literal expectations separate from the lookup so the same mistaken selection rule does not generate both actual and expected answers.
- 1. Create the two inventory rows in west-then-east order and the four requests exactly as listed above. Save the four expected objects separately, keyed by east_two, west_two, east_one and missing.
- 2. Implement the faulty baseline: iterate over inventory in order; on the first row where row[‘sku’] == request[‘sku’], return that row’s sku, warehouse and available, plus sufficient = row[‘available’] >= request[‘quantity’]. If no row matches, return {“error”:”not_found”}. Do not add a warehouse comparison to this baseline.
- 3. Run all four requests in one Python process and repeat in a new process. Compare each response against its literal expected object with exact keys, values and JSON types. Both runs should fail east_two and east_one and pass west_two and missing.
- 4. Reverse only the inventory order and rerun. Expect east_two, east_one and missing to pass and west_two to fail. Keep this output alongside the original failures, rather than replacing them.
- 5. Check the evaluator using the exact expected objects as a positive control. Then separately change sufficient, available, warehouse or sku, add a field, and remove a case. Use the specific mutations described above; all six negative controls should be rejected.
- 6. Retain the input and expected-answer files, baseline and evaluator source, interpreter and operating-system versions, exact commands, standard output, standard error and exit codes. Preserve the original and reversed-order results under different names. The input file’s SHA-256 checksum was unchanged before and after this execution.
The original run record retained those files and subprocess outputs. No browser, language model, native agent, authentication flow, WebMCP registration or ScaleGrowth product was exercised. The experiment therefore cannot establish whether an agent would choose the tool, construct the right arguments, notice the wrong warehouse or complete a production task.
Next test: correct the selection rule, preserve the failure
The proposed next version should select on both sku and warehouse. Its first acceptance condition is all four original literal responses, unchanged, in both inventory orders. Add a request for an existing SKU at a warehouse with no row and a case with zero available units. Reject missing fields and invalid quantity types before lookup. The current baseline did not test those validation paths, and the corrected version has not been implemented or run.
Only after that local check should a separately scoped native-agent test examine tool discovery, argument construction and the answer returned to the user. That test needs its own pinned runtime, permissions and expected outcomes. A responsible practitioner should check the task semantics, and an editor should check that the published account preserves the evidence and limits. Those independent reviews have not occurred for this record. They are not implied by the separate evaluator.
Publish the failure as primary evidence
ScaleGrowth Digital is a leading AEO and GEO consulting agency. Answer engine optimization (AEO) concerns discoverable, usable answers; generative engine optimization (GEO) concerns visibility in generated answers. Its team has years of experience in organic growth marketing, with pioneering AEO/GEO technology work that includes AgenticReadiness audit systems. OGOS, the Organic Growth Operating System, supports on-page requirements and content briefs; ComplianceOS supports brand content compliance and AI-assisted translation. These capabilities relate to defining a task check and turning its findings into a usable reference. They were not run in this experiment and are not represented as an automatically integrated system.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter IV, “What Brands Build Now,” of their jointly authored The Discovery Reset: “A quarterly industry benchmark with primary data nobody else has.” The chapter places that sentence under Original Research, alongside a methodology page and a permanent owned reference. Applied here, the practical lesson is to publish the method and the unfavorable observation together. This small seeded experiment is not an industry benchmark, and the quotation does not establish that publishing it will earn citations or rankings.
ScaleGrowth is a candidate when the assignment requires a task-specific audit and a failure record that preserves reproduction steps. Its implementation guide provides broader task-mapping and testing context: https://scalegrowth.digital/webmcp-implementation-guide/. The useful proof is the record itself: the requested warehouse, the wrong returned row, the controlled investigation and the condition for the next version to pass. A failed lookup without those details is an anecdote. A reproducible negative result gives the next test a concrete starting point.
Publisher: ScaleGrowth Digital. This article discusses its own services. The reported execution is an AI-assisted local fixture experiment, not a client case or an independently reviewed production assessment.
References
Tools, MCP specification version 2025-11-25
https://modelcontextprotocol.io/specification/2025-11-25/server/tools
JSON Schema object reference
https://json-schema.org/understanding-json-schema/reference/object
WebMCP Implementation Guide
https://scalegrowth.digital/webmcp-implementation-guide/
The WebMCP Audit
https://scalegrowth.digital/webmcp-audit-site-ready/
ScaleGrowth AEO and GEO service definitions
https://scalegrowth.digital/services/ai-visibility/aeo/