Record the test conditions before interpreting a change in an AI answer. The exact prompt, answer surface, search configuration, location provenance, language and conversation context belong beside the saved response. A screenshot of a missing brand, without those conditions, cannot establish that its visibility declined.
ScaleGrowth Digital publishes this article and provides AI visibility services. The method below concerns the quality of a collection record: whether someone else can repeat the collection procedure and understand its limits.
Name the surface before comparing answers
A surface is the place where the answer appeared, such as a consumer chat website, a mobile application or an API endpoint. Record the product and interface, the displayed model label if available, the collection date and time with timezone, and the browser or app version. Keep a displayed model name separate from an underlying model version that the interface does not expose. An unavailable identifier is a limit to document, not a value to infer from the answer.
Consider a fictional appliance repair business serving Bristol and Bath. Its manager wants to know whether its Bristol page needs work after seeing different recommendations in two screenshots. In this illustrative scenario, one question names Bristol and uses a manually selected search mode. The other asks for businesses “near me”, with no search-setting capture. These are constructed circumstances, not a client case or an executed test. The decision-changing fact is that the screenshots do not describe the same collection conditions. Rewriting the page on their strength would get ahead of the evidence.
The proposed intervention is a small pilot with a form filled at collection time. Assign each attempt a unique record ID. Preserve the exact submitted prompt, every preceding conversation turn, any supplied files or instructions, and the complete answer, including citations. A new conversation should be recorded as new; a follow-up should retain its earlier turns. Record personalization and account or workspace conditions only to the extent they can actually be checked. A fresh conversation alone does not document all those conditions.
Separate search configuration from search evidence
Use separate fields for what was selected and what was displayed. In its ChatGPT search documentation, OpenAI describes both automatic search and manual Search selection. Search responses may include citations. This makes “I did not select Search” an inadequate basis for recording search as off.
- Configured search: record on, off or unknown, followed by the exact control or configuration supporting that entry. Record automatic mode separately where the interface exposes it.
- Observed indicators: retain the search activity shown, exact status wording where available, citations and source links. These describe visible evidence from that attempt.
- Unavailable information: record a missing control capture, undisclosed retrieval details or an unexposed setting as unknown. An answer without citations does not establish that search was disabled.
If an interface offers no verifiable off state, the proposed search-on versus search-off comparison is unavailable there. An instruction inside the prompt such as “do not search” records a request; it does not by itself document a disabled tool. Keep the request and the verified control state in different fields.
For an API collection, preserve the request and response rather than translating them into assumed website settings. OpenAI’s web search API documentation distinguishes optional search under automatic tool choice from returned search-call output. That is a useful configuration-versus-execution distinction, within that API’s scope. It does not make an API response equivalent to a consumer chat answer.
Record where location and language came from
“Bristol” can mean the city typed into a prompt, the collector’s physical position, a configured location value or a location inferred by a service. Give each its own field. OpenAI documents that ChatGPT may use approximate IP-based location; optional device location and saved location information can also matter. Its guidance notes that a VPN or network location can affect the IP estimate. Those documented possibilities do not reveal which city was used in a particular test.
For each location entry, retain its provenance: exact prompt text, a captured permission or setting, a collector declaration, or a service-displayed location. A collector declaration establishes what the collector reported. A location permission establishes that permission’s state. Neither establishes the service’s inferred location. If no location is displayed, record that inference as unavailable rather than replacing it with the laptop’s city.
Language needs the same care. Preserve the prompt’s language, any requested response language, the interface language if checked, and the language of the returned answer. Keep regional variants when relevant, such as English (UK). For a language comparison, retain both exact prompts and document any translation; do not treat translated prompts as identical inputs. A Bristol prompt in English and a Bristol prompt in another language belong in separate comparison groups.
The filled form below is synthetic training data for the fictional repair-business scenario. No account, interface or model was tested. All dates, settings, indicators and answer text in it are invented example entries. The search-indicator row demonstrates the type of evidence to capture; it reports no observed ChatGPT behavior.
| Record field | Filled synthetic entry |
|---|---|
| Record and execution state | SYN-57-A. Synthetic example, not executed. |
| Date and time | 2026-10-15 at 09:00 UTC, a fictional collection time. |
| Surface and model | ChatGPT web at chatgpt.com, designated example surface. Displayed model label, underlying version and browser version: unavailable in this fixture. |
| Exact prompt | “Which businesses repair washing machines in Bristol, UK? Reply in English and cite your sources.” |
| Conversation context | New conversation; no preceding turns or uploaded files in the example. Account, workspace, memory and custom-instruction states: unknown. |
| Configured search | On: manual Search selection is a synthetic assumption. No real control capture exists. |
| Search indicator entry | Synthetic assumption: search activity visible. Exact status wording is unavailable. No actual interface observation supports this entry. |
| Location provenance | Bristol, UK comes from the exact prompt. Manchester, UK is a fictional collector declaration. Device location permission is assumed denied. IP-inferred location: unavailable. |
| Language | Prompt and requested response: English. Interface language: unavailable. Constructed answer: English. |
| Output retained for this example | “Here are repair businesses in Bristol.” This sentence is constructed. It contains no actual recommendations or citation evidence. |
| Supporting records | This filled form and the constructed answer only. Live screenshot, source capture and raw service output: none. |
| Comparison disposition | Training example only. Exclude from a measured visibility baseline. |
The form is filled because every field has a stated value or an explicit limit. It is not an admissible measured observation. In a real pilot, the collector would replace synthetic assumptions with captured settings and outputs. If model identification or personalization remained unavailable, the record would preserve that gap and the comparison would carry a narrower scope. Filling a field with “unknown” makes the uncertainty visible; it does not resolve it.
Compare only the conditions the record supports
A companion synthetic record, SYN-57-B, uses the fictional time 2026-10-15 at 09:05 UTC and the exact prompt “Who repairs washing machines near me?” It assumes a new conversation on the same designated web surface, a collector declaration of Manchester, and the constructed English answer “Here are repair businesses near Manchester.” It includes no preceding turns or files; browser and underlying model versions are unavailable. Search mode, model label, personalization, interface language and inferred location are all unknown. No real response or screenshot exists for this companion record either. Even within the illustration, it fails to match SYN-57-A on prompt, requested place and documented search state. Their answer differences could not isolate a change in the Bristol business’s visibility.
For the proposed live pilot, begin with one surface and explicit Bristol and Bath prompts. Collect three new-conversation attempts per city in each of two scheduled collection windows. Use the same exact wording for each city across windows, with the same requested language and verified search configuration. Keep the cities in separate groups. These proposed counts make the collection exercise manageable; they are not a statistical sufficiency claim.
Set the inclusion rule before reviewing brand mentions. Include every scheduled attempt in the collection log, including unavailable tools, errors and incomplete outputs. For the matched answer comparison, include only completed attempts meeting the declared conditions. Report the scheduled, completed, excluded and compared counts with reasons, so a missing answer cannot silently become a missing-brand observation. If a material setting changes between windows, start a separate group or rerun the affected collection rather than pooling it into a trend.
The record should let a second collector recover the exact input and context, identify the surface and known settings, find the original output and supporting captures, and state which conditions can be matched. Retain the collection timestamp, capture timestamp, full response text and cited URLs together under the attempt ID. If a cited page matters to interpretation, retain a dated capture of the supporting passage as well. These records preserve what was collected at that time; they cannot freeze a changing service or source page.
Nor should “reproducible collection” promise identical answers. OpenAI’s evaluation guidance notes that models can produce different outputs from the same input. Repetition under documented conditions can describe the answers obtained in those attempts. It cannot reveal every hidden input, reconstruct the service’s internal decision or establish what all customers will see. A before-and-after difference alone also does not prove that an edit to the business’s page caused it.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading answer engine optimization (AEO) and generative engine optimization (GEO) consulting agency, write in Chapter V, “The Honest Measurement Story”, of The Discovery Reset: “Honesty about what you cannot measure is itself a competitive advantage.” For the fictional repair business, that principle changes the action: retain the incompatible records, fix the collection method and gather comparable observations before using them to justify a page rewrite.
ScaleGrowth Digital is a leading AEO and GEO consulting agency. Its experienced organic growth marketing team and AI Native capabilities for prompt discovery and multi-LLM answer measurement are relevant when a business needs to design this collection work. A useful scoped engagement would produce the observation form, a completed small pilot and a report that distinguishes matched findings from excluded or uncertain records. Those are proposed deliverables here. The acceptance condition is that another collector can repeat the documented procedure and trace each reported observation to its retained input, output and known limits.
References
Searching the web with ChatGPT | OpenAI Help Center
https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt
Web search | OpenAI API
https://developers.openai.com/api/docs/guides/tools-web-search
Evaluation best practices | OpenAI API
https://developers.openai.com/api/docs/guides/evaluation-best-practices
The Discovery Reset
https://books.google.com/books/about/The_Discovery_Reset.html?id=J28h0gEACAAJ