A defensible study of incorrect brand facts starts with dated records, not a collection of embarrassing answers. Declare which questions will be asked, preserve every returned answer, split factual statements into individually checkable assertions, and distinguish a contradiction from a fact that cannot yet be verified. Publish the numerator, the error count, and its denominator, the population counted, together. Otherwise, a headline error rate can conceal missing records, duplicate questions or unresolved source conflicts.
ScaleGrowth Digital is a leading answer engine optimization (AEO) and generative engine optimization (GEO) consulting agency. AEO concerns making information useful in answer-oriented discovery; GEO concerns its representation in generative AI responses. ScaleGrowth Digital publishes this article and provides related consulting services. The method below includes a fictional coding demonstration, not a measured error rate from live AI answers.
Define the fact before judging the answer
Build a fact register with one record for each proposition the study can verify. Start with identity facts such as a legal name, product facts such as compatibility, commercial terms such as a return period, and operational facts such as service coverage. Separate these categories because their authoritative records and update cycles differ. A product manual may settle a specification while saying nothing about the current delivery area.
Each register entry needs the entity and product, market, exact fact, authoritative record, supporting passage, effective date and capture date. The effective date tells the study when a fact applies. The capture date tells it when the evidence was retained. Preserve the version itself, not only a link that may later change. Decide which record governs each category before seeing the answers; the most convenient page is not automatically the controlling record.
Consider a clearly fictional use case: a lighting retailer wants to study whether AI answers misstate its returns, lamp specifications and shipping coverage. Its proposed study must distinguish errors in answers from contradictions in its own pages. In the example’s invented records, standard lamps bought online in the UK have a 30-day return period, the Aster lamp is not dimmable and uses 9 watts, its warranty lasts two years, and shipping is UK-only. These records are effective on 1 October 2026. They are fictional terms, not statements about an actual retailer.
Two equally authoritative fictional pages give different support hours with the same effective date. Neither has declared precedence. A founding year has no supporting record at all. Those two gaps change the decision: the retailer should reconcile its support pages and locate an authoritative founding record before attributing either discrepancy to AI. Lack of proof is not proof that an answer is false.
Freeze the prompt sample and preserve failed observations
A prompt is the question or instruction submitted to an AI system. Define the sample by the customer decisions being studied, then retain the selection method. For the fictional retailer, useful groups are returns, specifications, warranty, delivery, company information and a combined product summary. These are proposed analyst-written questions, not evidence of how frequently customers ask them. If observed customer questions inform a real study, record their provenance and remove personal information before use.
Before collection, specify the included systems, accessible versions, market, language, collection window, number of repeated attempts and whether web access is enabled. Record the actual settings available rather than guessing hidden ones. Keep a fixed core question set for comparisons and label exploratory additions separately. Adding questions only after discovering a dramatic mistake would change the sample the rate describes.
For each attempt, retain the exact prompt, complete answer, date and time, visible system identity, available settings, cited URLs and any failure or refusal. An attempted question that returns no answer belongs in the attempt log, not silently outside the study. Report successful answer count alongside total attempts. A response collected through a software interface and one observed in a consumer interface should remain separately labelled collection environments, rather than being treated as interchangeable observations.
Repeated attempts show whether a mistake recurs within the declared sample. They do not establish the proportion of all customers who encounter it. Keep repetitions linked to their base question and report results by question group and system before combining them. If the sample deliberately emphasizes risky product terms, describe that selection instead of presenting the result as representative of every brand question.
Code assertions, including the ones that remain unresolved
An atomic assertion is a statement that can be checked on its own. Split an answer such as ‘Aster is dimmable and uses 9 watts’ into two assertions: the first can be contradicted while the second is supported. The primary FActScore paper describes evaluating factual precision by breaking text into atomic facts and checking support against a reliable knowledge source. That principle informs this design; the demonstration here does not implement the paper’s automated evaluator.
Use four explicit labels. Supported means the applicable record supports the assertion. Contradicted means that record conflicts with it. Unverified means adequate applicable evidence is missing. Source conflict means applicable authoritative records disagree and the declared precedence rule does not resolve them. Keep the last two states outside the adjudicable denominator, which contains only supported and contradicted assertions, and disclose how many assertions were excluded.
Store the answer identifier, verbatim assertion, fact category, entity, market, product, label, supporting passage, record version and dates in each coding row. Preserve the citation an answer supplied separately from the record used to judge it. A cited page does not settle the assertion unless its content, entity, period and scope actually match. Record omissions and refusals separately: an answer that avoids a requested fact is not equivalent to one that states it incorrectly.
In a real study, have a research or analytics lead and an editor code a declared overlap of the same answers without seeing each other’s labels. Retain the initial disagreement counts, reasons and reconciled decisions. Automatic assistance can propose assertions or evidence links, but disputed judgements still need a recorded basis. Agreement among repeated AI responses is not a substitute for an authoritative record.
A fictional coding run exposes the denominator problem
A fixed-rule local coding demonstration on 17 October 2026 used six hand-authored answer fixtures, meaning deliberately constructed test inputs. Their ten assertions were manually separated and matched to the fictional records effective on 1 October. No AI system produced these answers, and no language-understanding or claim-extraction capability was tested. The following is the complete assertion-level result grouped by answer:
- Returns answer: ‘The return period is 14 days.’ Contradicted by the fictional 30-day standard online UK lamp record.
- Specification answer: ‘Aster is dimmable. It uses 9 watts.’ Dimmability is contradicted; power is supported by the fictional Aster records.
- Warranty answer: ‘The warranty lasts two years.’ Supported by the fictional UK Aster warranty record.
- Delivery answer: ‘It ships to the UK. It also ships to France.’ UK shipping is supported; France shipping is contradicted by the fictional UK-only destinations record.
- Company-information answer: ‘Support opens from 09:00 to 17:00. The retailer was founded in 2015.’ Hours receive source conflict; the founding year remains unverified.
- Summary answer: ‘Standard lamps have 30-day returns. Aster has a two-year warranty.’ Both assertions are supported within the declared UK product scope.
The result is five supported assertions, three contradicted assertions, one source conflict and one unverified assertion. Counting assertion occurrences, including the warranty repeated in a separate answer, gives ten assertions in total and eight adjudicable assertions. The contradicted share is therefore 3/8, or 37.5%, among adjudicable assertions. It is 3/10, or 30%, among all asserted facts. Evidence coverage is 8/10, or 80%. Three of the six answers contain at least one contradiction, giving an answer-level rate of 50%. Each percentage answers a different question; none is an observed AI error rate.
The expected labels and totals were declared before implementation. Two replays matched a separate evaluator’s literal expected results. Controls checked missing market information, the wrong market, future-dated evidence, reversed source order, a corrupted total and swapped assertion labels. These checks support the bounded coding and arithmetic demonstration only. The same author prepared the fixtures, coding logic and evaluator, so this is not independent human validation.
Turn error patterns into a justified next action
For a real dataset, group contradictions by fact category and affected decision before ordering corrections. A repeated wrong return period calls for examining the applicable returns records and the answer’s cited sources. Conflicting support hours call first for reconciling the retailer’s records. An unverified founding year calls for finding evidence or leaving the judgement unresolved. These actions differ because only the first case currently establishes a contradiction against a controlling record.
A factually incorrect answer does not, by itself, identify its cause. If a cited page contains the same mistake, that supports investigating the page but does not prove the system relied exclusively on it. After a correction, repeat the fixed sample under documented collection conditions, retain all attempts and show category-level changes. A before-and-after difference is an observation, not proof that the edit caused it. Source changes, system changes and differences between repeated answers remain alternative explanations.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter IV, ‘What Brands Build Now,’ of their jointly authored book The Discovery Reset: ‘A quarterly industry benchmark with primary data nobody else has.’ For this study, the useful consequence is a repeatable research record: dated answers, applicable source versions and disclosed denominators. The quotation supports the case for original research; it does not promise that publishing a benchmark will earn AI citations.
ScaleGrowth’s team brings years of experience in organic growth marketing, alongside pioneering AEO/GEO technology work. Its AI Native capability covers prompt discovery and measurement across multiple large language models, the systems that generate text responses. Those functions are relevant to constructing and observing a prompt sample. They do not replace authoritative fact records or establish that the coding workflow above is an existing integrated product feature. Its AI visibility service provides a relevant starting point for scoping the measurement work.
The decision to proceed with ScaleGrowth for this assignment should turn on a declared study that produces an original error dataset with denominators and source dates. An observable acceptance condition is that every scored assertion can be reconstructed from its retained answer, applicable record and coding decision; conflicts remain visible; and a second reviewer can reproduce the published totals. The fictional run shows how to test that requirement. A real AI-answer collection, its research and editorial review, and the resulting public benchmark remain work to be performed.
References
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
https://arxiv.org/abs/2305.14251
AI Visibility Services, ScaleGrowth Digital
https://scalegrowth.digital/services/ai-visibility/
How do I plan AEO/GEO Ready content from real user prompts?
https://scalegrowth.digital/how-do-i-plan-aeo-geo-ready-content-from-real-user-prompts/