A Keyword Research Framework That Goes Beyond Volume And Difficulty

Search volume and keyword difficulty are the two metrics that built the SEO industry and the two metrics most likely to mislead a 2026 keyword research programme. Volume reports an outdated demand surface that no longer reflects how AI Overview, ChatGPT, and Perplexity intercept the click. Difficulty estimates competitive density on the ten-blue-link SERP, which has stopped being the only ranking surface that matters. The keyword research that works now scores each candidate query across six dimensions: search intent class, AI answer panel presence, ranking surface count, commercial value per click, internal capability to win, and entity coverage gap. This piece sets out the framework and shows how we applied it across a 71,000-keyword wealth-platform RFP and a 4,962-keyword gold-loan classification project.

What Volume And Difficulty Stopped Measuring

Search volume, as reported by every major tool, is a historical metric derived from logged queries inside Google Ads. The number does not adjust for the proportion of those queries that now resolve inside the AI Overview answer panel without a click, the proportion that resolve in Bing Copilot or Perplexity outside Google’s measurement entirely, or the proportion that have shifted to LLM chat surfaces that no major tool measures. A keyword reporting 12,000 monthly searches in 2024 may produce 4,000 actionable clicks in 2026 because the rest are absorbed by zero-click answer surfaces. The same keyword may simultaneously produce 2,500 citations across ChatGPT, Claude, and Perplexity that the tool does not see.

Keyword difficulty is worse. The metric estimates the average backlink strength of pages ranking on the first page of Google for the query. It says nothing about how an AI engine selects which of those pages to cite, which featured snippet Google extracts, or which subset of pages AI Overview surfaces alongside its generated answer. A keyword can show low difficulty by the classical measure and high competitive density at the AI answer surface, or the reverse. The teams who continue to prioritise on volume and difficulty alone are scoring against a SERP that no longer exists.

The Six-Dimension Framework We Use

Across the audits and content-recommendation programmes we run, every keyword is scored on six dimensions before it earns a slot in the production queue. The framework is not theoretical; it is the operational filter we use to compress a 50,000-keyword universe to a 200-page content programme without dropping commercial intent on the floor.

Dimension one: intent class. The query is classified into one of four intent buckets: informational, navigational, commercial-investigation, or transactional. The classification is automated by an LLM pass with a deterministic rule fallback for ambiguous cases. Intent class dictates the page type the keyword maps to and the conversion behaviour the page can produce. A 5,000-volume informational keyword feeding a sales page wastes the opportunity; a 500-volume commercial-investigation keyword feeding a comparison page may convert 20 times harder.

Dimension two: AI answer panel presence. Does the query produce an AI Overview block, a Bing Copilot panel, a Perplexity answer, or a ChatGPT Search citation? A keyword that produces no AI answer surface today still rewards classical SEO behaviour. A keyword that produces a full AI Overview answer panel requires the page to be cited inside that panel to capture meaningful traffic, which is a different optimisation problem.

Dimension three: ranking surface count. How many surfaces (organic SERP, local pack, AI Overview, AI Mode, Featured Snippet, People Also Ask, image pack, video pack) does the query produce? More surfaces means more opportunities to capture the click, but it also means more competition for attention and a more complex production target.

Dimension four: commercial value per click. The classical CPC metric, adjusted for the brand’s own conversion rate on the closest analogous page, and weighted against the lifetime value the buyer represents. This dimension prevents the team from chasing high-volume queries with low actual revenue contribution.

Dimension five: internal capability to win. Honest self-assessment. Does the brand have the entity authority, the content team capacity, the technical infrastructure, and the link profile to win this query inside the budget window? A high-value query the brand cannot win is not a candidate, it is a distraction.

Dimension six: entity coverage gap. The query is mapped against the brand’s existing entity coverage in the model’s knowledge graph. A query in a strong entity area (where the brand already has citation and recall) compounds existing equity. A query in a weak entity area requires entity-building work first, before page-level work makes sense.

How The Six Dimensions Combine

How It Applied On Two Real Programmes

On a wealth-management platform RFP we ran across 71,000 organic keywords, the classical volume-and-difficulty filter would have produced a top-200 keyword list dominated by head terms in home loans, life insurance, and stocks. The framework filter, applied across 11,920 high-volume keywords classified in 25 AI-driven batches, produced a different shortlist. Home loan dropped out (out-of-scope LOB subdomain). Business loan rose to the top (49,500 search volume head term, position 26, 180,000-keyword gap to the leader). The scope-alignment sweep excluded approximately 38 percent of the original information architecture blueprint because dimensions five and six (internal capability and entity gap) caught what dimensions one and four had missed.

On a gold-loan keyword classification programme spanning 4,962 keywords for a major BFSI lender, the framework was applied to bucket every query into one of 12 deterministic categories. Zero rows ended up requiring manual review because the rules covered the edge cases through a tiebreaker table. The classification fed directly into a content production engine that shipped 794 schema-validated briefs across four batches in five weeks, every brief tagged with its scored dimensions so the writer knew which page type, which surface, and which entity area the brief was targeting. The full content-engine architecture sits inside the content engineering brief, and the BFSI-specific patterns inside the BFSI growth engineering coverage. The AI-visibility layer that informs dimensions two and six is documented inside the AI visibility audit.

A separate observation from the same BFSI portfolio: a 25,000-page NBFC technical audit found that the brand was already ranking on 94,100 keywords but mention rates across ChatGPT, AI Overview, and AI Mode sat at 8 percent, 15.6 percent, and 19 percent. The volume universe was solid. The entity coverage was incomplete. Dimension six (entity coverage gap) is the dimension that surfaced this gap. Volume and difficulty alone would have continued recommending new ranking keywords and missed the citation problem entirely.

Practitioner Takeaway

1. Stop ranking your keyword universe on a single metric. Build a six-dimension score card and re-rank. The top 200 keywords on the new score card will not be the top 200 on volume and difficulty alone.
2. Run an AI answer panel pass on every shortlisted query. Open the query in incognito Google, look for the AI Overview block, run it through ChatGPT, Claude, and Perplexity, record which engines produce a full answer panel. This data is dimension two.
3. Score commercial value with your own conversion rates, not generic CPC. A query worth twenty cents in CPC may be worth thirty rupees per acquisition on your funnel, or the reverse.
4. Be honest about internal capability. A keyword you cannot win inside the budget window is a distraction. Move it to a future-quarter parking lot, do not abandon the framework by chasing it now.
5. Map entity coverage before adding new content. If the brand is weak on an entity area, page-level content is premature. Run entity-building work first, then return to the keyword shortlist.

Frequently Asked Questions

Is search volume still useful for keyword research?

Yes, as one of six inputs. Volume is a useful demand signal when adjusted for AI Overview interception and zero-click answer absorption. Used alone, volume produces shortlists that ignore commercial value, entity gaps, and the changed retrieval landscape. Used as one dimension among six, it provides scale context.

How do we score AI answer panel presence at scale?

For a few hundred keywords, manual SERP inspection is workable. For larger universes, the DataForSEO SERP API exposes AI Overview detection for Google, and a custom panel of queries can be run through ChatGPT, Claude, and Perplexity in parallel to record citation behaviour. We typically run 300 to 500 keyword AI visibility panels per audit and extrapolate.

What if our brand has no entity coverage in a target area?

Build coverage before pages. Publish primary research that the knowledge graph picks up, get cited in neutral publications, file structured data that disambiguates the entity, secure analyst or community mentions. Entity-building is slower than content publishing and runs on a separate timeline.

Should keyword difficulty be ignored entirely?

No, but it should not be a top-two factor. Difficulty maps to one part of dimension five (internal capability to win). It is useful as a signal of competitive intensity on the classical SERP, and useless as a signal of citation difficulty on AI surfaces.

How long does the six-dimension framework take to run on a large keyword universe?

For a 5,000-keyword universe, one to two weeks with an automated classification pipeline. For a 50,000-keyword universe, three to five weeks with parallel agent classification. The bottleneck is dimension two (AI answer panel presence) because it requires live SERP and live LLM queries that cannot be batched indefinitely.

If you want a keyword research programme that scores every candidate on intent, AI panel presence, surface count, commercial value, internal capability, and entity coverage, request the audit that delivers a scored shortlist mapped to a 12-week content production plan.

Request a six-dimension keyword research audit

By ScaleGrowth Digital Editorial

{
“@context”: “https://schema.org”,
“@graph”: [
{
“@type”: “Article”,
“headline”: “A Keyword Research Framework That Goes Beyond Volume And Difficulty”,
“description”: “Volume and difficulty mislead in 2026. The six-dimension framework we use to compress 50,000-keyword universes into actionable shortlists, applied across a 71K wealth-platform RFP and a 4,962-keyword BFSI gold-loan programme.”,
“author”: {
“@type”: “Organization”,
“name”: “ScaleGrowth Digital Editorial”,
“url”: “https://scalegrowth.digital/about/”
},
“publisher”: {
“@type”: “Organization”,
“name”: “ScaleGrowth Digital”,
“logo”: {
“@type”: “ImageObject”,
“url”: “https://scalegrowth.digital/logo.png”
}
},
“mainEntityOfPage”: “https://scalegrowth.digital/keyword-research-framework-beyond-volume-and-difficulty/”,
“datePublished”: “2026-09-17”,
“dateModified”: “2026-09-17”
},
{
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “Is search volume still useful for keyword research?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Yes, as one of six inputs. Volume is a useful demand signal when adjusted for AI Overview interception and zero-click answer absorption. Used alone, volume produces shortlists that ignore commercial value, entity gaps, and the changed retrieval landscape.”
}
},
{
“@type”: “Question”,
“name”: “How do we score AI answer panel presence at scale?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “For a few hundred keywords, manual SERP inspection is workable. For larger universes, the DataForSEO SERP API exposes AI Overview detection for Google, and a custom panel of queries can be run through ChatGPT, Claude, and Perplexity in parallel.”
}
},
{
“@type”: “Question”,
“name”: “What if our brand has no entity coverage in a target area?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Build coverage before pages. Publish primary research that the knowledge graph picks up, get cited in neutral publications, file structured data that disambiguates the entity, secure analyst or community mentions.”
}
},
{
“@type”: “Question”,
“name”: “Should keyword difficulty be ignored entirely?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “No, but it should not be a top-two factor. Difficulty maps to one part of internal capability. It is useful as a signal of competitive intensity on the classical SERP, and useless as a signal of citation difficulty on AI surfaces.”
}
},
{
“@type”: “Question”,
“name”: “How long does the six-dimension framework take to run on a large keyword universe?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “For a 5,000-keyword universe, one to two weeks with an automated classification pipeline. For a 50,000-keyword universe, three to five weeks with parallel agent classification.”
}
}
]
}
]
}