Discovery is no longer confined to a page of blue links. People now ask ChatGPT, Perplexity, Google AI Overviews and other answer engines to research products, compare suppliers and explain unfamiliar topics. These systems assemble answers from multiple sources and may cite only a small part of any individual page.
That changes the shape of the work, but it does not make traditional search fundamentals obsolete. A website still has to be crawlable, indexable, useful and trustworthy before an AI system can reliably retrieve and cite it.
The practical goal is not to “rank inside an LLM”. It is to make your information easy to discover, easy to extract, easy to verify and easy to attribute to the correct entity.
What AI-search visibility actually means
AI-search visibility is the likelihood that a brand, page or passage is retrieved and used when an answer engine responds to a relevant question.
You may see this described as Generative Engine Optimization (GEO), Answer Engine Optimization (AEO) or AI-search optimization. The labels overlap. The durable work beneath them is straightforward:
- Search and answer engines can access the page
- The page answers a specific question clearly
- Important facts are available in readable HTML
- Claims are supported by evidence
- The brand and its relationships are unambiguous
- Other relevant pages and sources reinforce the same entity
The original academic work on Generative Engine Optimization found that methods such as adding citations, quotations and statistics could improve source visibility in experimental generative-engine settings, in some cases by up to 40%. That is useful evidence, but it is not a universal ranking formula. Real products use different indexes, models and retrieval systems that continue to change.
How an answer engine gets from a question to a citation
The implementation differs by platform, but most web-grounded answer systems have two broad gates.
Gate 1: discovery and retrieval
The system first needs candidate documents. Those may come from a traditional search index, a platform-specific crawler, live retrieval or a combination of sources.
If the page is blocked, returns an error, requires a login or hides its useful content behind client-side interactions, it may never enter the candidate set.
Google states that pages eligible for AI Overviews and AI Mode must already be indexed and eligible for a normal search snippet. Its official AI-features guidance says there are no additional technical requirements beyond established Search fundamentals.
Gate 2: passage selection and synthesis
Once a page has been retrieved, the system needs to identify a passage that answers the question. Clear definitions, focused sections, descriptive headings and evidence make that extraction easier.
The page is not evaluated only as one document. Individual passages may compete with passages from many other sources. A strong section should therefore make sense when read on its own.
1. Make crawler access an explicit decision
Do not treat every AI-related user agent as the same thing. Search retrieval, user-triggered browsing and model training are separate purposes.
For ChatGPT search, OpenAI recommends allowing OAI-SearchBot. Its publisher guidance distinguishes that crawler from GPTBot, which publishers can block when they want to exclude content from potential model training.
Perplexity similarly documents PerplexityBot as its search-results crawler and separates it from user-triggered fetching. See the official Perplexity crawler documentation.
A simple permissive example looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xmlThat file is only one layer. A CDN, web application firewall or bot-protection service can still return a 403 even when robots.txt allows access. Test the complete request path.
For Google AI features, normal Googlebot controls apply. Google explicitly says no separate AI crawler permission is required for inclusion in AI Overviews or AI Mode.
2. Put important information in the initial HTML
A page can be visually complete for a human and structurally empty to a simple crawler.
If the core answer appears only after a client-side API request, inside an interaction that must be clicked or in a canvas-based interface, retrieval becomes less reliable. Server-side rendering or static generation makes the main content available in the first response.
This does not mean every website must use one particular framework. It means the page source should contain:
- The primary heading and explanatory copy
- Product or service details
- Prices, specifications and policies when relevant
- Internal links to related pages
- Canonical and metadata signals
- Structured data that reflects visible content
Use JavaScript to enhance the experience, not to conceal the meaning of the page.
3. Structure sections so they survive extraction
Answer engines frequently retrieve a passage rather than an entire article. Write sections that remain useful outside their original layout.
Lead with the answer
When a heading asks “What is GEO?”, define it immediately. Do not spend three paragraphs creating suspense before stating the answer.
Keep one primary idea per section
A section about crawler access should explain crawler access. A separate section can cover structured data. Clear boundaries make passages easier to select and reduce the risk that important context is split across unrelated chunks.
Use descriptive headings
“Technical requirements for AI-search visibility” is more useful than “The foundations”. Specific headings help people scan and provide retrieval systems with better context.
Make references explicit
Avoid chains of pronouns where “it”, “this” and “they” could refer to several entities. Name the product, organisation or concept when ambiguity would weaken an extracted passage.
Good content architecture is not robotic writing. It is clear writing with strong information boundaries.
4. Publish evidence, not unsupported confidence
Generative answers need claims they can corroborate. A page becomes more useful when it distinguishes opinion, experience and externally verifiable fact.
Practical evidence includes:
- Original research or anonymised product data
- Clear methodology and sample size
- Links to primary documentation
- Named authors with relevant experience
- Dates for information that changes over time
- Concrete examples, screenshots and implementation details
Do not add statistics merely because numbers look authoritative. A weak number copied through ten derivative articles remains a weak source. Link to the original research or remove the claim.
This is also where genuinely original material wins. A page that repeats the same generic summary as every competitor gives an answer engine little reason to cite it.
5. Use structured data for entity clarity
Structured data helps machines interpret relationships already visible on the page. It can clarify that an article has an author, a service belongs to an organisation or a product has a particular offer.
Useful practices include:
- Giving the organisation a stable
@id - Reusing that identifier across page-level JSON-LD graphs
- Linking to genuine official profiles with
sameAs - Keeping names, addresses and descriptions consistent
- Matching every structured claim to visible page content
Do not invent external entity identifiers or stuff knowsAbout with every keyword you want to rank for. Structured data should reduce ambiguity, not manufacture authority.
Google’s guidance is especially important here: structured data remains useful, but there is no special AI schema required for its AI search features. Existing markup must follow normal policies and match what users can see.
6. Build a connected information architecture
One excellent page is easier to trust when the surrounding site supports it.
Connect related concepts deliberately:
- A service page should link to relevant case studies and technical guides
- An article should link to the product or capability it explains
- Author and organisation pages should establish ownership
- Navigation and sitemaps should make important pages discoverable
- Duplicate or contradictory pages should be consolidated
Internal links help crawlers discover content, but they also create context. They show how the organisation relates to a topic and which page is the canonical explanation.
Think of the website as a small knowledge system rather than a collection of landing pages.
7. Treat llms.txt as an experiment, not a ranking requirement
The llms.txt specification describes itself as a proposal for giving agents a concise Markdown index of a website. It can be useful for documentation-heavy sites and tools that explicitly consume it.
It is not a substitute for crawlable HTML, a sitemap, internal linking or robots.txt. It is also not a requirement for Google AI Overviews or AI Mode. Google’s documentation explicitly says you do not need to create new AI text files or special markup to appear in those features.
If the maintenance cost is low, an llms.txt file can still be a sensible experiment:
- Include only canonical, high-value pages
- Describe each link honestly and concisely
- Keep the file aligned with the live website
- Measure whether relevant agents request it
- Do not claim adoption that the consuming platforms have not documented
We built a free llms.txt generator for teams that want to test the format without turning it into a large implementation project.
8. Measure what the systems actually send
AI-search optimization becomes guesswork when measurement stops at rankings.
Track:
- Referral traffic from answer engines
- Landing pages receiving that traffic
- Queries and topics associated with cited pages
- Branded mentions and citation accuracy
- Conversions or assisted conversions from those visits
- Crawler access and server errors in logs
OpenAI adds utm_source=chatgpt.com to ChatGPT search referral URLs, according to its publisher documentation. Google reports AI-feature traffic within Search Console’s overall web-search reporting. Other platforms expose different levels of detail.
The aim is not to create one universal “AI visibility score”. Measure whether the right audiences discover accurate information and take useful next steps.
A practical AI-search readiness checklist
Before investing in novel optimization tactics, confirm the basics:
- Important pages return a successful response and are indexable.
- Search crawlers are not accidentally blocked by
robots.txt, the CDN or the firewall. - The main content exists in the initial HTML response.
- Every important page answers a clear set of user questions.
- Headings and sections make sense when extracted independently.
- Claims link to primary evidence where appropriate.
- Organisation, author, product and service entities are consistent.
- Structured data matches visible content and validates correctly.
- Internal links connect supporting pages into a coherent topic structure.
- Referral traffic, crawler requests and conversions are being measured.
Only after these are in place should optional experiments such as llms.txt, alternate Markdown views or platform-specific integrations become priorities.
Build for people and machines at the same time
AI-search visibility is not a separate website layer that can be installed with a plugin. It is the outcome of clear content, accessible architecture, consistent entities, strong evidence and reliable delivery.
The good news is that the same work improves the website for people. Fast pages, understandable navigation, direct answers and trustworthy claims create a better buying experience whether the visitor arrives from Google, ChatGPT, Perplexity or a colleague’s link.
COMMS Digital designs and builds websites and digital products with these foundations in the architecture from the start: crawlable content, clean information models, structured data and performance that supports discovery rather than fighting it.