llms.txt and AI Crawlers for SaaS Sites

Technical AEO starts with a simple question: can answer engines reach and understand the pages that explain your product?

For SaaS sites, the answer depends on four files and signals:

• robots.txt
• sitemap.xml
• canonical tags
• /llms.txt

Each one has a different job. Mixing them up leads to bad SEO decisions.

What llms.txt Is For

llms.txt is a proposed Markdown file placed at /llms.txt. The original llms.txt proposal describes it as a way to provide information that helps LLMs use a website at inference time.

In practical terms, it is a curated map. It tells AI systems which pages best explain the company, product, use cases, documentation, pricing, and important guides.

It should not contain secrets. It should not include private app routes. It should not replace your sitemap.

There is also an important Google-specific caveat. Google's generative AI optimization guide says you do not need llms.txt or other special AI text files to appear in Google Search generative AI features, and that Google Search ignores llms.txt as a special signal. That does not make the file useless. It means teams should treat it as a helpful convention for AI systems and site documentation, not as a Google ranking lever.

A SaaS /llms.txt file can include:

• Company and product summary.
• Homepage.
• Use-case pages.
• Pricing page.
• Product documentation or help center.
• Comparison pages.
• Core educational guides.
• Sitemap link.
• Support or sales contact page.

For AEO Table, that means pages such as AI search monitoring, Google AI Overview monitoring, AI citation tracking, Search Console AI Reports vs AEO Table, and the AI search visibility baseline.

What robots.txt Is For

robots.txt is an access policy for crawlers that choose to follow it.

Use it to say which paths crawlers may or may not fetch. For SaaS sites, the common pattern is:

• Allow public marketing pages.
• Allow public educational content.
• Allow public documentation if it supports discoverability.
• Disallow app routes, account pages, onboarding flows, private reports, API routes, and auth pages.

OpenAI documents crawler user agents such as OAI-SearchBot, GPTBot, and ChatGPT-User in its crawler overview. Perplexity documents PerplexityBot and Perplexity-User in its crawler docs. Anthropic documents ClaudeBot, Claude-User, and Claude-SearchBot in its crawler help article.

The policy choice is yours. The important part is to make it intentional.

Separate Search, Training, And User Fetching

Do not treat every AI bot as one thing.

This split matters because the business decision is different for each role.

• Search and retrieval bots affect whether your public pages can appear as sources.
• Training bots affect whether future model training may use public content.
• User fetchers support user-directed browsing or answer generation.

A common SaaS policy is to allow search and retrieval access for public pages, keep private app routes blocked, and make a separate legal decision about training crawlers.

Google Search now has another control surface to keep separate. Google's site-owner update describes a Search Console toggle for whether a site can appear in and help ground responses in AI Overviews, AI Mode, and AI Overviews in Discover. Google says opted-out sites will not receive traffic or impressions from those generative AI features, and that the setting is not used as a ranking signal outside those features.

That is an appearance policy inside Google Search. It is not a crawl policy, not an llms.txt convention, and not the same as Google-Extended.

What A Good SaaS Policy Looks Like

For most B2B SaaS sites that want AI search visibility, a reasonable policy is:

• Public marketing pages: allowed.
• Public blog and guides: allowed.
• Public use cases and comparison pages: allowed.
• Pricing, terms, privacy, support: allowed.
• Search-oriented AI crawlers for public resources: allowed when visibility is a goal.
• Dashboard, auth, onboarding, API, private reports: disallowed.
• Training-oriented crawlers: decided separately based on legal, licensing, and data policy.

This gives crawlers access to source pages without exposing app surfaces.

Crawler access is only one part of the implementation. Use the agent-friendly website checklist to review whether the public pages themselves are structured, attributable, and easy for answer systems to retrieve.

Where Canonicals Fit

Canonical tags tell search systems which URL should represent a page.

If /llms.txt, sitemap, internal links, and canonical tags point to different versions of a URL, you create unnecessary ambiguity. Use one canonical URL pattern.

For example:

• Sitemap: https://www.aeotable.com/en/blog/ai-search-monitoring
• Canonical: https://www.aeotable.com/en/blog/ai-search-monitoring
• Internal links: /en/blog/ai-search-monitoring
• llms.txt: https://www.aeotable.com/en/blog/ai-search-monitoring

That consistency matters more than adding many crawler-specific tricks.

Where Structured Data Fits

Structured data helps search engines understand page content. Google's structured data introduction recommends using supported formats such as JSON-LD where appropriate.

For SaaS public pages, common types include:

• SoftwareApplication for the homepage or product page.
• Article for blog posts.
• FAQPage where visible FAQ content exists and the page meets Google's FAQ structured data guidance.
• BreadcrumbList for article hierarchy.

Schema is not a substitute for useful content. It is a clarity layer.

Pages To Include In llms.txt

Start small. Do not list every URL.

Recommended sections:

Product

• Homepage.
• Pricing.
• Core use cases.
• Security or trust page if available.

Use Cases

• AI search monitoring.
• ChatGPT brand monitoring.
• Google AI Overview monitoring.
• Competitor AI visibility.
• AI citation tracking.

Guides

• Answer Engine Optimization guide.
• Google Search Console generative AI reports.
• AI search visibility baseline.
• AI search query set.
• AI visibility score.
• AI search visibility audit checklist.

Comparison

• Manual AI visibility tracking comparison.
• Search Console AI Reports vs AEO Table.
• Competitor or alternative pages where they exist.

Technical Context

• Sitemap.
• Robots policy.
• Support contact.

Technical Checklist

Use this before publishing /llms.txt:

• The file is available at /llms.txt.
• It is plain text or Markdown.
• Links use canonical absolute URLs.
• Listed pages return 200.
• Listed pages are not blocked by robots.
• Listed pages appear in sitemap when they are indexable.
• Private app routes are not listed.
• The file is updated when major public content changes.

Common Mistakes

Do not treat /llms.txt as an ad page. Keep it factual.

Do not list pages that are blocked in robots.txt.

Do not include every blog post. List the pages that explain the product, category, and most important use cases.

Do not promise crawler behavior you cannot control. llms.txt is a proposal and a helpful convention, not a guaranteed inclusion mechanism.

Do not claim llms.txt is required for Google AI Overviews or AI Mode. Google's current guidance says it is not.

Do not use robots.txt, /llms.txt, Google-Extended, and Google's AI features opt-out as interchangeable controls. They govern different layers: fetching, curated discovery, model-use policy, and Google Search AI appearance.

A Simple llms.txt Draft

Here is a practical starting structure for a SaaS site:

```markdown # AEO Table

AEO Table helps teams monitor how AI answers mention, cite, and compare their brand across ChatGPT, Google AI, Perplexity, and other answer engines.

Product

• Homepage: AI search visibility monitoring for B2B teams.
• Pricing: Plans and launch credits.
• Support: Contact and product questions.

Use Cases

• AI search monitoring: Repeatable monitoring across answer engines.
• Google AI Overview monitoring: Monitor Google AI Overviews with answer-level evidence.
• Competitor AI visibility: Compare brand and competitor mentions.
• AI citation tracking: Review cited domains and source quality.

Comparison

• Search Console AI Reports vs AEO Table: Compare Google page visibility reports with cross-channel answer monitoring.
• AEO Table vs manual AI visibility tracking: Compare spreadsheets with repeatable Tasks, Runs, and Reports.

Guides

• Answer Engine Optimization
• Google Search Console generative AI reports
• AI search visibility baseline
• AI search visibility audit checklist

Technical

- Sitemap - Robots policy ```

Keep this file shorter than the site itself. It is a hand-curated index, not a second sitemap.

The Bottom Line

For SaaS teams, technical AEO is mostly clarity and access control.

Use robots.txt to manage crawler access. Use sitemap and canonical tags to keep indexable URLs consistent. Use /llms.txt to point AI systems to the public resources that best explain your product and category.

Then measure whether those pages actually show up in answers with an AI search visibility audit and a repeatable AI search monitoring workflow.