Skip to main content
← Back to Blog

Claude Search

Claude SEO: How to Get Cited in Claude Search

A current guide to Anthropic's three crawler roles, source-ready page design, and prompt-level measurement for Claude visibility.

Why Claude Visibility Needs Its Own Measurement

Claude can answer from model knowledge, use web search, or retrieve a page at a user's direction. Those paths create different visibility problems. A page can be available to search while the brand remains absent from a category prompt. A brand can also appear in an answer while its own site receives no citation.

Anthropic does not publish a brand-ranking formula. The practical job is to make accurate pages available to the right retrieval paths, then measure whether Claude mentions and cites them for a stable set of buyer questions.

Foglift signal, July 14, 2026

Across 1,000 recent monitored responses, Claude mentioned foglift.io in 25% of checks. Claude cited otterly.ai 68 times, while the other four tracked engines cited that domain 260 times. The counts describe the source layer. They do not rank products. This is why brand mentions, cited domains, and answer position need separate fields.

The Three Anthropic Crawler Roles

Anthropic's current crawler documentation separates model development, search indexing, and user-directed retrieval. Treating every bot as “ClaudeBot” produces the wrong robots.txt advice.

User agentDocumented roleVisibility implication
Claude-SearchBotIndexes and analyzes online content to improve Claude search results.Blocking it can reduce a site's visibility and accuracy in Claude search results.
Claude-UserRetrieves web content when a Claude user initiates a request.Blocking it prevents user-directed retrieval from your site.
ClaudeBotCollects public web content that could contribute to potential model training.This control concerns future training data collection rather than current search indexing.

Choose access by role and policy. A publisher can allow Claude-SearchBot and Claude-User for retrieval while making a separate decision about ClaudeBot. Anthropic says all three bots honor robots.txt directives and supports the non-standard Crawl-delay extension.

8 Steps to Improve Claude Search Visibility

1. Configure the crawler roles you intend to support

Review robots.txt for all three Anthropic user agents. Then check CDN, firewall, and bot-management rules for 403 responses. Anthropic does not publish fixed IP ranges, so user-agent and access-log evidence are more useful than an IP allowlist.

2. Put essential product facts in crawlable HTML

State what the product does, who it serves, current pricing, important limits, and the date of the information. Keep these facts in server-rendered HTML. A vague homepage gives retrieval systems little evidence to use in a comparison answer.

3. Keep entity statements consistent

Use the same brand name, product category, company description, and canonical URL across the homepage, About page, documentation, profiles, and review listings. Add Organization or SoftwareApplication JSON-LD where it accurately represents visible content.

Structured data does not guarantee a Claude citation. It can clarify page entities and relationships, but Anthropic has not published schema markup as a direct ranking factor.

4. Answer the exact questions buyers ask

Build one page around a coherent buyer job. Use descriptive headings, concise answers, current examples, and tables where the comparison fields repeat. Cite primary sources for competitor pricing, specifications, and research claims.

5. Make claims easy to verify

Attach a source and date to factual claims. Explain the sample behind original data. Keep product documentation aligned with the live interface and plan limits. Retrieval helps only when the page gives Claude reliable evidence to summarize.

6. Publish honest comparison context

Comparison prompts need facts such as price, engine coverage, developer access, and best-fit use case. Use the same fields for every vendor and disclose your publisher relationship. Foglift's Otterly alternatives guide shows the source and evaluation boundaries directly.

7. Earn independent corroboration

A brand's own page establishes its claim. Independent industry coverage, review listings, documentation references, and community discussions provide corroboration. SE Ranking's 2025 study of 129,000 domains found referring domains were the strongest measured predictor of AI citations in its dataset.

8. Measure prompt-level outcomes

Track category, comparison, problem, and branded prompts separately. For each response, store whether the brand appeared, its answer position, sentiment, cited URLs, and competing domains. Repeat the same prompt set because a single answer is too variable to establish a trend.

Claude and ChatGPT Retrieval Controls

RoleClaudeChatGPT
Search indexingClaude-SearchBotOAI-SearchBot
User-requested fetchClaude-UserChatGPT-User
Potential trainingClaudeBotGPTBot

The same operating model works across both providers: configure access by crawler role, publish verifiable answers, and measure actual prompt outcomes. Read the complete AI crawler guide for current controls.

How to Monitor Claude Brand Mentions

Manual spot checks are useful for inspecting an answer. Recurring measurement needs a fixed prompt set, timestamps, response evidence, and competitor fields. Foglift's paid plans monitor Claude with ChatGPT, Perplexity, Gemini, and Google AI Overview. Active Free workspaces monitor Google AI Overview weekly and support on-demand checks for that engine while the workspace remains active.

  • Separate category, comparison, problem, and branded prompts
  • Record mention status, answer position, sentiment, and cited URLs
  • Track competitor domains by engine and prompt
  • Compare rolling windows rather than isolated responses
  • Annotate page changes so movement can be investigated

Track Anthropic Bot Visits Separately

Server logs or the Foglift Tracker can show whether Anthropic's bots reached a page. That evidence confirms access. It does not prove indexing, retrieval, a citation, or a brand mention.

// Next.js middleware
import { trackAICrawlers } from '@foglift/tracker/nextjs';

export const middleware = trackAICrawlers({
  apiKey: 'your-api-key'
});

// Inspect Claude-SearchBot, Claude-User, and ClaudeBot
// as separate roles in crawler reporting.

Common Claude Visibility Mistakes

  1. Using one ClaudeBot rule for every goal. Search, user retrieval, and potential training have separate user agents.
  2. Treating crawler visits as citations. Access is one step in the measurement chain.
  3. Publishing vague product claims. Claude needs concrete facts and sources to construct a useful comparison.
  4. Assuming schema creates rankings. Structured data clarifies facts but does not guarantee selection.
  5. Reading one prompt as a trend. Prompt wording and response variance require repeated measurement.

Claude Search Action Plan

1Choose which Anthropic bot roles your site intends to support
2Verify robots.txt, CDN, and firewall access for those bots
3State your product, audience, pricing, and limits in crawlable HTML
4Add valid structured data that matches the visible page
5Publish source-backed answers for the prompts buyers actually ask
6Create factual comparison pages for your closest alternatives
7Track Anthropic bot visits separately from Claude answer mentions
8Measure a stable Claude prompt set on a recurring cadence

Frequently Asked Questions

Which Anthropic crawler should I allow for Claude Search?

Allow Claude-SearchBot when you want Anthropic to index your pages for search quality. Claude-User handles retrieval initiated by a person using Claude. ClaudeBot collects public web content that could contribute to future model training. Anthropic documents separate robots.txt controls for all three bots.

How does Claude decide which brands to mention?

Anthropic does not publish a brand-ranking formula. A web-grounded Claude answer can use search results and cite sources, while an answer without web retrieval can draw on the model's existing knowledge and the conversation context. Measure a stable prompt set over time instead of treating a single response as a ranking.

Does structured data guarantee a Claude citation?

Structured data does not guarantee a Claude citation. Valid Organization, Product, Article, or FAQPage markup can make page facts easier for machines to identify, but Anthropic has not published evidence that schema markup is a direct Claude ranking factor. Keep markup consistent with visible page content.

Can I track whether Claude mentions my brand?

Yes. Track a defined set of category, comparison, and brand prompts on a fixed cadence. Record mention status, position, sentiment, cited URLs, and competing domains for each response. Foglift paid plans include Claude alongside ChatGPT, Perplexity, Gemini, and Google AI Overview; Free monitors Google AI Overview weekly while the workspace is active.

Sources and Further Reading

  • Anthropic Help Center, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” Published April 7, 2026. Defines ClaudeBot, Claude-User, and Claude-SearchBot roles. support.claude.com
  • Anthropic Transparency Hub, 2026. Documents the public-web and other data mix used for current Claude model families. anthropic.com
  • Aggarwal et al., “GEO: Generative Engine Optimization,” KDD 2024. Evaluates source presentation tactics across generative engines; it does not establish a Claude-specific ranking formula. arxiv.org
  • SE Ranking, “Do LLMs Really Cite Sources? Analysis of 129,000 Domains,” 2025. Reports referring domains as the strongest citation predictor in its sample. seranking.com
  • SparkToro, “AIs Are Highly Inconsistent When Recommending Brands,” 2025. Analyzes 2,961 recommendation queries and motivates repeated prompt measurement. sparktoro.com

Start with a measurable baseline.

Check how AI engines describe your brand, then monitor Claude on a recurring paid plan.

Fundamentals: Learn about GEO (Generative Engine Optimization) and AEO (Answer Engine Optimization) (the two frameworks for optimizing your content for AI search engines).

Related reading

Free tool

Run a free Technical Audit for your AI Readiness Score

Audit any URL in 30 seconds. See scores for SEO, AI Readiness, performance, security, and accessibility.

Free Technical Audit

No signup required. Results in 30 seconds.