AI Crawler Role Taxonomy & Visibility Evidence Standards

Open CC BY 4.0 dataset of 14 major AI crawler roles (training, search index, live user retrieval), 4 attribution evidence layers, and public static GEO readiness weights.

CitationGraph classifies AI bot traffic into three operational roles—model training crawlers, AI search indexers, and user-triggered live retrieval agents—across four distinct evidence layers, while weighting public static GEO previews equally across Infrastructure (25%), Metadata (25%), Structured Data (25%), and AI Guidance (25%).

1. AI Crawler & Retrieval Agent Role Classification (14 Reference Bots)

Separating background corpus training from search indexing and real-time user prompt fetches prevents robots.txt policies from accidentally blocking AI citations.

Bot TokenOperatorOperational Rolerobots.txt TokenSignal LayerVerification
GPTBotOpenAImodel_trainingGPTBotserver_crawler_discoveryPublished IP CIDR ranges + User-Agent
OAI-SearchBotOpenAIsearch_indexOAI-SearchBotserver_crawler_discoveryPublished IP CIDR ranges + User-Agent
ChatGPT-UserOpenAIlive_user_retrievalChatGPT-Userlive_retrieval_fetchPublished IP CIDR ranges + User-Agent
ClaudeBotAnthropicmodel_trainingClaudeBotserver_crawler_discoveryUser-Agent + ASN attribution
Claude-SearchBotAnthropicsearch_indexClaude-SearchBotserver_crawler_discoveryUser-Agent + ASN attribution
Claude-UserAnthropiclive_user_retrievalClaude-Userlive_retrieval_fetchUser-Agent + request header pattern
PerplexityBotPerplexity AIsearch_indexPerplexityBotserver_crawler_discoveryPublished IP JSON list + User-Agent
Perplexity-UserPerplexity AIlive_user_retrievalPerplexity-Userlive_retrieval_fetchPublished IP JSON list + User-Agent
Google-ExtendedGooglemodel_trainingGoogle-Extendedserver_crawler_discoveryReverse/Forward DNS googlebot.com + policy token
GoogleOtherGooglesearch_indexGoogleOtherserver_crawler_discoveryReverse/Forward DNS googlebot.com
Applebot-ExtendedApplemodel_trainingApplebot-Extendedserver_crawler_discoveryReverse/Forward DNS applebot.apple.com + policy token
AmazonbotAmazonsearch_indexAmazonbotserver_crawler_discoveryReverse/Forward DNS crawl.amazonbot.amazon
Meta-ExternalAgentMetamodel_trainingMeta-ExternalAgentserver_crawler_discoveryUser-Agent + Meta ASN verification
CCBotCommon Crawlmodel_trainingCCBotserver_crawler_discoveryUser-Agent + Common Crawl AWS egress metadata

2. Four-Layer First-Party AI Visibility Evidence Model

Each visibility claim is scoped to its underlying deterministic signal source rather than collapsed into a single opaque score.

Layer IDLayer NamePrimary Signal SourceDeterministic Scope Boundary
L1_server_crawler_truthServer Crawler & Retrieval TelemetryEdge/runtime HTTP request headers, status codes, and verified bot identityProves crawler discovery and live answer-engine page fetches without client JS
L2_citation_surface_readinessCitation Surface & Entity Graph ReadinessCanonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrorsEvaluates structural eligibility for verifiable citations; does not guarantee citation
L3_verified_ai_referral_sessionFirst-Party AI Referral Session AttributionDocument referrer, UTM/citation parameters, and session landing pageSeparates human clicks arriving from AI answer engines from generic organic/direct visits
L4_connected_outcome_contextConnected Search & Conversion ContextAuthorized GSC query matrices, waitlist/lead events, and commerce outcomesAvailable in connected workspaces only; never inferred from public static previews

3. Public Static GEO Readiness Preview Weights (100-Point Scale)

Applies strictly to the public single-URL static checker (/tools/geo-score). Connected hosted workspaces evaluate live crawler and referral telemetry separately.

DimensionWeightEvaluated Static SignalsScope Boundary
infrastructure25%HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presencePublic static scan dimension; excludes private server logs
metadata25%Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open GraphPublic static scan dimension; checks single-URL HTML head tags
structured_data25%Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schemaPublic static scan dimension; validates machine-parsable entity definitions
ai_guidance25%llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headingsPublic static scan dimension; does not represent hosted multi-engine citation tracking

Published under Creative Commons Attribution 4.0 International (CC BY 4.0) by Global Gravity Technology. Cite https://citationgraph.ai/en/data/ai-crawler-and-visibility-taxonomy.

License: https://creativecommons.org/licenses/by/4.0/

Why separate model training bots from AI search and user retrieval bots?

Blocking training crawlers such as GPTBot or Google-Extended controls corpus training, whereas blocking OAI-SearchBot, ChatGPT-User, or PerplexityBot removes your pages from live AI search citations.

Does this public dataset include private customer analytics?

No. Every table in this dataset documents public protocol definitions, bot verification rules, and static preview weights. Hosted workspace telemetry remains strictly private.