AI Crawler Role Taxonomy & Visibility Evidence Standards
Open CC BY 4.0 dataset of 14 major AI crawler roles (training, search index, live user retrieval), 4 attribution evidence layers, and public static GEO readiness weights.
1. AI Crawler & Retrieval Agent Role Classification (14 Reference Bots)
Separating background corpus training from search indexing and real-time user prompt fetches prevents robots.txt policies from accidentally blocking AI citations.
| Bot Token | Operator | Operational Role | robots.txt Token | Signal Layer | Verification |
|---|---|---|---|---|---|
| GPTBot | OpenAI | model_training | GPTBot | server_crawler_discovery | Published IP CIDR ranges + User-Agent |
| OAI-SearchBot | OpenAI | search_index | OAI-SearchBot | server_crawler_discovery | Published IP CIDR ranges + User-Agent |
| ChatGPT-User | OpenAI | live_user_retrieval | ChatGPT-User | live_retrieval_fetch | Published IP CIDR ranges + User-Agent |
| ClaudeBot | Anthropic | model_training | ClaudeBot | server_crawler_discovery | User-Agent + ASN attribution |
| Claude-SearchBot | Anthropic | search_index | Claude-SearchBot | server_crawler_discovery | User-Agent + ASN attribution |
| Claude-User | Anthropic | live_user_retrieval | Claude-User | live_retrieval_fetch | User-Agent + request header pattern |
| PerplexityBot | Perplexity AI | search_index | PerplexityBot | server_crawler_discovery | Published IP JSON list + User-Agent |
| Perplexity-User | Perplexity AI | live_user_retrieval | Perplexity-User | live_retrieval_fetch | Published IP JSON list + User-Agent |
| Google-Extended | model_training | Google-Extended | server_crawler_discovery | Reverse/Forward DNS googlebot.com + policy token | |
| GoogleOther | search_index | GoogleOther | server_crawler_discovery | Reverse/Forward DNS googlebot.com | |
| Applebot-Extended | Apple | model_training | Applebot-Extended | server_crawler_discovery | Reverse/Forward DNS applebot.apple.com + policy token |
| Amazonbot | Amazon | search_index | Amazonbot | server_crawler_discovery | Reverse/Forward DNS crawl.amazonbot.amazon |
| Meta-ExternalAgent | Meta | model_training | Meta-ExternalAgent | server_crawler_discovery | User-Agent + Meta ASN verification |
| CCBot | Common Crawl | model_training | CCBot | server_crawler_discovery | User-Agent + Common Crawl AWS egress metadata |
2. Four-Layer First-Party AI Visibility Evidence Model
Each visibility claim is scoped to its underlying deterministic signal source rather than collapsed into a single opaque score.
| Layer ID | Layer Name | Primary Signal Source | Deterministic Scope Boundary |
|---|---|---|---|
| L1_server_crawler_truth | Server Crawler & Retrieval Telemetry | Edge/runtime HTTP request headers, status codes, and verified bot identity | Proves crawler discovery and live answer-engine page fetches without client JS |
| L2_citation_surface_readiness | Citation Surface & Entity Graph Readiness | Canonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrors | Evaluates structural eligibility for verifiable citations; does not guarantee citation |
| L3_verified_ai_referral_session | First-Party AI Referral Session Attribution | Document referrer, UTM/citation parameters, and session landing page | Separates human clicks arriving from AI answer engines from generic organic/direct visits |
| L4_connected_outcome_context | Connected Search & Conversion Context | Authorized GSC query matrices, waitlist/lead events, and commerce outcomes | Available in connected workspaces only; never inferred from public static previews |
3. Public Static GEO Readiness Preview Weights (100-Point Scale)
Applies strictly to the public single-URL static checker (/tools/geo-score). Connected hosted workspaces evaluate live crawler and referral telemetry separately.
| Dimension | Weight | Evaluated Static Signals | Scope Boundary |
|---|---|---|---|
| infrastructure | 25% | HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presence | Public static scan dimension; excludes private server logs |
| metadata | 25% | Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open Graph | Public static scan dimension; checks single-URL HTML head tags |
| structured_data | 25% | Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schema | Public static scan dimension; validates machine-parsable entity definitions |
| ai_guidance | 25% | llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headings | Public static scan dimension; does not represent hosted multi-engine citation tracking |
Published under Creative Commons Attribution 4.0 International (CC BY 4.0) by Global Gravity Technology. Cite https://citationgraph.ai/en/data/ai-crawler-and-visibility-taxonomy.
Why separate model training bots from AI search and user retrieval bots?
Blocking training crawlers such as GPTBot or Google-Extended controls corpus training, whereas blocking OAI-SearchBot, ChatGPT-User, or PerplexityBot removes your pages from live AI search citations.
Does this public dataset include private customer analytics?
No. Every table in this dataset documents public protocol definitions, bot verification rules, and static preview weights. Hosted workspace telemetry remains strictly private.