---
title: "AI Crawler Role Taxonomy & Visibility Evidence Standards"
description: "Open CC BY 4.0 dataset of 14 major AI crawler roles (training, search index, live user retrieval), 4 attribution evidence layers, and public static GEO readiness weights."
canonical_url: "https://citationgraph.ai/en/data/ai-crawler-and-visibility-taxonomy"
markdown_url: "https://citationgraph.ai/en/data/ai-crawler-and-visibility-taxonomy.md"
locale: "en"
entity_id: "https://citationgraph.ai/#software-application"
parent_organization: "https://www.global-gravity.com/#organization"
sibling_product: "https://getgravityads.ai/#software-application"
---

# AI Crawler Role Taxonomy & Visibility Evidence Standards

> CitationGraph classifies AI bot traffic into three operational roles—model training crawlers, AI search indexers, and user-triggered live retrieval agents—across four distinct evidence layers, while weighting public static GEO previews equally across Infrastructure (25%), Metadata (25%), Structured Data (25%), and AI Guidance (25%).

- **License**: https://creativecommons.org/licenses/by/4.0/
- **CSV Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.csv
- **JSON Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.json

## 1. AI Crawler & Retrieval Agent Role Classification (14 Reference Bots)

| Bot Token | Operator | Role | robots.txt Token | Signal Layer | Verification |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `GPTBot` | OpenAI | `model_training` | `GPTBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `OAI-SearchBot` | OpenAI | `search_index` | `OAI-SearchBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `ChatGPT-User` | OpenAI | `live_user_retrieval` | `ChatGPT-User` | `live_retrieval_fetch` | Published IP CIDR ranges + User-Agent |
| `ClaudeBot` | Anthropic | `model_training` | `ClaudeBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-SearchBot` | Anthropic | `search_index` | `Claude-SearchBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-User` | Anthropic | `live_user_retrieval` | `Claude-User` | `live_retrieval_fetch` | User-Agent + request header pattern |
| `PerplexityBot` | Perplexity AI | `search_index` | `PerplexityBot` | `server_crawler_discovery` | Published IP JSON list + User-Agent |
| `Perplexity-User` | Perplexity AI | `live_user_retrieval` | `Perplexity-User` | `live_retrieval_fetch` | Published IP JSON list + User-Agent |
| `Google-Extended` | Google | `model_training` | `Google-Extended` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com + policy token |
| `GoogleOther` | Google | `search_index` | `GoogleOther` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com |
| `Applebot-Extended` | Apple | `model_training` | `Applebot-Extended` | `server_crawler_discovery` | Reverse/Forward DNS applebot.apple.com + policy token |
| `Amazonbot` | Amazon | `search_index` | `Amazonbot` | `server_crawler_discovery` | Reverse/Forward DNS crawl.amazonbot.amazon |
| `Meta-ExternalAgent` | Meta | `model_training` | `Meta-ExternalAgent` | `server_crawler_discovery` | User-Agent + Meta ASN verification |
| `CCBot` | Common Crawl | `model_training` | `CCBot` | `server_crawler_discovery` | User-Agent + Common Crawl AWS egress metadata |

## 2. Four-Layer First-Party AI Visibility Evidence Model

| Layer ID | Layer Name | Signal Source | Deterministic Scope |
| :--- | :--- | :--- | :--- |
| `L1_server_crawler_truth` | Server Crawler & Retrieval Telemetry | Edge/runtime HTTP request headers, status codes, and verified bot identity | Proves crawler discovery and live answer-engine page fetches without client JS |
| `L2_citation_surface_readiness` | Citation Surface & Entity Graph Readiness | Canonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrors | Evaluates structural eligibility for verifiable citations; does not guarantee citation |
| `L3_verified_ai_referral_session` | First-Party AI Referral Session Attribution | Document referrer, UTM/citation parameters, and session landing page | Separates human clicks arriving from AI answer engines from generic organic/direct visits |
| `L4_connected_outcome_context` | Connected Search & Conversion Context | Authorized GSC query matrices, waitlist/lead events, and commerce outcomes | Available in connected workspaces only; never inferred from public static previews |

## 3. Public Static GEO Readiness Preview Weights (100-Point Scale)

| Dimension | Weight | Evaluated Signals | Boundary Note |
| :--- | :--- | :--- | :--- |
| `infrastructure` | 25% | HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presence | Public static scan dimension; excludes private server logs |
| `metadata` | 25% | Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open Graph | Public static scan dimension; checks single-URL HTML head tags |
| `structured_data` | 25% | Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schema | Public static scan dimension; validates machine-parsable entity definitions |
| `ai_guidance` | 25% | llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headings | Public static scan dimension; does not represent hosted multi-engine citation tracking |
