---
title: "AI 爬虫角色分类体系与可见度归因证据标准数据集"
description: "基于 CC BY 4.0 协议开放的结构化数据集：涵盖 14 类主流 AI 爬虫角色（模型训练、搜索索引、实时检索）、四层首方归因信号标准与公开静态 GEO 评分权重。"
canonical_url: "https://citationgraph.ai/zh/data/ai-crawler-and-visibility-taxonomy"
markdown_url: "https://citationgraph.ai/zh/data/ai-crawler-and-visibility-taxonomy.md"
locale: "zh"
entity_id: "https://citationgraph.ai/#software-application"
parent_organization: "https://www.global-gravity.com/#organization"
sibling_product: "https://getgravityads.ai/#software-application"
---

# AI 爬虫角色分类体系与可见度归因证据标准数据集

> CitationGraph 将 AI 机器访问严格区分为三类运行角色（模型训练抓取、AI 搜索索引、用户提问触发的实时检索），建立四层可核验归因证据模型，并在公开静态 GEO 预览中按基础设施（25%）、元数据（25%）、结构化数据（25%）与 AI 引导协议（25%）等权评估技术就绪度。

- **License**: https://creativecommons.org/licenses/by/4.0/
- **CSV Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.csv
- **JSON Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.json

## 1. 主流 AI 爬虫与检索 Agent 角色分类标准（14 类核心标识）

| Bot Token | Operator | Role | robots.txt Token | Signal Layer | Verification |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `GPTBot` | OpenAI | `model_training` | `GPTBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `OAI-SearchBot` | OpenAI | `search_index` | `OAI-SearchBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `ChatGPT-User` | OpenAI | `live_user_retrieval` | `ChatGPT-User` | `live_retrieval_fetch` | Published IP CIDR ranges + User-Agent |
| `ClaudeBot` | Anthropic | `model_training` | `ClaudeBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-SearchBot` | Anthropic | `search_index` | `Claude-SearchBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-User` | Anthropic | `live_user_retrieval` | `Claude-User` | `live_retrieval_fetch` | User-Agent + request header pattern |
| `PerplexityBot` | Perplexity AI | `search_index` | `PerplexityBot` | `server_crawler_discovery` | Published IP JSON list + User-Agent |
| `Perplexity-User` | Perplexity AI | `live_user_retrieval` | `Perplexity-User` | `live_retrieval_fetch` | Published IP JSON list + User-Agent |
| `Google-Extended` | Google | `model_training` | `Google-Extended` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com + policy token |
| `GoogleOther` | Google | `search_index` | `GoogleOther` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com |
| `Applebot-Extended` | Apple | `model_training` | `Applebot-Extended` | `server_crawler_discovery` | Reverse/Forward DNS applebot.apple.com + policy token |
| `Amazonbot` | Amazon | `search_index` | `Amazonbot` | `server_crawler_discovery` | Reverse/Forward DNS crawl.amazonbot.amazon |
| `Meta-ExternalAgent` | Meta | `model_training` | `Meta-ExternalAgent` | `server_crawler_discovery` | User-Agent + Meta ASN verification |
| `CCBot` | Common Crawl | `model_training` | `CCBot` | `server_crawler_discovery` | User-Agent + Common Crawl AWS egress metadata |

## 2. 四层首方 AI 可见度归因证据层级规范

| Layer ID | Layer Name | Signal Source | Deterministic Scope |
| :--- | :--- | :--- | :--- |
| `L1_server_crawler_truth` | Server Crawler & Retrieval Telemetry | Edge/runtime HTTP request headers, status codes, and verified bot identity | Proves crawler discovery and live answer-engine page fetches without client JS |
| `L2_citation_surface_readiness` | Citation Surface & Entity Graph Readiness | Canonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrors | Evaluates structural eligibility for verifiable citations; does not guarantee citation |
| `L3_verified_ai_referral_session` | First-Party AI Referral Session Attribution | Document referrer, UTM/citation parameters, and session landing page | Separates human clicks arriving from AI answer engines from generic organic/direct visits |
| `L4_connected_outcome_context` | Connected Search & Conversion Context | Authorized GSC query matrices, waitlist/lead events, and commerce outcomes | Available in connected workspaces only; never inferred from public static previews |

## 3. 公开静态 GEO 就绪度预览评分权重表（100 分制）

| Dimension | Weight | Evaluated Signals | Boundary Note |
| :--- | :--- | :--- | :--- |
| `infrastructure` | 25% | HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presence | Public static scan dimension; excludes private server logs |
| `metadata` | 25% | Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open Graph | Public static scan dimension; checks single-URL HTML head tags |
| `structured_data` | 25% | Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schema | Public static scan dimension; validates machine-parsable entity definitions |
| `ai_guidance` | 25% | llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headings | Public static scan dimension; does not represent hosted multi-engine citation tracking |
