公开权威基准数据集 · CC BY 4.0
AI 爬虫角色分类体系与可见度归因证据标准数据集
基于 CC BY 4.0 协议开放的结构化数据集:涵盖 14 类主流 AI 爬虫角色(模型训练、搜索索引、实时检索)、四层首方归因信号标准与公开静态 GEO 评分权重。
CitationGraph 将 AI 机器访问严格区分为三类运行角色(模型训练抓取、AI 搜索索引、用户提问触发的实时检索),建立四层可核验归因证据模型,并在公开静态 GEO 预览中按基础设施(25%)、元数据(25%)、结构化数据(25%)与 AI 引导协议(25%)等权评估技术就绪度。
1. 主流 AI 爬虫与检索 Agent 角色分类标准(14 类核心标识)
区分后台语料训练、AI 搜索建索引与用户实时提问抓取,避免在配置 robots.txt 时误伤品牌在生成式回答中的实时引用资格。
| 标识名称 | 所属机构 | 核心职责角色 | robots 控制令牌 | 归属信号层 | 核验方式 |
|---|---|---|---|---|---|
| GPTBot | OpenAI | model_training | GPTBot | server_crawler_discovery | Published IP CIDR ranges + User-Agent |
| OAI-SearchBot | OpenAI | search_index | OAI-SearchBot | server_crawler_discovery | Published IP CIDR ranges + User-Agent |
| ChatGPT-User | OpenAI | live_user_retrieval | ChatGPT-User | live_retrieval_fetch | Published IP CIDR ranges + User-Agent |
| ClaudeBot | Anthropic | model_training | ClaudeBot | server_crawler_discovery | User-Agent + ASN attribution |
| Claude-SearchBot | Anthropic | search_index | Claude-SearchBot | server_crawler_discovery | User-Agent + ASN attribution |
| Claude-User | Anthropic | live_user_retrieval | Claude-User | live_retrieval_fetch | User-Agent + request header pattern |
| PerplexityBot | Perplexity AI | search_index | PerplexityBot | server_crawler_discovery | Published IP JSON list + User-Agent |
| Perplexity-User | Perplexity AI | live_user_retrieval | Perplexity-User | live_retrieval_fetch | Published IP JSON list + User-Agent |
| Google-Extended | model_training | Google-Extended | server_crawler_discovery | Reverse/Forward DNS googlebot.com + policy token | |
| GoogleOther | search_index | GoogleOther | server_crawler_discovery | Reverse/Forward DNS googlebot.com | |
| Applebot-Extended | Apple | model_training | Applebot-Extended | server_crawler_discovery | Reverse/Forward DNS applebot.apple.com + policy token |
| Amazonbot | Amazon | search_index | Amazonbot | server_crawler_discovery | Reverse/Forward DNS crawl.amazonbot.amazon |
| Meta-ExternalAgent | Meta | model_training | Meta-ExternalAgent | server_crawler_discovery | User-Agent + Meta ASN verification |
| CCBot | Common Crawl | model_training | CCBot | server_crawler_discovery | User-Agent + Common Crawl AWS egress metadata |
2. 四层首方 AI 可见度归因证据层级规范
每一项可见度指标均绑定明确的确定性信号来源,拒绝用单一黑盒估算分替代真实服务端抓取与引荐证据。
| 层级编码 | 证据层名称 | 核心信号来源 | 确定性适用边界 |
|---|---|---|---|
| L1_server_crawler_truth | Server Crawler & Retrieval Telemetry | Edge/runtime HTTP request headers, status codes, and verified bot identity | Proves crawler discovery and live answer-engine page fetches without client JS |
| L2_citation_surface_readiness | Citation Surface & Entity Graph Readiness | Canonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrors | Evaluates structural eligibility for verifiable citations; does not guarantee citation |
| L3_verified_ai_referral_session | First-Party AI Referral Session Attribution | Document referrer, UTM/citation parameters, and session landing page | Separates human clicks arriving from AI answer engines from generic organic/direct visits |
| L4_connected_outcome_context | Connected Search & Conversion Context | Authorized GSC query matrices, waitlist/lead events, and commerce outcomes | Available in connected workspaces only; never inferred from public static previews |
3. 公开静态 GEO 就绪度预览评分权重表(100 分制)
仅适用于公开单页静态检测工具(/tools/geo-score);商业托管版工作台另行结合真实爬虫日志与引荐会话进行纵深评估。
| 评估维度 | 权重占比 | 静态核验信号项 | 能力边界说明 |
|---|---|---|---|
| infrastructure | 25% | HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presence | Public static scan dimension; excludes private server logs |
| metadata | 25% | Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open Graph | Public static scan dimension; checks single-URL HTML head tags |
| structured_data | 25% | Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schema | Public static scan dimension; validates machine-parsable entity definitions |
| ai_guidance | 25% | llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headings | Public static scan dimension; does not represent hosted multi-engine citation tracking |
本数据集由 Global Gravity Technology(重力科技)按 CC BY 4.0 国际许可协议免费发布。引用请注明 https://citationgraph.ai/zh/data/ai-crawler-and-visibility-taxonomy。
为什么要把训练爬虫与 AI 搜索索引、实时检索爬虫拆开统计?
限制 GPTBot 或 Google-Extended 仅影响大模型预训练语料采集;若误拦截 OAI-SearchBot、ChatGPT-User 或 PerplexityBot,则会直接导致网站无法出现在实时 AI 回答引用中。
这份公开数据集是否包含客户站点的私有遥测数据?
不包含。本数据集仅公开行业爬虫识别规范、归因层级定义与公开静态检测权重,零私有客户流量数据暴露。