AI 爬虫角色分类体系与可见度归因证据标准数据集

基于 CC BY 4.0 协议开放的结构化数据集:涵盖 14 类主流 AI 爬虫角色(模型训练、搜索索引、实时检索)、四层首方归因信号标准与公开静态 GEO 评分权重。

CitationGraph 将 AI 机器访问严格区分为三类运行角色(模型训练抓取、AI 搜索索引、用户提问触发的实时检索),建立四层可核验归因证据模型,并在公开静态 GEO 预览中按基础设施(25%)、元数据(25%)、结构化数据(25%)与 AI 引导协议(25%)等权评估技术就绪度。

1. 主流 AI 爬虫与检索 Agent 角色分类标准(14 类核心标识)

区分后台语料训练、AI 搜索建索引与用户实时提问抓取,避免在配置 robots.txt 时误伤品牌在生成式回答中的实时引用资格。

标识名称所属机构核心职责角色robots 控制令牌归属信号层核验方式
GPTBotOpenAImodel_trainingGPTBotserver_crawler_discoveryPublished IP CIDR ranges + User-Agent
OAI-SearchBotOpenAIsearch_indexOAI-SearchBotserver_crawler_discoveryPublished IP CIDR ranges + User-Agent
ChatGPT-UserOpenAIlive_user_retrievalChatGPT-Userlive_retrieval_fetchPublished IP CIDR ranges + User-Agent
ClaudeBotAnthropicmodel_trainingClaudeBotserver_crawler_discoveryUser-Agent + ASN attribution
Claude-SearchBotAnthropicsearch_indexClaude-SearchBotserver_crawler_discoveryUser-Agent + ASN attribution
Claude-UserAnthropiclive_user_retrievalClaude-Userlive_retrieval_fetchUser-Agent + request header pattern
PerplexityBotPerplexity AIsearch_indexPerplexityBotserver_crawler_discoveryPublished IP JSON list + User-Agent
Perplexity-UserPerplexity AIlive_user_retrievalPerplexity-Userlive_retrieval_fetchPublished IP JSON list + User-Agent
Google-ExtendedGooglemodel_trainingGoogle-Extendedserver_crawler_discoveryReverse/Forward DNS googlebot.com + policy token
GoogleOtherGooglesearch_indexGoogleOtherserver_crawler_discoveryReverse/Forward DNS googlebot.com
Applebot-ExtendedApplemodel_trainingApplebot-Extendedserver_crawler_discoveryReverse/Forward DNS applebot.apple.com + policy token
AmazonbotAmazonsearch_indexAmazonbotserver_crawler_discoveryReverse/Forward DNS crawl.amazonbot.amazon
Meta-ExternalAgentMetamodel_trainingMeta-ExternalAgentserver_crawler_discoveryUser-Agent + Meta ASN verification
CCBotCommon Crawlmodel_trainingCCBotserver_crawler_discoveryUser-Agent + Common Crawl AWS egress metadata

2. 四层首方 AI 可见度归因证据层级规范

每一项可见度指标均绑定明确的确定性信号来源,拒绝用单一黑盒估算分替代真实服务端抓取与引荐证据。

层级编码证据层名称核心信号来源确定性适用边界
L1_server_crawler_truthServer Crawler & Retrieval TelemetryEdge/runtime HTTP request headers, status codes, and verified bot identityProves crawler discovery and live answer-engine page fetches without client JS
L2_citation_surface_readinessCitation Surface & Entity Graph ReadinessCanonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrorsEvaluates structural eligibility for verifiable citations; does not guarantee citation
L3_verified_ai_referral_sessionFirst-Party AI Referral Session AttributionDocument referrer, UTM/citation parameters, and session landing pageSeparates human clicks arriving from AI answer engines from generic organic/direct visits
L4_connected_outcome_contextConnected Search & Conversion ContextAuthorized GSC query matrices, waitlist/lead events, and commerce outcomesAvailable in connected workspaces only; never inferred from public static previews

3. 公开静态 GEO 就绪度预览评分权重表(100 分制)

仅适用于公开单页静态检测工具(/tools/geo-score);商业托管版工作台另行结合真实爬虫日志与引荐会话进行纵深评估。

评估维度权重占比静态核验信号项能力边界说明
infrastructure25%HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presencePublic static scan dimension; excludes private server logs
metadata25%Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open GraphPublic static scan dimension; checks single-URL HTML head tags
structured_data25%Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schemaPublic static scan dimension; validates machine-parsable entity definitions
ai_guidance25%llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headingsPublic static scan dimension; does not represent hosted multi-engine citation tracking

本数据集由 Global Gravity Technology(重力科技)按 CC BY 4.0 国际许可协议免费发布。引用请注明 https://citationgraph.ai/zh/data/ai-crawler-and-visibility-taxonomy。

License: https://creativecommons.org/licenses/by/4.0/

为什么要把训练爬虫与 AI 搜索索引、实时检索爬虫拆开统计?

限制 GPTBot 或 Google-Extended 仅影响大模型预训练语料采集;若误拦截 OAI-SearchBot、ChatGPT-User 或 PerplexityBot,则会直接导致网站无法出现在实时 AI 回答引用中。

这份公开数据集是否包含客户站点的私有遥测数据?

不包含。本数据集仅公开行业爬虫识别规范、归因层级定义与公开静态检测权重,零私有客户流量数据暴露。