---
title: "AI 크롤러 역할 분류 및 가시성 근거 표준 데이터셋"
description: "CC BY 4.0 라이선스로 제공되는 공개 데이터셋: 14개 주요 AI 크롤러 역할 분류(학습, 검색 인덱스, 실시간 사용자 조회), 4단계 귀속 근거 계층 및 정적 GEO 가중치 수록."
canonical_url: "https://citationgraph.ai/ko/data/ai-crawler-and-visibility-taxonomy"
markdown_url: "https://citationgraph.ai/ko/data/ai-crawler-and-visibility-taxonomy.md"
locale: "ko"
entity_id: "https://citationgraph.ai/#software-application"
parent_organization: "https://www.global-gravity.com/#organization"
sibling_product: "https://getgravityads.ai/#software-application"
---

# AI 크롤러 역할 분류 및 가시성 근거 표준 데이터셋

> CitationGraph는 AI 봇 트래픽을 모델 학습 크롤러, AI 검색 인덱서, 실시간 사용자 프롬프트 조회의 3가지 역할로 분류하고 4단계 퍼스트파티 근거 모델과 4차원 균등 가중치(인프라 25%, 메타데이터 25%, 구조화 데이터 25%, AI 가이던스 25%)로 정적 준비도를 평가합니다.

- **License**: https://creativecommons.org/licenses/by/4.0/
- **CSV Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.csv
- **JSON Download**: https://citationgraph.ai/data/ai-crawler-and-visibility-taxonomy.json

## 1. 주요 AI 크롤러 및 조회 에이전트 역할 분류 (14개 기준 봇)

| Bot Token | Operator | Role | robots.txt Token | Signal Layer | Verification |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `GPTBot` | OpenAI | `model_training` | `GPTBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `OAI-SearchBot` | OpenAI | `search_index` | `OAI-SearchBot` | `server_crawler_discovery` | Published IP CIDR ranges + User-Agent |
| `ChatGPT-User` | OpenAI | `live_user_retrieval` | `ChatGPT-User` | `live_retrieval_fetch` | Published IP CIDR ranges + User-Agent |
| `ClaudeBot` | Anthropic | `model_training` | `ClaudeBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-SearchBot` | Anthropic | `search_index` | `Claude-SearchBot` | `server_crawler_discovery` | User-Agent + ASN attribution |
| `Claude-User` | Anthropic | `live_user_retrieval` | `Claude-User` | `live_retrieval_fetch` | User-Agent + request header pattern |
| `PerplexityBot` | Perplexity AI | `search_index` | `PerplexityBot` | `server_crawler_discovery` | Published IP JSON list + User-Agent |
| `Perplexity-User` | Perplexity AI | `live_user_retrieval` | `Perplexity-User` | `live_retrieval_fetch` | Published IP JSON list + User-Agent |
| `Google-Extended` | Google | `model_training` | `Google-Extended` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com + policy token |
| `GoogleOther` | Google | `search_index` | `GoogleOther` | `server_crawler_discovery` | Reverse/Forward DNS googlebot.com |
| `Applebot-Extended` | Apple | `model_training` | `Applebot-Extended` | `server_crawler_discovery` | Reverse/Forward DNS applebot.apple.com + policy token |
| `Amazonbot` | Amazon | `search_index` | `Amazonbot` | `server_crawler_discovery` | Reverse/Forward DNS crawl.amazonbot.amazon |
| `Meta-ExternalAgent` | Meta | `model_training` | `Meta-ExternalAgent` | `server_crawler_discovery` | User-Agent + Meta ASN verification |
| `CCBot` | Common Crawl | `model_training` | `CCBot` | `server_crawler_discovery` | User-Agent + Common Crawl AWS egress metadata |

## 2. 4단계 퍼스트파티 AI 가시성 근거 모델

| Layer ID | Layer Name | Signal Source | Deterministic Scope |
| :--- | :--- | :--- | :--- |
| `L1_server_crawler_truth` | Server Crawler & Retrieval Telemetry | Edge/runtime HTTP request headers, status codes, and verified bot identity | Proves crawler discovery and live answer-engine page fetches without client JS |
| `L2_citation_surface_readiness` | Citation Surface & Entity Graph Readiness | Canonical HTML, JSON-LD @id graph, llms.txt, and .md dual-stack mirrors | Evaluates structural eligibility for verifiable citations; does not guarantee citation |
| `L3_verified_ai_referral_session` | First-Party AI Referral Session Attribution | Document referrer, UTM/citation parameters, and session landing page | Separates human clicks arriving from AI answer engines from generic organic/direct visits |
| `L4_connected_outcome_context` | Connected Search & Conversion Context | Authorized GSC query matrices, waitlist/lead events, and commerce outcomes | Available in connected workspaces only; never inferred from public static previews |

## 3. 공개 정적 GEO 준비도 미리보기 가중치 (100점 만점)

| Dimension | Weight | Evaluated Signals | Boundary Note |
| :--- | :--- | :--- | :--- |
| `infrastructure` | 25% | HTTPS enforcement, HTTP 200 status, robots.txt crawl rules, XML sitemap presence | Public static scan dimension; excludes private server logs |
| `metadata` | 25% | Title & meta description boundaries, self-referencing canonical, hreflang reciprocity, Open Graph | Public static scan dimension; checks single-URL HTML head tags |
| `structured_data` | 25% | Valid JSON-LD syntax, Organization/SoftwareApplication/Dataset @id entities, FAQ/Article schema | Public static scan dimension; validates machine-parsable entity definitions |
| `ai_guidance` | 25% | llms.txt / llms-full.txt availability, Markdown alternate Link headers, clean section headings | Public static scan dimension; does not represent hosted multi-engine citation tracking |
