Original research - 3 July 2026
Crawler and AI Retrieval Access Audit 2026
A public audit of Visa Atlas robots policy, AI/search crawler access rules, retrieval entry points, JSON headers and local server-log proof commands.
This report proves access intent and crawler-facing discovery wiring. It does not claim that any engine has fetched, indexed, ranked or cited the site; that proof comes from server logs, Search Console and reviewed citation evidence.
17
Allowed crawler rows
13
Explicit allow rules
8
Denied scraper rules
8
Technical checks
Robots policy matrix
src/lib/crawler-access-audit.ts exports the allow and deny rule constants consumed by src/app/robots.ts; this feed, the research report and generated robots.txt share the same policy list.
- Explicit allow
OpenAI - ai-crawler
GPTBot
Explicitly allowed for OpenAI crawler access; server logs are still required to prove fetches.
robots directive: allow /
- Explicit allow
OpenAI - ai-retrieval
OAI-SearchBot
Explicitly allowed for ChatGPT search retrieval; citation presence is measured separately from access.
robots directive: allow /
- Explicit allow
OpenAI - ai-retrieval
ChatGPT-User
Explicitly allowed for user-triggered ChatGPT browsing and retrieval requests.
robots directive: allow /
- Explicit allow
Anthropic - ai-crawler
ClaudeBot
Explicitly allowed for Anthropic crawler access.
robots directive: allow /
- Explicit allow
Anthropic - ai-retrieval
Claude-Web
Explicitly allowed for Claude web retrieval.
robots directive: allow /
- Explicit allow
Anthropic - ai-crawler
anthropic-ai
Explicitly allowed for Anthropic AI crawler access.
robots directive: allow /
- Explicit allow
Perplexity - ai-crawler
PerplexityBot
Explicitly allowed for Perplexity crawler access.
robots directive: allow /
- Explicit allow
Perplexity - ai-retrieval
Perplexity-User
Explicitly allowed for user-triggered Perplexity retrieval requests.
robots directive: allow /
- Explicit allow
Google - ai-training-token
Google-Extended
Explicit robots token allowance; live Gemini or Googlebot fetches must be read from server logs and Search Console.
robots directive: allow /
- Explicit allow
Apple - ai-training-token
Applebot-Extended
Explicit robots token allowance for Apple extended crawler controls.
robots directive: allow /
- Explicit allow
Common Crawl - common-crawl
CCBot
Explicitly allowed because open dataset reuse and downstream AI retrieval are part of the public data strategy.
robots directive: allow /
- Explicit allow
Meta - ai-crawler
Meta-ExternalAgent
Explicitly allowed for Meta external agent access.
robots directive: allow /
- Explicit allow
Amazon - ai-crawler
Amazonbot
Explicitly allowed for Amazon crawler access.
robots directive: allow /
- Wildcard allow
Google - search
Googlebot
Allowed by the wildcard rule and discovered through the sitemap, indexable page manifest and rendered HTML.
robots directive: allow /
- Wildcard allow
Microsoft Bing - search
bingbot
Allowed by the wildcard rule and discovered through the same sitemap and page-manifest surfaces as Googlebot.
robots directive: allow /
- Wildcard allow
Apple - search
Applebot
Allowed by the wildcard rule; Applebot-Extended is also explicitly allowed.
robots directive: allow /
- Wildcard allow
Anthropic - ai-retrieval
Claude-SearchBot
Allowed by the wildcard rule; monitored by the bot-log parser as an Anthropic retrieval user agent.
robots directive: allow /
- Disallow
Competitive SEO crawler - competitive-tool
SemrushBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
AhrefsBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
MJ12bot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
DotBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
DataForSeoBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
BLEXBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
SeznamBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
- Disallow
Competitive SEO crawler - competitive-tool
PetalBot
Disallowed because the access strategy welcomes search and AI retrieval while blocking aggressive competitive scraping.
robots directive: disallow /
Evidence surfaces
Retrieval and discovery gates
instrumented
robots.txt allows search and AI retrievers
npm run audit:visibility-metricsThe generated robots file consumes the same allow and deny rule constants published by this dataset.
Evidence surfaces
- /robots.txt
- src/app/robots.ts
- src/lib/crawler-access-audit.ts
instrumented
Sitemap and page manifest expose indexable URLs
npm run audit:indexable-page-manifestThe sitemap is advertised from robots.txt and mirrored into a JSON manifest for crawler reconciliation.
Evidence surfaces
instrumented
Public JSON emits X-Robots-Tag: noindex, follow
npm run audit:data-front-doorAPI routes stay out of search result pages while still allowing crawlers to follow discovery links.
Evidence surfaces
- /api/public
- /api/openapi.json
- next.config.ts
instrumented
Public API responses advertise discovery Link headers
npm run audit:data-front-doorLink headers connect JSON feeds to human docs, OpenAPI, license guidance and LLM indexes.
Evidence surfaces
instrumented
Human research and data pages expose alternate JSON headers
npm run audit:data-citationThe report page has an alternate JSON endpoint for crawlers and data consumers.
Evidence surfaces
instrumented
LLM indexes point to crawler access evidence
npm run audit:data-front-doorPlain-text retrieval indexes include the report and JSON endpoint alongside the wider open-data graph.
Evidence surfaces
instrumented
Live fetch evidence remains a local server-log workflow
npm run seo:botlog -- logs.txt --jsonGA4 cannot prove crawler fetches; server logs are required for live bot-hit evidence and are not published by this endpoint. The parser is regression-tested with npm run audit:botlog.
Evidence surfaces
instrumented
Crawler identity and WAF rules require official IP or DNS checks
npm run audit:crawler-accessUser-agent strings can be spoofed. WAF allow rules and verified-fetch claims must pair the user-agent with official IP range JSON or Google reverse/forward DNS evidence where available.
Identity and WAF verification
User-agent strings alone are not proof. Where official IP ranges or DNS verification methods exist, local WAF allow rules and verified-fetch claims must match the user-agent to those official identity sources before publishing aggregate evidence.
- ip-range-json
OpenAI
GPTBot
If a CDN/WAF rule blocks AI crawlers, allow GPTBot only when the user-agent matches and the source IP is present in the official OpenAI GPTBot range file.
Server-log proof must preserve timestamp, normalized path, user-agent, status code and a source IP matched against https://openai.com/gptbot.json before the hit is treated as verified OpenAI crawler access.
The JSON range file is an identity aid, not evidence of indexing, ranking, answer use or training inclusion.
Official identity sources
- ip-range-json
OpenAI
OAI-SearchBot
If a CDN/WAF rule blocks ChatGPT search retrieval, allow OAI-SearchBot only when the user-agent matches and the source IP is present in the official OpenAI search bot range file.
Server-log proof must preserve timestamp, normalized path, user-agent, status code and a source IP matched against https://openai.com/searchbot.json before the hit is treated as verified ChatGPT search retrieval.
Verified fetch access does not prove that ChatGPT cited Visa Atlas or sent human referral sessions.
Official identity sources
- ip-range-json
OpenAI
ChatGPT-User
If a CDN/WAF rule blocks user-triggered ChatGPT browsing, allow ChatGPT-User only when the user-agent matches and the source IP is present in the official OpenAI ChatGPT-User range file.
Server-log proof must preserve timestamp, normalized path, user-agent, status code and a source IP matched against https://openai.com/chatgpt-user.json before the hit is treated as verified user-triggered ChatGPT retrieval.
User-triggered fetch evidence is separate from automated crawler access and from analytics referral attribution.
Official identity sources
- ip-range-json
Perplexity
PerplexityBot
If a CDN/WAF rule blocks Perplexity crawler access, allow PerplexityBot only when the user-agent matches and the source IP is present in the official PerplexityBot range file.
Server-log proof must preserve timestamp, normalized path, user-agent, status code and a source IP matched against https://www.perplexity.ai/perplexitybot.json before the hit is treated as verified Perplexity crawler access.
Verified PerplexityBot access is not the same thing as a Perplexity answer citation or referral session.
Official identity sources
- reverse-dns-and-ip-range
Google
Googlebot
If a CDN/WAF rule blocks Googlebot, allow Googlebot only after reverse DNS and forward DNS checks agree or the source IP is present in the official Google common-crawlers range file.
Server-log proof must preserve timestamp, normalized path, user-agent, status code and either matching reverse/forward DNS for googlebot.com or an IP match against the official Google common crawler ranges.
Verified Googlebot access does not prove indexing, ranking, Search Console coverage, Discover inclusion or AI Overview citation.
- robots-token-only
Google
Google-Extended
Keep Google-Extended as a robots token control, not a server-log bot identity; do not create WAF allow or proof rows from this token alone.
The only public proof for Google-Extended here is robots.txt policy intent. Live Google fetch proof must use Googlebot, Search Console or verified Google crawler logs.
Google-Extended is not counted as a loggable crawler hit by the local bot-log parser.
Official identity sources
- robots-token-only
Apple
Applebot-Extended
Keep Applebot-Extended as a robots token control, not a server-log bot identity; do not create WAF allow or proof rows from this token alone.
The only public proof for Applebot-Extended here is robots.txt policy intent. Live Apple fetch proof must come from actual Applebot log rows.
Applebot-Extended is not counted as a loggable crawler hit by the local bot-log parser.
Official identity sources
Live proof limits
This dataset proves access intent and discovery wiring. It does not prove that a crawler fetched the site; live proof requires server access logs or Search Console evidence.
Raw server logs, IP addresses, Search Console exports, model transcripts and unpublished referral analytics remain local and are not exposed by this public endpoint.
npm run audit:visibility-metricsnpm run audit:data-front-doornpm run audit:data-citationnpm run audit:indexable-page-manifestnpm run audit:rendered-seonpm run audit:crawler-accessnpm run audit:botlognpm run seo:botlog -- logs.txt --json- Allowed in robots.txt is not a ranking, indexing, citation or traffic guarantee.
- Some AI systems use search-engine crawlers or robots tokens rather than a distinct user-agent in server logs.
- User-agent strings can be spoofed, so verified-fetch claims need official IP range or reverse/forward DNS checks where the crawler publisher provides them.
- Public JSON endpoints emit noindex, follow so dataset links remain discoverable without competing with canonical human pages in search results.
Cite or reuse this dataset
This audit is free to reuse under CC BY 4.0. Cite the JSON endpoint for the machine-readable crawler policy matrix and the report page for methodology and live-proof limits.
Suggested citation
Visa Atlas, "Visa Atlas crawler and AI retrieval access audit", https://visaatlas.org/research/crawler-ai-retrieval-access-audit-2026. Last verified 3 July 2026.
- JSON endpoint
- https://visaatlas.org/api/public/crawler-access