Skip to content
SIbotaccess

wikipedia.org

Scanned

/api/site/wikipedia.org

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBotinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
OAI-SearchBotinherited
Search / answer index
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
ChatGPT-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

Anthropic

ClaudeBotinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
Claude-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
Claude-SearchBotinherited
Search / answer index
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

Google AI

Google-Extendedinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

Perplexity

PerplexityBotinherited
Search / answer index
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
Perplexity-Userinherited
On-demand user fetch
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

ByteDance

Bytespiderinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

Common Crawl

CCBotinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial

Meta

Meta-ExternalAgentinherited
Training crawler
inherits User-agent: *
Disallow: /w/
Disallow: /api/
Disallow: /trap/
Disallow: /wiki/Special:
Disallow: /wiki/Spezial:
Disallow: /wiki/Spesial:
Allow: /w/api.php?action=mobileview&
Allow: /w/load.php?
Allow: /api/rest_v1/?doc
Allow: /w/rest.php/site/v1/sitemap
+422 more disallow rules
Unspecifiedeffective: partial
Back to the directory