github.com

Scanned 2026-10-22 11:28:35 UTC

/api/site/github.com

Access by AI user-agent

Parsed from the site’s robots.txt. “Inherited” means no rule names the agent and it falls back to User-agent: *.

OpenAI

GPTBot

Training crawler

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Partial

OAI-SearchBot

Search / answer index

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Partial

ChatGPT-Userinherited

On-demand user fetch

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

Anthropic

ClaudeBot

Training crawler

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Partial

Claude-Userinherited

On-demand user fetch

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

Claude-SearchBotinherited

Search / answer index

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

Google AI

Google-Extendedinherited

Training crawler

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

Perplexity

PerplexityBot

Search / answer index

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Partial

Perplexity-Userinherited

On-demand user fetch

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

ByteDance

Bytespider

Training crawler

Disallow: /

Blocked

Common Crawl

CCBotinherited

Training crawler

inherits User-agent: *

Disallow: /account-login

Disallow: /copilot/

Disallow: /copilot/c/

Disallow: /ekansa/Open-Context-Data

Disallow: /ekansa/opencontext-*

Disallow: /Explodingstuff/

Allow: /*?tab=achievements&achievement=*

Allow: /marketplace/*?tab=readme$

Allow: /marketplace/*?tab=transparency$

+156 more disallow rules

Unspecifiedeffective: partial

Files found

/robots.txt

Crawl rules

HTTP 200present

/llms.txt

Content index for LLMs

HTTP 200present

/.well-known/agents.json

Agent manifest (A2A)

HTTP 404absent

/.well-known/ai-catalog.json

AI content catalog

HTTP 200present

/.well-known/botaccess.json

Action declaration (this spec)

HTTP 404absent

Relevant robots.txt

Only the groups matching the AI user-agents we track (plus the wildcard).

User-agent: gptbot

Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$

User-agent: oai-searchbot

Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$

User-agent: claudebot

Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$

User-agent: perplexitybot

Crawl-delay: 1
Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$

User-agent: bytespider

Disallow: /

User-agent: *

Allow: /*?tab=achievements&achievement=*
Allow: /marketplace/*?tab=readme$
Allow: /marketplace/*?tab=transparency$
Disallow: /account-login
Disallow: /copilot/
Disallow: /copilot/c/
Disallow: /ekansa/Open-Context-Data
Disallow: /ekansa/opencontext-*
Disallow: /Explodingstuff/
Disallow: /gist/
Disallow: /search/advanced
Disallow: /search$
Disallow: /*/*/pulse_committer_data.json$
Disallow: /*/*/pulse_committer_data.json?*
Disallow: /*/archive/
Disallow: /*/blame/
Disallow: /*/cache/
Disallow: /*/raw/
Disallow: /*/tree/
Disallow: */tarball/
Disallow: */zipball/
Disallow: /.git/
Disallow: */.git/
Disallow: /*.git$
Disallow: /*.atom$
Disallow: /*report-abuse?report=*
Disallow: /*/*/commits/
Disallow: /marketplace/*/*?tab=readme$
Disallow: /marketplace/*/*?tab=transparency$
Disallow: /marketplace/*?*?tab=readme$
Disallow: /marketplace/*?*?tab=transparency$
Disallow: /marketplace/actions?tab=readme$
Disallow: /marketplace/actions?tab=transparency$
Disallow: /marketplace/free-trials?tab=readme$
Disallow: /marketplace/free-trials?tab=transparency$
Disallow: /marketplace/manage?tab=readme$
Disallow: /marketplace/manage?tab=transparency$
Disallow: /marketplace/models?tab=readme$
Disallow: /marketplace/models?tab=transparency$
Disallow: /marketplace/new?tab=readme$

llms.txt (excerpt)

/llms.txt

# GitHub

> GitHub is a developer platform for building, shipping, and maintaining software. It provides cloud-based Git repository hosting, CI/CD via GitHub Actions, project management with Issues and Projects, code review via pull requests, AI-powered development with GitHub Copilot, and APIs (REST and GraphQL) for automation and integration.

## How to use

To retrieve full article content, pag
…(truncated)

Back to the directory

github.com

Access by AI user-agent

OpenAI

Anthropic

Google AI

Perplexity

ByteDance

Common Crawl

Meta