What is robots.txt?

Robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may or may not visit. It manages crawler access, not whether a page appears in search results.

Why it matters

A single mistaken rule can block search engines or AI crawlers from an entire site. It is also where many sites now choose whether assistants such as ChatGPT can read their content.

How to use it

Keep it minimal, allow the crawlers you want, reference your sitemap and never use it to hide pages that should simply be noindexed instead.

Why it matters

A single mistaken rule can block search engines or AI crawlers from an entire site. It is also where many sites now choose whether assistants such as ChatGPT can read their content.

How to use it

Keep it minimal, allow the crawlers you want, reference your sitemap and never use it to hide pages that should simply be noindexed instead.

← Back to the glossary

Put it into practice

Run Your Content Marketing in One Place

Awareness Marketing turns these ideas into a weekly workflow: research, writing, publishing and measurement.

See How It Works

Browse the Glossary →