Robots.txt and other website text files: what you actually need

A small text file can change which parts of a website a crawler visits, help a security researcher reach the right person, or point an AI assistant towards useful documentation. These files have different jobs. Adding every filename you have heard about is rarely a useful starting point.

For most business websites, begin by checking crawl access and your sitemap. Add the other files when you have a real use for them and someone responsible for maintaining them.

Which files belong on your website?

File	Main purpose	When to consider it
/robots.txt	Give cooperating crawlers access rules	When you need crawl rules or want to declare your sitemap
/sitemap.txt or /sitemap.xml	List URLs you want search engines to discover	When discovery needs help; choose a supported format
/.well-known/security.txt	Provide a vulnerability-reporting contact	When you can receive and handle security reports
/llms.txt	Offer concise guidance and documentation links for AI tools	When you can maintain useful public documentation
/ads.txt	Declare authorised sellers of your website's advertising inventory	When you monetise through relevant advertising platforms
/app-ads.txt	Declare authorised advertising sellers for an app	When your app's advertising setup requires it
/humans.txt	Credit the people behind a website	When your team wants public credits

The sections below explain the relevant specifications and proposals. Their existence does not make every file a requirement for every website.

robots.txt: decide what cooperating crawlers may visit

Publish this file at the root of the relevant website, such as https://example.com/robots.txt. Use plain UTF-8 text. Rules for that HTTPS host do not automatically apply to its HTTP version, another subdomain or another port. Your CMS may provide its own controls instead of a physical file you upload. Google's robots.txt setup guide

A small illustrative file could look like this:

User-agent: *
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

Here the wildcard addresses crawlers without a more specific applicable group, the rule asks them to avoid the internal-search path, and the last line declares a sitemap. Replace the example domain and paths with your own. A sitemap declaration belongs outside your assumptions about which pages need blocking: it identifies a discovery file, rather than granting access to everything listed there. Google documents the supported fields and crawler-group matching in its robots.txt specification.

An accidental Disallow: / asks the relevant crawlers to avoid the entire site. Check for that after moving a staging website into production. The underlying Robots Exclusion Protocol describes voluntary crawler instructions, not an authentication system.

Crawling, indexing and privacy are separate decisions

Blocking a URL in robots.txt does not reliably remove that URL from Google Search. Google may know about it through links elsewhere. Private documents need access control; listing a private path in a public file does not protect it. Also avoid blocking CSS or JavaScript that a search engine needs to understand important pages. Google's explanation of robots.txt limitations

For a public page you want excluded from Google's index, a supported noindex meta tag or HTTP response header is the appropriate mechanism. Google must be able to fetch the page to see that instruction. Blocking it in robots.txt can prevent this. Putting Noindex: into robots.txt is not supported by Google. Google's noindex guidance

Do not assume every crawler reads every extension. Google, for example, does not support Crawl-delay in robots.txt. Check each crawler's documentation before relying on crawler-specific behaviour. Google's supported-rule guidance

sitemap.txt: a simple alternative to an XML sitemap

Google accepts a plain-text sitemap containing one complete URL per line:

https://example.com/
https://example.com/services/
https://example.com/contact/

Use UTF-8, include only URLs, and leave out headings or commentary. Google supports up to 50,000 URLs or 50 MB uncompressed per sitemap. XML is useful when you need richer sitemap features; a text file can be sufficient for a straightforward URL list. A sitemap helps discovery but does not guarantee crawling or indexing. Google's sitemap formats and limits

Choose URLs you actually want in search: the preferred versions of your pages, rather than an assortment of redirects, duplicates and obsolete addresses. If your CMS already maintains a good XML sitemap, keep using it. A second text sitemap is not an extra requirement. You can declare the actual sitemap URL in robots.txt using the Sitemap: field described above.

For the relationship between these URL choices, see our guide to internal links, canonical URLs and sitemaps.

security.txt: make security reports reach a real person

The recommended web location is https://example.com/.well-known/security.txt, served over HTTPS. The file must include a contact and one expiry timestamp. Additional fields can link to your disclosure policy or state preferred languages. The specification recommends an expiry less than a year ahead. RFC 9116

Contact: mailto:[email protected]
Expires: 2027-04-25T00:00:00Z
Preferred-Languages: en

This is an example, not a working contact. Replace the mailbox and choose a future date when you publish; set a reminder to review it before expiry. Check that someone receives and handles incoming reports. A security.txt file provides a reporting route. It does not inspect your code or certify your website's security. The security.txt project provides a generator and explanations of the fields.

llms.txt: a maintained guide for AI tools

The llms.txt project proposes a Markdown file with concise website context and links to useful documentation. Its purpose is to help AI tools find and use information without working through a whole page layout. It remains a proposal; do not assume that every assistant reads it. The llms.txt proposal

# Example Studio

> Website design and maintenance for small businesses.

## Useful pages

- [Services](https://example.com/services/): What we offer.
- [Support](https://example.com/support/): How to get help.

Use accurate descriptions and public links. Assign an owner to update them when services or URLs change. Treat the file as documentation, not a promise of AI citations, search rankings or training exclusion. It does not replace robots.txt or access controls. If you want to manage a particular AI crawler, review that provider's current crawler instructions separately.

ads.txt and app-ads.txt: relevant to advertising publishers

Ads.txt lets publishers declare the companies authorised to sell their digital advertising inventory. App-ads.txt extends this approach to app inventory, with discovery through the developer website. These files concern advertising supply-chain transparency. A business that only buys ads should not add unrelated seller declarations to its website. IAB Tech Lab's ads.txt and app-ads.txt guidance

If you monetise with an advertising provider, obtain the exact entries and publisher account IDs from that provider. Do not copy another site's values. Google describes ads.txt as a root-level list of authorised publisher IDs in its implementation guidance.

humans.txt: optional credits for your team

Humans.txt is a voluntary way to describe who designed, developed or contributed to a website. The project's guidance uses UTF-8 text and suggests team and site sections. The humans.txt quick-start guide

Publish names or contact details only with the contributors' agreement. Team roles and a general contact page may be more appropriate than personal email addresses. Keep the information current when responsibility changes. This is a credits file, with no need to present it as a search-ranking tactic.

A practical check before and after publishing

Give each file an owner and record why you need it. Then check the public URL after deployment:

1. Use the correct location. Check the actual production host, including whether visitors use www. Security.txt has its own .well-known location; files are not interchangeable.
2. Check the HTTP response. A browser showing a page is not enough. Confirm the expected status and content type, and inspect the response body. A generic HTML error page returned with status 200 is still the wrong content.
3. Check the content. Replace example domains, contact addresses, expiry dates and provider IDs. Remove any private information.
4. Check the intended effect. Test important URLs against your crawler rules. For Google indexing questions, use Search Console rather than inferring index status from a local file alone.
5. Keep a recovery copy. Save the previous version so an accidental crawl restriction can be corrected quickly. Include the files in reviews after a redesign, domain move or CMS migration.

Prioritise a functioning website and clear crawl instructions before maintaining optional files with no defined purpose. Start with our technical SEO checklist if you need a wider review. LimeSeek can help you inspect crawlability and on-page SEO signals after changes; an SEO scan does not replace security review or the validation tools for every file described here.