How to Block Search Engines with X-Robots-Tag HTTP Header | moakh
moakh
Your progress

Crawling and indexing · 16

16. X-Robots-Tag HTTP Headers

If you want to block search engines from indexing certain files, you can use an HTTP header called X-Robots-Tag.

Because HTML meta tags only work in HTML pages, there needed to be a way for other files to send a message to bots that blocks them from being indexed.

That method is an HTTP header called the X-Robots-Tag.

How to implement this header requires programming knowledge so we won’t go into the details yet, but when a Web server includes a tag with the name X-Robots-Tag (case-insensitive), search engines are supposed to respect the values.

Here’s an example of an X-Robots-Tag telling search engines not to include the page in the search results:

x-robots-tag: noindex

To learn more about the possible values of the header, see this page.

Takeaways

  • It’s possible to tell search engines not to index files on the Web by including an X-Robots-Tag in the HTTP response headers.