robots.txt

Most small sites need very little in it. Google says that having no robots.txt is fine and means it can crawl every address on your site. The file matters when it goes wrong: a single Disallow: / line under User-agent: * tells crawlers to stay away from the whole site.

It is also not a way to hide a page. Google says a blocked page can still appear in results, without a description, if other sites link to it. To keep a page out, use noindex or a password.

On builders such as Wix or Blogger you may not be able to edit the file at all: look for the search settings in the builder instead.

1. Open yoursite.com/robots.txt in a private browser window. If you get a not found page, there is no file, and Google can crawl everything.
2. Read the Disallow lines. Each one blocks the paths that start with it, for the crawlers named in the User-agent line above it.
3. In Google Search Console, open Settings and then the robots.txt report. It shows the files Google found, when it last fetched them and any errors. After removing a block, choose Request a recrawl from the menu next to the file.

• A Disallow: / copied over from the test version of the site, which blocks every page once the site goes live. It is one of the causes in our guide to why a site is not on Google.
• Blocking the image, style or script files a page needs. Google says not to block them if the page is harder to understand without them.
• Putting the file in a subfolder. It only works at the root, and only for that host: a shop on shop.yoursite.com needs its own.