Uisdom
Start your blog
en

Crawling

Also called: Googlebot, web crawler, crawl

Updated November 9, 2026

Crawling is how Google finds and downloads pages: an automated program called Googlebot visits addresses it has discovered, mostly through links and sitemaps, and downloads their text, images and videos.

Why it matters on a small site

Google can only read what is on a page by crawling it, and only then decide whether to index it. It finds most new pages by following links from pages it already knows, and others through the sitemap you submit, so a page nothing links to is easy to miss.

Most of the crawling is done by Googlebot's smartphone version, because Google mainly indexes the mobile version of most sites.

What stops Google from crawling a page

  • Server problems: the site is down, or fails to answer, when Googlebot calls.
  • A robots.txt rule that blocks the page or the whole site.
  • Pages that only open after a login.
  • No links to the page and no sitemap entry, so Google never learns the address.

How to check yours

Our guide to Discovered - currently not indexed covers pages Google knows about but has not crawled yet. For any single page:

  1. In Google Search Console, type the full address of the page into the inspection bar at the top.
  2. Read the crawl details: Last crawl is when Googlebot last fetched the page, Crawl allowed? says whether robots.txt lets it in, and Page fetch whether your server delivered the page.
  3. Click Test live URL to have Google try the page right now. A block or a server error in the result is the thing to fix.

Often confused with

  • Indexing

    Crawling is Googlebot downloading the page. Indexing is the next step, where Google analyses it and decides whether to store it. A page can be crawled and still not be indexed.

Frequently asked questions

›How often does Googlebot visit my site?

There is no fixed schedule. The URL Inspection tool shows when Google last crawled each page, and Google says Googlebot should not visit most sites more than once every few seconds on average.

›Can I stop Googlebot from crawling a page?

Yes, with a robots.txt rule. That stops the crawl, not the listing: Google says a blocked address can still appear in results, so use noindex to keep a page out.

›Can I pay Google to crawl my site more often?

No. Google says it does not accept payment to crawl a site more frequently or to rank it higher.

Read more

Sources

  1. Google Search Central In-depth guide to how Google Search works
  2. Google Search Central What is Googlebot
  3. Search Console Help URL Inspection tool