Overview
Googlebot is the generic name for Google's smartphone and desktop web crawlers. Google Search primarily uses the mobile version of a site's content, so most Googlebot requests use the smartphone user agent.
Both crawler variants use the Googlebot product token in robots.txt. You can identify the smartphone or desktop variant from the full HTTP user-agent string, but you cannot give them separate robots.txt rules.
A user-agent string is not proof that a request came from Google. Verify suspicious requests with reverse and forward DNS or match the source address against Google's published crawler IP ranges. Traffic records can show that a token was used, but they cannot prove who sent it.
Googlebot User-Agent Strings
Googlebot identifies itself with the following user-agent strings:
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Googlebot/2.1 (+http://www.google.com/bot.html)Googlebot robots.txt
Googlebot follows robots.txt rules for automatic crawling. A disallow rule stops crawling, but it does not guarantee that the URL cannot appear in search results. Use a crawlable noindex directive when the goal is to prevent indexing.
User-agent: Googlebot
Allow: /
# Example: prevent crawling a private section
# Disallow: /private/How Googlebot Handles OG Images
Googlebot crawls pages for Google Search; it does not generate social share cards. Google Search may use og:title as one source for a result's title link, so keep it accurate and consistent with the page title and main heading.
Meta Tags Read
og:title| Preferred Size | Not applicable to Googlebot web crawling |
| Cache Duration | No fixed interval |
Caching Behavior
Google does not publish a fixed recrawl interval. Recrawling depends on the page, site, and changes Google detects.
Fallback Behavior
Google can build title links from the HTML title, visible page title, headings, other page text, anchor text, and structured data. Open Graph metadata does not replace crawlable page content.
Things to Know
- Googlebot Smartphone and Googlebot Desktop obey the same Googlebot robots.txt token.
- Googlebot user-agent strings can be spoofed; verify the source before allowlisting or blocking by identity.
- Blocking Googlebot affects Google Search, including Discover and other Search features.
- Image discovery and indexing also involve Googlebot-Image, which has its own robots.txt token.
How to Test
Use URL Inspection in Google Search Console to test a URL, inspect Google's indexed view, and request recrawling after an important change.
- Test the exact URL in Search Console and check whether crawling and indexing are allowed.
- Use reverse DNS on the request IP, then resolve the returned Google hostname forward and confirm it maps back to the same IP.
- For automated verification, match the request IP against Google's published common-crawler IP ranges.
- Check the rendered page, canonical URL, robots meta tag, and robots.txt rules together.
You can also use the MyOG OG Preview tool to check how your OG tags are configured before testing with Googlebot.
FAQ
What is the Googlebot user agent?
Googlebot is the robots.txt product token shared by Google's smartphone and desktop web crawlers. Their full HTTP user-agent strings differ, but both include Googlebot/2.1.
Does a bare Googlebot user-agent prove the request came from Google?
No. Google documents Googlebot as the robots.txt token, while its published HTTP user-agent strings include Googlebot/2.1. Any client can send the bare word Googlebot, so verify the request IP before treating it as a Google crawler.
How do I verify a Googlebot request?
Do not trust the user-agent string alone. Run a reverse DNS lookup on the source IP, confirm the hostname matches googlebot.com or geo.googlebot.com, then resolve that hostname and confirm it returns the original IP. You can also match the IP against Google's published common-crawler ranges.
Does Googlebot use Open Graph tags?
Googlebot is not a social preview crawler. Google Search may use og:title as one source when it creates a title link, but it also uses the HTML title, visible headings, page text, links, and structured data. Keep Open Graph tags accurate for sharing and consistent with the visible page.
Can I block only Googlebot Desktop?
No. Googlebot Smartphone and Googlebot Desktop obey the same Googlebot product token in robots.txt, so robots.txt cannot target one variant without the other.
Related Bots
Check what crawlers see
Paste your URL to test OG tags, image fetches, crawler access, and the preview card before you share it.
Already have an account?