Overview
ExaSearchBot is the crawler for Exa, a search engine that people and applications query through Exa's API. ExaSearchBot fetches public pages so they can be found, retrieved, and cited through Exa.
Exa uses a second agent, ExaUser, when a user or an app built on the Exa API asks for a page's current content. ExaUser fetches on demand, does not crawl, and is not used to decide whether a page can appear in Exa search. Requests labeled ExaSearchBot are the crawler.
The user-agent string alone does not prove a request came from Exa. Exa signs every ExaSearchBot request with HTTP Message Signatures (RFC 9421) under the Web Bot Auth scheme, so you can check the signature instead of trusting the header.
ExaSearchBot User-Agent String
ExaSearchBot identifies itself with the following user-agent string:
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)ExaSearchBot robots.txt
ExaSearchBot checks robots.txt before it fetches. It uses a group addressed to ExaSearchBot first. If there is none, it follows your rules for major search engines, then the * group. Exa does not say which search engine groups it reads, so if your Googlebot or Bingbot rules differ from *, add an ExaSearchBot group instead of guessing which one applies.
# Give ExaSearchBot its own rules (takes precedence)
User-agent: ExaSearchBot
Disallow: /private/
# Or block ExaSearchBot from the whole site
# User-agent: ExaSearchBot
# Disallow: /How ExaSearchBot Handles OG Images
ExaSearchBot crawls pages for search and does not build link previews. Exa does not document which meta tags it reads or how it uses og:image. MyOG has logged requests with the ExaSearchBot user agent to OG image URLs that MyOG serves, so an og:image URL on your page can receive ExaSearchBot traffic too.
| Preferred Size | Not documented by Exa |
| Cache Duration | Not documented |
Caching Behavior
Exa does not publish a recrawl interval or cache duration for ExaSearchBot.
Fallback Behavior
Exa does not document a fallback order for titles, descriptions, or images. ExaSearchBot only accesses public content. Exa says it does not try to get past logins, paywalls, or CAPTCHAs and does not submit forms, so keep the content you want indexed in the public HTML.
Things to Know
- Without an ExaSearchBot group in robots.txt, ExaSearchBot follows the rules you set for major search engines, and only then the * group.
- A robots.txt disallow stops crawling but does not remove a page that is already in Exa's index. Use a noindex robots meta tag or an X-Robots-Tag: noindex header instead.
- Exa verifies identity with signed requests, not published IP ranges. Exa's crawler page does not list IP ranges.
- Exa says ExaSearchBot is rate-limited per site.
How to Test
Exa has no public URL tester for ExaSearchBot. Check your access logs for the user agent, then verify the request signature before you treat it as Exa.
- Look for Signature, Signature-Input, and Signature-Agent headers on requests that claim to be ExaSearchBot.
- Cloudflare, Akamai, and other providers that support Web Bot Auth validate ExaSearchBot signatures at the edge without extra setup.
- To verify a request yourself, validate the signature per RFC 9421 against Exa's Ed25519 keys at https://crawler.exa.ai/.well-known/http-message-signatures-directory.
- Read your robots.txt as ExaSearchBot would: its own group first, then your rules for major search engines, then *.
- To get a page out of Exa's index, add noindex and wait for ExaSearchBot to fetch the page again.
FAQ
What is ExaSearchBot?
ExaSearchBot is Exa's web crawler. It fetches public pages so they can be found, retrieved, and cited through Exa search and the Exa API. It identifies itself as Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/).
What is the difference between ExaSearchBot and ExaUser?
ExaSearchBot crawls the web automatically and obeys robots.txt. ExaUser fetches a single page live when a user or an Exa API app asks for it, and Exa says robots.txt rules may not apply to those user-initiated fetches. Use the ExaSearchBot token in robots.txt to manage Exa search opt-outs.
How do I verify an ExaSearchBot request?
Check the request signature, not the user agent or IP address. Exa signs each ExaSearchBot request under Web Bot Auth (RFC 9421) and publishes its Ed25519 keys at crawler.exa.ai/.well-known/http-message-signatures-directory. Cloudflare, Akamai, and other providers that support Web Bot Auth check the signature for you.
Does blocking ExaSearchBot in robots.txt remove my pages from Exa?
No. robots.txt controls crawling, not indexing. To remove a page that Exa has already indexed, serve a noindex robots meta tag or an X-Robots-Tag: noindex header. Exa drops the page after ExaSearchBot fetches it again, so the page must stay crawlable until then.
Does ExaSearchBot use Open Graph tags?
Exa does not document Open Graph handling for ExaSearchBot, and ExaSearchBot does not build social share cards. Requests with the ExaSearchBot user agent have reached OG image URLs served by MyOG, so expect it in the logs of any host that serves your og:image files.
Related Bots
Check what crawlers see
Paste your URL to test OG tags, image fetches, crawler access, and the preview card before you share it.
Already have an account?