Overview
AhrefsBot is Ahrefs' web crawler. If your server logs show AhrefsBot or http://ahrefs.com/robot/, this is usually the crawler fetching pages for Ahrefs backlink, keyword, and site audit data.
For most public sites, AhrefsBot is not a security problem and does not affect Google rankings directly. The practical decision is whether you want fresh Ahrefs data, less crawler load, or no Ahrefs crawling at all.
AhrefsBot follows links, requests pages with GET, extracts website content and metadata, and stores what it finds for Ahrefs tools such as Site Explorer and Site Audit. It also captures Open Graph tags, but it is not a social preview crawler.
If crawl volume is the problem, slowing AhrefsBot with a robots.txt Crawl-delay is usually better than blocking it completely. Blocking it makes Ahrefs' view of your site stale or incomplete.
AhrefsBot User-Agent String
AhrefsBot identifies itself with the following user-agent string:
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)AhrefsBot robots.txt
Ahrefs says AhrefsBot respects robots.txt allow, disallow, and Crawl-delay rules. Many sites slow AhrefsBot before blocking it completely, because a full block makes Ahrefs data about the site stale.
# To allow AhrefsBot:
User-agent: AhrefsBot
Allow: /
# To block AhrefsBot:
User-agent: AhrefsBot
Disallow: /
# To limit crawl rate:
User-agent: AhrefsBot
Crawl-delay: 5How AhrefsBot Handles OG Images
AhrefsBot reads OG tags as part of its SEO analysis. The data appears in Ahrefs' tools — it is not used for link previews.
Meta Tags Read
og:imageog:titleog:descriptionog:url| Preferred Size | No specific preference — captures whatever is defined |
| Cache Duration | Days to weeks, varies by page authority |
Caching Behavior
Regular crawl schedule. Popular pages are re-crawled more frequently. Ahrefs' freshness depends on your site's authority and update frequency.
Fallback Behavior
AhrefsBot captures page data regardless of OG tags. Missing OG tags will show as empty fields in Ahrefs' tools.
Things to Know
- AhrefsBot is one of the highest-volume third-party crawlers — it can significantly impact server resources on smaller sites.
- You can verify AhrefsBot requests are genuine by reverse-DNS lookup — legitimate requests come from *.ahrefs.com.
- Ahrefs supports robots.txt Crawl-delay for AhrefsBot.
- Blocking AhrefsBot does not affect your search engine rankings — it only affects Ahrefs' data about your site.
How to Test
Use Ahrefs Site Explorer or Site Audit to see how AhrefsBot views your site. Enter a URL to inspect crawled metadata, backlinks, internal links, and technical SEO issues.
- Search server logs for AhrefsBot and http://ahrefs.com/robot/ to confirm which URLs it is requesting.
- If crawl load is high, add Crawl-delay for AhrefsBot before reaching for a full Disallow rule.
- Use reverse-DNS lookup before trusting or blocking requests that claim to be AhrefsBot.
You can also use the MyOG OG Preview tool to check how your OG tags are configured before testing with AhrefsBot.
FAQ
What is AhrefsBot?
AhrefsBot is the web crawler for Ahrefs, a popular SEO analysis tool. It crawls the web to build an index of pages, backlinks, and keywords used by SEO professionals. It reads page content, follows links, and captures metadata including Open Graph tags.
What is http://ahrefs.com/robot/ in my logs?
http://ahrefs.com/robot/ is the URL shown in the AhrefsBot user agent string. A typical request identifies itself as Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/). Use reverse-DNS lookup to confirm the request is genuinely from Ahrefs before allowing, slowing, or blocking it.
How do I block AhrefsBot in robots.txt?
Add User-agent: AhrefsBot and Disallow: / to robots.txt. Blocking AhrefsBot stops Ahrefs from crawling your site, but it does not affect Google, Bing, or other search engine rankings.
How do I slow AhrefsBot with Crawl-delay?
Add User-agent: AhrefsBot and Crawl-delay with the number of seconds between requests. For example, Crawl-delay: 5 asks AhrefsBot to wait 5 seconds between requests. Slowing is often a better first step than blocking if the main issue is server load.
How do I test the AhrefsBot user agent?
Search your server logs for AhrefsBot or http://ahrefs.com/robot/ requests, then verify suspicious requests with reverse DNS before trusting or blocking them. To see what Ahrefs has already crawled, test the URL in Ahrefs Site Explorer or Site Audit.
Is AhrefsBot a website extractor?
Yes. AhrefsBot works like a website extractor for Ahrefs' SEO tools: it fetches pages, follows links, reads page content, and extracts metadata such as titles, descriptions, canonical URLs, and Open Graph tags. It uses that data for Ahrefs reports, not for social media link previews.
What are crawled pages on Ahrefs?
Crawled pages on Ahrefs are URLs AhrefsBot has visited and stored in Ahrefs' index or reports. Ahrefs uses those crawls to show technical SEO issues, internal links, backlinks, metadata, and organic keyword data. The count is Ahrefs' view of your site, not Google's index count.
Is AhrefsBot using too much of my server resources?
AhrefsBot is one of the most active third-party crawlers on the web. If it's consuming too many resources, you can add a Crawl-delay directive in robots.txt (e.g., Crawl-delay: 5 for 5 seconds between requests), block specific paths, or contact Ahrefs at ahrefs.com/robot/ to request a reduced crawl rate.
Does blocking AhrefsBot affect my SEO?
No. AhrefsBot is not a search engine crawler. Blocking it will not affect your rankings on Google, Bing, or any search engine. It only means that Ahrefs' data about your site (backlink profile, organic keywords, etc.) will become stale or incomplete.
Related Bots
Check what crawlers see
Paste your URL to test OG tags, image fetches, crawler access, and the preview card before you share it.
Already have an account?