Overview
AhrefsBot is the web crawler for Ahrefs, one of the most widely used SEO tools. It continuously crawls the web to build and maintain Ahrefs' index of pages, backlinks, and keywords — used by SEO professionals worldwide to analyze website performance.
AhrefsBot is one of the most active crawlers on the web. It maintains a massive index of billions of pages and processes millions of pages daily. This high crawl volume means it can consume noticeable server resources, especially on smaller sites.
The crawler reads page content, follows links to discover backlinks, and captures metadata including OG tags. This data powers Ahrefs' Site Explorer, Content Explorer, and other SEO analysis tools.
User-Agent String
AhrefsBot identifies itself with the following user-agent string:
Mozilla/5.0 (compatible; AhrefsBot/7.0; +https://ahrefs.com/robot/)How AhrefsBot Handles OG Images
AhrefsBot reads OG tags as part of its SEO analysis. The data appears in Ahrefs' tools — it is not used for link previews.
Meta Tags Read
og:imageog:titleog:descriptionog:url| Preferred Size | No specific preference — captures whatever is defined |
| Cache Duration | Days to weeks, varies by page authority |
Caching Behavior
Regular crawl schedule. Popular pages are re-crawled more frequently. Ahrefs' freshness depends on your site's authority and update frequency.
Fallback Behavior
AhrefsBot captures page data regardless of OG tags. Missing OG tags will show as empty fields in Ahrefs' tools.
Things to Know
- AhrefsBot is one of the highest-volume third-party crawlers — it can significantly impact server resources on smaller sites.
- You can verify AhrefsBot requests are genuine by reverse-DNS lookup — legitimate requests come from *.ahrefs.com.
- Ahrefs provides a way to control crawl rate at ahrefs.com/robot/ for verified site owners.
- Blocking AhrefsBot does not affect your search engine rankings — it only affects Ahrefs' data about your site.
robots.txt
Fully respects robots.txt. Many sites choose to block AhrefsBot to save server resources, though this means Ahrefs data about your site will become stale.
# To allow AhrefsBot:
User-agent: AhrefsBot
Allow: /
# To limit crawl rate:
User-agent: AhrefsBot
Crawl-delay: 5How to Test
Use Ahrefs' Site Explorer to see how AhrefsBot views your site. Enter any URL to see the crawled OG tags, backlinks, and other metadata.
- Check your server logs to measure AhrefsBot's crawl volume and impact on server resources.
- If AhrefsBot is consuming too many resources, use robots.txt crawl-delay or contact Ahrefs to reduce crawl rate.
- Verify that requests claiming to be AhrefsBot are genuine using reverse-DNS lookup on the IP address.
You can also use the MyOG OG Preview tool to check how your OG tags are configured before testing with AhrefsBot.
FAQ
What is AhrefsBot?
AhrefsBot is the web crawler for Ahrefs, a popular SEO analysis tool. It crawls the web to build an index of pages, backlinks, and keywords used by SEO professionals. It reads page content, follows links, and captures metadata including Open Graph tags.
Is AhrefsBot using too much of my server resources?
AhrefsBot is one of the most active third-party crawlers on the web. If it's consuming too many resources, you can add a Crawl-delay directive in robots.txt (e.g., Crawl-delay: 5 for 5 seconds between requests), block specific paths, or contact Ahrefs at ahrefs.com/robot/ to request a reduced crawl rate.
Does blocking AhrefsBot affect my SEO?
No. AhrefsBot is not a search engine crawler. Blocking it will not affect your rankings on Google, Bing, or any search engine. It only means that Ahrefs' data about your site (backlink profile, organic keywords, etc.) will become stale or incomplete.
Related Bots
Test Your OG Images
Check how your Open Graph images appear to bots and crawlers. Preview your link cards before sharing.
Already have an account?