Crawl Activity
SEO Compare — seocompare.co.uk
Bing crawl ran 12h ago · 24 Jun, 08:02 · runs daily 06:00
Cloudflare AI Crawl Control (14d)
Edge-level bot counts, true hits — not sampled
6,623
13 bots tracked
06-05 06-12
ClaudeBot 1,445
Bingbot 1,125
Amazonbot 985
ChatGPT-User 845
Googlebot 649
4460
Total Bing Crawls
395
Pages Indexed (Bing)
13
AI Bots Allowed
0
AI Bots Blocked
Crawler Activity
Bing (bars, left axis) vs AI bots (lines, right axis)
AI Crawler Access
GPTBot allowed
ChatGPT-User allowed
Google-Extended allowed
GoogleOther allowed
ClaudeBot allowed
Claude-Web not_mentioned
PerplexityBot allowed
Bytespider allowed
CCBot allowed
cohere-ai allowed
Amazonbot allowed
anthropic-ai not_mentioned
FacebookBot allowed
Applebot-Extended allowed
Bingbot allowed
Server-Side Bot Hits
Real crawler visits detected at the server level (last 30 days)
17889
Total Bot Hits
3977
AI Crawler Hits
4598
Search Crawler Hits
22
Unique Bots
Bot Breakdown
Other Bot 3949
SemrushBot 2429
Bingbot 2247
ChatGPT-User 1386
Googlebot 1164
ClaudeBot 1045
Other Crawler 872
AhrefsBot 693
Other Spider 647
MajesticBot 636
YandexBot 591
Applebot 567
Meta-Agent 528
GPTBot 441
PerplexityBot 430
Daily Hits
24 Jun
20 AI 62 search 109 seo 239
23 Jun
60 AI 171 search 1 social 40 seo 526
22 Jun
70 AI 86 search 4 social 88 seo 533
21 Jun
126 AI 158 search 1 social 217 seo 797
20 Jun
48 AI 107 search 8 social 201 seo 664
19 Jun
149 AI 131 search 3 social 30 seo 477
18 Jun
297 AI 154 search 3 social 166 seo 1092
17 Jun
153 AI 127 search 4 social 20 seo 1009
16 Jun
53 AI 197 search 5 social 32 seo 833
15 Jun
65 AI 162 search 8 social 37 seo 425
14 Jun
124 AI 137 search 1 social 401 seo 690
13 Jun
124 AI 142 search 2 social 58 seo 463
12 Jun
177 AI 96 search 2 social 20 seo 391
11 Jun
193 AI 111 search 2 social 26 seo 425
Live Crawler Access Test (HTTP Response)
Tests if each bot actually gets a 200 response when visiting the homepage
GPTBot
ClaudeBot
PerplexityBot
Googlebot
Bingbot
Topic Clusters (221 detected)
/research/
4
pages in cluster
/ai-search-questions/
1
pages in cluster
/what-is-ai-search-optimisation/
1
pages in cluster
/ai-seo-solicitors/
1
pages in cluster
/ai-seo-private-healthcare/
1
pages in cluster
/ai-seo-law-firms-uk/
1
pages in cluster
/insights/
8
pages in cluster
/pure-ai-search-agencies/
1
pages in cluster
/best-ai-seo-audit-services/
1
pages in cluster
/ai-search-visibility/
1
pages in cluster
/ai-search-agencies-london/
1
pages in cluster
/ai-seo-audit-birmingham/
1
pages in cluster
/best-ai-website-builders-for-ecommerce-uk/
1
pages in cluster
/how-to-get-cited-ai-overviews/
1
pages in cluster
/affordable-ai-seo-uk/
1
pages in cluster
/ai-search-agencies-chelmsford/
1
pages in cluster
/ai-search-agencies-enterprise/
1
pages in cluster
/best-geo-agencies-b2b/
1
pages in cluster
/ai-seo-b2b-guide/
1
pages in cluster
/ai-seo-tools-saas/
1
pages in cluster
/b2b-ai-overviews/
1
pages in cluster
/chatgpt-vs-perplexity-vs-gemini/
1
pages in cluster
/contact/
1
pages in cluster
/ai-seo-law-firms/
1
pages in cluster
/tools/
5
pages in cluster
/best-llmo-agencies/
1
pages in cluster
/best-ai-seo-agencies-healthcare/
1
pages in cluster
/ai-seo-agencies-usa/
1
pages in cluster
/ai-seo-recruitment-agencies/
1
pages in cluster
/ai-seo-audit-newcastle/
1
pages in cluster
/ai-search-agencies-cardiff/
1
pages in cluster
/ai-search-agencies-edinburgh/
1
pages in cluster
/best-seo-agencies-uk-2026/
1
pages in cluster
/ai-search-agencies-canada/
1
pages in cluster
/show-on-ai-search/
1
pages in cluster
/privacy/
1
pages in cluster
/ai-search-agencies-saas/
1
pages in cluster
/getcito-review/
1
pages in cluster
/ai-seo-audit-sheffield/
1
pages in cluster
/otterly-ai-review/
1
pages in cluster
/disclaimer/
1
pages in cluster
/ai-search-agencies-birmingham/
1
pages in cluster
/agency-comparison-matrix/
1
pages in cluster
/how-to-rank-b2b-chatgpt/
1
pages in cluster
/ai-seo-audit-chelmsford/
1
pages in cluster
/chatgpt-seo-healthcare/
1
pages in cluster
/best-perplexity-seo-agencies/
1
pages in cluster
/ai-seo-ecommerce-guide/
1
pages in cluster
/ai-seo-tools-uk/
1
pages in cluster
/geo-for-healthcare/
1
pages in cluster
/best-ai-seo-agencies-saas/
1
pages in cluster
/ai-seo-audit-oxford/
1
pages in cluster
/best-ai-search-website-builders-uk/
1
pages in cluster
/geo-for-law-firms/
1
pages in cluster
/authors/
1
pages in cluster
/agencies-with-case-studies/
1
pages in cluster
/ai-seo-audit-brighton/
1
pages in cluster
/ai-seo-vs-geo-vs-aeo-agencies/
1
pages in cluster
/find-an-agency/
1
pages in cluster
/chatgpt-seo-financial/
1
pages in cluster
/chatgpt-seo-law-firms/
1
pages in cluster
/ai-seo-audit-cambridge/
1
pages in cluster
/saas-ai-overviews/
1
pages in cluster
/ai-seo-audit-cardiff/
1
pages in cluster
/ai-search-agencies-europe/
1
pages in cluster
/ai-seo-tools-ecommerce/
1
pages in cluster
/ai-seo-tools-b2b/
1
pages in cluster
/how-we-are-funded/
1
pages in cluster
/ai-seo-dentists-uk/
1
pages in cluster
/ai-search-agencies-brighton/
1
pages in cluster
/best-ai-visibility-agencies/
1
pages in cluster
/best-ai-seo-agencies-b2b/
1
pages in cluster
/methodology/
1
pages in cluster
/best-geo-agencies/
1
pages in cluster
/ai-seo-ifa-financial-advisors/
1
pages in cluster
/best-free-ai-website-builders-uk/
1
pages in cluster
/ai-search-agencies-southampton/
1
pages in cluster
/site-index/
1
pages in cluster
/chatgpt-seo-ecommerce/
1
pages in cluster
/best-aeo-agencies-ecommerce/
1
pages in cluster
/ai-seo-agencies-saas-uk/
1
pages in cluster
/schema-markup-ai-search/
1
pages in cluster
/terms/
1
pages in cluster
/ai-seo-audit-southampton/
1
pages in cluster
/peec-ai-review/
1
pages in cluster
/best-aeo-agencies-uk/
1
pages in cluster
/ai-seo-saas-vs-b2b/
1
pages in cluster
/ai-seo-healthcare/
1
pages in cluster
/ai-search-hospitality/
1
pages in cluster
/best/
16
pages in cluster
/ai-seo-professional-services/
1
pages in cluster
/ai-seo-nhs/
1
pages in cluster
/agencies-offering-audits/
1
pages in cluster
/ai-seo-audit-leeds/
1
pages in cluster
/about/
1
pages in cluster
/get-found-on-chatgpt/
1
pages in cluster
/best-chatgpt-seo-agencies/
1
pages in cluster
/ai-search-statistics-2026/
1
pages in cluster
/ai-search-agencies-liverpool/
1
pages in cluster
/best-ai-seo-agencies-uk/
1
pages in cluster
/ai-seo-banking/
1
pages in cluster
/ai-citation-gaps/
1
pages in cluster
/ai-search-agencies-leeds/
1
pages in cluster
/blog/
46
pages in cluster
/ai-search-agencies-colchester/
1
pages in cluster
/semrush-vs-ahrefs-ai-visibility/
1
pages in cluster
/how-to-rank-ecommerce-chatgpt/
1
pages in cluster
/chatgpt-seo-agency-uk/
1
pages in cluster
/ai-seo-agency-manchester/
1
pages in cluster
/ai-seo-vs-traditional-seo/
1
pages in cluster
/ai-seo-shopify/
1
pages in cluster
/best-ai-seo-companies/
1
pages in cluster
/ai-overviews-agencies/
1
pages in cluster
/ai-seo-audit-glasgow/
1
pages in cluster
/ai-search-agencies-oxford/
1
pages in cluster
/ai-seo-audit-nottingham/
1
pages in cluster
/best-aeo-agencies/
1
pages in cluster
/uk-ai-overviews-agencies/
1
pages in cluster
/how-to-get-cited-gemini/
1
pages in cluster
/chatgpt-optimisation-agencies/
1
pages in cluster
/ai-visibility-agency-uk/
1
pages in cluster
/ai-seo-agencies-ecommerce-uk/
1
pages in cluster
/chatgpt-visibility/
1
pages in cluster
/best-ai-search-tools/
1
pages in cluster
/agency-directory/
1
pages in cluster
/ai-instructions/
1
pages in cluster
/aeo-vs-geo-vs-llmo/
1
pages in cluster
/se-ranking-ai-tracking-review/
1
pages in cluster
/ai-search-agencies-middle-east/
1
pages in cluster
/ai-seo-barristers/
1
pages in cluster
/ai-seo-audit/
1
pages in cluster
/ai-search-agencies-uk/
1
pages in cluster
/best-llmo-agencies-2026/
1
pages in cluster
/build-brand-ai-trust/
1
pages in cluster
/ai-seo-dental/
1
pages in cluster
/guides/
1
pages in cluster
/ai-seo-audit-london/
1
pages in cluster
/best-geo-agencies-2026/
1
pages in cluster
/editorial-policy/
1
pages in cluster
/ai-search-agencies-reading/
1
pages in cluster
/structure-website-ai-crawlers/
1
pages in cluster
/ai-seo-audit-colchester/
1
pages in cluster
/affordable-ai-seo-ecommerce/
1
pages in cluster
/ai-seo-manufacturing/
1
pages in cluster
/best-geo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-norwich/
1
pages in cluster
/ai-search-agencies-asia/
1
pages in cluster
/ai-visibility-check/
1
pages in cluster
/ai-search-local-business/
1
pages in cluster
/ai-search-agencies-nottingham/
1
pages in cluster
/how-to-get-cited-perplexity/
1
pages in cluster
/best-ai-search-agencies/
1
pages in cluster
/ai-search-agencies-ecommerce/
1
pages in cluster
/ai-seo-financial-uk/
1
pages in cluster
/compare/
67
pages in cluster
/agency-vs-tool/
1
pages in cluster
/how-to-rank-law-firm-chatgpt/
1
pages in cluster
/ai-search-agencies-sheffield/
1
pages in cluster
/ai-seo-travel-tourism-uk/
1
pages in cluster
/affordable-ai-seo-agencies/
1
pages in cluster
/best-geo-agencies-uk/
1
pages in cluster
/ai-seo-linkedin-b2b/
1
pages in cluster
/ai-seo-audit-bristol/
1
pages in cluster
/ai-search-agencies-newcastle/
1
pages in cluster
/chatgpt-seo-b2b/
1
pages in cluster
/ai-seo-london/
1
pages in cluster
/ai-seo-audit-edinburgh/
1
pages in cluster
/perplexity-optimisation-agencies/
1
pages in cluster
/best-ai-overviews-agencies-2026/
1
pages in cluster
/ai-search-agencies-small-business/
1
pages in cluster
/ai-search-agencies-b2b/
1
pages in cluster
/best-geo-agencies-saas/
1
pages in cluster
/ai-seo-audit-norwich/
1
pages in cluster
/uk-ai-seo-agency-market-benchmarks-2026/
1
pages in cluster
/ai-seo-audit-reading/
1
pages in cluster
/restaurant-hospitality-ai-search-guide/
1
pages in cluster
/ai-seo-agency-london/
1
pages in cluster
/ai-search-agencies-australia/
1
pages in cluster
/ai-seo-agencies-b2b-uk/
1
pages in cluster
/best-ai-seo-agencies/
1
pages in cluster
/how-to-get-cited-chatgpt/
1
pages in cluster
/ai-seo-insurance/
1
pages in cluster
/best-gemini-seo-agencies/
1
pages in cluster
/ai-seo-saas-guide/
1
pages in cluster
/ai-search-agencies-bristol/
1
pages in cluster
/ai-seo-healthcare-uk/
1
pages in cluster
/ai-seo-agency-uk/
1
pages in cluster
/alternatives/
13
pages in cluster
/geo-for-fintech/
1
pages in cluster
/geo-vs-seo/
1
pages in cluster
/ai-visibility-score/
1
pages in cluster
/ai-seo-audit-manchester/
1
pages in cluster
/ai-search-agencies-cambridge/
1
pages in cluster
/accessibility/
1
pages in cluster
/data/
4
pages in cluster
/ai-seo-hotels-accommodation/
1
pages in cluster
/best-ai-seo-agencies-ecommerce/
1
pages in cluster
/ai-search-agencies-glasgow/
1
pages in cluster
/ai-search-glossary/
1
pages in cluster
/ai-seo-manchester/
1
pages in cluster
/ai-search-optimisation-cost/
1
pages in cluster
/ai-seo-recruitment/
1
pages in cluster
/ai-seo-manufacturing-companies/
1
pages in cluster
/ai-seo-audit-liverpool/
1
pages in cluster
/free-ai-visibility-audit/
1
pages in cluster
/ai-seo-financial-services/
1
pages in cluster
/how-to-get-cited-claude/
1
pages in cluster
/stats/
23
pages in cluster
/chatgpt-seo-agencies-uk/
1
pages in cluster
/chatgpt-seo-saas/
1
pages in cluster
/ai-search-restaurants-uk/
1
pages in cluster
/ai-search-manufacturing-companies/
1
pages in cluster
/ecommerce-ai-overviews/
1
pages in cluster
/questions/
72
pages in cluster
/accountants-ai-search/
1
pages in cluster
/ai-seo-product-led-growth/
1
pages in cluster
/best-ai-seo-agencies-financial/
1
pages in cluster
/best-ai-seo-agencies-law-firms/
1
pages in cluster
/ai-search-agencies-manchester/
1
pages in cluster
/best-aeo-agencies-2026/
1
pages in cluster
/gemini-optimisation-agencies/
1
pages in cluster
Terminology Consistency (H1 Analysis)
2026
x (%)
search
x (%)
agencies
x (%)
seocompare
x (%)
best
x (%)
chatgpt
x (%)
agency
x (%)
visibility
x (%)
side
x (%)
platform
x (%)
perplexity
x (%)
coverage
x (%)
optimisation
x (%)
compare
x (%)
compared
x (%)