Bot policy

ARIA-Crawler/0.1

ARIA's web crawler fetches pages on behalf of verified domain owners so they can measure their AI search presence. If you've seen ARIA-Crawler/0.1 in your server logs, this page explains why and what you can do.

What it is

ARIA is an Answer Engine Optimization platform. When a business verifies they own a domain on ARIA, our crawler reads their site on a schedule so we can score how cite-ready their pages are, surface content gaps, and power an on-site chat widget. The crawler only visits domains whose owners have explicitly verified ownership — it does not crawl the open web.

User-agent
ARIA-Crawler/0.1
Operator
QED Code Ltd · hello@qedcode.io
Egress IPs
Published at /.well-known/crawler-ips.txt
Purpose
Reads pages to score AI-search readiness for the verified domain owner — not for training data or ad targeting.
How it behaves
robots.txt
Fetched and honoured before every crawl. Disallow rules and Crawl-delay directives are both respected.
Crawl rate
A polite delay between page fetches; longer if your robots.txt sets a Crawl-delay.
Depth
Breadth-first, at most two links deep from the homepage and sitemaps. Stays on the verified domain only.
Frequency
At most once every six hours per domain; only re-crawls when the owner requests a refresh or the TTL has elapsed.
Fetch mode
Both a plain HTTP fetch (what citation crawlers see) and a headless Chromium render. No form submissions, no logins, no writes.
Allow ARIA-Crawler

If you use Cloudflare or another WAF that blocks unknown bots by default, add a rule that passes requests whose User-Agent contains ARIA-Crawler/0.1. You can also whitelist our published IP range at /.well-known/crawler-ips.txt.

Cloudflare

Dashboard → Security → Security rules → create a rule: User Agent contains "ARIA-Crawler/0.1" → Allow.

Block ARIA-Crawler

Add the following to your robots.txt and we will not fetch any page on your site:

User-agent: ARIA-Crawler/0.1
Disallow: /

You can also block specific paths by listing them under the Disallow: directive instead of /.

Contact

Questions or concerns about a crawl? Email hello@qedcode.io and we'll respond promptly.