Setup · crawling

Crawling your site

Once you've verified a domain, I fetch your website in the background and keep a copy of each page's raw HTML — one row per URL, the whole document. That stored copy is the foundation everything else reads from: your readiness score, your chat answers, the questions worth tracking, and whether engines can reach you at all.

What a crawl feeds

I don't crawl for its own sake. A single pass over your site produces the raw material for four separate parts of the product, so one read keeps them all current.

input

Crawl

raw HTML, one row per URL

→

feeds

Four products

Readiness · Access · Chat context · Coverage

Stored pages feed Readiness, chat answers, and question discovery; the robots result and JavaScript-only flags feed Access.

How I read your site

I start from your homepage and walk same-site links breadth-first, adding your sitemap. I stay on your own domain and its www host, go no more than two links deep, and identify myself with a clear ARIA crawler user agent so you can spot me in your logs. I'm polite — a short delay between fetches — and I cap each page at about 1.5 MB so one enormous document can't stall a pass.

Depth

A breadth-first walk, at most two links deep, never leaving your domain or its www host.

Stored cap

I keep up to 200 pages per domain — a safety cap, not a plan limit.

Discovered

I look a little further than I store — up to 500 pages — to report how big your site is.

Freshness

A verified domain is re-read when it has never been crawled or its last crawl is older than 6 hours. Unchanged HTML isn't rewritten — a page's capture time is tracked apart from the crawl's freshness.

Stored content

The full rendered HTML of each page (post-JavaScript), one row per URL — the document, not a summary.

Blind spots

If a site has more pages than the 200-page safety cap, the unread pages are blind spots in every score — I can't measure what I never stored.

Two ways I fetch a page

For every page I can fetch it two ways. The two views rarely disagree — but when they do, that disagreement is itself a signal.

Raw fetch

The JavaScript-free view a citation crawler sees. It's also how I read your robots.txt and your sitemaps.

Rendered fetch

The full page DOM after a headless Chromium render runs your JavaScript.

Note

Comparing the two is how I spot a JavaScript-only page: one that's rich after a render but nearly empty in the raw view. To a citation crawler that page reads as blank, so I flag it for Access.

When I re-read, and what state means

A background worker loops continuously, sleeping about 10 seconds between passes, and picks up any domain that's due — never crawled, or last crawled more than 6 hours ago. So a freshly verified or refreshed domain is picked up within seconds.

Crawl state isn't stored anywhere — I derive it from what the worker is doing right now:

before

Queued

due, not yet read

→

during

Reading

a URL in flight

→

after

Idle

a pass has finished

The three states ARIA shows for a crawl, derived live.

Refresh now simply marks the domain due — the actual rendering is the worker's job, never your web request, because rendering is slow. If your subscription lapses, I stop crawling that domain until it's restored.

Where this fits

Crawling sits between proving a domain is yours and everything I measure on it. Verification is the prerequisite; what a crawl stores feeds Readiness, Access, and chat; and the 200-page safety cap limits how much of your site I store in one pass.