Crawler policy

This is a beta draft, written to be complete enough for legal review, not yet reviewed by counsel. Do not treat it as final or binding until that review is complete and this frontmatter's status changes.

Identity

suo's crawler identifies itself as SuoBot in its User-Agent header on every request.

Verified domains only

SuoBot only crawls a domain after its owner completes domain verification — a DNS TXT record or a meta tag. It will not crawl a domain on anyone's behalf without that proof of control, so it cannot be pointed at a site you do not own.

The proof is checked again about once a week. If it has been gone for 14 days the domain stops being verified and SuoBot stops crawling it, scheduled or not. Asking for verification instructions for a domain does not reserve it, and an account that passes verification for a domain takes it over from any account that verified it before.

Where it goes

Every request SuoBot makes is to the verified host or one of its subdomains, over https on port 443: robots.txt, each sitemap, each page and each redirect between them. It does not follow a sitemap entry, a link or a redirect to any other host, and it refuses a host that resolves to a private or non-public address.

What it obeys

robots.txt, checked before any page on a domain is fetched, including a User-agent: SuoBot block scoped specifically to it.
crawl-delay, if your robots.txt specifies one for SuoBot or for *, up to 30 seconds. A longer value is treated as 30 seconds. The delay holds between requests and between one group of pages and the next.
Rate limits: independent of robots.txt, SuoBot self-limits its request rate per domain to avoid meaningfully adding to your traffic, and runs only one crawl of a domain at a time.

What it never does

Runs JavaScript. It reads server-rendered HTML only — content that only appears after client-side rendering is invisible to it.
Sends credentials, cookies, or authentication headers of any kind.
Submits a form, clicks a button, or otherwise interacts with a page beyond following <a href> links and reading a sitemap.
Crawls a domain that has not been verified.

How much it crawls

At most 10,000 pages in one crawl.
One scheduled crawl a day for each index, plus a small number of crawls started by hand (3 a day).
It reads at most the first 500 kB of robots.txt, 50 MB of any one sitemap, and 2 MB of any one page, and gives up on a response that stalls.
A site that asks for a long crawl-delay is crawled only as far as fits in 20 hours.

Blocking SuoBot

Add a rule to your robots.txt:

User-agent: SuoBot
Disallow: /

Or scope it to specific paths:

User-agent: SuoBot
Disallow: /internal/
Disallow: /drafts/

Changes to robots.txt are picked up before the next crawl; there is no need to contact us to have a disallow rule respected.

Rate limits

SuoBot limits itself to a conservative number of requests per second per domain, backing off further if your server responds slowly or with error statuses, so a crawl should never be mistaken for load pressure on your site. The limits above bound how often and how far it crawls.

Contact

Questions about SuoBot's behavior or a request to have it re-crawl sooner: abuse@usesuo.com. Security reports: security@usesuo.com.

Placeholders for legal review: an explicit statement of what happens to already-indexed content if verification is later revoked (today, crawling stops and the records stay until the index is deleted), and whether a domain-wide Disallow: / in robots.txt before verification is itself sufficient to block a crawl attempt, are intentionally left for confirmation with engineering and counsel together.