We crawl Shopify stores on behalf of our merchants to find broken links, SEO issues, and accessibility problems. Every request is rate-limited, signed, and robots.txt-compliant. If you're seeing us in your logs, this page explains who we are and how to interact with us.
How we identify ourselves
User-Agent
TinySEOBot
TinySEOBot/1.0 (+https://audit.tinyseo.com)
From (RFC 9110 §10.1.2 contact)
audit@tinyseo.com
Signature-Agent
https://audit.tinyseo.com
We treat every origin we touch as if it belonged to someone we might need to email tomorrow. These commitments are implemented in code, not just policy.
RFC 9309 compliant. Every outbound request is gated against the target host's robots.txt, cached per-host, and Crawl-delay directives are honored up to 60s per request.
Every request carries an Ed25519 signature over the method, authority, target URI, and signature agent (RFC 9421 HTTP Message Signatures). No spoofable User-Agent tricks — verifiers can cryptographically prove the request came from us.
A minimum interval is enforced between requests to the same host, shared across all of our queue workers. The merchant's own storefront is always fetched directly (never via proxies) so we don't look like DDoS traffic.
On 401/403 we treat the host as fully disallowed until cache expiry. We retry on transient 5xx with exponential backoff and honor Retry-After headers up to 10 seconds.
Every request carries an Ed25519 signature over its
method, authority, target URI, and
Signature-Agent.
Receivers fetch our JWK directory, look the key up by
its keyid thumbprint,
and verify the Signature
header against the canonicalized base string.
For operators who'd rather firewall-allow us than
implement RFC 9421, we also publish a Google-shape
JSON list of the IPs we fetch your storefront and
robots.txt from.
Third-party URLs referenced by your site (CDNs,
external links) are fetched through a rotating proxy
pool and won't match that list — those still
carry the signature instead.
Signature-Agent: "https://audit.tinyseo.com" Signature-Input: sig1=("@authority" "@method" "@target-uri" "signature-agent");created=...;expires=...;keyid="...";alg="ed25519";nonce="...";tag="web-bot-auth" Signature: sig1=:<base64 Ed25519 signature>:
JWK directory
https://audit.tinyseo.com/.well-known/http-message-signatures-directory
Served with
Content-Type: application/http-message-signatures-directory,
24h cache.
Direct-egress IPs
https://audit.tinyseo.com/.well-known/tinyseobot-ips.json
Google-shape prefixes[] JSON, 1h cache.
Scope: storefront + robots.txt fetches
only. Third-party URLs use the rotating proxy pool
above.
That's fine. Add a rule for our product token to your
robots.txt
and we'll stop within the cache TTL (up to an hour).
If you need an immediate cutoff, email us and we'll
remove your host from our active audit queues inside
an hour.
Example robots.txt entries
Block everything
User-agent: TinySEOBot Disallow: /
Block a specific path
User-agent: TinySEOBot Disallow: /admin/ Disallow: /drafts/
Slow us down instead
User-agent: TinySEOBot Crawl-delay: 5