For website owners
About the SourceFinch crawler
Requests identifying as SourceFinch come from SourceFinch, a service operated by The Schoeller Group LLC that monitors public pages and data feeds on behalf of its customers and keeps a dated record of what each page said. Requests sent before our public launch identified as Source-Engine-Cloud.
How it behaves
- It honours
robots.txtfor theSourceFinchuser agent (and the olderSource-Engine-Cloud) and for*, on every page it fetches. - It fetches only public HTTPS pages, a few pages per check, on schedules no faster than every 5 minutes.
- It never signs in, submits forms, or works around access controls, CAPTCHAs or bot challenges.
- It runs at most one check against your site at a time, spaces page requests by your
Crawl-delay(at least one second), backs off on429/503andRetry-After, and treats an unreachablerobots.txtas “do not crawl”. - It never pays for access on its own: an HTTP
402(pay-per-crawl) response stops the request. - Each customer is responsible for their rights to the material they monitor.
Verifying our requests
Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). They carry Signature-Agent: "https://sourcefinch.com", and the public key is published at /.well-known/http-message-signatures-directory (key id knplk8U6caKMlv8UuB_OWLP7T0IHKWuSRG1vnNxjJKE). A request that claims to be SourceFinch without a valid signature is not from us.
Opting out
Add a rule to your robots.txt:
User-agent: SourceFinch Disallow: /
Or email hello@sourcefinch.com and we will block your domain for all customers.