For website owners

About the SourceFinch crawler

Requests identifying as SourceFinch come from SourceFinch, a service operated by The Schoeller Group LLC that monitors public pages and data feeds on behalf of its customers and keeps a dated record of what each page said. Requests sent before our public launch identified as Source-Engine-Cloud.

How it behaves

Verifying our requests

Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). They carry Signature-Agent: "https://sourcefinch.com", and the public key is published at /.well-known/http-message-signatures-directory (key id knplk8U6caKMlv8UuB_OWLP7T0IHKWuSRG1vnNxjJKE). A request that claims to be SourceFinch without a valid signature is not from us.

Opting out

Add a rule to your robots.txt:

User-agent: SourceFinch
Disallow: /

Or email hello@sourcefinch.com and we will block your domain for all customers.