For website owners

About the SourceFinch crawler

Requests identifying as SourceFinch come from SourceFinch, a service operated by The Schoeller Group LLC that monitors public pages and data feeds on behalf of its customers and keeps a dated record of what each page said. Requests sent before our public launch identified as Source-Engine-Cloud.

How it behaves

• It honours robots.txt for the SourceFinch user agent (and the older Source-Engine-Cloud) and for *, on every page it fetches.
• It fetches only public HTTPS pages, a few pages per check, on schedules no faster than hourly.
• It never signs in, submits forms, or works around access controls, CAPTCHAs or bot challenges.
• Each customer is responsible for their rights to the material they monitor.

Opting out

Add a rule to your robots.txt:

User-agent: SourceFinch
Disallow: /

Or email hello@sourcefinch.com and we will block your domain for all customers.