ObitsDailyBot

ObitsDailyBot is the program ObitsDaily uses to read funeral homes' public obituary pages. It identifies itself with this user agent:

ObitsDailyBot/1.0 (+https://obitsdaily.com/bot/)

What it visits

Only a funeral home's own website: its robots.txt, its sitemaps and obituary list pages, and the obituary pages listed there. It never visits other obituary sites, and never asks for photos, logins or forms.

How it behaves

• It reads robots.txt before anything else and follows it, Crawl-delay included; a change there takes effect within 24 hours.
• It makes one request at a time to a site, at least 10 seconds apart (longer when your Crawl-delay asks for more).
• It makes at most 2 requests a second across every site it visits.
• It asks only for pages that have changed (If-Modified-Since and ETag) whenever your site allows it.
• It stops when a site turns it away or asks it to slow down, and waits days before it tries again. It never works around a block: it never changes its user agent or its address, and never uses proxies.
• It sends no cookies and never signs in.

What it keeps

Only what a listing needs. The text of each obituary page is kept privately for at most 30 days, usually 7, and used only to make a short listing of facts, which appears automatically; a person checks new listings and takes down any that are wrong. The obituary's own wording is never published, and photos are never downloaded.

Stopping it

To keep it out of your site, add this to your robots.txt:

User-agent: ObitsDailyBot
Disallow: /

Or write to [email protected] from your funeral home: we will stop it for your site and list none of your notices.