Crawler policy

The planned crawler identifies itself as WargamingEventsBot/0.1 (+https://wargamingevents.com/bot). It reads public event pages so small shops do not have to retype their calendars.

It fetches /robots.txt first, follows the group that names WargamingEventsBot, and otherwise follows the * group. A disallow rule blocks the fetch. A failed robots fetch also blocks the fetch. It waits at least one second between requests to a host, honors crawl-delay up to 15 seconds, and sends If-None-Match when it has an ETag.

A store can opt out by disallowing this user-agent in robots.txt. There is no mailbox yet. When José opens one, the contact address will be bot@wargamingevents.com. Until then, robots.txt is the opt-out.

The crawler in this repository is a library with tests. It is not scheduled, and it has not visited any site.