HallAtlasBot
The crawler behind HallAtlas.
What it does
HallAtlasBot builds a weekly census of the permitting and agenda software each US county, city and township runs. It reads governments' own public pages and public DNS. It also reads two open datasets: the Census Bureau's list of US local governments and CISA's list of .gov domains.
User agent
Every request carries exactly this user agent:
HallAtlasBot/0.1 (+https://hallatlas.com/bot; bot@hallatlas.com)
How to opt out
HallAtlasBot follows robots.txt and answers to the token HallAtlasBot. To stop it crawling your
site, add:
User-agent: HallAtlasBot
Disallow: /
Or email an opt-out request to bot@hallatlas.com. Opt-outs are honoured within 7 days.
Where requests come from
All requests come from one address: 20.98.74.217 (IPv4 only).
How fast
- At most 1 request per 10 seconds per host.
- At most 1 request per second per hosting provider identified from DNS.
- Crawl-delay in robots.txt is honoured.
- At most 900 requests in any 24 hours to sites already identified as CivicPlus-hosted. A site identified later is counted from then on, with the requests already sent to it, so the total can go above 900 after the fact.
How it behaves
- GET requests only.
- It checks robots.txt before every URL.
- No logins, forms, search boxes or challenge solving.
- No requests to vendor-hosted portals or personal-data portals.
What it keeps
Non-personal link records, plus what it needs to follow these rules: DNS answers, robots.txt rules, its terms decisions and a log of its requests (kept 90 days). No page text, apart from short, name-scrubbed link text and short, name-scrubbed excerpts of a site's terms page.