project-management-only / Scraper Snitch Bot / Milestone: v0.12

Milestone: v0.12

Status: closed

Created: 19 Jan 2023 08:27:21

Last Updated: 29 Jan 2023 15:52:48

Release v0.12:

Log-Agent Changes

Moves to YAML based configuration
Introduces adjusting scoring based on sensitivity of path requested
Improves robots.txt behaviour handling
Support for Dry-run mode: detailing scores and reasons for loglines, but not writing upstream
Referer blocklist support (allowing sane handling of the justmytoots.com examples)

Snitch Bot Changes

Check whether bot is using a tor exit node
Add support for an allowlist
Improves robots.txt behaviour detection
Added flags Fetches-robots.txt, Does-not-fetch-robots.txt and Ignores-robots.txt

Issues

Key	Title	Status	Created
#4	Restrict disclosures to certain times of day Fixed/Done Task bot	closed	20 Jan 23
#3	Log-Agent Dry Run Mode Fixed/Done New Feature log-agent	closed	19 Jan 23
#2	Revisit Legal Basis Fixed/Done Task	closed	19 Jan 23
#1	robots.txt Fetch Flag Fixed/Done New Feature bot log-agent	closed	19 Jan 23