jira-projects/ADBLK#1: Replacement of lists

Currently, I provide quite a range of options at https://www.bentasker.co.uk/adblock/:

• Full Autolist (Unbound format)
• Manually blocked zones / Unbound format
• ABP/Ublock compatible list of blocked domains/zones
• add Manual Blocks to ABP
• ABP/Ublock compatible list of blocked domains/zones without Social Media tracker domains
• add Manual Blocks (no SM) to ABP
• Modified version of EasyList (ABP/Ublock compatible)
• add Modified Easylist Blocks to ABP
• Modified version of EasyList (ABP/Ublock compatible) without Social Media tracker domains
• add Modified Easylist Blocks (no SM) to ABP
• ABP compatible list of Social Media tracker domains
• Add Social Media Trackers to ABP
• Pi-Hole compatible blocklist
• Pi-Hole compatible blocklist with Social Media tracking domains

(There are some Greasemonkey scripts too, but I'll ignore those as they're largely static).

Workflow

These scripts are refreshed, and often compiled, by a fairly clunk workchain, initially triggered by update_addomains.sh.

The workflow was quickly hacked together quite some time ago, and hasn't really had the attention it needs to be less crap.

It has had some minor improvements, like the amendment to allow blocks to be broken out into dedicated files (allowing categorisation of domains), but as workflows go it's still pretty shoddy.

Delivery

My adblock lists are delivered via www.bentasker.co.uk using my standard CDN resources. This is unusual, in that most adblock lists tend to be delivered straight from Github.

The problem with delivering via www.bentasker.co.uk is that the cache needs to be invalidated whenever an update is made to the lists. This makes automation hard (as the server either needs creds to interact with the CDN, or you have to have quite short TTLs).

Management

Blocks are managed via a number of config files, and then compiled into the published files.

But, there's some legacy cruft from the old management approach, so lines may not always be consistently added in the correct place.

This could probably be addressed independently if needed, but worth including in the context of wider changes.

Option 1: Discontinue Lists entirely

Had it been just me using the lists, then it's possible that I might have considered this option (and instead locally hosted something for pihole to consume).

However, looking at my access log, there are thousands upon thousands of requests a month for the adblock lists.

Whilst I don't want to continue the lists in their current form, it feels like offering an alternative would be the decent thing

Option 2: Discontinue (most) automation

This is currently my preferred option.

In this option, we'd spin out a new set of adblock lists, but with certain elements of the automation removed.

In particular, we'd no longer retrieve and rewrite easylist lists (so the config files easylist_append_lines.txt, easylist_strip.txt and easylist_strip_absolute.txt would be deprecated and removed).

Configuration would be revised, but wouldn't be that dissimilar to now.

However, the project would no longer be reliant on a cron script somewhere - individual lists would be compiled, on commit, by a git hook.

This change in build process means that external lists (such as the cname-trackers list) shouldn't be included. Whilst they could trivially be pulled in during a hook run, it doesn't make logical sense to present a "complete" list knowing that it'll only be updated when I add something to my own lists - inclusion of third party scripts needs to have a regular refresh cadence, which I don't want to commit to in this project.

Delivery of the lists would be via Github rather than my CDN

The idea here is that I should be able to more quickly publish updates to lists without changes in my own infra necessarily impacting it.

The existing adblock files would be left where they are - where possible redirects would be added to the new version, but only where the new version is directly compatible with what the user-agent thinks they're requesting (i.e. if they're requesting something with no social media domains in it, we can't direct to a generic list).

Option 3: Do Nothing

The final option is to carry on as we are now - do nothing.

But, it's likely to impact the cadence of updates: I've had a few infra changes this year, and the cron to retrieve and publish updates needs revising (in part due to my move to using nikola for my main site).

My intention is to look at moving to Option 2.

Ideally, I'll continue to track changes in this project, and given the option I'd like commits to be made into this project too (potentially with some sort of dual-remote setup in order to publish into Github).

The logical approach would be to fork the existing repo to make the necessary changes so that (where possible) a record of when domains were added (and why) is retained in the commit history.

OK, the easy bit is done - the next stage is to look at writing a pre-commit hook that can build lists for us.

We first need to define what lists we want to continue to provide. To a certain extent, that's going to be defined by which are actually in use.

• The unbound compatible autolist.txt gets a few hundred requests a month, should probably port that over
• adblock_compiled.txt (ABP/Ublock compatible list of blocked domains/zones)
• blockeddomains.txt Pihole compatible list
• manualzones.txt: Used for regex blocks in Pihole

Not being ported:

• zoneblocks.unbound.txt: Unbound format version of manualzones.txt
• adblock_compiled_no_sm.txt
• easylist_modified.txt - Modified easylist support is being deprecated
• easylist_modified_no_sm.txt - Modified easylist support is being deprecated
• social_media_trackers.txt - very limited use

It doesn't look like there's any particular current interest in lists that identify social media trackers separately (or those that exclude them).

So, V2 will generate a much simpler subset of lists, which can be summarised as

• Unbound format list of domains
• ABP/Ublock compatible list of domains
• Pihole compatible list of domains
• List of zones blocked for use in Pihole regexes

We now have our 4 formats - the script is much simplified compared to it's predecessor (largely because we're not having to mess about with modifying easylist).

The list directory in the repository contains more or less a single adblock list published in a number of different formats formats

• adblock_plus.txt: Adblock Plus and UBlock Origin compatible format
• unbound.txt: Unbound config compatible format
• blockeddomains.txt: A simple list of Blocked domains
• regexes.txt: A list of zone wide blocks
• zones.txt: A list of zone wide blocks

The list of blocked zones can be used with a parser to generate regexes to feed into PiHole.

Still TODO

• Update original page/README
• Site news post?
• Add redirects for old lists to new

Need to think carefully before doing any redirects, as we don't want to accidentally remove people's existing protection.

For example, because the original rewrites easylist (and pulls in a list of miner domains and cname trackers), there are quite a few entries in blockeddomains.txt

$ wc -l files/adblock/blockeddomains.txt 
30664 files/adblock/blockeddomains.txt

Whereas the repo version has much fewer

$ wc -l lists/blockeddomains.txt 
755 lists/blockeddomains.txt

So, it'd be unwise to redirect blockdomains.txt away as it'd have the effect of unexpectedly removing ~30K domains from people's blocklists.

The same logic applies to the ABP format (5676 vs 1019).

The unbound format (previously autolist.txt) is less clear - the previous version had 394, the current has 390. Should look into why that is.

Similarly the list of blockedzones is close (195 vs 197), again, we should look into why

The list of regex blocks can be redirected - they're identical.

The unbound format (previously autolist.txt) is less clear - the previous version had 394, the current has 390. Should look into why that is.

The missing lines are

local-zone: "pecult.com" redirect
local-data: "pecult.com A 127.0.0.1"
local-zone: "vibuin.com" redirect
local-data: "vibuin.com A 127.0.0.1"

It looks like the build script is skipping it because it'll be blocked as a zone, but then the zone block never makes it into the file...

That happens because the check runs

egrep -v -e "^${domain#*.}|^$domain" $blocked_zones

Which returns true.

There's no good reason for the inversion (-v) there, so I'm removing that.

It returns true because the statement evaluates to

$ egrep -e "^com|^pecult.com" lists/zones.txt 
commoncannon.com

The initial check was intended to strip a subdomain off (so that a block for foo.bar.com wouldn't collide with a zone block for bar.com), obviously that doesn't work well if there is no subdomain in the name.

It's not trivially addressable either: we could check for a depth of 2, but foo.co.uk would fail that check and still return the same result.

I'll spin out a seperate ticket to track that one (#2)

#2 is resolved. It also accounts for (and has corrected) the difference between blockedzones.txt and zones.txt

The new lists now have the same content as their predecessors, except for the third-party entries pulled in. So, I think we're good in that respect.

It's occurred to me that implementing any redirects is probably unwise. A number of my scripts do something like

curl -s "https://www.bentasker.co.uk/adblock/regex_blocks.txt"

Those would be broken by a redirect (because curl won't follow it by default). I can fix my own scripts, but that'd still leave the risk of breaking other people's deployments.

So, no redirects to implement.

Writeup published at https://www.bentasker.co.uk/posts/blog/general/replacing-my-ad-block-lists-with-a-newer-version.html.

I've disabled all automation around version 1 - I think we're done.