ADBLK-21: Expose list of Regexes

Issue Information

Issue Type: New Feature
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: Adblock Lists (ADBLK)
Resolution: Done (2020-05-02 11:44:19)

Created: 2020-05-02 11:24:26
Time Spent Working


Description
Xiaomi's fast snooping estate (ADBLK-19) highlights that there's a real benefit to creating a regex based list to be consumed by PiHole


Issue Links

Discovered In ADBLK-19: Xiaomi Data-Grubbing
Toggle State Changes

Activity


2020-05-02 11:28:09
In ADBLK-12 I created a script to turn zone blocks into regexes for Pihole, so it'd probably be prudent to take and adapt that for this - otherwise one may overwrite the other

2020-05-02 11:28:15

Repo: adblocklists
Commit: 37515ac56f82da49140d5e2c20b1315ea66da36f
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:26:37 2020 +0100
Commit Message: ADBLK-21 Expose a Regex list to be consumed by PiHole



Added (+)
-------
config/regexes/xiaomi.txt


Modified (-)(+)
-------
bin/update_addomains.sh




Webhook User-Agent 

GitHub-Hookshot/7431eee


View Commit


2020-05-02 11:34:16

Repo: adblocklists
Commit: d841c2572a60804f1627c3f49be6e7f0f1aa352b
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:33:40 2020 +0100
Commit Message: ADBLK-21 add script to pull down and consume the regex list



Added (+)
-------
bin/pihole_apply_regexes.sh




Webhook User-Agent 

GitHub-Hookshot/7431eee


View Commit


2020-05-02 11:44:15

Repo: adblocklists
Commit: 64be443e3e84d43a1685e9511f5cf18882796239
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:40:34 2020 +0100
Commit Message: ADBLK-21 Update index page to include Pihole information



Modified (-)(+)
-------
index.html




Webhook User-Agent 

GitHub-Hookshot/7431eee


View Commit


2020-05-02 11:44:19
btasker changed status from 'Open' to 'Resolved'

2020-05-02 11:44:19
btasker added 'Done' to resolution

2020-05-02 11:44:22
btasker changed status from 'Resolved' to 'Closed'

2020-05-02 12:03:58
The ADBLK-12 script has been adapted and then committed into the repo (available at https://www.bentasker.co.uk/adblock/bin/pihole_apply_regexes.sh )

It should be run as a cron, so install/setup is 
sudo -i
cd /root
wget https://www.bentasker.co.uk/adblock/bin/pihole_apply_regexes.sh
chmod +x pihole_apply_regexes.sh
echo "0 */2 * * *    root    /root/pihole_apply_regexes.sh" | tee /etc/cron.d/update_ads


This will check for updates to the regex and zone list every 2 hours.

The script itself pulls 2 files using curl

- https://www.bentasker.co.uk/adblock/regex_blocks.txt (the focus of this issue)
- https://www.bentasker.co.uk/adblock/blockedzones.txt (the list created in ADBLK-12)

The latter is processed to turn the list of zones into regexes, and then the two are combined (with duplicates removed). If the result differs from the copy stored on disk (i.e. the config has changed) then pihole is told to recompile it's regex list