##########################################################################################

       ADBLK-21: Expose list of Regexes

##########################################################################################


Issue Type: New Feature 
-----------------------------------------------------------------------------------------

Issue Information
====================

Priority: Major					Status:      Closed
Resolution:  Done (2020-05-02 11:44:19)
Project: Adblock Lists (ADBLK)


Reported By: btasker					
Assigned To: btasker





Time Estimate: 0 minutes
Time Logged:   0 minutes


-----------------------------------------------------------------------------------------

Issue Description
==================

Xiaomi's fast snooping estate (ADBLK-19) highlights that there's a real benefit to
creating a regex based list to be consumed by PiHole

-----------------------------------------------------------------------------------------

Issue Relations
================
	
	- Discovered In ADBLK-19: Xiaomi Data-Grubbing


-----------------------------------------------------------------------------------------

Activity
==========

	
-----------------------------------------------------------------------------------------
2020-05-02 11:28:09              btasker
-----------------------------------------------------------------------------------------

In ADBLK-12 I created a script to turn zone blocks into regexes for Pihole, so it'd
probably be prudent to take and adapt that for this - otherwise one may overwrite the
other
	
-----------------------------------------------------------------------------------------
2020-05-02 11:28:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 37515ac56f82da49140d5e2c20b1315ea66da36f
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:26:37 2020 +0100
Commit Message: ADBLK-21 Expose a Regex list to be consumed by PiHole



Added (+)
-------
config/regexes/xiaomi.txt


Modified (-)(+)
-------
bin/update_addomains.sh



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/37515ac56f82da49140d5e2c20b1315ea66da36f


	
-----------------------------------------------------------------------------------------
2020-05-02 11:34:16              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: d841c2572a60804f1627c3f49be6e7f0f1aa352b
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:33:40 2020 +0100
Commit Message: ADBLK-21 add script to pull down and consume the regex list



Added (+)
-------
bin/pihole_apply_regexes.sh



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/d841c2572a60804f1627c3f49be6e7f0f1aa352b


	
-----------------------------------------------------------------------------------------
2020-05-02 11:44:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 64be443e3e84d43a1685e9511f5cf18882796239
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:40:34 2020 +0100
Commit Message: ADBLK-21 Update index page to include Pihole information



Modified (-)(+)
-------
index.html



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/64be443e3e84d43a1685e9511f5cf18882796239


	
-----------------------------------------------------------------------------------------
2020-05-02 11:44:19              
-----------------------------------------------------------------------------------------

btasker changed status from 'Open' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2020-05-02 11:44:19              
-----------------------------------------------------------------------------------------

btasker added 'Done' to resolution
	
-----------------------------------------------------------------------------------------
2020-05-02 11:44:22              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'
	
-----------------------------------------------------------------------------------------
2020-05-02 12:03:58              btasker
-----------------------------------------------------------------------------------------

The ADBLK-12 script has been adapted and then committed into the repo (available at
https://www.bentasker.co.uk/adblock/bin/pihole_apply_regexes.sh )

It should be run as a cron, so install/setup is

-- BEGIN SNIPPET --

sudo -i
cd /root
wget https://www.bentasker.co.uk/adblock/bin/pihole_apply_regexes.sh
chmod +x pihole_apply_regexes.sh
echo "0 */2 * * *    root    /root/pihole_apply_regexes.sh" | tee /etc/cron.d/update_ads

 -- END SNIPPET --

This will check for updates to the regex and zone list every 2 hours.

The script itself pulls 2 files using curl

- https://www.bentasker.co.uk/adblock/regex_blocks.txt (the focus of this issue)
- https://www.bentasker.co.uk/adblock/blockedzones.txt (the list created in ADBLK-12)

The latter is processed to turn the list of zones into regexes, and then the two are
combined (with duplicates removed). If the result differs from the copy stored on disk
(i.e. the config has changed) then pihole is told to recompile it's regex list