It's reasonably obvious from various replies to this tweet (
https://twitter.com/bentasker/status/1122088753495969793) that there's an appetite for some documentation on how to set up your own ad-blocking DNS over HTTPS server.
So, the aim of this task is to spin up a new publicly accessible one and document the process with a focus on
- Blocking ads/trackers by pulling my adblock lists (
https://www.bentasker.co.uk/adblock/)
- Configuring in such a way that it should be reasonably safe to make publicly available
- (Optionally) configuring to make accessible via an Onion name
As with my initial attempt, I want to have a DoH handler sat in front of
unbound (which will handle the adblock lists, caching and forwarding queries onto an upstream recursor where they cannot be handled locally).
I previously used this DoH server -
https://github.com/m13253/dns-over-https - to handle the DoH part, and see no reason not to again.
The ultimate outcome should be a working server
and a step-by-step guide on how to build it. A working
ansible playbook would be a bonus.
However, there is a secondary aspect to this that I'd like to look into (and should, theoretically, be possible):
I'd like to look into the possibility of writing some LUA to accept DoH requests, translate them into DNS requests (to
Unbound) and then translate the response back into something to be returned to the client. That way, I can potentially deploy a DoH service across the entire edge of my personal (and admittedly small) CDN.
I'll raise a subtask for that aspect of it nearer the time, it's definitely not the primary aim, and is simply being noted so that I don't forget.
Activity
2019-04-27 19:49:23
We do still need to worry about things like our IP's reputation and rate of upstream queries though, so there is still a need for some protection. Plus of course worries about attempts to exploit the software
2019-04-27 19:51:54
2019-04-27 19:56:45
2019-04-27 19:56:45
2019-04-27 23:35:07
View Commit | View Changes
2019-04-27 23:48:18
- has 1GB RAM, 1VPU, 25Gb SSD (basically, Digital Ocean's smallest droplet)
- Running Debian 9.7
- IPv6 enabled
2019-04-27 23:48:44
Install the software
Make a copy of the default config and change for setting live later. We're going to leave the default to test the chain first
2019-04-27 23:48:58
Verify everything is listening
Place a test request to check that Nginx passes on correctly, and that we get a response back from the DNS-over-Https server
2019-04-27 23:49:14
Currently we have DNS-over-HTTP which isn't much of an improvement (and isn't supported by any clients you might point at your DoH server).
Next step is to sort out HTTPS. To begin with, we'll get SSL stapling set up
Now we need to set up certbot, which also means I need to make sure I've set up a DNS record for dns.bentasker.co.uk
Right, onwards
Agree all the terms etc. When prompted, choose redirect
We should now have Nginx listening on port 443
We probably want to tune the SSL options that certbot uses though. At the very least we should review them
Nudge the cipher suites and families up to date
Restart Nginx
Certbot should autorenew every 30 days
At this point, we have a working DoH server (though Nginx will bitch in it's logs about not being able to reach the resolver for Cert stapling). We can place queries against the server:
The next step will to be get Unbound setup to handle Ad blocking (as well as providing resolution for some internal names)
2019-04-28 00:21:37
First things first, we want to make sure no-one outside can start hitting unbound while we configure it
Verify that Unbound (in it's default state) is working
Edit the config
root@debian-9-DoH-instance:~# cd /etc/unbound/ root@debian-9-DoH-instance:/etc/unbound# cp unbound.conf unbound.conf.old root@debian-9-DoH-instance:/etc/unbound# mkdir local.d cat << EOM > unbound.conf server: interface: 127.0.0.1 port: 5353 # Randomise case to make poisioning harder use-caps-for-id: yes # Minimise QNAMEs qname-minimisation: yes do-daemonize: yes verbosity: 1 logfile: "/var/log/unbound/unbound.log" log-queries: yes # Enable UDP, "yes" or "no". do-udp: yes # Enable TCP, "yes" or "no". do-tcp: yes # This is where we'll put our adblock config include: /etc/unbound/local.d/*.conf include: "/etc/unbound/unbound.conf.d/*.conf" EOMIf you want to simply forward queries onto Google and Cloudflare rather than have unbound handle them itself, you can also run
cat << EOM > /etc/unbound/unbound.conf.d/forwarders.conf forward-zone: name: "." forward-addr: 8.8.8.8 forward-addr: 8.8.4.4 forward-addr: 1.1.1.1 EOMCreate the log directory and set up log rotation
root@debian-9-DoH-instance:/etc/unbound# mkdir /var/log/unbound root@debian-9-DoH-instance:/etc/unbound# touch /var/log/unbound/unbound.log root@debian-9-DoH-instance:/etc/unbound# chown unbound -R /var/log/unbound cat << EOM > /etc/logrotate.d/unbound /var/log/unbound/*.log { daily missingok rotate 52 copytruncate compress delaycompress } EOMEither way, we then need to restart unbound
And it should now be listening on 5353
Verify it's working:
OK, we should now be ready to swap out the DoH server config
Time for another test request
Now we need to set up the adblocking
Manually populate the first instance of the blocklist
Test request of a currently adblocked domain
2019-04-28 01:24:19
First thing to do is to set a sensible set of firewall rules
The next thing we want to do is to set up some query rate limiting in Nginx
We then need to adjust our config for the DoH property to reference this limit. We're also going to allow a bit of bursting and ensure queries aren't delayed. Adding this to
At this point, as it's getting late, I figured I'd configure it in Firefox and tail the NGinx logs
The format's a bit naff for assessing performance though, so configuring a different log format too
log_format main '$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t' '$status\t$body_bytes_sent\t"$http_referer"\t' '"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\tCACHE_$upstream_cache_status\t$request_time\t$hostname';Giving us
There are definitely some improvements that can be made (including allowing some caching without screwing over ECS too much), and we may also need to tweak unbound to ensure it's sending ECS information on (and the correct information at that)
Firefox (at least) also allows you to set the value of the
2019-04-28 01:33:54
2019-04-28 11:19:24
There are a few things we might want to do to improve performance. For a start we should enable HTTP/2
upstream dns-backend { server 127.0.0.1:8053; } server { server_name dns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { # Apply rate limiting limit_req zone=doh_limit burst=50 nodelay; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; } location / { return 404; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dns.bentasker.co.uk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dns.bentasker.co.uk/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot }We can also cut out some latency on the queries by having Nginx use keep-alive connections to the backend, and return them to a pool once each query's done (meaning we don't have to wait for a 3 way each time). We need to add the keepalive directive to the upstream and override the upstream
upstream dns-backend { server 127.0.0.1:8053; keepalive 30; } server { server_name dns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { # Apply rate limiting limit_req zone=doh_limit burst=50 nodelay; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection ""; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; } location / { return 404; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dns.bentasker.co.uk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dns.bentasker.co.uk/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot }Testing that, we can see what looks like an improvement in response times, but there are also some failed requests (NGinx logged a 499 because the client went away - presumably because of the long response time)
Unscientifically comparing average response times between this set of lines and the one in the previous comment though, shows this is actually a little slower
ben@milleniumfalcon:~$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6 == 200) t=t+$13; c=c+1; end}END{print t/c}' /tmp/before 0.0739524 ben@milleniumfalcon:~$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6 == 200) t=t+$13; c=c+1; end}END{print t/c}' /tmp/after 0.0902727Not really enough log lines to draw any conclusions there, particularly as I was visiting different sites in Firefox, so it only takes an authoritative to be slower (or likely in the case of the 499's, down).
The next thing I want to do is introduce some basic caching in NGinx. To do this, we're going to break the IP (I'm ignoring IPv6 for the moment) down to being a /24 and inject that into the cache key
First we need to create a cache directory
and add the following into the
This will create a 800M cache zone. Now we need to tell the
See https://projectsstatic.bentasker.co.uk/MISC/MISC27/serverblock.txt for the amended NGinx config (as the regex in it screws with JIRA's formatting). Basically it takes the client's IP, kicks the last octet off the end and injects it into the cache key.
This will, of course, only help in clients that use GET (i.e. in Firefox,
Pulling a line from the access logs and testing with curl:
Generating some new lines with Firefox, response times are still higher than I'd like
Switching over to have Unbound just act as a forwarder (onto Google DNS in this case) gives a massive improvement.
ben@milleniumfalcon:~/Documents/Notes/Request_Router$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6 == 200) t=t+$13; c=c+1; end}END{print t/c}' /tmp/after 0.0324127Given this is being built on a single core instance, it's probably a bit much to ask for that much stuff to all happen at once. Will look at bumping the cores up later to verify.
Want to look at enabling ECS next (and we can't have unbound acting as a forwarder for that, because Google DNS etc ignore incoming ECS and set their own - i.e. act as if we never included ECS in our query).
2019-04-28 12:56:51
Seems this isn't actually an option, at least not without installing out-of-band.
Debian's unbound package is compiled without the ECS module
Otherwise, it would (should?) simply have been a case of adding the following to the
send-client-subnet: yes client-subnet-zone: "." client-subnet-always-forward: yes max-client-subnet-ipv4: 24 max-client-subnet-ipv6: 48Not having ECS enabled means most CDNs are going to send me to the PoP nearest to my resolver rather than one nearest me (they may differ vastly). That's not entirely optimal.
That said, it could be mitigated to some extent, as I could run a number of servers in different locations and then CNAME
That, of course, won't address instances where a CDN might have pops within the users ISP (because their routing system will never find out that they're on that ISP). It also relies on the idea that I've done such a good job of choosing locations for my servers that it's either comparable, or better, than the choices other CDNs have made - otherwise that's a degradation in end user service.
Essentially, without that ECS support, what we've now built is something that's no better for the end user than Cloudflare's offering on 1.1.1.1. We can do better than that.
So, lets build a newer version of the DoH server so that it can insert ECS (which unbound will then hopefully just pass through)
At which point, we can see in packet captures that the DoH server is including ECS in it's queries to unbound
Frame 1: 101 bytes on wire (808 bits), 101 bytes captured (808 bits) Linux cooked capture Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1 User Datagram Protocol, Src Port: 52907 (52907), Dst Port: 5353 (5353) Multicast Domain Name System (query) [Response In: 10] Transaction ID: 0xf198 Flags: 0x0100 Standard query Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 1 Queries Additional records <Root>: type OPT Name: <Root> Type: OPT (41) .001 0000 0000 0000 = UDP payload size: 0x1000 0... .... .... .... = Cache flush: False Higher bits in extended RCODE: 0x00 EDNS0 version: 0 Z: 0x0000 0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs .000 0000 0000 0000 = Reserved: 0x0000 Data length: 11 Option: CSUBNET - Client subnet Option Code: CSUBNET - Client subnet (8) Option Length: 7 Option Data: 0001180056af2f Family: IPv4 (1) Source Netmask: 24 Scope Netmask: 0 Client Subnet: 86.175.47.0Unfortunately, Unbound isn't passing that through. Looks like we are going to have to go out of band with unbound then (the previous changes weren't in vain though as we do need the client's subnet to be included in the query that Unbound receives)
Before we start it, need to drop the ECS config in. Adding the following into the
module-config: "subnetcache validator iterator" client-subnet-zone: "." client-subnet-always-forward: yes max-client-subnet-ipv4: 24 max-client-subnet-ipv6: 48To get this working, I've also had to disable Unbound's chroot (for now). Need to sit down and sort it out properly later though. Adding the following to the
And restarting unbound
As before, we see ECS going from the DoH server into unbound on UDP 5353
Frame 23: 108 bytes on wire (864 bits), 108 bytes captured (864 bits) Linux cooked capture Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1 User Datagram Protocol, Src Port: 53388 (53388), Dst Port: 5353 (5353) Multicast Domain Name System (query) [Response In: 43] Transaction ID: 0xa65c Flags: 0x0100 Standard query Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 1 Queries Additional records <Root>: type OPT Name: <Root> Type: OPT (41) .001 0000 0000 0000 = UDP payload size: 0x1000 0... .... .... .... = Cache flush: False Higher bits in extended RCODE: 0x00 EDNS0 version: 0 Z: 0x0000 Data length: 11 Option: CSUBNET - Client subnet Option Code: CSUBNET - Client subnet (8) Option Length: 7 Option Data: 0001180056af2f Family: IPv4 (1) Source Netmask: 24 Scope Netmask: 0 Client Subnet: 86.175.47.0But now, we also see the query going out (in this case to 8.8.4.4 which will have ignored it). But the important thing is it's there
Frame 25: 108 bytes on wire (864 bits), 108 bytes captured (864 bits) Linux cooked capture Internet Protocol Version 4, Src: 178.128.166.130, Dst: 8.8.4.4 User Datagram Protocol, Src Port: 44225 (44225), Dst Port: 53 (53) Domain Name System (query) [Response In: 29] Transaction ID: 0x6faa Flags: 0x0100 Standard query Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 1 Queries Additional records <Root>: type OPT Name: <Root> Type: OPT (41) UDP payload size: 4096 Higher bits in extended RCODE: 0x00 EDNS0 version: 0 Z: 0x8000 1... .... .... .... = DO bit: Accepts DNSSEC security RRs .000 0000 0000 0000 = Reserved: 0x0000 Data length: 11 Option: CSUBNET - Client subnet Option Code: CSUBNET - Client subnet (8) Option Length: 7 Option Data: 0001180056af2f Family: IPv4 (1) Source Netmask: 24 Scope Netmask: 0 Client Subnet: 86.175.47.0So, if we move the forwarder config out of the way, we should see Unbound sending ECS information to authoritatives
Woot!
Frame 23: 116 bytes on wire (928 bits), 116 bytes captured (928 bits) Linux cooked capture Internet Protocol Version 4, Src: 178.128.166.130, Dst: 94.23.157.104 User Datagram Protocol, Src Port: 21409 (21409), Dst Port: 53 (53) Domain Name System (query) [Response In: 29] Transaction ID: 0x8691 Flags: 0x0010 Standard query Questions: 1 Answer RRs: 0 Authority RRs: 0 Additional RRs: 1 Queries Additional records <Root>: type OPT Name: <Root> Type: OPT (41) UDP payload size: 4096 Higher bits in extended RCODE: 0x00 EDNS0 version: 0 Z: 0x8000 Data length: 11 Option: CSUBNET - Client subnet Option Code: CSUBNET - Client subnet (8) Option Length: 7 Option Data: 0001180056af2f Family: IPv4 (1) Source Netmask: 24 Scope Netmask: 0 Client Subnet: 86.175.47.0So now we have proper working ECS support too.
Will look at scaling the VM up to have more cores later so that can check whether that improves performance a bit.
2019-04-28 15:14:17
And automated pulling those changes
There's no need to push records for my LAN out to the DoH server:
- I use
- Where it's not
(Obviously, the latter may give rise to concerns of DNS leaks, so some care is needed).
If I were to want to push records for my LAN out to the DoH server I could stick them into my
2019-04-29 10:35:06
Actually, this is very much client and configuration specific.
Firefox will fall through to OS DNS only if
However, outside of Firefox, other DoH implementations don't offer a fallback at all. Google's (well, Jigsaw's) Intra - https://play.google.com/store/apps/details?id=app.intra&hl=en_GB - is all or nothing, for example. Which means, if on Android you've got
So, there may actually be a need to make these records available so that a wider range of support is achievable. Within Firefox, for them to be accepted
Worth noting that Intra does not offer configuration to support this, so it wouldn't be possible to use Intra with a server protected in this manner
2019-04-29 11:16:45
It's commonly held that DoH means you cannot do split-horizon DNS, but this is only actually true if you're using someone else's DoH service.
If you use a DoH server (or a proxy to one) where you control the name (and can therefore get a SSL certificate) then split-horizon can still be achieved by using split-horizon at the UDP level for that name.
I.e.
If I were to deploy a new DoH server into the LAN, and furnish it with a certificate for dns.bentasker.co.uk then it'd simply be a case of having the LAN DNS return
When out and about, they'd still resolve to the main one.
A little off topic, but just to finish that thought process.
Neither of those DoH servers need to actually technically be a DoH server, they could simply be a proxy onto a public one
server { server_name cfdns.bentasker.co.uk; root /tmp/NOEXIST; location /dns-query { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host 1.1.1.1; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_read_timeout 86400; proxy_pass https://1.1.1.1/dns-query; } location / { return 404; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/pki/cert/cert.pem; ssl_certificate_key /etc/pki/private/cert.key; }(though you might want to secure it to make sure you don't get banned by CF when people start trying to abuse via you).
So split horizon is still totally do-able, it's just that rather than simply running DNS on the LAN, you also need to run a publicly routable DoH instance or proxy.
2019-04-30 19:30:59
They note some of the limitations in the README, but those all look addressable enough. Whether moving it to LUA would actually constitute an improvement over deploying
2019-04-30 19:33:59
The easiest way is probably to take what's been learnt (don't use the repo's
Although it's not something I want for my setup, once the setup's done, it's probably worth removing
2019-05-01 12:59:34
Not done the pihole swap out yet though.
2019-05-01 14:47:18
To use Pihole instead of unbound, we need to skip the unbound step (see below if you want to use unbound as well) and instead do
When prompted, do not install Pi-hole default firewall rules
Make a note of the admin password
Once the installer's complete, we need to reconfigure lighttpd to bind to a different port
Edit the DNS-over-HTTPS server config to change the upstream
vi /etc/dns-over-https/doh-server.conf upstream = [ "127.0.0.1:53" ] systemctl restart doh-serverNow we'll edit our NGinx config to add a server block to proxy through
cat doh-admin.conf server { server_name dnsadmin.bentasker.co.uk; root /tmp/NOEXIST; #In production, definitely want to add some auth to this location /admin/ { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://127.0.0.1:8080; } listen 443 ssl http2; # managed by Certbot ssl_certificate /etc/letsencrypt/live/dnsadmin.bentasker.co.uk/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/dnsadmin.bentasker.co.uk/privkey.pem; # managed by Certbot } systemctl restart nginxYou should now be able to login to the admin page at https://dnsadmin.bentasker.co.uk/admin and see reports on how your queries have been handled.
Running Unbound Alongside
If you still want your box to handle resolution rather than forwarding, there's a little more customisation that can be done. We need to rebind unbound to 127.0.1.1 (as pihole won't accept a custom port on a custom upstream resolver), which also means we need to move pihole out of the way
First we need to tell pihole only to bind to a specific address
Edit
interface: 127.0.1.1 port: 53 systemctl restart unboundNow, in pihole's admin page, we can set an upstream resolver of 127.0.1.1 and it should all just work (pihole seems to forward ECS data through if it was in the query it received)
To add my adblock list into pihole's blocklists, use URL https://www.bentasker.co.uk/adblock/blockeddomains.txt
2019-05-02 16:33:17
Get the hostnames for the new onion
Edit the NGinx config to inject an Alt-Svc header containing both the V2 and the V3
nano /etc/nginx/conf.d/doh.conf location /dns-query { add_header Alt-Svc 'h2="7t2po4wsbkmu2vwqmib3kzkocydlh3yn675zvi24rhezinzc6yrtkeyd.onion:443", h2="6ohomagiytkiro4l.onion:443"; ma=900'; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $http_host; proxy_set_header X-NginX-Proxy true; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_redirect off; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400; proxy_pass http://dns-backend/dns-query; }We'll also adjust the log format in nginx.conf so that we log the alt_used header and can see if browsers are actually using it (at this point, I genuinely don't know whether Tor Browser Bundle will or not - at least not once network.proxy.socks_remote_dns is disabled)
log_format main '$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t' '$status\t$body_bytes_sent\t"$http_referer"\t' '"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\tCACHE_$upstream_cache_status\t$request_time\t$hostname\t$http_alt_used';Reload Nginx
Configured in Firefox and TBB.
Seems TBB won't use Alt-Used, no. I couldn't get my vanilla Firefox install to do it either.
It's probably possible to convince clients to use plain HTTP, but then they won't be able to use HTTP/2 so lookups will likely end up taking quite a while (just checked, Jigsaw's Intra won't take a http URI).
It might also be that's somethings not right in my test itself of course.
2019-05-02 17:19:51
It's a bit lengthy because I've included some of the options I looked at here rather than doing a one-size-fits-all implementation. I've not had the time to look (in too much depth) into what else would need to be done to make this "safe" (or a given measure of) to advertise as a public and open DoH resolver, so I've put a warning into the article instead.
2019-05-02 17:22:44
In the meantime, for the paranoid (and for my own sanity if I need to look back at this), having published the article, I've
- Moved my DoH recursor off dns.bentasker.co.uk/dns-query
- Moved the Pi-Hole Admin interface away
- Locked down access to both
- Removed Tor
- Disposed of the initial build VM where stuff isn't
I am now using a self-hosted DoH server, it's just not reachable at any of the addresses listed in this issue :)
2019-05-02 17:24:15
2019-05-02 17:24:15
2019-05-02 17:24:19
2019-05-10 23:25:47
2019-05-10 23:25:47
2019-05-10 23:27:36
2019-05-10 23:27:36
2019-05-10 23:27:41