##########################################################################################

       MISC-27: Build and document a publicly accessible DoH Server

##########################################################################################


Issue Type: Task 
-----------------------------------------------------------------------------------------

Issue Information
====================

Priority: Major					Status:      Closed
Resolution:  Done (2019-05-10 23:27:36)
Project: Miscellaneous (MISC)


Reported By: btasker					
Assigned To: btasker





Time Estimate: 0 minutes
Time Logged:   0 minutes


-----------------------------------------------------------------------------------------

Issue Description
==================

It's reasonably obvious from various replies to this tweet
(https://twitter.com/bentasker/status/1122088753495969793) that there's an appetite for
some documentation on how to set up your own ad-blocking DNS over HTTPS server.

So, the aim of this task is to spin up a new publicly accessible one and document the
process with a focus on

- Blocking ads/trackers by pulling my adblock lists
(https://www.bentasker.co.uk/adblock/)
- Configuring in such a way that it should be reasonably safe to make publicly available
- (Optionally) configuring to make accessible via an Onion name

As with my initial attempt, I want to have a DoH handler sat in front of unbound (which
will handle the adblock lists, caching and forwarding queries onto an upstream recursor
where they cannot be handled locally).

I previously used this DoH server - https://github.com/m13253/dns-over-https - to handle
the DoH part, and see no reason not to again.

The ultimate outcome should be a working server *and* a step-by-step guide on how to build
it. A working ansible playbook would be a bonus.

However, there is a secondary aspect to this that I'd like to look into (and should,
theoretically, be possible): 

I'd like to look into the possibility of writing some LUA to accept DoH requests,
translate them into DNS requests (to Unbound) and then translate the response back into
something to be returned to the client. That way, I can potentially deploy a DoH service
across the entire edge of my personal (and admittedly small) CDN.

I'll raise a subtask for that aspect of it nearer the time, it's definitely not the
primary aim, and is simply being noted so that I don't forget.

-----------------------------------------------------------------------------------------

Issue Relations
================
	
	- DNS over HTTPS server (https://github.com/m13253/dns-over-https)



-----------------------------------------------------------------------------------------

Activity
==========

	
-----------------------------------------------------------------------------------------
2019-04-27 19:49:23              btasker
-----------------------------------------------------------------------------------------

One thing to note (it should be obvious, but just so I remember to explicitly point it out
all the same) - although there are various risks involved with making any DNS service
publicly available, because DoH requires a TCP connection (which implies a 3-way must be
successfully completed before any queries are even submitted, let alone responded to),
there isn't the same worry about DNS reflection attacks that there are when making a UDP
DNS server available for public use.

We do still need to worry about things like our IP's reputation and rate of upstream
queries though, so there is still a need for some protection. Plus of course worries about
attempts to exploit the software
	
-----------------------------------------------------------------------------------------
2019-04-27 19:51:54              btasker
-----------------------------------------------------------------------------------------

Actually, I don't think there's any particular need for this issue to sit in the STAGING
project, it's not likely to contain any sensitive info at any point, so I'm going to move
it over to MISC (as that'll be it's ultimate destination anyway).
	
-----------------------------------------------------------------------------------------
2019-04-27 19:56:45              
-----------------------------------------------------------------------------------------

btasker changed Project from 'STAGING' to 'Miscellaneous'
	
-----------------------------------------------------------------------------------------
2019-04-27 19:56:45              
-----------------------------------------------------------------------------------------

btasker changed Key from 'STGNG-8' to 'MISC-27'
	
-----------------------------------------------------------------------------------------
2019-04-27 23:35:07              git
-----------------------------------------------------------------------------------------


-- BEGIN SNIPPET --

-------------------------
From: git@<Domain Hidden>
To: jira@<Domain Hidden>
Date: None
Subject: Add DNS record for dns.bentasker.co.uk for MISC-27
-------------------------

 -- END SNIPPET --


-- BEGIN QUOTE --

Repo: chaos_dns
Host:hiyori

commit a02adf80b22a64188feb2cafd85b47dd5394d852
Author: root <root@<Domain Hidden>>
Date:   Sat Apr 27 23:54:15 2019 +0100

Commit Message: Add DNS record for dns.bentasker.co.uk for MISC-27

 bentasker.co.uk.zone | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- END QUOTE --

http://chessrepos.bentasker.co.uk/?p=chaos_dns.git&a=commit&h=a02adf80b22a64188feb2cafd85b47dd5394d852
|
http://chessrepos.bentasker.co.uk/projects/chaos_dns.git/commitdiff/a02adf80b22a64188feb2cafd85b47dd5394d852


	
-----------------------------------------------------------------------------------------
2019-04-27 23:48:18              btasker
-----------------------------------------------------------------------------------------

*VM Details*

- has 1GB RAM, 1VPU, 25Gb SSD (basically, Digital Ocean's smallest droplet)
- Running Debian 9.7
- IPv6 enabled
	
-----------------------------------------------------------------------------------------
2019-04-27 23:48:44              btasker
-----------------------------------------------------------------------------------------

*DoH Server Software Setup*


Install the software


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# apt-get update
root@debian-9-DoH-instance:~# apt-get install curl software-properties-common
root@debian-9-DoH-instance:~# curl
"https://www.aaflalo.me/doh-server/doh-server_2.0.1_amd64.deb" -o
doh-server_2.0.1_amd64.deb
root@debian-9-DoH-instance:~# dpkg -i doh-server_2.0.1_amd64.deb

 -- END SNIPPET --

Make a copy of the default config and change for setting live later. We're going to leave
the default to test the chain first


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# cp /etc/dns-over-https/doh-server.conf
/etc/dns-over-https/doh-server.conf.new
root@debian-9-DoH-instance:~# vi /etc/dns-over-https/doh-server.conf.new
root@debian-9-DoH-instance:~# diff -u /etc/dns-over-https/doh-server.conf
/etc/dns-over-https/doh-server.conf.new
--- /etc/dns-over-https/doh-server.conf.orig	2019-04-27 22:06:11.614790000 +0000
+++ /etc/dns-over-https/doh-server.conf	2019-04-27 22:07:30.550790000 +0000
@@ -21,10 +21,8 @@
 # Upstream DNS resolver
 # If multiple servers are specified, a random one will be chosen each time.
 upstream = [
-    "1.1.1.1:53",
-    "1.0.0.1:53",
-    "8.8.8.8:53",
-    "8.8.4.4:53",
+    "127.0.0.1:5353"
 ]
 
 # Upstream timeout

 -- END SNIPPET --
	
-----------------------------------------------------------------------------------------
2019-04-27 23:48:58              btasker
-----------------------------------------------------------------------------------------

*Nginx Setup*


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# apt install gnupg2 ca-certificates lsb-release
root@debian-9-DoH-instance:~# echo "deb http://nginx.org/packages/debian `lsb_release -cs`
nginx" >> /etc/apt/sources.list.d/nginx.list
root@debian-9-DoH-instance:~# curl -fsSL https://nginx.org/keys/nginx_signing.key |
apt-key add -
root@debian-9-DoH-instance:~# apt-key fingerprint ABF5BD827BD9BF62
root@debian-9-DoH-instance:~# apt-get update; apt-get -y install nginx


cat << EOM > /etc/nginx/conf.d/doh.conf
upstream dns-backend {
    server 127.0.0.1:8053;
}
server {
        listen 80;
        server_name dns.bentasker.co.uk;
        root /tmp/NOEXIST;
        location /dns-query {
                proxy_set_header X-Real-IP \$remote_addr;
                proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
                proxy_set_header Host \$http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_http_version 1.1;
                proxy_set_header Upgrade \$http_upgrade;
                proxy_redirect off;
                proxy_set_header        X-Forwarded-Proto \$scheme;
                proxy_read_timeout 86400;
                proxy_pass http://dns-backend/dns-query;
        }


        location / {
            return 404;
        }
}
EOM

root@debian-9-DoH-instance:~# service nginx restart

 -- END SNIPPET --

Verify everything is listening


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# netstat -lnp | grep 80
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
3230/nginx: master  
tcp        0      0 127.0.0.1:8053          0.0.0.0:*               LISTEN     
1951/doh-server     
tcp6       0      0 ::1:8053                :::*                    LISTEN     
1951/doh-server     
udp6       0      0 fe80::7ca7:d3ff:fe7:123 :::*                                845/ntpd  
         /run/user/0/gnupg/S.gpg-agent

 -- END SNIPPET --

Place a test request to check that Nginx passes on correctly, and that we get a response
back from the DNS-over-Https server


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# curl -s -H "Host: dns.bentasker.co.uk"
"http://127.0.0.1/dns-query?name=www.bentasker.co.uk&type=A" | python -m json.tool
{
    "AD": false,
    "Answer": [
        {
            "Expires": "Sat, 27 Apr 2019 22:42:07 UTC",
            "TTL": 925,
            "data": "wwwsite.balanced.bentasker.co.uk.",
            "name": "www.bentasker.co.uk.",
            "type": 5
        },
        {
            "Expires": "Sat, 27 Apr 2019 22:27:11 UTC",
            "TTL": 29,
            "data": "206.189.120.26",
            "name": "wwwsite.balanced.bentasker.co.uk.",
            "type": 1
        }
    ],
    "CD": false,
    "Question": [
        {
            "name": "www.bentasker.co.uk.",
            "type": 1
        }
    ],
    "RA": true,
    "RD": true,
    "Status": 0,
    "TC": false
}

 -- END SNIPPET --
	
-----------------------------------------------------------------------------------------
2019-04-27 23:49:14              btasker
-----------------------------------------------------------------------------------------

*SSL Setup*


Currently we have DNS-over-HTTP which isn't much of an improvement (and isn't supported by
any clients you might point at your DoH server).

Next step is to sort out HTTPS. To begin with, we'll get SSL stapling set up

-- BEGIN SNIPPET --

cat << EOM > /etc/nginx/conf.d/00-cert-stapling.conf
ssl_stapling on;
ssl_stapling_verify on;
resolver 127.0.0.1:5353;
EOM

 -- END SNIPPET --

Now we need to set up certbot, which also means I need to make sure I've set up a DNS
record for dns.bentasker.co.uk


-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ host dns.bentasker.co.uk
dns.bentasker.co.uk has address 178.128.166.130
dns.bentasker.co.uk has IPv6 address 2a03:b0c0:1:e0::3f7:6001

 -- END SNIPPET --

Right, onwards


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# apt-get install certbot python-certbot-nginx
root@debian-9-DoH-instance:~# certbot --nginx -d dns.bentasker.co.uk

 -- END SNIPPET --

Agree all the terms etc. When prompted, choose redirect 

We *should* now have Nginx listening on port 443


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# netstat -lnp | grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN     
3230/nginx: master  
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN     
3230/nginx: master  

 -- END SNIPPET --

We probably want to tune the SSL options that certbot uses though. At the very least we
should review them


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# cp /etc/letsencrypt/options-ssl-nginx.conf
/etc/letsencrypt/options-ssl-nginx.conf.orig

 -- END SNIPPET --

Nudge the cipher suites and families up to date

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# diff -u /etc/letsencrypt/options-ssl-nginx.conf.orig
/etc/letsencrypt/options-ssl-nginx.conf
--- /etc/letsencrypt/options-ssl-nginx.conf.orig	2019-04-27 22:38:36.572009098 +0000
+++ /etc/letsencrypt/options-ssl-nginx.conf	2019-04-27 22:39:41.346457231 +0000
@@ -7,7 +7,12 @@
 ssl_session_cache shared:le_nginx_SSL:1m;
 ssl_session_timeout 1440m;
 
-ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
+ssl_protocols TLSv1.1 TLSv1.2 TLSv1.3;
 ssl_prefer_server_ciphers on;
 
-ssl_ciphers
"ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";
+ssl_ciphers
'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
+
+
+
+
+add_header Strict-Transport-Security "max-age=31536000;" always;

 -- END SNIPPET --

Restart Nginx

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# service nginx restart

 -- END SNIPPET --

Certbot should autorenew every 30 days

At this point, we have a working DoH server (though Nginx will bitch in it's logs about
not being able to reach the resolver for Cert stapling). We can place queries against the
server:

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ curl -s
"https://dns.bentasker.co.uk/dns-query?name=www.bentasker.co.uk&type=A" | python -m
json.tool
{
    "AD": false,
    "Answer": [
        {
            "Expires": "Sat, 27 Apr 2019 23:37:59 UTC",
            "TTL": 3329,
            "data": "wwwsite.balanced.bentasker.co.uk.",
            "name": "www.bentasker.co.uk.",
            "type": 5
        },
        {
            "Expires": "Sat, 27 Apr 2019 22:42:59 UTC",
            "TTL": 29,
            "data": "206.189.120.26",
            "name": "wwwsite.balanced.bentasker.co.uk.",
            "type": 1
        }
    ],
    "CD": false,
    "Question": [
        {
            "name": "www.bentasker.co.uk.",
            "type": 1
        }
    ],
    "RA": true,
    "RD": true,
    "Status": 0,
    "TC": false,
    "edns_client_subnet": "86.175.47.0/0"
}

 -- END SNIPPET --

The next step will to be get Unbound setup to handle Ad blocking (as well as providing
resolution for some internal names)
	
-----------------------------------------------------------------------------------------
2019-04-28 00:21:37              btasker
-----------------------------------------------------------------------------------------

*Unbound setup*

First things first, we want to make sure no-one outside can start hitting unbound while we
configure it

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# iptables -I INPUT -p tcp --dport 53 -j REJECT
root@debian-9-DoH-instance:~# iptables -I INPUT -p udp --dport 53 -j REJECT
root@debian-9-DoH-instance:~# iptables -I INPUT -i lo -j ACCEPT

root@debian-9-DoH-instance:~# apt-get install unbound dnsutils

 -- END SNIPPET --

Verify that Unbound (in it's default state) is working

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound/unbound.conf.d# dig @127.0.0.1 www.google.com

; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64286
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com.			IN	A

;; ANSWER SECTION:
www.google.com.		300	IN	A	216.58.206.36

;; Query time: 555 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Apr 27 22:54:45 UTC 2019
;; MSG SIZE  rcvd: 59

 -- END SNIPPET --

Edit the config

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# cd /etc/unbound/
root@debian-9-DoH-instance:/etc/unbound# cp unbound.conf unbound.conf.old
root@debian-9-DoH-instance:/etc/unbound# mkdir local.d


cat << EOM > unbound.conf
server:

    interface: 127.0.0.1
    port: 5353

    # Randomise case to make poisioning harder
    use-caps-for-id: yes

    # Minimise QNAMEs
    qname-minimisation: yes


    do-daemonize: yes
    verbosity: 1
    logfile: "/var/log/unbound/unbound.log"
    log-queries: yes

    # Enable UDP, "yes" or "no".
    do-udp: yes

    # Enable TCP, "yes" or "no".
    do-tcp: yes

    # This is where we'll put our adblock config 
    include: /etc/unbound/local.d/*.conf

include: "/etc/unbound/unbound.conf.d/*.conf"

EOM

 -- END SNIPPET --

If you want to simply forward queries onto Google and Cloudflare rather than have unbound
handle them itself, you can also run

-- BEGIN SNIPPET --

cat << EOM > /etc/unbound/unbound.conf.d/forwarders.conf
forward-zone:
        name: "."
        forward-addr: 8.8.8.8
        forward-addr: 8.8.4.4
        forward-addr: 1.1.1.1
EOM

 -- END SNIPPET --

Create the log directory and set up log rotation

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# mkdir /var/log/unbound
root@debian-9-DoH-instance:/etc/unbound# touch /var/log/unbound/unbound.log
root@debian-9-DoH-instance:/etc/unbound# chown unbound -R /var/log/unbound
cat << EOM > /etc/logrotate.d/unbound
/var/log/unbound/*.log {
        daily
        missingok
        rotate 52
        copytruncate
        compress
        delaycompress
}
EOM


 -- END SNIPPET --

Either way, we then need to restart unbound

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# systemctl restart unbound

 -- END SNIPPET --

And it should now be listening on 5353

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# netstat -lnp | grep unbound
tcp        0      0 127.0.0.1:5353          0.0.0.0:*               LISTEN     
5878/unbound        
tcp        0      0 127.0.0.1:8953          0.0.0.0:*               LISTEN     
5878/unbound        
tcp6       0      0 ::1:8953                :::*                    LISTEN     
5878/unbound        
udp        0      0 127.0.0.1:5353          0.0.0.0:*                          
5878/unbound 

 -- END SNIPPET --

Verify it's working:

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# dig @127.0.0.1 -p 5353 www.bentasker.co.uk

; <<>> DiG 9.10.3-P4-Debian <<>> @127.0.0.1 -p 5353 www.bentasker.co.uk
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44469
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.bentasker.co.uk.		IN	A

;; ANSWER SECTION:
www.bentasker.co.uk.	3600	IN	CNAME	wwwsite.balanced.bentasker.co.uk.
wwwsite.balanced.bentasker.co.uk. 30 IN	A	206.189.120.26

;; Query time: 382 msec
;; SERVER: 127.0.0.1#5353(127.0.0.1)
;; WHEN: Sat Apr 27 23:03:19 UTC 2019
;; MSG SIZE  rcvd: 95

 -- END SNIPPET --


OK, we should now be ready to swap out the DoH server config

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# cd /etc/dns-over-https/
root@debian-9-DoH-instance:/etc/dns-over-https# ls
doh-server.conf  doh-server.conf.new
root@debian-9-DoH-instance:/etc/dns-over-https# mv doh-server.conf doh-server.conf.orig
root@debian-9-DoH-instance:/etc/dns-over-https# mv doh-server.conf.new doh-server.conf

root@debian-9-DoH-instance:/etc/dns-over-https# systemctl restart doh-server
root@debian-9-DoH-instance:/etc/dns-over-https# systemctl status doh-server

 -- END SNIPPET --

Time for another test request


-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ curl -s
"https://dns.bentasker.co.uk/dns-query?name=projects.bentasker.co.uk&type=A" | python -m
json.tool

 -- END SNIPPET --


Now we need to set up the adblocking

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound/local.d# curl -o /root/update_ads.sh
https://www.bentasker.co.uk/adblock/static/update_ads.sh
root@debian-9-DoH-instance:/etc/unbound/local.d# chmod +x /root/update_ads.sh
root@debian-9-DoH-instance:/etc/unbound/local.d# echo "0 0 * * *    root   
/root/update_ads.sh" > /etc/cron.d/ad_domains_update

 -- END SNIPPET --

Manually populate the first instance of the blocklist

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound/local.d# cd /etc/unbound/local.d/
root@debian-9-DoH-instance:/etc/unbound/local.d# wget -O adblock.new
"https://www.bentasker.co.uk/adblock/autolist.txt"
root@debian-9-DoH-instance:/etc/unbound/local.d# mv adblock.new adblock.conf
root@debian-9-DoH-instance:/etc/unbound/local.d# systemctl restart unbound

 -- END SNIPPET --

Test request of a currently adblocked domain 

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ curl -s
"https://dns.bentasker.co.uk/dns-query?name=google-analytics.com&type=A" | python -m
json.tool
{
    "AD": false,
    "Answer": [
        {
            "Expires": "Sun, 28 Apr 2019 00:20:41 UTC",
            "TTL": 3600,
            "data": "127.0.0.1",
            "name": "google-analytics.com.",
            "type": 1
        }
    ],
    "CD": false,
    "Question": [
        {
            "name": "google-analytics.com.",
            "type": 1
        }
    ],
    "RA": true,
    "RD": true,
    "Status": 0,
    "TC": false
}

 -- END SNIPPET --
	
-----------------------------------------------------------------------------------------
2019-04-28 01:24:19              btasker
-----------------------------------------------------------------------------------------

*Initial Security/Safety Tweaks*


First thing to do is to set a sensible set of firewall rules

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# iptables -F INPUT
root@debian-9-DoH-instance:~# iptables -A INPUT -m state --state RELATED,ESTABLISHED -j
ACCEPT
root@debian-9-DoH-instance:~# iptables -A INPUT -i lo -s 127.0.0.0/8 -j ACCEPT
root@debian-9-DoH-instance:~# iptables -A INPUT -p tcp --dport 443 -j ACCEPT
root@debian-9-DoH-instance:~# iptables -A INPUT -p tcp --dport 22 -j ACCEPT
root@debian-9-DoH-instance:~# iptables -A INPUT -j REJECT
root@debian-9-DoH-instance:~# ip6tables -F INPUT
root@debian-9-DoH-instance:~# ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j
ACCEPT
root@debian-9-DoH-instance:~# ip6tables -A INPUT -i lo -s ::1 -j ACCEPT
root@debian-9-DoH-instance:~# ip6tables -A INPUT -p tcp --dport 443 -j ACCEPT
root@debian-9-DoH-instance:~# ip6tables -A INPUT -p tcp --dport 22 -j ACCEPT
root@debian-9-DoH-instance:~# ip6tables -A INPUT -j REJECT
root@debian-9-DoH-instance:~# apt-get -y install iptables-persistent

 -- END SNIPPET --

The next thing we want to do is to set up some query rate limiting in Nginx

-- BEGIN SNIPPET --

cat << EOM > /etc/nginx/conf.d/00-rate-limits.conf
limit_req_zone \$binary_remote_addr zone=doh_limit:10m rate=300r/s;
EOM

 -- END SNIPPET --

We then need to adjust our config for the DoH property to reference this limit. We're also
going to allow a bit of bursting and ensure queries aren't delayed. Adding this to
/etc/nginx/conf.d/doh.conf within the location /dns-query section

-- BEGIN SNIPPET --

limit_req zone=doh_limit burst=50 nodelay;

 -- END SNIPPET --


At this point, as it's getting late, I figured I'd configure it in Firefox and tail the
NGinx logs

-- BEGIN SNIPPET --

86.175.47.235 - - [28/Apr/2019:00:15:12 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABBHdpa2kGdWJ1bnR1A2NvbQAAAQABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 198 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:12 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABBHdpa2kGdWJ1bnR1A2NvbQAAHAABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 131 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:21 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABA3d3dxBnb29nbGUtYW5hbHl0aWNzA2NvbQAAHAABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 61 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:21 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABA3d3dxBnb29nbGUtYW5hbHl0aWNzA2NvbQAAAQABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 93 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:30 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABBnB1YnN1YgpydHNjaGFubmVsA2NvbQAAAQABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 87 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:30 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAABBnB1YnN1YgpydHNjaGFubmVsA2NvbQAAHAABAAApEAAAAAAAAAgACAAEAAEAAA
HTTP/1.1" 200 99 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:43 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAATAOY2xpZW50LWNoYW5uZWwGZ29vZ2xlA2NvbQAAAQAB HTTP/1.1" 200
99 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:44 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAATAOY2xpZW50LWNoYW5uZWwGZ29vZ2xlA2NvbQAAHAAB HTTP/1.1" 200
111 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0"
"-"
86.175.47.235 - - [28/Apr/2019:00:15:54 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAA3NzbAdnc3RhdGljA2NvbQAAHAAB HTTP/1.1" 200 87 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:15:55 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAA3NzbAdnc3RhdGljA2NvbQAAAQAB HTTP/1.1" 499 0 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:02 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAA3d3dwZnb29nbGUDY29tAAABAAE HTTP/1.1" 200 73 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:02 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAAA3d3dwZnb29nbGUDY29tAAAcAAE HTTP/1.1" 200 85 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:08 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAABnZpZGVvcwliZW50YXNrZXICY28CdWsAAAEAAQ HTTP/1.1" 200 167
"-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:08 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAABnZpZGVvcwliZW50YXNrZXICY28CdWsAABwAAQ HTTP/1.1" 200 179
"-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:13 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAABG1haWwGZ29vZ2xlA2NvbQAAHAAB HTTP/1.1" 200 147 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"
86.175.47.235 - - [28/Apr/2019:00:16:13 +0000] "GET
/dns-query?dns=AAABAAABAAAAAAAABG1haWwGZ29vZ2xlA2NvbQAAAQAB HTTP/1.1" 200 135 "-"
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101 Firefox/64.0" "-"

 -- END SNIPPET --
The format's a bit naff for assessing performance though, so configuring a different log
format too

-- BEGIN SNIPPET --

    log_format  main  '$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t'
                      '$status\t$body_bytes_sent\t"$http_referer"\t'
                     
'"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\tCACHE_$upstream_cache_status\t$request_time\t$hostname';

 -- END SNIPPET --

Giving us

-- BEGIN SNIPPET --

86.175.47.235	-	-	[28/Apr/2019:00:18:45 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABHNzbzIPc3dpZnRmZWRlcmF0aW9uA2NvbQAAAQAB
HTTP/1.1"	200	577	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.015	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:46 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AHZ29kYWRkeQNjb20AAAEAAQ
HTTP/1.1"	200	145	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.428	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:46 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AHZ29kYWRkeQNjb20AABwAAQ
HTTP/1.1"	200	188	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.417	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:46 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AHZ29kYWRkeQNjb20GYWthZG5zA25ldAAAHAAB
HTTP/1.1"	200	142	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.002	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAB3N0YXRpYzEJYmVudGFza2VyAmNvAnVrAAABAAE
HTTP/1.1"	200	169	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.027	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AGaW50LXgzC2xldHNlbmNyeXB0A29yZwAAAQAB
HTTP/1.1"	200	285	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.001	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAACHNuaXBwZXRzCWJlbnRhc2tlcgJjbwJ1awAAAQAB
HTTP/1.1"	200	171	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.138	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AGaW50LXgzC2xldHNlbmNyeXB0A29yZwAAHAAB
HTTP/1.1"	200	309	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.001	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AGaW50LXgzC2xldHNlbmNyeXB0A29yZwllZGdlc3VpdGUDbmV0AAABAAE
HTTP/1.1"	200	217	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.001	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AGaW50LXgzC2xldHNlbmNyeXB0A29yZwllZGdlc3VpdGUDbmV0AAAcAAE
HTTP/1.1"	200	241	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.001	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAB3N0YXRpYzEJYmVudGFza2VyAmNvAnVrAAAcAAE
HTTP/1.1"	200	181	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.207	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:47 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAACHNuaXBwZXRzCWJlbnRhc2tlcgJjbwJ1awAAHAAB
HTTP/1.1"	200	183	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.262	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:48 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABXBpd2lrCWJlbnRhc2tlcgJjbwJ1awAAAQAB
HTTP/1.1"	200	253	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.010	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:48 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABXBpd2lrCWJlbnRhc2tlcgJjbwJ1awAAAQAB
HTTP/1.1"	200	253	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.000	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:48 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABXBpd2lrCWJlbnRhc2tlcgJjbwJ1awAAHAAB
HTTP/1.1"	200	140	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.019	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:49 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAB3BhZ2VhZDIRZ29vZ2xlc3luZGljYXRpb24DY29tAAAcAAE
HTTP/1.1"	499	0	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	1.501	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:49 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAB3BhZ2VhZDIRZ29vZ2xlc3luZGljYXRpb24DY29tAAABAAE
HTTP/1.1"	499	0	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	1.501	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:49 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAEWdvb2dsZXN5bmRpY2F0aW9uA2NvbQAAAgAB
HTTP/1.1"	200	246	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.012	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:53 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAA3d3dwliZW50YXNrZXICY28CdWsAAAEAAQ
HTTP/1.1"	200	161	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.011	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:00:18:53 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAAA3d3dwliZW50YXNrZXICY28CdWsAABwAAQ
HTTP/1.1"	200	173	"-"	"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:64.0) Gecko/20100101
Firefox/64.0"	"-"	"dns.bentasker.co.uk"	CACHE_-	0.001	debian-9-doh-instance

 -- END SNIPPET --

There are definitely some improvements that can be made (including allowing some caching
without screwing over ECS too much), and we may also need to tweak unbound to ensure it's
sending ECS information on (and the correct information at that)

Firefox (at least) also allows you to set the value of the Authorization header in
network.trr.credentials, so it's also possible to have this publicly accessible, but not
publicly usable. Want to look at setting that up too.
	
-----------------------------------------------------------------------------------------
2019-04-28 01:33:54              btasker
-----------------------------------------------------------------------------------------

That's it for tonight, will pick back up on this at some other point (possibly for a bit
tomorrow)
	
-----------------------------------------------------------------------------------------
2019-04-28 11:19:24              btasker
-----------------------------------------------------------------------------------------

*Performance Tweaks*

There are a few things we might want to do to improve performance. For a start we should
enable HTTP/2


-- BEGIN SNIPPET --

upstream dns-backend {
    server 127.0.0.1:8053;
}
server {
        server_name dns.bentasker.co.uk;
        root /tmp/NOEXIST;
        location /dns-query {

                # Apply rate limiting
                limit_req zone=doh_limit burst=50 nodelay;

                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_redirect off;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_read_timeout 86400;
                proxy_pass http://dns-backend/dns-query;
        }


        location / {
            return 404;
        }

    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/dns.bentasker.co.uk/fullchain.pem; # managed by
Certbot
    ssl_certificate_key /etc/letsencrypt/live/dns.bentasker.co.uk/privkey.pem; # managed
by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

 -- END SNIPPET --

We can also cut out some latency on the queries by having Nginx use keep-alive connections
to the backend, and return them to a pool once each query's done (meaning we don't have to
wait for a 3 way each time). We need to add the keepalive directive to the upstream *and*
override the upstream Connection header 


-- BEGIN SNIPPET --

upstream dns-backend {
    server 127.0.0.1:8053;
    keepalive 30;
}
server {
        server_name dns.bentasker.co.uk;
        root /tmp/NOEXIST;
        location /dns-query {

                # Apply rate limiting
                limit_req zone=doh_limit burst=50 nodelay;

                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection "";
                proxy_redirect off;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_read_timeout 86400;
                proxy_pass http://dns-backend/dns-query;
        }


        location / {
            return 404;
        }

    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/dns.bentasker.co.uk/fullchain.pem; # managed by
Certbot
    ssl_certificate_key /etc/letsencrypt/live/dns.bentasker.co.uk/privkey.pem; # managed
by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

 -- END SNIPPET --

Testing that, we can see what looks like an improvement in response times, but there are
also some failed requests (NGinx logged a 499 because the client went away - presumably
because of the long response time)

Unscientifically comparing average response times between this set of lines and the one in
the previous comment though, shows this is actually a little slower

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6 == 200) t=t+$13; c=c+1;
end}END{print t/c}' /tmp/before
0.0739524
ben@milleniumfalcon:~$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6 == 200) t=t+$13; c=c+1;
end}END{print t/c}' /tmp/after
0.0902727

 -- END SNIPPET --

Not really enough log lines to draw any conclusions there, particularly as I was visiting
different sites in Firefox, so it only takes an authoritative to be slower (or likely in
the case of the 499's, down).

The next thing I want to do is introduce some basic caching in NGinx. To do this, we're
going to break the IP (I'm ignoring IPv6 for the moment) down to being a /24 and inject
that into the cache key 

First we need to create a cache directory

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/nginx/conf.d# mkdir /mnt/cache

 -- END SNIPPET --

and add the following into the http section of nginx.conf

-- BEGIN SNIPPET --

proxy_cache_path  /mnt/cache levels=1:2 keys_zone=doh-cache:8m max_size=800m
inactive=30d;
proxy_temp_path /mnt/cache/tmp; 
proxy_cache_use_stale updating invalid_header error timeout http_502; # Use a stale entry
if origin unavailable

 -- END SNIPPET --

This will create a 800M cache zone. Now we need to tell the doh server block to use it,
and amend the cache key.

See https://projectsstatic.bentasker.co.uk/MISC/MISC27/serverblock.txt for the amended
NGinx config (as the regex in it screws with JIRA's formatting). Basically it takes the
client's IP, kicks the last octet off the end and injects it into the cache key.

This will, of course, only help in clients that use GET (i.e. in Firefox,
network.trr.useGET would need to be changed from it's default of False. But the main aim
is to reduce the chances of the backend getting clogged up if a client repeatedly
re-places the same query.

Pulling a line from the access logs and testing with curl:

-- BEGIN SNIPPET --

86.175.47.235	-	-	[28/Apr/2019:10:08:28 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AIZGlnaWNlcnQDY29tAAAcAAE
HTTP/1.1"	200	180	"-"	"curl/7.47.0"	"-"	"dns.bentasker.co.uk"	CACHE_MISS	0.001	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:10:08:29 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AIZGlnaWNlcnQDY29tAAAcAAE
HTTP/1.1"	200	180	"-"	"curl/7.47.0"	"-"	"dns.bentasker.co.uk"	CACHE_HIT	0.000	debian-9-doh-instance
86.175.47.235	-	-	[28/Apr/2019:10:08:32 +0000]	"GET
/dns-query?dns=AAABAAABAAAAAAAABG9jc3AIZGlnaWNlcnQDY29tAAAcAAE
HTTP/1.1"	200	180	"-"	"curl/7.47.0"	"-"	"dns.bentasker.co.uk"	CACHE_HIT	0.000	debian-9-doh-instance

 -- END SNIPPET --

Generating some new lines with Firefox, response times are still higher than I'd like

Switching over to have Unbound just act as a forwarder (onto Google DNS in this case)
gives a massive improvement.

-- BEGIN SNIPPET --

ben@milleniumfalcon:~/Documents/Notes/Request_Router$ awk -F'\t' 'BEGIN{t=0; c=0}{if ($6
== 200) t=t+$13; c=c+1; end}END{print t/c}' /tmp/after
0.0324127

 -- END SNIPPET --

Given this is being built on a single core instance, it's probably a bit much to ask for
that much stuff to all happen at once. Will look at bumping the cores up later to verify.

Want to look at enabling ECS next (and we can't have unbound acting as a forwarder for
that, because Google DNS etc ignore incoming ECS and set their own - i.e. act as if we
never included ECS in our query).
	
-----------------------------------------------------------------------------------------
2019-04-28 12:56:51              btasker
-----------------------------------------------------------------------------------------

*Unbound ECS*

Seems this isn't actually an option, at least not without installing out-of-band.

Debian's unbound package is compiled without the ECS module


-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/etc/unbound# unbound -h
usage:  unbound [options]
	start unbound daemon DNS resolver.
-h	this help
-c file	config file to read instead of /etc/unbound/unbound.conf
	file format is described in unbound.conf(5).
-d	do not fork into the background.
-v	verbose (more times to increase verbosity)
Version 1.6.0
linked libs: libevent 2.0.21-stable (it uses epoll), OpenSSL 1.1.0j  20 Nov 2018
linked modules: dns64 python validator iterator
BSD licensed, see LICENSE in source package for details.
Report bugs to unbound-bugs@<Domain Hidden>

 -- END SNIPPET --


Otherwise, it would (should?) simply have been a case of adding the following to the
server section

-- BEGIN SNIPPET --

    send-client-subnet: yes
    client-subnet-zone: "." 
    client-subnet-always-forward: yes
    max-client-subnet-ipv4: 24
    max-client-subnet-ipv6: 48

 -- END SNIPPET --

Not having ECS enabled means most CDNs are going to send me to the PoP nearest to my
resolver rather than one nearest me (they may differ vastly). That's not entirely
optimal.

That said, it could be mitigated to some extent, as I could run a number of servers in
different locations and then CNAME dns.bentasker.co.uk into my CDN's geo-routing system so
that when browsers perform a lookup of the name to send DoH queries to they're routed to
the server closest to them. That way the results would still, mostly, be more appropriate
to them than they would otherwise. 

That, of course, won't address instances where a CDN might have pops within the users ISP
(because their routing system will never find out that they're on that ISP). It also
relies on the idea that I've done such a good job of choosing locations for my servers
that it's either comparable, or better, than the choices other CDNs have made - otherwise
that's a degradation in end user service.

Essentially, without that ECS support, what we've now built is something that's no better
for the end user than Cloudflare's offering on 1.1.1.1. We can do better than that.


So, lets build a newer version of the DoH server so that *it* can insert ECS (which
unbound will then hopefully just pass through)

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# apt-get install build-essential git
root@debian-9-DoH-instance:~# apt-get remove doh-server
root@debian-9-DoH-instance:~# mkdir build
root@debian-9-DoH-instance:~# cd build

# Install Go >= 1.10
root@debian-9-DoH-instance:~/build# wget
https://dl.google.com/go/go1.12.2.linux-amd64.tar.gz
root@debian-9-DoH-instance:~/build# tar xzf go1.12.2.linux-amd64.tar.gz 
root@debian-9-DoH-instance:~/build# mv go /usr/local/
root@debian-9-DoH-instance:~/build# export GOROOT=/usr/local/go
root@debian-9-DoH-instance:~/build# export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
root@debian-9-DoH-instance:~/build# go version
go version go1.12.2 linux/amd64

# Build the App
root@debian-9-DoH-instance:~# mkdir ~/gopath
root@debian-9-DoH-instance:~# export GOPATH=~/gopath
root@debian-9-DoH-instance:~/build# wget
https://github.com/m13253/dns-over-https/archive/v2.0.1.tar.gz
root@debian-9-DoH-instance:~/build# tar xzf v2.0.1.tar.gz 
root@debian-9-DoH-instance:~/build# cd dns-over-https-2.0.1/
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# # Backup the original config,
just in case
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# cp -r /etc/dns-over-https/
/etc/dns-over-https.orig
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# 
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# make
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# make install
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# cat
/etc/dns-over-https/doh-server.conf # Check my modifications are there
root@debian-9-DoH-instance:~/build/dns-over-https-2.0.1# systemctl restart doh-server

 -- END SNIPPET --

At which point, we can see in packet captures that the DoH server is including ECS in it's
queries to unbound

-- BEGIN SNIPPET --

Frame 1: 101 bytes on wire (808 bits), 101 bytes captured (808 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 52907 (52907), Dst Port: 5353 (5353)
Multicast Domain Name System (query)
    [Response In: 10]
    Transaction ID: 0xf198
    Flags: 0x0100 Standard query
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 1
    Queries
    Additional records
        <Root>: type OPT
            Name: <Root>
            Type: OPT (41)
            .001 0000 0000 0000 = UDP payload size: 0x1000
            0... .... .... .... = Cache flush: False
            Higher bits in extended RCODE: 0x00
            EDNS0 version: 0
            Z: 0x0000
                0... .... .... .... = DO bit: Cannot handle DNSSEC security RRs
                .000 0000 0000 0000 = Reserved: 0x0000
            Data length: 11
            Option: CSUBNET - Client subnet
                Option Code: CSUBNET - Client subnet (8)
                Option Length: 7
                Option Data: 0001180056af2f
                Family: IPv4 (1)
                Source Netmask: 24
                Scope Netmask: 0
                Client Subnet: 86.175.47.0

 -- END SNIPPET --

Unfortunately, Unbound *isn't* passing that through. Looks like we are going to have to go
out of band with unbound then (the previous changes weren't in vain though as we do need
the client's subnet to be included in the query that Unbound receives)

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~# cp -r /etc/unbound/ /etc/unbound.orig #back up the config
root@debian-9-DoH-instance:~# cp /lib/systemd/system/unbound.service ~
root@debian-9-DoH-instance:~# apt-get remove unbound # Remove the packaged one
root@debian-9-DoH-instance:~# apt-get install libssl-dev libexpat1-dev

root@debian-9-DoH-instance:/tmp# cd ~/build/
root@debian-9-DoH-instance:~/build# wget
http://www.unbound.net/downloads/unbound-latest.tar.gz
root@debian-9-DoH-instance:~/build# tar xzf unbound-latest.tar.gz
root@debian-9-DoH-instance:~/build# cd unbound-1.9.1/
root@debian-9-DoH-instance:~/build/unbound-1.9.1# ./configure --enable-subnet

root@debian-9-DoH-instance:~/build/unbound-1.9.1# make
root@debian-9-DoH-instance:~/build/unbound-1.9.1# make install
root@debian-9-DoH-instance:~/build/unbound-1.9.1# vi ~/unbound.service
root@debian-9-DoH-instance:~/build/unbound-1.9.1# cat ~/unbound.service 
[Unit]
Description=Unbound DNS server
Documentation=man:unbound(8)
After=network.target
Before=nss-lookup.target
Wants=nss-lookup.target

[Service]
User=unbound
Type=simple
Restart=on-failure
EnvironmentFile=-/etc/default/unbound
ExecStartPre=/usr/local/sbin/unbound-anchor -a /var/lib/unbound/root.key -c
/etc/unbound/unbound_server.pem
ExecStart=/usr/local/sbin/unbound -c /etc/unbound/unbound.conf -d $DAEMON_OPTS
ExecReload=/usr/local/sbin/unbound-control reload

[Install]
WantedBy=multi-user.target


root@debian-9-DoH-instance:~/build/unbound-1.9.1# useradd -r unbound
root@debian-9-DoH-instance:~/build/unbound-1.9.1# chown unbound /var/log/unbound/
root@debian-9-DoH-instance:~/build/unbound-1.9.1# ldconfig
root@debian-9-DoH-instance:~/build/unbound-1.9.1# mkdir /var/lib/unbound
root@debian-9-DoH-instance:~/build/unbound-1.9.1# chown unbound /var/lib/unbound/

root@debian-9-DoH-instance:~/build/unbound-1.9.1# cp /root/unbound.service
/lib/systemd/system/
root@debian-9-DoH-instance:~/build/unbound-1.9.1# systemctl daemon-reload
root@debian-9-DoH-instance:~/build/unbound-1.9.1# systemctl unmask unbound
root@debian-9-DoH-instance:~/build/unbound-1.9.1# systemctl enable unbound

 -- END SNIPPET --

Before we start it, need to drop the ECS config in. Adding the following into the server
section of /etc/unbound/unbound.conf

-- BEGIN SNIPPET --

    module-config: "subnetcache validator iterator"
    client-subnet-zone: "." 
    client-subnet-always-forward: yes
    max-client-subnet-ipv4: 24
    max-client-subnet-ipv6: 48

 -- END SNIPPET --



To get this working, I've also had to disable Unbound's chroot (for now). Need to sit down
and sort it out properly later though. Adding the following to the server section too

-- BEGIN SNIPPET --

    chroot: ""

 -- END SNIPPET --

And restarting unbound

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:~/build/unbound-1.9.1# systemctl restart unbound

 -- END SNIPPET --

As before, we see ECS going from the DoH server into unbound on UDP 5353

-- BEGIN SNIPPET --

Frame 23: 108 bytes on wire (864 bits), 108 bytes captured (864 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 53388 (53388), Dst Port: 5353 (5353)
Multicast Domain Name System (query)
    [Response In: 43]
    Transaction ID: 0xa65c
    Flags: 0x0100 Standard query
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 1
    Queries
    Additional records
        <Root>: type OPT
            Name: <Root>
            Type: OPT (41)
            .001 0000 0000 0000 = UDP payload size: 0x1000
            0... .... .... .... = Cache flush: False
            Higher bits in extended RCODE: 0x00
            EDNS0 version: 0
            Z: 0x0000
            Data length: 11
            Option: CSUBNET - Client subnet
                Option Code: CSUBNET - Client subnet (8)
                Option Length: 7
                Option Data: 0001180056af2f
                Family: IPv4 (1)
                Source Netmask: 24
                Scope Netmask: 0
                Client Subnet: 86.175.47.0

 -- END SNIPPET --

But now, we also see the query going out (in this case to 8.8.4.4 which will have ignored
it). But the important thing is it's there

-- BEGIN SNIPPET --

Frame 25: 108 bytes on wire (864 bits), 108 bytes captured (864 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 178.128.166.130, Dst: 8.8.4.4
User Datagram Protocol, Src Port: 44225 (44225), Dst Port: 53 (53)
Domain Name System (query)
    [Response In: 29]
    Transaction ID: 0x6faa
    Flags: 0x0100 Standard query
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 1
    Queries
    Additional records
        <Root>: type OPT
            Name: <Root>
            Type: OPT (41)
            UDP payload size: 4096
            Higher bits in extended RCODE: 0x00
            EDNS0 version: 0
            Z: 0x8000
                1... .... .... .... = DO bit: Accepts DNSSEC security RRs
                .000 0000 0000 0000 = Reserved: 0x0000
            Data length: 11
            Option: CSUBNET - Client subnet
                Option Code: CSUBNET - Client subnet (8)
                Option Length: 7
                Option Data: 0001180056af2f
                Family: IPv4 (1)
                Source Netmask: 24
                Scope Netmask: 0
                Client Subnet: 86.175.47.0

 -- END SNIPPET --

So, if we move the forwarder config out of the way, we should see Unbound sending ECS
information to authoritatives

-- BEGIN SNIPPET --

root@debian-9-DoH-instance:/tmp# mv /etc/unbound/unbound.conf.d/forwarders.conf /root/
root@debian-9-DoH-instance:/tmp# systemctl restart unbound

 -- END SNIPPET --

Woot! 


-- BEGIN SNIPPET --

Frame 23: 116 bytes on wire (928 bits), 116 bytes captured (928 bits)
Linux cooked capture
Internet Protocol Version 4, Src: 178.128.166.130, Dst: 94.23.157.104
User Datagram Protocol, Src Port: 21409 (21409), Dst Port: 53 (53)
Domain Name System (query)
    [Response In: 29]
    Transaction ID: 0x8691
    Flags: 0x0010 Standard query
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 1
    Queries
    Additional records
        <Root>: type OPT
            Name: <Root>
            Type: OPT (41)
            UDP payload size: 4096
            Higher bits in extended RCODE: 0x00
            EDNS0 version: 0
            Z: 0x8000
            Data length: 11
            Option: CSUBNET - Client subnet
                Option Code: CSUBNET - Client subnet (8)
                Option Length: 7
                Option Data: 0001180056af2f
                Family: IPv4 (1)
                Source Netmask: 24
                Scope Netmask: 0
                Client Subnet: 86.175.47.0

 -- END SNIPPET --

So now we have proper working ECS support too.

Will look at scaling the VM up to have more cores later so that can check whether that
improves performance a bit.
	
-----------------------------------------------------------------------------------------
2019-04-28 15:14:17              btasker
-----------------------------------------------------------------------------------------

As this issue is public I won't detail the exact procedure (plus it's well recorded
elsewhere), but I've also now allowed the DoH server to pull down config files containing
override records, so that I can commit overrides (to block a new ad domain, for example)
and have it take effect on my LANs UDP DNS server as well as this DoH one.


-- BEGIN SNIPPET --

    include: /etc/unbound/local.d/dns.d/*.conf

 -- END SNIPPET --

And automated pulling those changes 

-- BEGIN SNIPPET --

cat << EOM > /etc/cron.d/dns-zones
*/30 * * * * root /root/edge_scripts/bin/action_on_git_changes.sh
/etc/unbound/local.d/dns.d/ 'service unbound restart'
EOM

 -- END SNIPPET --

There's no need to push records for my LAN out to the DoH server:

- I use .home which Firefox doesn't use DoH for anyway
- Where it's not .home if resolution fails using DoH, FF will fall back on the OS
configured DNS

(Obviously, the latter may give rise to concerns of DNS leaks, so some care is needed).

If I were to want to push records for my LAN out to the DoH server I could stick them into
my dns.d repo, but would also need to set network.trr.allow-rfc1918 to true in Firefox
	
-----------------------------------------------------------------------------------------
2019-04-29 10:35:06              btasker
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --

There's no need to push records for my LAN out to the DoH server:

- I use .home which Firefox doesn't use DoH for anyway
- Where it's not .home if resolution fails using DoH, FF will fall back on the OS
configured DNS

(Obviously, the latter may give rise to concerns of DNS leaks, so some care is needed).

-- END QUOTE --

Actually, this is very much client and configuration specific.

Firefox will fall through to OS DNS only if network.trr.mode is set to 2. If it's 3 then
it won't.

However, outside of Firefox, other DoH implementations don't offer a fallback at all.
Google's (well, Jigsaw's) Intra -
https://play.google.com/store/apps/details?id=app.intra&hl=en_GB - is all or nothing, for
example. Which means, if on Android you've got Intra running, it'll intercept the queries
when Firefox falls back and send them out to the DoH server (despite being .home
addresses) and you won't be able to resolve those resources.

So, there may actually be a need to make these records available so that a wider range of
support is achievable. Within Firefox, for them to be accepted network.trr.allow-rfc1918
would need to be toggled to true (though it looks like Intra will just accept them without
question)



-- BEGIN QUOTE --

Firefox (at least) also allows you to set the value of the Authorization header in
network.trr.credentials, so it's also possible to have this publicly accessible, but not
publicly usable. Want to look at setting that up too

-- END QUOTE --

Worth noting that Intra does not offer configuration to support this, so it wouldn't be
possible to use Intra with a server protected in this manner
	
-----------------------------------------------------------------------------------------
2019-04-29 11:16:45              btasker
-----------------------------------------------------------------------------------------

*Split Horizon*

It's commonly held that DoH means you cannot do split-horizon DNS, but this is only
actually true if you're using someone else's DoH service.

If you use a DoH server (or a proxy to one) where you control the name (and can therefore
get a SSL certificate) then split-horizon can still be achieved by using split-horizon at
the UDP level for that name.

I.e.

-- BEGIN SNIPPET --

Lan DNS         Lan DoH                    |                Auth NS             Web DoH

 -- END SNIPPET --

If I were to deploy a new DoH server into the LAN, and furnish it with a certificate for
dns.bentasker.co.uk then it'd simply be a case of having the LAN DNS return 192.168.1.3
(or whatever) for dns.bentasker.co.uk so that Firefox and Intra connect to the local DoH
server (which would contain config for whatever records I wanted to override on this side
of the horizon).

When out and about, they'd still resolve to the main one.

A little off topic, but just to finish that thought process.

Neither of those DoH servers need to actually technically be a DoH server, they could
simply be a proxy onto a public one

-- BEGIN SNIPPET --

server {
        server_name cfdns.bentasker.co.uk;
        root /tmp/NOEXIST;
        location /dns-query {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host 1.1.1.1;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_redirect off;
                proxy_read_timeout 86400;
                proxy_pass https://1.1.1.1/dns-query;
        }


        location / {
            return 404;
        }

    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/pki/cert/cert.pem;
    ssl_certificate_key /etc/pki/private/cert.key;
}

 -- END SNIPPET --
(though you might want to secure it to make sure you don't get banned by CF when people
start trying to abuse via you).

So split horizon is still totally do-able, it's just that rather than simply running DNS
on the LAN, you also need to run a publicly routable DoH instance or proxy.
	
-----------------------------------------------------------------------------------------
2019-04-30 19:30:59              btasker
-----------------------------------------------------------------------------------------

Just taking a quick scan over the net, there is a rudimentary DoH implementation in LUA
here - https://github.com/fanf2/doh101/blob/master/roles/doh101/files/doh.lua

They note some of the limitations in the README, but those all look addressable enough.
Whether moving it to LUA would actually constitute an improvement over deploying
doh-server is something else.
	
-----------------------------------------------------------------------------------------
2019-04-30 19:33:59              btasker
-----------------------------------------------------------------------------------------

Anyway, odd tweaks aside, I think I'm at the point now where the next step is to take the
comments on this issue and compress them down into some documentation on how to set all
this up.

The easiest way is probably to take what's been learnt (don't use the repo's unbound etc)
and build another server from scratch so that it's right first time, whilst documenting
the exact process for writing up.

Although it's not something I want for my setup, once the setup's done, it's probably
worth removing unbound and installing pihole instead so that can also document what's
involved with that (pihole install is relatively straight forward and easy).
	
-----------------------------------------------------------------------------------------
2019-05-01 12:59:34              btasker
-----------------------------------------------------------------------------------------

The "right first time" version of the process is noted down here -
https://projectsstatic.bentasker.co.uk/MISC/MISC27/MISC27_DoH_Server_Build.txt

Not done the pihole swap out yet though.
	
-----------------------------------------------------------------------------------------
2019-05-01 14:47:18              btasker
-----------------------------------------------------------------------------------------

*Pihole Install*

To use Pihole instead of unbound, we need to skip the unbound step (see below if you want
to use unbound as well) and instead do


-- BEGIN SNIPPET --

curl -sSL https://install.pi-hole.net | bash

 -- END SNIPPET --

When prompted, do not install Pi-hole default firewall rules

Make a note of the admin password

Once the installer's complete, we need to reconfigure lighttpd to bind to a different
port

-- BEGIN SNIPPET --

vi /etc/lighttpd/lighttpd.conf

change server.port to 8080
systemctl start lighttpd

root@debian-9-doh-newbuild:~# netstat -lnp | grep light
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN     
27041/lighttpd      
tcp6       0      0 :::8080                 :::*                    LISTEN     
27041/lighttpd      
unix  2      [ ACC ]     STREAM     LISTENING     291926   27053/php-cgi       
/var/run/lighttpd/php.socket-0

 -- END SNIPPET --

Edit the DNS-over-HTTPS server config to change the upstream

-- BEGIN SNIPPET --

vi /etc/dns-over-https/doh-server.conf

upstream = [
    "127.0.0.1:53"
]


systemctl restart doh-server

 -- END SNIPPET --


Now we'll edit our NGinx config to add a server block to proxy through

-- BEGIN SNIPPET --

cat doh-admin.conf 
server {
        server_name dnsadmin.bentasker.co.uk;
        root /tmp/NOEXIST;

        #In production, definitely want to add some auth to this
        location /admin/ {
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_redirect off;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_read_timeout 86400;
                proxy_pass http://127.0.0.1:8080;
        }


    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/dnsadmin.bentasker.co.uk/fullchain.pem; #
managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dnsadmin.bentasker.co.uk/privkey.pem; #
managed by Certbot

}

systemctl restart nginx

 -- END SNIPPET --

You should now be able to login to the admin page at
https://dnsadmin.bentasker.co.uk/admin and see reports on how your queries have been
handled.


*Running Unbound Alongside*

If you still want your box to handle resolution rather than forwarding, there's a little
more customisation that can be done. We need to rebind unbound to 127.0.1.1 (as pihole
won't accept a custom port on a custom upstream resolver), which also means we need to
move pihole out of the way

First we need to tell pihole only to bind to a specific address 

-- BEGIN SNIPPET --

cat << EOM > /etc/dnsmasq.d/99-mysettings.conf
listen-address=::1,127.0.0.1
bind-interfaces
EOM

systemctl restart pihole-FTL

 -- END SNIPPET --

Edit /usr/local/etc/unbound/unbound.conf and set 

-- BEGIN SNIPPET --

    interface: 127.0.1.1
    port: 53

systemctl restart unbound

 -- END SNIPPET --

Now, in pihole's admin page, we can set an upstream resolver of 127.0.1.1 and it should
all just work (pihole seems to forward ECS data through if it was in the query it
received)


To add my adblock list into pihole's blocklists, use URL
https://www.bentasker.co.uk/adblock/blockeddomains.txt
	
-----------------------------------------------------------------------------------------
2019-05-02 16:33:17              btasker
-----------------------------------------------------------------------------------------

*DoH via Tor*


-- BEGIN SNIPPET --

apt-get install apt-transport-https
cat << EOM > /etc/apt/sources.list.d/tor.list
deb https://deb.torproject.org/torproject.org stretch main
deb-src https://deb.torproject.org/torproject.org stretch main
EOM


curl
https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc |
gpg --import
gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add -

apt-get update
apt-get -y install tor deb.torproject.org-keyring


# You may also want to restrict access with HiddenServiceAuthorizeClient

cat << EOM > /etc/tor/torrc
RunAsDaemon 1
DataDirectory /var/lib/tor

HiddenServiceDir /var/lib/tor/doh_service/
HiddenServicePort 443 127.0.0.1:443

HiddenServiceDir /var/lib/tor/doh_service_v2
HiddenServiceVersion 2
HiddenServicePort 443 127.0.0.1:443



EOM


systemctl reload tor

 -- END SNIPPET --

Get the hostnames for the new onion

-- BEGIN SNIPPET --

cat /var/lib/tor/doh_service/hostname
cat /var/lib/tor/doh_service_v2/hostname

 -- END SNIPPET --

Edit the NGinx config to inject an Alt-Svc header containing both the V2 and the V3

-- BEGIN SNIPPET --

nano /etc/nginx/conf.d/doh.conf

        location /dns-query {
                add_header Alt-Svc
'h2="7t2po4wsbkmu2vwqmib3kzkocydlh3yn675zvi24rhezinzc6yrtkeyd.onion:443",
h2="6ohomagiytkiro4l.onion:443"; ma=900';
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $http_host;
                proxy_set_header X-NginX-Proxy true;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_redirect off;
                proxy_set_header        X-Forwarded-Proto $scheme;
                proxy_read_timeout 86400;
                proxy_pass http://dns-backend/dns-query;

        }

 -- END SNIPPET --

We'll also adjust the log format in nginx.conf so that we log the alt_used header and can
see if browsers are actually using it (at this point, I genuinely don't know whether Tor
Browser Bundle will or not - at least not once network.proxy.socks_remote_dns is
disabled)

-- BEGIN SNIPPET --

    log_format  main  '$remote_addr\t-\t$remote_user\t[$time_local]\t"$request"\t'
                      '$status\t$body_bytes_sent\t"$http_referer"\t'
                     
'"$http_user_agent"\t"$http_x_forwarded_for"\t"$http_host"\tCACHE_$upstream_cache_status\t$request_time\t$hostname\t$http_alt_used';




 -- END SNIPPET --
Reload Nginx

-- BEGIN SNIPPET --

systemctl reload nginx 

 -- END SNIPPET --

Configured in Firefox and TBB.

Seems TBB won't use Alt-Used, no. I couldn't get my vanilla Firefox install to do it
either.

It's probably possible to convince clients to use plain HTTP, but then they won't be able
to use HTTP/2 so lookups will likely end up taking quite a while (just checked, Jigsaw's
Intra won't take a http URI).

It might also be that's somethings not right in my test itself of course.
	
-----------------------------------------------------------------------------------------
2019-05-02 17:19:51              btasker
-----------------------------------------------------------------------------------------

OK, I've taken my various notes and created an article -
https://www.bentasker.co.uk/documentation/linux/407-building-and-running-your-own-dns-over-https-server

It's a bit lengthy because I've included some of the options I looked at here rather than
doing a one-size-fits-all implementation. I've not had the time to look (in too much
depth) into what else would need to be done to make this "safe" (or a given measure  of)
to advertise as a public and open DoH resolver, so I've put a warning into the article
instead.
	
-----------------------------------------------------------------------------------------
2019-05-02 17:22:44              btasker
-----------------------------------------------------------------------------------------

I do still want to look at that LUA though, but I'll raise a task for it in STAGING for
now, and then turn it into a subtask of this issue if and when I get around to doing it.

In the meantime, for the paranoid (and for my own sanity if I need to look back at this),
having published the article, I've

- Moved my DoH recursor off dns.bentasker.co.uk/dns-query
- Moved the Pi-Hole Admin interface away
- Locked down access to both
- Removed Tor
- Disposed of the initial build VM where stuff isn't chrooted etc.

I am now using a self-hosted DoH server, it's just not reachable at any of the addresses
listed in this issue :)
	
-----------------------------------------------------------------------------------------
2019-05-02 17:24:15              
-----------------------------------------------------------------------------------------

btasker changed status from 'Open' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2019-05-02 17:24:15              
-----------------------------------------------------------------------------------------

btasker added 'Done' to resolution
	
-----------------------------------------------------------------------------------------
2019-05-02 17:24:19              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'
	
-----------------------------------------------------------------------------------------
2019-05-10 23:25:47              
-----------------------------------------------------------------------------------------

btasker removed 'Done' from resolution
	
-----------------------------------------------------------------------------------------
2019-05-10 23:25:47              
-----------------------------------------------------------------------------------------

btasker changed status from 'Closed' to 'Reopened'
	
-----------------------------------------------------------------------------------------
2019-05-10 23:27:36              
-----------------------------------------------------------------------------------------

btasker changed status from 'Reopened' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2019-05-10 23:27:36              
-----------------------------------------------------------------------------------------

btasker added 'Done' to resolution
	
-----------------------------------------------------------------------------------------
2019-05-10 23:27:41              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'