MISC-19: Design Investigatory Powers Act workarounds

Issue Information

Issue Type: Task
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: Miscellaneous (MISC)
Resolution: Done (2019-12-10 15:50:15)
Affects Version: Draft IPB Reading,
Target version: Draft IPB Reading,
Labels: IPB, Privacy, Routing, Security, VPN,

Created: 2016-11-17 19:48:52
Time Spent Working
120 minutes
1 minutes
119 minutes

The fuckwits in Westminister have signed off on the Investigatory Powers Bill, including all the privacy-invasive provisions such as ICRs. The bill is simply awaiting Royal Assent now.

Someone on Ars has helpfully listed the agencies that will be able to request access (WITHOUT a warrant) to Internet Connection Records

Air Accidents Investigation Branch
Charity Commission
Commission for Healthcare Audit and Inspection
Commissioners of Revenue and Customs
Common Services Agency for the Scottish Health Service
Criminal Cases Review Commission
Department for Environment, Food and Rural Affairs (for the purposes of the Marine Fisheries Agency)
Department for Transport (for the purposes of transport security, Vehicle and Operator Services Agency, Driving Standards Agency and Maritime and Coastguard Agency)
Department for Work and Pensions
Department of Agriculture and Rural Development for Northern Ireland
Department of Enterprise, Trade and Investment for Northern Ireland (for the purposes of Trading Standards)
Department of Health (for the purposes of the Medicines and Healthcare Products Regulatory Agency)
Environment Agency
Financial Services Authority
Fire Authority for Northern Ireland
Food Standards Agency
Gambling Commission
Gangmasters Licensing Authority
General Pharmaceutical Council
Government Communications Headquarters
Health & Safety Executive
Her Majesty¬'s Chief Inspector of Schools in England
HM Revenue and Customs
Home Office (for the purposes of HM Prison Service and the UK Border Agency)
Information Commissioner
Marine Accident Investigation Branch
Maritime and Coastguard Agency
Ministry of Defence
NHS ambulance service Trust
NHS Counter Fraud and Security Management Service
Northern Ireland Ambulance Service Health and Social Services Trust
Northern Ireland Health and Social Services Central Services Agency
Northern Ireland Office (for the purposes of the Northern Ireland Prison Service)
Office of Fair Trading
Office of the Deputy Prime Minister
Office of the Police Ombudsman for Northern Ireland
Port of Dover Police
Port of Liverpool Police
Post Office Investigation Branch
Postal Services Commission
Rail Accident Investigation Branch
Royal Air Force Police
Royal Military Police
Royal Navy Regulating Branch
Scottish Ambulance Service Board
Scottish Environment Protection Agency
Secret Intelligence Service
Security Service
Serious Fraud Office
The Armed Forces
The Pensions Regulator
Special Police Forces (including the Scottish Drug Enforcement Agency)
Territorial Police Forces
Welsh Ambulance Services NHS Trust
Welsh Government (for the purposes of the NHS Directorate, NHS Finance Division, Common Agricultural Policy Management Division and Care Standards Inspectorate for Wales)

The Government, quite frankly, can go fuck itself.

I cannot, in good conscience, allow my family's data to be recorded on a list that will, inevitably, be misused/leaked. Managed by incompetent ISPs and scope and access "controlled" by bungling politicians. No thanks.

Raising this issue to start the process of moving all relevant traffic over to using a VPN and/or Tor.

There are a number of requirements

- The solution must support failover, so if an endpoint goes down, another should be used
- Responses to inbound connections should not transit the link
- Policies should be used to control what can and cannot egress the network (for example, the current VPN links should be allowed to leave as they do now)
- Port 80 traffic should still go via the transparent proxy and then transit the VPN link
- The system should fail closed. If all endpoints are unreachable, block all traffic (but provide me a way to override that for troubleshooting)

Most DNS queries already leave via a VPN, but this may also be a good opportunity to look at setting up DNSCrypt

Once the provisional infrastructure is in place, I'll run an audit with PAS to check that egress traffic is as expected

Use of a VPN won't be enough to stop a targeted investigation, but that's explicitly not the aim of this issue. The aim is purely to ensure that the ISP held Internet Connection Records for my connection don't contain anything of relevance.

Issue Links

Gov Factsheet on ICRs
PCAP Analyse and Report
Toggle State Changes


Whether or not the VPN endpoints can be in the UK is going to depend on the Government actually defining who they consider to be a "Communication Service Provider". If possible, I'd prefer it was UK based to avoid geo-blocking headaches

For example, France have a similar law on the books, but it only applies to ISPs that have residential customers, so a number of datacentres don't (officially at least) have the capability deployed.

An alternate approach, which may be suitable for some of the traffic, is to run a transparent Tor instance and configure it only to use exit nodes in the UK
ExitNodes {GB}

There'll be a few headaches there though with some services blocking known exit nodes
For the initial development/testing, I'll spin up a couple of DigitalOcean droplets in London to act as the endpoints.

To begin with I'll use DNS round-robin for failover purposes and set something a little more intelligent up later.
btasker changed timespent from '0 minutes' to '10 minutes'
Given the Bulk Equipment Interference (funny how it's just "hacking" when it's anyone else) provisions, also a good time to review the IDS setup and see if there are any improvements that can be made.
Will raise a separate internal issue for it later, but need to keep a close eye on what they consider to be a CSP to see if any of the services I offer are likely to be impacted. Already part way there, but it may also be worth accelerating the project to reduce the amount of data I'm actually able to capture in the first place.
btasker added 'IPB Privacy Routing Security VPN' to labels
For reference, DigitalOcean don't give exact link speeds for Droplets, only that the hypervisor has multiple 1GB nics in a bond. They say you should plan on being able to achieve around 300Mbps of throughput, which should be more than sufficient.
btasker changed status from 'Open' to 'In Progress'
OK, so endpoint setup.

512Mb droplet running Debian 8

Essentially following the steps here - https://www.bentasker.co.uk/documentation/linux/259-openvpn-on-debian - though the easy-rsa steps are different on Jessie

root@debian-512mb-lon1-01:~# apt-get update
root@debian-512mb-lon1-01:~# apt-get install openvpn easy-rsa
root@debian-512mb-lon1-01:~# cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
root@debian-512mb-lon1-01:~# cd /etc/openvpn/ 
root@debian-512mb-lon1-01:/etc/openvpn# gunzip server.conf.gz

Using subnet, so setting the following in server.conf. Also changing the DH param size to 2K

dh dh2048.pem
push "redirect-gateway def1 bypass-dhcp" # Divert all traffic over the tunnel
user nobody # Drop privileges
group nobody 

Setting up easy-rsa
root@debian-512mb-lon1-01:/etc/openvpn# openssl dhparam -out dh2048.pem 2048
root@debian-512mb-lon1-01:/etc/openvpn# mkdir easy-rsa/keys -p
root@debian-512mb-lon1-01:/etc/openvpn# cd easy-rsa/
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp -r /usr/share/easy-rsa/* ./
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# nano vars # Setting the various defaults

KEY_SIZE was already 2048, so leaving that

Building the certs
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# source vars
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./clean-all
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-ca 
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server

Hit a slight issue here
Using configuration from /etc/openvpn/easy-rsa/openssl-1.0.0.cnf
/etc/openvpn/easy-rsa/keys/index.txt: No such file or directory
unable to open '/etc/openvpn/easy-rsa/keys/index.txt'
140002835130000:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/openvpn/easy-rsa/keys/index.txt','r')
140002835130000:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:

root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# > keys/index.txt 
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server

Using configuration from /etc/openvpn/easy-rsa/openssl-1.0.0.cnf
/etc/openvpn/easy-rsa/keys/serial: No such file or directory
error while loading serial number
140583156987536:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/openvpn/easy-rsa/keys/serial','r')
140583156987536:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:

root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# echo 01 > keys/serial
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server


Copying the keys/certs to the right place
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp keys/{server.crt,server.key,ca.crt} ..

Also going to set up HMAC
root@debian-512mb-lon1-01:/etc/openvpn# openvpn --genkey --secret ta.key

So, using the following config in server.conf
root@debian-512mb-lon1-01:/etc/openvpn# cat server.conf 
port 1194

;proto tcp # Not using, for now
proto udp

dev tun

ca ca.crt
cert server.crt
key server.key
dh dh2048.pem
tls-auth ta.key 0

ifconfig-pool-persist ipp.txt

push "redirect-gateway def1 bypass-dhcp"
keepalive 10 120


user nobody
group nogroup

status openvpn-status.log
verb 3

Setting up the firewall for masquerading and enabling forwarding
root@debian-512mb-lon1-01:/etc/openvpn# iptables -t nat -A POSTROUTING -s -o eth0 -j MASQUERADE
root@debian-512mb-lon1-01:/etc/openvpn# echo 1 > /proc/sys/net/ipv4/ip_forward
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf 

Restarting OpenVPN and making sure it starts at boot
root@debian-512mb-lon1-01:/etc/openvpn# service openvpn restart
root@debian-512mb-lon1-01:/etc/openvpn# update-rc.d openvpn defaults

Generating a client config so it can be tested
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# . vars 
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key lan1
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# mkdir /root/lan1
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp keys/{lan1.crt,lan1.key,ca.crt} /root/lan1/
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp ../ta.key /root/lan1/
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cat << EOM > /root/lan1/vpn.ovpn
dev tun
port 1194
proto udp
remote $MYIP 1194

ca ca.crt
cert lan1.crt
key lan1.key
tls-auth ta.key 1


root@milleniumfalcon:~/VPNS/lanbridge# openvpn vpn.ovpn
ben@milleniumfalcon:~/VPNS/lanbridge$ ip route get via dev tun0  src 

ben@milleniumfalcon:~/VPNS/lanbridge$ ping -c 3
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=60 time=12.7 ms
64 bytes from icmp_seq=2 ttl=60 time=12.9 ms
64 bytes from icmp_seq=3 ttl=60 time=13.1 ms

--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms

So the server's up and running (though I could do with adding some additional firewall rules to lock down access). For now, making sure the masquerade is restored on boot
root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# apt-get install iptables-persistent

Need to build the lan side concentrator next then.
btasker changed status from 'In Progress' to 'Open'
btasker changed timespent from '10 minutes' to '49 minutes'
Concentrator VM is currently spinning up and installing Debian 8, so in the meantime, creating firewall rules on the endpoint

root@debian-512mb-lon1-01:~# iptables -N sshin
root@debian-512mb-lon1-01:~# for src in $PERMSOURCES; do iptables -A sshin -s $src -j ACCEPT; done
root@debian-512mb-lon1-01:~# iptables -A sshin -j REJECT
root@debian-512mb-lon1-01:~# iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -i lo -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -p tcp --dport 22 -j sshin
root@debian-512mb-lon1-01:~# iptables -A INPUT -p udp --dport 1194 -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -p tcp --dport 80 -j ACCEPT
root@debian-512mb-lon1-01:~# iptables -A INPUT -j REJECT
root@debian-512mb-lon1-01:~# iptables-save > /etc/iptables/rules.v4

Leaving port 80 open as the intention is to have a default page up explaining that this IP is just a VPN endpoint. Plus I'll probably use it for status checks in the DNS load balancer

I've not enabled IPv6 on the droplet, and don't currently intend to, so have left ipv6 tables untouched (for now)
btasker changed timespent from '49 minutes' to '51 minutes'
As the intention is to have failover support between endpoints, I've created a new RR in DNS which'll point towards my DNS load balancer
vpnbridge.bentasker.co.uk.      CNAME   vpnbridge.balanced.bentasker.co.uk.

For now the balancer will only have the one entry in it. Until RR-7 is done I can't add the protocol VPN, so as "SSH" isn't in use anywhere else I'll use that for the time being
Have created the record in the balancer
router> find_cache vpnbridge.balanced.bentasker.co.uk

Query took 0.00398 seconds

Client:	 Location: g.us
Domain: vpnbridge.balanced.bentasker.co.uk
Required Protocols: p.ssh 

| Hostname                 | Address        | Rack     | Cost | Cost Src |
| vpn1.sys.bentasker.co.uk | | r.gb.lon | 100  | d        |

Seems to be working
ben@milleniumfalcon:~$ host vpnbridge.bentasker.co.uk
vpnbridge.bentasker.co.uk is an alias for vpnbridge.balanced.bentasker.co.uk.
vpnbridge.balanced.bentasker.co.uk has address
btasker changed timespent from '51 minutes' to '61 minutes'
Looks like the Debian install has completed, so I'll start setting the concentrator up shortly
btasker changed status from 'Open' to 'In Progress'
Installing Openvpn
root@vpn1:/home/ben# apt-get install openvpn

Moving config over and testing connectivity
root@vpn1:~# mv /tmp/lanbridge/ ./
root@vpn1:~/lanbridge# openvpn vpn.ovpn 
root@vpn1:~# ip route get via dev tun0  src 

So, next we want to set it up as a router so it'll forward packets on
root@vpn1:~# echo 1 > /proc/sys/net/ipv4/ip_forward
root@vpn1:~# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
root@vpn1:~# iptables -t nat -I POSTROUTING -j MASQUERADE

Quick test
root@milleniumfalcon:~# ip route add via
root@milleniumfalcon:~# ping -c 3
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=59 time=13.5 ms
64 bytes from icmp_seq=2 ttl=59 time=14.0 ms
64 bytes from icmp_seq=3 ttl=59 time=13.8 ms

--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 13.552/13.812/14.017/0.216 ms

root@milleniumfalcon:~# traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  0.523 ms  0.558 ms  0.591 ms
 2 (  12.540 ms  12.666 ms  12.961 ms

It's more or less ready to go, though need to set up a script to make sure that VPN's restarted if it breaks. Will re-use part of this script - https://github.com/bentasker/RaspbPiVPNRoutingMgr.git (originally from https://www.bentasker.co.uk/documentation/linux/260-usurping-the-bthomehub-with-a-raspberry-pi-part-4-using-a-vpn-to-tunnel-connections-to-specific-ips )
root@vpn1:~# echo "INSTDIR='/root/vpns'" > .vpn_config
root@vpn1:~# git clone https://github.com/bentasker/RaspbPiVPNRoutingMgr.git
root@vpn1:~/RaspbPiVPNRoutingMgr# mkdir config/VPNs
root@vpn1:~/RaspbPiVPNRoutingMgr# cp -r ../lanbridge/ config/VPNs/
root@vpn1:~/RaspbPiVPNRoutingMgr# mv config/VPNs/lanbridge/vpn.ovpn config/VPNs/landbridge/lanbridge.conf
root@vpn1:~/RaspbPiVPNRoutingMgr# echo "daemon" >> config/VPNs/landbridge/lanbridge.conf 
root@vpn1:~/RaspbPiVPNRoutingMgr# echo lanbridge > config/VPNS
root@vpn1:~/RaspbPiVPNRoutingMgr# echo "iptables -t nat -I POSTROUTING -j MASQUERADE" >> config/firewall 
root@vpn1:~# mv RaspbPiVPNRoutingMgr/ vpns
root@vpn1:~# /root/vpns/scripts/monitor_vpns.sh

Then running again to check it doesn't try and start a new process
root@vpn1:~# /root/vpns/scripts/monitor_vpns.sh 
# Adding a custom route, just as an example
root@vpn1:~# echo "route add gw" >> vpns/config/routes 

Cronning it up
root@vpn1:~# echo "* * * * * root /root/vpns/scripts/monitor_vpns.sh" > /etc/cron.d/vpn-tunnels

Should be good to go
btasker changed status from 'In Progress' to 'Open'
btasker changed timespent from '61 minutes' to '82 minutes'
For obvious reasons I don't want to just cut-over to using it for all traffic, so will start by creating a policy to redirect my desktop via the new gateway. Assuming that works I'll then gradually move stuff over.
As it's currently running on a Virtualbox host, I've configured it to start at boot (in case of power interruptions)
$ vboxmanage modifyvm "VPN Concentrator" --autostart-enabled on
btasker changed status from 'Open' to 'In Progress'
OK, next up is creating some policies on the firewall to direct traffic via the new gateway

ben@milleniumfalcon:~$ traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  1.525 ms  1.652 ms  1.750 ms
 2 (  13.709 ms  13.828 ms  14.178 ms
 3 (  14.559 ms (  14.291 ms (  14.651 ms

Looks good so far then.

Access to the stuff VPN'd from the main router still works as well.

Next step, I guess, will be to look at configuring the transparent proxy to transit the link. That'll knock a whole lot of cleartext off what the ISP can see in one hit.
That was fairly simple to set up, the proxy is now transiting the VPN.

Need to run a PCAP to check for certain, but there should now be no port 80 traffic hitting the WAN interface at all.
btasker changed status from 'In Progress' to 'Open'
btasker changed timespent from '82 minutes' to '99 minutes'
So the PBR in place on the firewall's LAN interface can be summarised as follows

- If TCP and dst port == 80, GW TransparentProxy
- if src == mydesktop, GW VPNConcentrator

On the DMZ interface (where the proxy lives)

- If src == TransparentProxy, GW VPNConcentrator

So port 80 connections from my desktop still hit the cache first, but everything else will be directed via the VPN.

Going to leave a PCAP running for about 15 minutes (whilst the network is fairly busy) to confirm. Did a quick speed test though, throuhput is reasonable
ben@milleniumfalcon:~$ wget -O /dev/null --header="Host: speedtest.london.linode.com"
--2016-11-20 15:52:37--
Connecting to connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: ¬'/dev/null¬'

100%[===================================================================================================================>] 104,857,600 4.67MB/s   in 20s    

2016-11-20 15:52:58 (4.89 MB/s) - ¬'/dev/null¬' saved [104857600/104857600]

So that's 39 megabits. Still a bit of a drop compared to what I'm used to.

Running a quick test from the VPN concentrator itself, it looks like it's a product of going through the cache though, as I get about 50Mb/s from there (still slightly slower).

Not a big enough drop to be too inconvenient, and haven't really done anything to tune the connection speed yet, so will leave for the time being.
Looking at the PCAP the only port 80 traffic I see is an incoming connection (coincidental timing)
ben@milleniumfalcon:/tmp/misc19$ tcpdump -r misc19.pcap port 80
reading from file misc19.pcap, link-type EN10MB (Ethernet)
15:59:17.202920 IP tor-limits-scanning.cl.cam.ac.uk.39250 > Flags [S], seq 2819030485, win 65535, options [mss 1452], length 0
15:59:17.205613 IP > tor-limits-scanning.cl.cam.ac.uk.39250: Flags [S.], seq 2671805212, ack 2819030486, win 29200, options [mss 1460], length 0
15:59:17.218515 IP tor-limits-scanning.cl.cam.ac.uk.39250 > Flags [R], seq 2819030486, win 0, length 0
15:59:17.295404 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [S], seq 1392928851, win 29200, options [mss 1452,sackOK,TS val 43886324 ecr 0,nop,wscale 9], length 0
15:59:17.298334 IP > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [S.], seq 2757475325, ack 1392928852, win 28960, options [mss 1460,sackOK,TS val 611594553 ecr 43886324,nop,wscale 6], length 0
15:59:17.310518 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [.], ack 1, win 58, options [nop,nop,TS val 43886327 ecr 611594553], length 0
15:59:17.310567 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [P.], seq 1:98, ack 1, win 58, options [nop,nop,TS val 43886327 ecr 611594553], length 97
15:59:17.312692 IP > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [.], ack 98, win 453, options [nop,nop,TS val 611594556 ecr 43886327], length 0
15:59:17.382037 IP > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [P.], seq 1:585, ack 98, win 453, options [nop,nop,TS val 611594574 ecr 43886327], length 584
15:59:17.393760 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [.], ack 585, win 60, options [nop,nop,TS val 43886348 ecr 611594574], length 0
15:59:17.394080 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [F.], seq 98, ack 585, win 60, options [nop,nop,TS val 43886348 ecr 611594574], length 0
15:59:17.396788 IP > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [F.], seq 585, ack 99, win 453, options [nop,nop,TS val 611594577 ecr 43886348], length 0
15:59:17.408632 IP tor-limits-scanning.cl.cam.ac.uk.48790 > Flags [.], ack 586, win 60, options [nop,nop,TS val 43886352 ecr 611594577], length 0
15:59:56.666643 IP tor-limits-scanning.cl.cam.ac.uk.39250 > Flags [S], seq 2819030485, win 65535, options [mss 1452], length 0
15:59:56.669796 IP > tor-limits-scanning.cl.cam.ac.uk.39250: Flags [S.], seq 3288439354, ack 2819030486, win 29200, options [mss 1460], length 0
15:59:56.681477 IP tor-limits-scanning.cl.cam.ac.uk.39250 > Flags [R], seq 2819030486, win 0, length 0

But nothing outgoing despite having generated rather a lot of traffic. So that's good.

Before I can look at cutting everything over I need to create some policies so that there are exceptions

- Existing VPNs from the main router should just go straight out
- Connections from the concentrator to the endpoint obviously need to go straight out

There might be other exceptions I want to make too, but that's a starting point.

Also need to spin up a second endpoint so I can do some failover testing. That'll mean taking the endpoint down, so better to do first.
btasker changed timespent from '99 minutes' to '104 minutes'
OK, second endpoint spun up by creating a snapshot of the first. I've added it to the load balancer and created a DNS name for it.

I've stopped openvpn on the first gateway, want to see whether it'll re-resolve and then try on the second (though the load balancer will still return both IP's as I haven't set up the status checks yet).

Doesn't seem to have done yet. There would only have been one IP when it last resolve it though, so killing the process and seeing what happens now.

Doesn't look like it does a DNS round-robin then, will need to get the status checks up and running. For now (as a test) I'll configure a status check on the load balancer for the first endpoint - should have the result of knocking it out of selection
router> find_cache vpnbridge.balanced.bentasker.co.uk

Query took 0.00452 seconds

Client:	 Location: g.us
Domain: vpnbridge.balanced.bentasker.co.uk
Required Protocols: p.ssh 

| Hostname                 | Address        | Rack        | Cost | Cost Src |
| vpn2.sys.bentasker.co.uk | | r.gb.vpnlon | 100  | d        |

So lets see whether the VPN switches over to trying to connect to the other.

No, y'know what, I'm an idiot. I forgot to update the config to use the DNS name. Possible round-robin does work then, will try that again shortly, will make sure the connection to ep2 works first.

OK, it's connected to ep2, so going to put ep1 back into selection, stop openvpn on ep2 and see what happens
OK, it's just failed over, took around a minute to do so, which lines up with the settings I configured in the client config.
ping 15
ping-restart 60
resolv-retry 60
Setting up NGinx so I can enable status checks

cat << EOM >> /etc/apt/sources.list
deb http://nginx.org/packages/debian/ jessie nginx
deb-src http://nginx.org/packages/debian/ jessie nginx

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ABF5BD827BD9BF62
apt-get update
apt-get install nginx

mkdir /usr/share/nginx/status

cat << EOM > /etc/nginx/conf.d/status.conf
server {
        listen 80;
        listen   [::]:80; 

        root /usr/share/nginx/status; 
        index index.php index.html index.htm;

        server_name status.self;

        add_header X-Clacks-Overhead "GNU Terry Pratchett";


cat << EOM > /usr/share/nginx/html/index.html
<title>VPN Endpoint</title>
<p>This node is a VPN endpoint. Traffic you see as originating from here probably didn't.</p>
<p>If you consider traffic problematic, please visit <a href="https://www.bentasker.co.uk/contact-me">Contact-Me</a> to report it (reports with packet captures and/or loglines as proof are preferred)</p>

root@debian-512mb-lon1-01:~# service nginx restart

Creating the status check script
root@debian-512mb-lon1-01:~# apt-get install python-psutil
cat << EOM > /root/status.py
#!/usr/bin/env python


import os,json,psutil

response = {
    "status" : "Online",
    "loadavg" : '',
    "processors" : 0,
    "loadperc" : 0,
    "loadperc5" : 0,
    "loadperc15" : 0


f = open('/proc/loadavg', 'r')

f = open('/proc/cpuinfo', 'r')

response['processors'] = cpucount.count('processor\t')

# Calculate the loadpercentage based on number of processors
loads = response['loadavg'].split(' ')

# Now
response['loadperc'] = round(float(loads[0]) / response['processors'],2)
# 5 min
response['loadperc5'] = round(float(loads[1]) / response['processors'],2)
# 15 min
response['loadperc15'] = round(float(loads[2]) / response['processors'],2)

for pid in psutil.pids():
    p = psutil.Process(pid)
    if p.name() == "openvpn":

if s == 0:
    response['status'] = "Offline"

if os.path.exists(OFFLINE_FILE):
    response['status'] = "Offline"

print json.dumps(response)

root@debian-512mb-lon1-01:~# chmod +x /root/status.py

root@debian-512mb-lon1-01:~# /root/status.py 
{"status": "Online", "processors": 1, "loadperc": 0.0, "loadperc15": 0.02, "loadavg": "0.00 0.01 0.02 1/69 1500\n", "loadperc5": 0.01}
root@debian-512mb-lon1-01:~# service openvpn stop
root@debian-512mb-lon1-01:~# /root/status.py 
{"status": "Offline", "processors": 1, "loadperc": 0.0, "loadperc15": 0.02, "loadavg": "0.00 0.01 0.02 1/72 1527\n", "loadperc5": 0.01}

Setting up a cronjob
root@debian-512mb-lon1-01:~# echo "* * * * * root /root/status.py > /usr/share/nginx/status/status.json" >> /etc/cron.d/status-checks

Have added the status pages to the load balancer
btasker changed timespent from '104 minutes' to '114 minutes'
OK, I've created some whitelists (probably in need of tweaking a bit over time) and cut all other traffic over to using the tunnel.

Going to set a monitoring pcap running on the WAN interface, to keep size down I'll exclude the VPN traffic.
# tcpdump -i eth0 -s0 -w misc19.pcap -v not port 1194

Will come back and analyse that later
Current state should be that all non-vpn (and non-tor) IPv4 traffic now transits the VPN.

IPv6 traffic will still hit the helium tunnelled connection as I haven't quite figured out exactly how I want to address that yet - supporting failover is going to be a little more tricky unless I use NAT64 (which will probably be the only option, in truth).

Most clients on the LAN aren't given an IPv6 address, so it won't affect the majority of the traffic anyway.

Using the VPN does mean that we won't see the benefit of any ISP level caches, but it's not the end of the world.

Looking at an analysis of the capture so far, the only traffic I'm seeing is one of the existing VPNs (which doesn't use port 1194) and the Helium IPv6 tunnel. Bodes well
btasker changed timespent from '114 minutes' to '119 minutes'
Hmmm, so first headache. Said earlier I'd prefer a UK based endpoint if possible to avoid geolocking issues, but it seems that the BBC have decided they don't like datacentre IPs. Not overly surprising (given they're paranoid about VPNs) but as a licensepayer I'm obviously a little indignant :)

I've added a new policy to allow the iplayer box to go out without transiting the VPN, though it comes at the cost of leaking anything else I stream too.

I've a suspicion they only do the geo-location stuff when fetching the manifest though, so it might be possible to work around by setting up a reverse proxy (that'll go out without transiting the VPN) and letting the stream use the VPN.
Yup, adding a reverse proxy for open.live.bbc.co.uk fixed it. I'm sure they'll break that at some point, but at least means I can watch Planet Earth later.
So, the base requirements are all satisfied to some extent

- The solution supports failover.
- Responses to inbound connections do not transit the link
- Policies control what can and cannot egress the network
- Port 80 traffic still goes via the transparent proxy and then transits the VPN link
- The system fails closed
- Iplayer still works (albeit with a bit of hacking)

However, it's worth nothing that both endpoints are currently in the same datacentre, so there's no geographic redundancy. I will look at addressing that, but as it's almost trivial to spin up a new endpoint now I may wait until the government get around to defining what's classed as a CSP.

I think I'll probably raise a separate issue to track getting DNSCrypt set up as this ones getting quite large.
The VPN now tunnel IPv6 as well

Was fairly simple to set up.

I've got a routed /48 so using a /64 from that inside the tunnel. Edited the OpenVPN config to include

server-ipv6 2001:470:69d7:4de::/64
push tun-ipv6
ifconfig-ipv6 2001:470:69d7:4de::1 2001:470:69d7:4de::2
push "route-ipv6 2000::/3"

As the VPN is load balanced, I either need NAT or to have the LAN addresses update whenever the tunnel fails over. The former is simpler, so NAT it is
Enabling NAT and forwarding at both ends
ip6tables -t nat -I POSTROUTING -j MASQUERADE

Enable IPv6 forwarding
echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf 
sysctl -p

Added policies to direct IPv6 traffic via the VPN concentrator on the LAN. Job done
The implementation has been working and stable for well over 6 months now. So closing this issue off.
btasker changed status from 'Open' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'
btasker removed 'Done' from resolution
btasker changed status from 'Closed' to 'Reopened'
btasker changed status from 'Reopened' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'
btasker removed 'Done' from resolution
btasker changed status from 'Closed' to 'Reopened'
btasker changed status from 'Reopened' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'
btasker removed 'Done' from resolution
btasker changed status from 'Closed' to 'Reopened'
btasker changed status from 'Reopened' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'

Work log

Ben Tasker
2016-11-17 19:57:41

Time Spent: 10 minutes
Log Entry: Mapping out architecture

Ben Tasker
2016-11-20 14:11:53

Time Spent: 39 minutes
Log Entry: Setting up and testing endpoint

Ben Tasker
2016-11-20 14:22:39

Time Spent: 2 minutes
Log Entry: Setting up iptables

Ben Tasker
2016-11-20 14:36:18

Time Spent: 10 minutes
Log Entry: Setting up DNS ready for failover support

Ben Tasker
2016-11-20 15:14:50

Time Spent: 21 minutes
Log Entry: Setting up LAN side VPN Concentrator

Ben Tasker
2016-11-20 15:45:06

Time Spent: 17 minutes
Log Entry: Setting up initial policy based routing

Ben Tasker
2016-11-20 16:12:29

Time Spent: 5 minutes
Log Entry: PCAP analysis

Ben Tasker
2016-11-20 17:31:41

Time Spent: 10 minutes
Log Entry: Setting up status pages

Ben Tasker
2016-11-20 18:01:39

Time Spent: 5 minutes
Log Entry: Pcap analysis