########################################################################################## MISC-19: Design Investigatory Powers Act workarounds ########################################################################################## Issue Type: Task ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Done (2019-12-10 15:50:15) Project: Miscellaneous (MISC) Reported By: btasker Assigned To: btasker Affected Versions: - Draft IPB Reading Targeted for fix in version: - Draft IPB Reading Labels: IPB, Privacy, Routing, Security, VPN, Time Estimate: 1 minutes Time Logged: 119 minutes ----------------------------------------------------------------------------------------- Issue Description ================== The fuckwits in Westminister have signed off on the Investigatory Powers Bill, including all the privacy-invasive provisions such as ICRs. The bill is simply awaiting Royal Assent now. Someone on Ars has helpfully listed the agencies that will be able to request access (WITHOUT a warrant) to Internet Connection Records -- BEGIN SNIPPET -- Air Accidents Investigation Branch Charity Commission Commission for Healthcare Audit and Inspection Commissioners of Revenue and Customs Common Services Agency for the Scottish Health Service Criminal Cases Review Commission Department for Environment, Food and Rural Affairs (for the purposes of the Marine Fisheries Agency) Department for Transport (for the purposes of transport security, Vehicle and Operator Services Agency, Driving Standards Agency and Maritime and Coastguard Agency) Department for Work and Pensions Department of Agriculture and Rural Development for Northern Ireland Department of Enterprise, Trade and Investment for Northern Ireland (for the purposes of Trading Standards) Department of Health (for the purposes of the Medicines and Healthcare Products Regulatory Agency) Environment Agency Financial Services Authority Fire Authority for Northern Ireland Food Standards Agency Gambling Commission Gangmasters Licensing Authority General Pharmaceutical Council Government Communications Headquarters Health & Safety Executive Her Majesty’s Chief Inspector of Schools in England HM Revenue and Customs Home Office (for the purposes of HM Prison Service and the UK Border Agency) Information Commissioner Marine Accident Investigation Branch Maritime and Coastguard Agency Ministry of Defence NHS ambulance service Trust NHS Counter Fraud and Security Management Service Northern Ireland Ambulance Service Health and Social Services Trust Northern Ireland Health and Social Services Central Services Agency Northern Ireland Office (for the purposes of the Northern Ireland Prison Service) Ofcom Office of Fair Trading Office of the Deputy Prime Minister Office of the Police Ombudsman for Northern Ireland Port of Dover Police Port of Liverpool Police Post Office Investigation Branch Postal Services Commission Rail Accident Investigation Branch Royal Air Force Police Royal Military Police Royal Navy Regulating Branch Scottish Ambulance Service Board Scottish Environment Protection Agency Secret Intelligence Service Security Service Serious Fraud Office The Armed Forces The Pensions Regulator Special Police Forces (including the Scottish Drug Enforcement Agency) Territorial Police Forces Welsh Ambulance Services NHS Trust Welsh Government (for the purposes of the NHS Directorate, NHS Finance Division, Common Agricultural Policy Management Division and Care Standards Inspectorate for Wales) -- END SNIPPET -- The Government, quite frankly, can go fuck itself. I cannot, in good conscience, allow my family's data to be recorded on a list that will, inevitably, be misused/leaked. Managed by incompetent ISPs and scope and access "controlled" by bungling politicians. No thanks. Raising this issue to start the process of moving all relevant traffic over to using a VPN and/or Tor. There are a number of requirements - The solution must support failover, so if an endpoint goes down, another should be used - Responses to inbound connections should not transit the link - Policies should be used to control what can and cannot egress the network (for example, the current VPN links should be allowed to leave as they do now) - Port 80 traffic should still go via the transparent proxy and then transit the VPN link - The system should fail closed. If all endpoints are unreachable, block all traffic (but provide me a way to override that for troubleshooting) Most DNS queries already leave via a VPN, but this may also be a good opportunity to look at setting up DNSCrypt Once the provisional infrastructure is in place, I'll run an audit with PAS to check that egress traffic is as expected Use of a VPN won't be enough to stop a targeted investigation, but that's explicitly not the aim of this issue. The aim is purely to ensure that the ISP held Internet Connection Records for my connection don't contain anything of relevance. ----------------------------------------------------------------------------------------- Issue Relations ================ - relates to MISC-11: Review Draft Investigatory Powers Bill - relates to LAN-69: IPv6 traffic is easily observable and bypasses VPNs - relates to LAN-127: Revisit LAN IPv6 support - Gov Factsheet on ICRs (https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/473745/Factsheet-Internet_Connection_Records.pdf) - PCAP Analyse and Report (https://github.com/bentasker/PCAPAnalyseandReport) ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2016-11-17 19:50:07 btasker ----------------------------------------------------------------------------------------- Whether or not the VPN endpoints can be in the UK is going to depend on the Government actually defining who they consider to be a "Communication Service Provider". If possible, I'd prefer it was UK based to avoid geo-blocking headaches For example, France have a similar law on the books, but it only applies to ISPs that have residential customers, so a number of datacentres don't (officially at least) have the capability deployed. An alternate approach, which may be suitable for some of the traffic, is to run a transparent Tor instance and configure it only to use exit nodes in the UK -- BEGIN SNIPPET -- ExitNodes {GB} -- END SNIPPET -- There'll be a few headaches there though with some services blocking known exit nodes ----------------------------------------------------------------------------------------- 2016-11-17 19:57:10 btasker ----------------------------------------------------------------------------------------- For the initial development/testing, I'll spin up a couple of DigitalOcean droplets in London to act as the endpoints. To begin with I'll use DNS round-robin for failover purposes and set something a little more intelligent up later. ----------------------------------------------------------------------------------------- 2016-11-17 19:57:41 ----------------------------------------------------------------------------------------- btasker changed timespent from '0 minutes' to '10 minutes' ----------------------------------------------------------------------------------------- 2016-11-17 20:28:49 btasker ----------------------------------------------------------------------------------------- Given the Bulk Equipment Interference (funny how it's just "hacking" when it's anyone else) provisions, also a good time to review the IDS setup and see if there are any improvements that can be made. ----------------------------------------------------------------------------------------- 2016-11-17 20:30:13 btasker ----------------------------------------------------------------------------------------- Will raise a separate internal issue for it later, but need to keep a close eye on what they consider to be a CSP to see if any of the services I offer are likely to be impacted. Already part way there, but it may also be worth accelerating the project to reduce the amount of data I'm actually able to capture in the first place. ----------------------------------------------------------------------------------------- 2016-11-17 20:30:39 ----------------------------------------------------------------------------------------- btasker added 'IPB Privacy Routing Security VPN' to labels ----------------------------------------------------------------------------------------- 2016-11-20 13:21:21 btasker ----------------------------------------------------------------------------------------- For reference, DigitalOcean don't give exact link speeds for Droplets, only that the hypervisor has multiple 1GB nics in a bond. They say you should plan on being able to achieve around 300Mbps of throughput, which should be more than sufficient. ----------------------------------------------------------------------------------------- 2016-11-20 13:31:57 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'In Progress' ----------------------------------------------------------------------------------------- 2016-11-20 14:11:18 btasker ----------------------------------------------------------------------------------------- OK, so endpoint setup. 512Mb droplet running Debian 8 Essentially following the steps here - https://www.bentasker.co.uk/documentation/linux/259-openvpn-on-debian - though the easy-rsa steps are different on Jessie -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:~# apt-get update root@debian-512mb-lon1-01:~# apt-get install openvpn easy-rsa root@debian-512mb-lon1-01:~# cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/ root@debian-512mb-lon1-01:~# cd /etc/openvpn/ root@debian-512mb-lon1-01:/etc/openvpn# gunzip server.conf.gz -- END SNIPPET -- Using subnet 10.17.0.0/24, so setting the following in server.conf. Also changing the DH param size to 2K -- BEGIN SNIPPET -- server 10.16.0.0 255.255.255.0 dh dh2048.pem push "redirect-gateway def1 bypass-dhcp" # Divert all traffic over the tunnel user nobody # Drop privileges group nobody -- END SNIPPET -- Setting up easy-rsa -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn# openssl dhparam -out dh2048.pem 2048 root@debian-512mb-lon1-01:/etc/openvpn# mkdir easy-rsa/keys -p root@debian-512mb-lon1-01:/etc/openvpn# cd easy-rsa/ root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp -r /usr/share/easy-rsa/* ./ root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# nano vars # Setting the various defaults -- END SNIPPET -- KEY_SIZE was already 2048, so leaving that Building the certs -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# source vars root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./clean-all root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-ca root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server -- END SNIPPET -- Hit a slight issue here -- BEGIN SNIPPET -- Using configuration from /etc/openvpn/easy-rsa/openssl-1.0.0.cnf /etc/openvpn/easy-rsa/keys/index.txt: No such file or directory unable to open '/etc/openvpn/easy-rsa/keys/index.txt' 140002835130000:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/openvpn/easy-rsa/keys/index.txt','r') 140002835130000:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400: root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# > keys/index.txt root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server Using configuration from /etc/openvpn/easy-rsa/openssl-1.0.0.cnf /etc/openvpn/easy-rsa/keys/serial: No such file or directory error while loading serial number 140583156987536:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('/etc/openvpn/easy-rsa/keys/serial','r') 140583156987536:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400: root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# echo 01 > keys/serial root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key-server server -- END SNIPPET -- Sorted. Copying the keys/certs to the right place -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp keys/{server.crt,server.key,ca.crt} .. -- END SNIPPET -- Also going to set up HMAC -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn# openvpn --genkey --secret ta.key -- END SNIPPET -- So, using the following config in server.conf -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn# cat server.conf port 1194 ;proto tcp # Not using, for now proto udp dev tun ca ca.crt cert server.crt key server.key dh dh2048.pem tls-auth ta.key 0 server 10.17.0.0 255.255.255.0 ifconfig-pool-persist ipp.txt push "redirect-gateway def1 bypass-dhcp" keepalive 10 120 comp-lzo user nobody group nogroup persist-key persist-tun status openvpn-status.log verb 3 -- END SNIPPET -- Setting up the firewall for masquerading and enabling forwarding -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn# iptables -t nat -A POSTROUTING -s 10.17.0.0/24 -o eth0 -j MASQUERADE root@debian-512mb-lon1-01:/etc/openvpn# echo 1 > /proc/sys/net/ipv4/ip_forward root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf -- END SNIPPET -- Restarting OpenVPN and making sure it starts at boot -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn# service openvpn restart root@debian-512mb-lon1-01:/etc/openvpn# update-rc.d openvpn defaults -- END SNIPPET -- Generating a client config so it can be tested -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# . vars root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# ./build-key lan1 root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# mkdir /root/lan1 root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp keys/{lan1.crt,lan1.key,ca.crt} /root/lan1/ root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cp ../ta.key /root/lan1/ root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# cat << EOM > /root/lan1/vpn.ovpn client dev tun port 1194 proto udp remote $MYIP 1194 nobind ca ca.crt cert lan1.crt key lan1.key tls-auth ta.key 1 comp-lzo persist-key persist-tun EOM -- END SNIPPET -- Testing -- BEGIN SNIPPET -- root@milleniumfalcon:~/VPNS/lanbridge# openvpn vpn.ovpn ben@milleniumfalcon:~/VPNS/lanbridge$ ip route get 8.8.8.8 8.8.8.8 via 10.17.0.5 dev tun0 src 10.17.0.6 cache ben@milleniumfalcon:~/VPNS/lanbridge$ ping -c 3 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=60 time=12.7 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=60 time=12.9 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=60 time=13.1 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms -- END SNIPPET -- So the server's up and running (though I could do with adding some additional firewall rules to lock down access). For now, making sure the masquerade is restored on boot -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:/etc/openvpn/easy-rsa# apt-get install iptables-persistent -- END SNIPPET -- Need to build the lan side concentrator next then. ----------------------------------------------------------------------------------------- 2016-11-20 14:11:30 ----------------------------------------------------------------------------------------- btasker changed status from 'In Progress' to 'Open' ----------------------------------------------------------------------------------------- 2016-11-20 14:11:53 ----------------------------------------------------------------------------------------- btasker changed timespent from '10 minutes' to '49 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 14:22:30 btasker ----------------------------------------------------------------------------------------- Concentrator VM is currently spinning up and installing Debian 8, so in the meantime, creating firewall rules on the endpoint -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:~# iptables -N sshin root@debian-512mb-lon1-01:~# for src in $PERMSOURCES; do iptables -A sshin -s $src -j ACCEPT; done root@debian-512mb-lon1-01:~# iptables -A sshin -j REJECT root@debian-512mb-lon1-01:~# iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -i lo -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -p tcp --dport 22 -j sshin root@debian-512mb-lon1-01:~# iptables -A INPUT -p udp --dport 1194 -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -p tcp --dport 80 -j ACCEPT root@debian-512mb-lon1-01:~# iptables -A INPUT -j REJECT root@debian-512mb-lon1-01:~# iptables-save > /etc/iptables/rules.v4 -- END SNIPPET -- Leaving port 80 open as the intention is to have a default page up explaining that this IP is just a VPN endpoint. Plus I'll probably use it for status checks in the DNS load balancer I've not enabled IPv6 on the droplet, and don't currently intend to, so have left ipv6 tables untouched (for now) ----------------------------------------------------------------------------------------- 2016-11-20 14:22:39 ----------------------------------------------------------------------------------------- btasker changed timespent from '49 minutes' to '51 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 14:27:53 btasker ----------------------------------------------------------------------------------------- As the intention is to have failover support between endpoints, I've created a new RR in DNS which'll point towards my DNS load balancer -- BEGIN SNIPPET -- vpnbridge.bentasker.co.uk. CNAME vpnbridge.balanced.bentasker.co.uk. -- END SNIPPET -- For now the balancer will only have the one entry in it. Until RR-7 is done I can't add the protocol VPN, so as "SSH" isn't in use anywhere else I'll use that for the time being ----------------------------------------------------------------------------------------- 2016-11-20 14:32:46 btasker ----------------------------------------------------------------------------------------- Have created the record in the balancer -- BEGIN SNIPPET -- router> find_cache vpnbridge.balanced.bentasker.co.uk Query took 0.00398 seconds Client: 8.8.8.8 Location: g.us Domain: vpnbridge.balanced.bentasker.co.uk Required Protocols: p.ssh | Hostname | Address | Rack | Cost | Cost Src | |------------------------------------------------------------------------| | vpn1.sys.bentasker.co.uk | 178.62.118.116 | r.gb.lon | 100 | d | -------------------------------------------------------------------------- -- END SNIPPET -- Seems to be working -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ host vpnbridge.bentasker.co.uk vpnbridge.bentasker.co.uk is an alias for vpnbridge.balanced.bentasker.co.uk. vpnbridge.balanced.bentasker.co.uk has address 178.62.118.116 -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2016-11-20 14:36:18 ----------------------------------------------------------------------------------------- btasker changed timespent from '51 minutes' to '61 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 14:36:50 btasker ----------------------------------------------------------------------------------------- Looks like the Debian install has completed, so I'll start setting the concentrator up shortly ----------------------------------------------------------------------------------------- 2016-11-20 14:53:03 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'In Progress' ----------------------------------------------------------------------------------------- 2016-11-20 15:14:22 btasker ----------------------------------------------------------------------------------------- Installing Openvpn -- BEGIN SNIPPET -- root@vpn1:/home/ben# apt-get install openvpn -- END SNIPPET -- Moving config over and testing connectivity -- BEGIN SNIPPET -- root@vpn1:~# mv /tmp/lanbridge/ ./ root@vpn1:~/lanbridge# openvpn vpn.ovpn root@vpn1:~# ip route get 8.8.8.8 8.8.8.8 via 10.17.0.5 dev tun0 src 10.17.0.6 cache -- END SNIPPET -- So, next we want to set it up as a router so it'll forward packets on -- BEGIN SNIPPET -- root@vpn1:~# echo 1 > /proc/sys/net/ipv4/ip_forward root@vpn1:~# echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf root@vpn1:~# iptables -t nat -I POSTROUTING -j MASQUERADE -- END SNIPPET -- Quick test -- BEGIN SNIPPET -- root@milleniumfalcon:~# ip route add 8.8.8.8 via 192.168.1.218 root@milleniumfalcon:~# ping -c 3 8.8.8.8 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=59 time=13.5 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=59 time=14.0 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=59 time=13.8 ms --- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 13.552/13.812/14.017/0.216 ms root@milleniumfalcon:~# traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1 192.168.1.218 (192.168.1.218) 0.523 ms 0.558 ms 0.591 ms 2 10.17.0.1 (10.17.0.1) 12.540 ms 12.666 ms 12.961 ms ...snip... -- END SNIPPET -- It's more or less ready to go, though need to set up a script to make sure that VPN's restarted if it breaks. Will re-use part of this script - https://github.com/bentasker/RaspbPiVPNRoutingMgr.git (originally from https://www.bentasker.co.uk/documentation/linux/260-usurping-the-bthomehub-with-a-raspberry-pi-part-4-using-a-vpn-to-tunnel-connections-to-specific-ips ) -- BEGIN SNIPPET -- root@vpn1:~# echo "INSTDIR='/root/vpns'" > .vpn_config root@vpn1:~# git clone https://github.com/bentasker/RaspbPiVPNRoutingMgr.git root@vpn1:~/RaspbPiVPNRoutingMgr# mkdir config/VPNs root@vpn1:~/RaspbPiVPNRoutingMgr# cp -r ../lanbridge/ config/VPNs/ root@vpn1:~/RaspbPiVPNRoutingMgr# mv config/VPNs/lanbridge/vpn.ovpn config/VPNs/landbridge/lanbridge.conf root@vpn1:~/RaspbPiVPNRoutingMgr# echo "daemon" >> config/VPNs/landbridge/lanbridge.conf root@vpn1:~/RaspbPiVPNRoutingMgr# echo lanbridge > config/VPNS root@vpn1:~/RaspbPiVPNRoutingMgr# echo "iptables -t nat -I POSTROUTING -j MASQUERADE" >> config/firewall root@vpn1:~# mv RaspbPiVPNRoutingMgr/ vpns root@vpn1:~# /root/vpns/scripts/monitor_vpns.sh -- END SNIPPET -- Then running again to check it doesn't try and start a new process -- BEGIN SNIPPET -- root@vpn1:~# /root/vpns/scripts/monitor_vpns.sh root@vpn1:~# # Adding a custom route, just as an example root@vpn1:~# echo "route add 192.168.4.0 gw 192.168.1.250" >> vpns/config/routes -- END SNIPPET -- Cronning it up -- BEGIN SNIPPET -- root@vpn1:~# echo "* * * * * root /root/vpns/scripts/monitor_vpns.sh" > /etc/cron.d/vpn-tunnels -- END SNIPPET -- Should be good to go ----------------------------------------------------------------------------------------- 2016-11-20 15:14:25 ----------------------------------------------------------------------------------------- btasker changed status from 'In Progress' to 'Open' ----------------------------------------------------------------------------------------- 2016-11-20 15:14:50 ----------------------------------------------------------------------------------------- btasker changed timespent from '61 minutes' to '82 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 15:16:33 btasker ----------------------------------------------------------------------------------------- For obvious reasons I don't want to just cut-over to using it for all traffic, so will start by creating a policy to redirect my desktop via the new gateway. Assuming that works I'll then gradually move stuff over. ----------------------------------------------------------------------------------------- 2016-11-20 15:18:52 btasker ----------------------------------------------------------------------------------------- As it's currently running on a Virtualbox host, I've configured it to start at boot (in case of power interruptions) -- BEGIN SNIPPET -- $ vboxmanage modifyvm "VPN Concentrator" --autostart-enabled on -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2016-11-20 15:26:50 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'In Progress' ----------------------------------------------------------------------------------------- 2016-11-20 15:38:56 btasker ----------------------------------------------------------------------------------------- OK, next up is creating some policies on the firewall to direct traffic via the new gateway Testing -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ traceroute 51.255.232.237 traceroute to 51.255.232.237 (51.255.232.237), 30 hops max, 60 byte packets 1 192.168.1.218 (192.168.1.218) 1.525 ms 1.652 ms 1.750 ms 2 10.17.0.1 (10.17.0.1) 13.709 ms 13.828 ms 14.178 ms 3 46.101.0.254 (46.101.0.254) 14.559 ms 46.101.0.253 (46.101.0.253) 14.291 ms 46.101.0.254 (46.101.0.254) 14.651 ms -- END SNIPPET -- Looks good so far then. Access to the stuff VPN'd from the main router still works as well. Next step, I guess, will be to look at configuring the transparent proxy to transit the link. That'll knock a whole lot of cleartext off what the ISP can see in one hit. ----------------------------------------------------------------------------------------- 2016-11-20 15:44:11 btasker ----------------------------------------------------------------------------------------- That was fairly simple to set up, the proxy is now transiting the VPN. Need to run a PCAP to check for certain, but there should now be no port 80 traffic hitting the WAN interface at all. ----------------------------------------------------------------------------------------- 2016-11-20 15:44:30 ----------------------------------------------------------------------------------------- btasker changed status from 'In Progress' to 'Open' ----------------------------------------------------------------------------------------- 2016-11-20 15:45:06 ----------------------------------------------------------------------------------------- btasker changed timespent from '82 minutes' to '99 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 15:58:59 btasker ----------------------------------------------------------------------------------------- So the PBR in place on the firewall's LAN interface can be summarised as follows - If TCP and dst port == 80, GW TransparentProxy - if src == mydesktop, GW VPNConcentrator On the DMZ interface (where the proxy lives) - If src == TransparentProxy, GW VPNConcentrator So port 80 connections from my desktop still hit the cache first, but everything else will be directed via the VPN. Going to leave a PCAP running for about 15 minutes (whilst the network is fairly busy) to confirm. Did a quick speed test though, throuhput is reasonable -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ wget -O /dev/null --header="Host: speedtest.london.linode.com" http://176.58.107.39/100MB-london.bin?foo --2016-11-20 15:52:37-- http://176.58.107.39/100MB-london.bin?foo Connecting to 176.58.107.39:80... connected. HTTP request sent, awaiting response... 200 OK Length: 104857600 (100M) [application/octet-stream] Saving to: ‘/dev/null’ 100%[===================================================================================================================>] 104,857,600 4.67MB/s in 20s 2016-11-20 15:52:58 (4.89 MB/s) - ‘/dev/null’ saved [104857600/104857600] -- END SNIPPET -- So that's 39 megabits. Still a bit of a drop compared to what I'm used to. Running a quick test from the VPN concentrator itself, it looks like it's a product of going through the cache though, as I get about 50Mb/s from there (still slightly slower). Not a big enough drop to be too inconvenient, and haven't really done anything to tune the connection speed yet, so will leave for the time being. ----------------------------------------------------------------------------------------- 2016-11-20 16:12:13 btasker ----------------------------------------------------------------------------------------- Looking at the PCAP the only port 80 traffic I see is an incoming connection (coincidental timing) -- BEGIN SNIPPET -- ben@milleniumfalcon:/tmp/misc19$ tcpdump -r misc19.pcap port 80 reading from file misc19.pcap, link-type EN10MB (Ethernet) 15:59:17.202920 IP tor-limits-scanning.cl.cam.ac.uk.39250 > 192.168.3.64.http: Flags [S], seq 2819030485, win 65535, options [mss 1452], length 0 15:59:17.205613 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.39250: Flags [S.], seq 2671805212, ack 2819030486, win 29200, options [mss 1460], length 0 15:59:17.218515 IP tor-limits-scanning.cl.cam.ac.uk.39250 > 192.168.3.64.http: Flags [R], seq 2819030486, win 0, length 0 15:59:17.295404 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [S], seq 1392928851, win 29200, options [mss 1452,sackOK,TS val 43886324 ecr 0,nop,wscale 9], length 0 15:59:17.298334 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [S.], seq 2757475325, ack 1392928852, win 28960, options [mss 1460,sackOK,TS val 611594553 ecr 43886324,nop,wscale 6], length 0 15:59:17.310518 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [.], ack 1, win 58, options [nop,nop,TS val 43886327 ecr 611594553], length 0 15:59:17.310567 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [P.], seq 1:98, ack 1, win 58, options [nop,nop,TS val 43886327 ecr 611594553], length 97 15:59:17.312692 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [.], ack 98, win 453, options [nop,nop,TS val 611594556 ecr 43886327], length 0 15:59:17.382037 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [P.], seq 1:585, ack 98, win 453, options [nop,nop,TS val 611594574 ecr 43886327], length 584 15:59:17.393760 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [.], ack 585, win 60, options [nop,nop,TS val 43886348 ecr 611594574], length 0 15:59:17.394080 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [F.], seq 98, ack 585, win 60, options [nop,nop,TS val 43886348 ecr 611594574], length 0 15:59:17.396788 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.48790: Flags [F.], seq 585, ack 99, win 453, options [nop,nop,TS val 611594577 ecr 43886348], length 0 15:59:17.408632 IP tor-limits-scanning.cl.cam.ac.uk.48790 > 192.168.3.64.http: Flags [.], ack 586, win 60, options [nop,nop,TS val 43886352 ecr 611594577], length 0 15:59:56.666643 IP tor-limits-scanning.cl.cam.ac.uk.39250 > 192.168.3.64.http: Flags [S], seq 2819030485, win 65535, options [mss 1452], length 0 15:59:56.669796 IP 192.168.3.64.http > tor-limits-scanning.cl.cam.ac.uk.39250: Flags [S.], seq 3288439354, ack 2819030486, win 29200, options [mss 1460], length 0 15:59:56.681477 IP tor-limits-scanning.cl.cam.ac.uk.39250 > 192.168.3.64.http: Flags [R], seq 2819030486, win 0, length 0 -- END SNIPPET -- But nothing outgoing despite having generated rather a lot of traffic. So that's good. Before I can look at cutting everything over I need to create some policies so that there are exceptions - Existing VPNs from the main router should just go straight out - Connections from the concentrator to the endpoint obviously need to go straight out There might be other exceptions I want to make too, but that's a starting point. Also need to spin up a second endpoint so I can do some failover testing. That'll mean taking the endpoint down, so better to do first. ----------------------------------------------------------------------------------------- 2016-11-20 16:12:29 ----------------------------------------------------------------------------------------- btasker changed timespent from '99 minutes' to '104 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 16:43:52 btasker ----------------------------------------------------------------------------------------- OK, second endpoint spun up by creating a snapshot of the first. I've added it to the load balancer and created a DNS name for it. I've stopped openvpn on the first gateway, want to see whether it'll re-resolve and then try on the second (though the load balancer will still return both IP's as I haven't set up the status checks yet). Doesn't seem to have done yet. There would only have been one IP when it last resolve it though, so killing the process and seeing what happens now. Doesn't look like it does a DNS round-robin then, will need to get the status checks up and running. For now (as a test) I'll configure a status check on the load balancer for the first endpoint - should have the result of knocking it out of selection -- BEGIN SNIPPET -- router> find_cache vpnbridge.balanced.bentasker.co.uk Query took 0.00452 seconds Client: 8.8.8.8 Location: g.us Domain: vpnbridge.balanced.bentasker.co.uk Required Protocols: p.ssh | Hostname | Address | Rack | Cost | Cost Src | |---------------------------------------------------------------------------| | vpn2.sys.bentasker.co.uk | 178.62.116.177 | r.gb.vpnlon | 100 | d | ----------------------------------------------------------------------------- -- END SNIPPET -- So lets see whether the VPN switches over to trying to connect to the other. No, y'know what, I'm an idiot. I forgot to update the config to use the DNS name. Possible round-robin does work then, will try that again shortly, will make sure the connection to ep2 works first. ----------------------------------------------------------------------------------------- 2016-11-20 16:56:51 btasker ----------------------------------------------------------------------------------------- OK, it's connected to ep2, so going to put ep1 back into selection, stop openvpn on ep2 and see what happens ----------------------------------------------------------------------------------------- 2016-11-20 17:02:07 btasker ----------------------------------------------------------------------------------------- OK, it's just failed over, took around a minute to do so, which lines up with the settings I configured in the client config. -- BEGIN SNIPPET -- ping 15 ping-restart 60 resolv-retry 60 -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2016-11-20 17:31:26 btasker ----------------------------------------------------------------------------------------- Setting up NGinx so I can enable status checks -- BEGIN SNIPPET -- cat << EOM >> /etc/apt/sources.list deb http://nginx.org/packages/debian/ jessie nginx deb-src http://nginx.org/packages/debian/ jessie nginx EOM apt-key adv --keyserver keyserver.ubuntu.com --recv-keys ABF5BD827BD9BF62 apt-get update apt-get install nginx mkdir /usr/share/nginx/status cat << EOM > /etc/nginx/conf.d/status.conf server { listen 80; listen [::]:80; root /usr/share/nginx/status; index index.php index.html index.htm; server_name status.self; add_header X-Clacks-Overhead "GNU Terry Pratchett"; } EOM cat << EOM > /usr/share/nginx/html/index.html VPN Endpoint

This node is a VPN endpoint. Traffic you see as originating from here probably didn't.

If you consider traffic problematic, please visit Contact-Me to report it (reports with packet captures and/or loglines as proof are preferred)

EOM root@debian-512mb-lon1-01:~# service nginx restart -- END SNIPPET -- Creating the status check script -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:~# apt-get install python-psutil cat << EOM > /root/status.py #!/usr/bin/env python # # # OFFLINE_FILE='/usr/share/nginx/offline.lock' import os,json,psutil response = { "status" : "Online", "loadavg" : '', "processors" : 0, "loadperc" : 0, "loadperc5" : 0, "loadperc15" : 0 } f = open('/proc/loadavg', 'r') response['loadavg']=f.read() f.close() f = open('/proc/cpuinfo', 'r') cpucount=f.read() f.close() response['processors'] = cpucount.count('processor\t') # Calculate the loadpercentage based on number of processors loads = response['loadavg'].split(' ') # Now response['loadperc'] = round(float(loads[0]) / response['processors'],2) # 5 min response['loadperc5'] = round(float(loads[1]) / response['processors'],2) # 15 min response['loadperc15'] = round(float(loads[2]) / response['processors'],2) s=0 for pid in psutil.pids(): p = psutil.Process(pid) if p.name() == "openvpn": s=1 break if s == 0: response['status'] = "Offline" if os.path.exists(OFFLINE_FILE): response['status'] = "Offline" print json.dumps(response) EOM root@debian-512mb-lon1-01:~# chmod +x /root/status.py -- END SNIPPET -- Testing -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:~# /root/status.py {"status": "Online", "processors": 1, "loadperc": 0.0, "loadperc15": 0.02, "loadavg": "0.00 0.01 0.02 1/69 1500\n", "loadperc5": 0.01} root@debian-512mb-lon1-01:~# service openvpn stop root@debian-512mb-lon1-01:~# /root/status.py {"status": "Offline", "processors": 1, "loadperc": 0.0, "loadperc15": 0.02, "loadavg": "0.00 0.01 0.02 1/72 1527\n", "loadperc5": 0.01} -- END SNIPPET -- Setting up a cronjob -- BEGIN SNIPPET -- root@debian-512mb-lon1-01:~# echo "* * * * * root /root/status.py > /usr/share/nginx/status/status.json" >> /etc/cron.d/status-checks -- END SNIPPET -- Have added the status pages to the load balancer ----------------------------------------------------------------------------------------- 2016-11-20 17:31:41 ----------------------------------------------------------------------------------------- btasker changed timespent from '104 minutes' to '114 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 17:39:53 btasker ----------------------------------------------------------------------------------------- OK, I've created some whitelists (probably in need of tweaking a bit over time) and cut all other traffic over to using the tunnel. Going to set a monitoring pcap running on the WAN interface, to keep size down I'll exclude the VPN traffic. -- BEGIN SNIPPET -- # tcpdump -i eth0 -s0 -w misc19.pcap -v not port 1194 -- END SNIPPET -- Will come back and analyse that later ----------------------------------------------------------------------------------------- 2016-11-20 18:01:24 btasker ----------------------------------------------------------------------------------------- Current state should be that all non-vpn (and non-tor) IPv4 traffic now transits the VPN. IPv6 traffic will still hit the helium tunnelled connection as I haven't quite figured out exactly how I want to address that yet - supporting failover is going to be a little more tricky unless I use NAT64 (which will probably be the only option, in truth). Most clients on the LAN aren't given an IPv6 address, so it won't affect the majority of the traffic anyway. Using the VPN does mean that we won't see the benefit of any ISP level caches, but it's not the end of the world. Looking at an analysis of the capture so far, the only traffic I'm seeing is one of the existing VPNs (which doesn't use port 1194) and the Helium IPv6 tunnel. Bodes well ----------------------------------------------------------------------------------------- 2016-11-20 18:01:39 ----------------------------------------------------------------------------------------- btasker changed timespent from '114 minutes' to '119 minutes' ----------------------------------------------------------------------------------------- 2016-11-20 20:02:52 btasker ----------------------------------------------------------------------------------------- Hmmm, so first headache. Said earlier I'd prefer a UK based endpoint if possible to avoid geolocking issues, but it seems that the BBC have decided they don't like datacentre IPs. Not overly surprising (given they're paranoid about VPNs) but as a licensepayer I'm obviously a little indignant :) I've added a new policy to allow the iplayer box to go out without transiting the VPN, though it comes at the cost of leaking anything else I stream too. I've a suspicion they only do the geo-location stuff when fetching the manifest though, so it might be possible to work around by setting up a reverse proxy (that'll go out without transiting the VPN) and letting the stream use the VPN. ----------------------------------------------------------------------------------------- 2016-11-20 20:29:31 btasker ----------------------------------------------------------------------------------------- Yup, adding a reverse proxy for open.live.bbc.co.uk fixed it. I'm sure they'll break that at some point, but at least means I can watch Planet Earth later. ----------------------------------------------------------------------------------------- 2016-11-20 23:47:05 btasker ----------------------------------------------------------------------------------------- So, the base requirements are all satisfied to some extent - The solution supports failover. - Responses to inbound connections do not transit the link - Policies control what can and cannot egress the network - Port 80 traffic still goes via the transparent proxy and then transits the VPN link - The system fails closed - Iplayer still works (albeit with a bit of hacking) However, it's worth nothing that both endpoints are currently in the same datacentre, so there's no geographic redundancy. I will look at addressing that, but as it's almost trivial to spin up a new endpoint now I may wait until the government get around to defining what's classed as a CSP. I think I'll probably raise a separate issue to track getting DNSCrypt set up as this ones getting quite large. ----------------------------------------------------------------------------------------- 2016-11-27 11:34:25 btasker ----------------------------------------------------------------------------------------- The VPN now tunnel IPv6 as well Was fairly simple to set up. I've got a routed /48 so using a /64 from that inside the tunnel. Edited the OpenVPN config to include -- BEGIN SNIPPET -- server-ipv6 2001:470:69d7:4de::/64 tun-ipv6 push tun-ipv6 ifconfig-ipv6 2001:470:69d7:4de::1 2001:470:69d7:4de::2 push "route-ipv6 2000::/3" -- END SNIPPET -- As the VPN is load balanced, I either need NAT or to have the LAN addresses update whenever the tunnel fails over. The former is simpler, so NAT it is Enabling NAT and forwarding at both ends -- BEGIN SNIPPET -- ip6tables -t nat -I POSTROUTING -j MASQUERADE -- END SNIPPET -- Enable IPv6 forwarding -- BEGIN SNIPPET -- echo "net.ipv6.conf.all.forwarding=1" >> /etc/sysctl.conf sysctl -p -- END SNIPPET -- Added policies to direct IPv6 traffic via the VPN concentrator on the LAN. Job done ----------------------------------------------------------------------------------------- 2017-07-06 10:14:17 btasker ----------------------------------------------------------------------------------------- The implementation has been working and stable for well over 6 months now. So closing this issue off. ----------------------------------------------------------------------------------------- 2017-07-06 10:14:17 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'Resolved' ----------------------------------------------------------------------------------------- 2017-07-06 10:14:17 ----------------------------------------------------------------------------------------- btasker added 'Done' to resolution ----------------------------------------------------------------------------------------- 2017-07-06 10:14:20 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- 2019-03-27 10:10:43 ----------------------------------------------------------------------------------------- btasker removed 'Done' from resolution ----------------------------------------------------------------------------------------- 2019-03-27 10:10:43 ----------------------------------------------------------------------------------------- btasker changed status from 'Closed' to 'Reopened' ----------------------------------------------------------------------------------------- 2019-03-27 10:11:05 ----------------------------------------------------------------------------------------- btasker changed status from 'Reopened' to 'Resolved' ----------------------------------------------------------------------------------------- 2019-03-27 10:11:05 ----------------------------------------------------------------------------------------- btasker added 'Done' to resolution ----------------------------------------------------------------------------------------- 2019-03-27 10:11:08 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- 2019-04-26 12:26:00 ----------------------------------------------------------------------------------------- btasker removed 'Done' from resolution ----------------------------------------------------------------------------------------- 2019-04-26 12:26:00 ----------------------------------------------------------------------------------------- btasker changed status from 'Closed' to 'Reopened' ----------------------------------------------------------------------------------------- 2019-04-26 12:50:57 ----------------------------------------------------------------------------------------- btasker changed status from 'Reopened' to 'Resolved' ----------------------------------------------------------------------------------------- 2019-04-26 12:50:57 ----------------------------------------------------------------------------------------- btasker added 'Done' to resolution ----------------------------------------------------------------------------------------- 2019-04-26 12:51:01 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- 2019-12-10 15:40:21 ----------------------------------------------------------------------------------------- btasker removed 'Done' from resolution ----------------------------------------------------------------------------------------- 2019-12-10 15:40:21 ----------------------------------------------------------------------------------------- btasker changed status from 'Closed' to 'Reopened' ----------------------------------------------------------------------------------------- 2019-12-10 15:50:15 ----------------------------------------------------------------------------------------- btasker changed status from 'Reopened' to 'Resolved' ----------------------------------------------------------------------------------------- 2019-12-10 15:50:15 ----------------------------------------------------------------------------------------- btasker added 'Done' to resolution ----------------------------------------------------------------------------------------- 2019-12-10 15:50:19 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- Worklog ======== ----------------------------------------------------------------------------------------- 2016-11-17 19:57:41 btasker 10 minutes ----------------------------------------------------------------------------------------- Mapping out architecture ----------------------------------------------------------------------------------------- 2016-11-20 14:11:53 btasker 39 minutes ----------------------------------------------------------------------------------------- Setting up and testing endpoint ----------------------------------------------------------------------------------------- 2016-11-20 14:22:39 btasker 2 minutes ----------------------------------------------------------------------------------------- Setting up iptables ----------------------------------------------------------------------------------------- 2016-11-20 14:36:18 btasker 10 minutes ----------------------------------------------------------------------------------------- Setting up DNS ready for failover support ----------------------------------------------------------------------------------------- 2016-11-20 15:14:50 btasker 21 minutes ----------------------------------------------------------------------------------------- Setting up LAN side VPN Concentrator ----------------------------------------------------------------------------------------- 2016-11-20 15:45:06 btasker 17 minutes ----------------------------------------------------------------------------------------- Setting up initial policy based routing ----------------------------------------------------------------------------------------- 2016-11-20 16:12:29 btasker 5 minutes ----------------------------------------------------------------------------------------- PCAP analysis ----------------------------------------------------------------------------------------- 2016-11-20 17:31:41 btasker 10 minutes ----------------------------------------------------------------------------------------- Setting up status pages ----------------------------------------------------------------------------------------- 2016-11-20 18:01:39 btasker 5 minutes ----------------------------------------------------------------------------------------- Pcap analysis