########################################################################################## PAS-4: HTTPS Referrer Search should only match on the Referrer field ########################################################################################## Issue Type: Bug ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Fixed (2015-11-27 13:17:10) Project: PCAP Analysis Script (PAS) Reported By: btasker Assigned To: btasker Components: - Processing Logic - HTTP - SSL/TLS Affected Versions: - 0.1 Targeted for fix in version: - 0.1 Labels: Referrer, SSLPathExtraction, Time Estimate: 80 minutes Time Logged: 10 minutes ----------------------------------------------------------------------------------------- Issue Description ================== The current means of searching for HTTPS sites within HTTP referrers is overly broad, the HTTP CSV is simply grepped. As a result, if a site has a cookie with a value matching the string we're searching for it will be matched and included in the known SSL paths section. For example -- BEGIN SNIPPET -- 1447885734.980645000 192.168.3.64 23.235.43.239 58834 80 www.wired.com GET /wp-content/uploads/2015/11/black-box-600x338-e1447374320621.jpg http://www.wired.com/2015/11/carnegie-mellon-denies-fbi-paid-for-tor-br eaking-research/ Mozilla/5.0 (Linux; Android 4.4.2; en-gb; SAMSUNG GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/1.5 Chrome/28.0.1500.94 Mobile Safari/537.36 original_referrer=https://www.google.co.uk/; _ga=GA1.2 .2062192190.1409172610; -- END SNIPPET -- Would match a search for https://www.google.co.uk despite not actually being related. Note: the fix for PAS-2 _might_ make this point moot ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2015-11-22 09:46:27 ----------------------------------------------------------------------------------------- btasker added 'Referrer SSLPathExtraction' to labels ----------------------------------------------------------------------------------------- 2015-11-22 12:12:16 btasker ----------------------------------------------------------------------------------------- The changes made in PAS-2 make this less likely, but do not eliminate it. For example, if there are genuine entries for https://www.google.co.uk the search would still pick out the example above. So additional work still needs to be done to implement this issue ----------------------------------------------------------------------------------------- 2015-11-22 13:04:19 btasker ----------------------------------------------------------------------------------------- It's a little less efficient, but matching is now performed exclusively against the referrer field. To avoid continually grepping the full HTTP request set, the httpsreferers temp file is still used. ----------------------------------------------------------------------------------------- 2015-11-22 13:05:42 git ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: PCAPAnalyseandReport Commit: 089c2bfe605f4a4df6f584b74ad0a8553ec3c01b Author: B Tasker