PAS-4: HTTPS Referrer Search should only match on the Referrer field



Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: PCAP Analysis Script (PAS)
Resolution: Fixed (2015-11-27 13:17:10)
Affects Version: 0.1,
Target version: 0.1,
Components: Processing Logic , HTTP , SSL/TLS ,
Labels: Referrer, SSLPathExtraction,

Created: 2015-11-22 09:46:13
Time Spent Working
Estimated:
 
90 minutes
Remaining:
  
80 minutes
Logged:
  
10 minutes


Description
The current means of searching for HTTPS sites within HTTP referrers is overly broad, the HTTP CSV is simply grepped.

As a result, if a site has a cookie with a value matching the string we're searching for it will be matched and included in the known SSL paths section.

For example
1447885734.980645000    192.168.3.64    23.235.43.239   58834   80      www.wired.com   GET     /wp-content/uploads/2015/11/black-box-600x338-e1447374320621.jpg        http://www.wired.com/2015/11/carnegie-mellon-denies-fbi-paid-for-tor-br
eaking-research/ Mozilla/5.0 (Linux; Android 4.4.2; en-gb; SAMSUNG GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Version/1.5 Chrome/28.0.1500.94 Mobile Safari/537.36    original_referrer=https://www.google.co.uk/;_ ga=GA1.2
.2062192190.1409172610;

Would match a search for https://www.google.co.uk despite not actually being related.

Note: the fix for PAS-2 _might make this point moot


Toggle State Changes

Activity


btasker added 'Referrer SSLPathExtraction' to labels
The changes made in PAS-2 make this less likely, but do not eliminate it. For example, if there are genuine entries for https://www.google.co.uk the search would still pick out the example above.

So additional work still needs to be done to implement this issue
It's a little less efficient, but matching is now performed exclusively against the referrer field. To avoid continually grepping the full HTTP request set, the httpsreferers temp file is still used.

Repo: PCAPAnalyseandReport
Commit: 089c2bfe605f4a4df6f584b74ad0a8553ec3c01b
Author: B Tasker <github@<Domain Hidden>>

Date: Sun Nov 22 13:03:24 2015 +0000
Commit Message: SSL Path extraction is now limited to matches on HTTP Referers. See PAS-4



Modified (-)(+)
-------
PCAP_Analysis.sh




Webhook User-Agent

GitHub-Hookshot/88897e7


View Commit

btasker changed timespent from '0 minutes' to '10 minutes'
Seems to be working, so closing as fixed.

Clearly I massively overestimated the work required to fix :)
btasker changed status from 'Open' to 'Resolved'
btasker added 'Fixed' to resolution
btasker changed status from 'Resolved' to 'Closed'
Re-opening to assign to a component
btasker removed 'Fixed' from resolution
btasker changed status from 'Closed' to 'Reopened'
btasker changed status from 'Reopened' to 'Resolved'
btasker added 'Fixed' to resolution
btasker changed status from 'Resolved' to 'Closed'

Work log


Ben Tasker
Permalink
2015-11-22 13:07:25

Time Spent: 10 minutes
Log Entry: Changing and testing