It wasn't originally designed that way, but really
ssltraffic.txt should be a CSV.
All the other metadata files are generated on the basis that they can be used as a "quick-read" reference, and additional information can then be extracted from webtraffic.csv (or whichever file is relevant) if needed.
The benefit of the approach above is that it's relatively easy to merge files generated from runs against multiple PCAPs. Extracting duplicates is pretty simple.
That's not possible with ssltraffic.txt though, and even leaving out that downside, I'm not sure it's doing a good job of providing the information that it needs to.
Essentially, it should give us the name of the site in question, and the path that's been identified. Duplicate paths should probably be removed too.
That can be achieved in CSV format (leading to the benefit above) by switching to a format of
FQDN, Identified path
Where paths are of interest, they can soon be located within webtraffic.csv using a quick grep.
Activity
2015-11-27 00:57:34
The old code was a bit opaque in terms of what it was doing and why, so I've made sure the new introductions are properly commented.
Test run going at the moment
2015-11-27 00:57:47
2015-11-27 01:44:13
2015-11-27 01:44:25
2015-11-27 01:44:34
2015-11-27 01:44:34
2015-11-27 01:44:51
2015-11-27 01:44:57
2015-11-27 01:45:46
Webhook User-Agent
View Commit
2015-11-27 13:12:41
2015-11-27 13:12:41
2015-11-27 13:12:41
2015-11-27 13:12:57
2015-11-27 13:12:57
2015-11-27 13:13:02