PAS-6: Reading of multiple PCAPs



Issue Information

Issue Type: New Feature
 
Priority: Minor
Status: Open

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: PCAP Analysis Script (PAS)
Resolution: Unresolved
Affects Version: 0.1,
Target version: 0.1,
Labels: Input, PCAP, Report,

Created: 2015-11-22 11:20:06
Time Spent Working
Estimated:
 
120 minutes
Remaining:
 
120 minutes
Logged:
 
0 minutes


Description
It'd be helpful to be able to generate a report sourced from multiple PCAPs.

Where a large PCAP is supplied, if the host running the processing doesn't have sufficient RAM to handle it, errors like the following will be seen
ben@milleniumfalcon:/tmp$ ~/Documents/src.old/System_Audit_utilities/Traffic/PCAP_Analysis.sh TEST-PCAP-sample-7.pcap 
Analysing Port 80 Traffic

(process:9236): GLib-ERROR **: /build/buildd/glib2.0-2.40.2/./glib/gmem.c:103: failed to allocate 41495301 bytes
/home/ben/Documents/src.old/System_Audit_utilities/Traffic/PCAP_Analysis.sh: line 18:  9236 Trace/breakpoint trap   (core dumped) tshark -q -r "$PCAP" -Y "http.host" -T fields -e frame.time_epoch -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport -e http.host -e http.request.method -e http.request.uri -e http.referer -e http.user_agent -e http.cookie > "${TMPDIR}/httprequests.txt"
Analysing HTTPS traffic

(process:9362): GLib-ERROR **: /build/buildd/glib2.0-2.40.2/./glib/gmem.c:103: failed to allocate 41495301 bytes
/home/ben/Documents/src.old/System_Audit_utilities/Traffic/PCAP_Analysis.sh: line 26:  9362 Trace/breakpoint trap   (core dumped) tshark -q -r "$PCAP" -Y "ssl.handshake" -T fields -e frame.time_epoch -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport -e ssl.handshake.extensions_server_name -e ssl.handshake.ciphersuite > "${TMPDIR}/sslrequests.txt"
Identifying HTTPS pages from HTTP Referrers
Looking for XMPP traffic

(process:10156): GLib-ERROR **: /build/buildd/glib2.0-2.40.2/./glib/gmem.c:103: failed to allocate 41495301 bytes
/home/ben/Documents/src.old/System_Audit_utilities/Traffic/PCAP_Analysis.sh: line 53: 10156 Trace/breakpoint trap   (core dumped) tshark -q -r "$PCAP" -Y "tcp.dstport == 5222" -T fields -e frame.time_epoch -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport > "${TMPDIR}/xmpprequests.txt"
Building reports
Done


If the PCAP is broken down into smaller chunks, recombining most of the current output is fairly straightforward
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/associatedhosts.csv | sort | uniq | less
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/observedcookies.csv | sort | uniq | less
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/observedhttpuseragents.csv | sort | uniq | less
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/visitedsites.csv | sort | uniq | less
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/webtraffic.csv | sort | uniq | less
ben@milleniumfalcon:/tmp$ cat report.TEST-PCAP-*/xmpppeers.csv | sort | uniq | less

However, there's no easy way to recombine the ssltraffic.txt files

What'd be ideal is if it were possible to tell the script to read in multiple PCAPs and then generate a single set of reports based on the content


Toggle State Changes

Activity


It will definitely require multiple runs of tshark, as it doesn't support reading from multiple files at once - https://www.wireshark.org/lists/wireshark-users/200808/msg00105.html

Using something like mergecap isn't an option given the reason the files were split in the first place is the system doing the processing has insufficient resources to process as a single PCAP.

I guess the simplest way will be to wrap each of the tshark calls in a for loop and cycle through any input files that have been specified