Need to re-consider the desirable traffic pattern.
The only considerations currently set in stone are
- Traffic generated by BASH Haystack Generator shouldn't be immediately identifiable (from outside the tunnel) as being from that source.
- There should be an option to generate random traffic in both directions (i.e. upstream and downstream)
- Traffic needs to look sufficiently like genuine use that it's indistinguisable (i.e. completely random is bad)
Issues were encountered (
BHAYSTACKG-2) with multiple runs overlapping, leading to exhaustion of File Descriptor limits on the remote server.
Whilst some overlap is probably still desirable, need to re-asses the base assumptions about what the traffic pattern should be
Activity
2014-12-08 14:19:45
A good starting place is probably to identify the different profiles of 'real' traffic (splitting into seperate comments to improve readability)
2014-12-08 14:20:45
- Very small amount upstream (potentially with an occasional spike if a file is POST'd somewhere), pretty much just request headers
- Small amounts downstream
- Batches of rapid requests (as page requisites such as CSS and JS are retrieved)
Note: You'd also normally see the destination IP's change regularly, but as the adversary is outside the tunnel they'd not see this
2014-12-08 14:20:54
Depends on the type of video being streamed, with a few potential complications if the stream is adaptive (see further down)
Non-adaptive HLS
- Small amount upstream (just web requests)
- Almost identically sized chunks coming downstream, with the occasional small response (the manifest file)
- if a live stream, Manifest file responses may vary in size, but generally not by much
- If a VOD stream, Manifest file size won't change, whether it's re-requested at all would depend on player behaviour
Smooth Streaming
More or less the same assumptions can be made for Microsoft's Smooth Streaming , though as well as requests for the video content, there'd be requests for the Audio.
If a stream was adaptive then the size of the responses may occasionally change (whether increase or decrease) if the player changes bitrate. The changes would be in batches though, so there'd be at least a few requests in a row for each size.
Progressive Download
The profile for a Progressive download, would be different, though it would depend on player behaviour (as some use byte-range requests instead of a single request)
- One request going out
- Sustained (and potentially very large) download
If byte range requests were used by the theoretical player, then the pattern would be closer to that of HLS (although there'd be no manifest requests)
2014-12-08 14:21:10
The exact behaviour obviously differs by protocol in use, but assuming we want to vaguely ape bit-torrent traffic
On a per-torrent basis:
- Small initial upstream with some downstream responses (joining the swarm)
- Lots of Small requests, larger downstream: fetching individual pieces - max 1MB each (but always multiples of 16KB as that's the bittorrent block size)
- As the 'download' progresses, the client would then start making retrieved pieces available to other peers, so we'd also get some small downstream requests, with larger response (again, max 1MB each but always a multiple of 16KB)
- A drop-off in traffic as the torrent completes (assuming the user has set their share ratio to 0)
Whilst an outside-the-tunnel adversary shouldn't be able to see the exact specifics, it's relatively important to ensure that the traffic grows in this nature.
It might also be worth looking at the traffic profile of other common uses (VoIP/Skype for example).
The aim isn't to swamp the tunnel with so much traffic that it's impossible to see anything but occasional spikes from normal usage, it's (in theory) far better, if possible, to ape standard usage patterns.
2014-12-09 13:51:32
2014-12-11 15:19:53
http://projectsstatic.bentasker.co.uk/BHAYSTACKG/BHAYSTACKG-3/generator_profiles/20141209_generator_profile.html
Whilst the profile for a short Web-browsing session can be seen here:
http://projectsstatic.bentasker.co.uk/BHAYSTACKG/BHAYSTACKG-3/real_usage_profiles/20141211_webbrowsing.html
The difference between the two is obviously very distinct, with the generator creating much higher levels of traffic than a single browsing session. This isn't necessarily a bad thing (as higher levels help ensure that only a small percentage increase in traffic is observed when 'real' traffic joins the mix).
The level of Granularity for the graphs is 5 minutes, however it'd also be worth generating graphs with higher levels of granularity (perhaps based on PCAPs rather than interface stats) to identify the bigger differences between the two that likely exist at a higher granularity.
Given the large difference in median ingress rates for the two profiles;
- Generator: ~250 KB/s
- Web Browsing: ~80 KB/s
I don't think you could reasonably assume that the traffic created by the generator could be considered by an outside observer to be a sign of multiple users browsing the net.
Similarly, the egress rate of the generator doesn't conform to pattern.
That said, it is reasonable to say that not all users of an Internet connection would be doing the same thing at once, some may be streaming video whilst others browse the net (for example), so the generator's traffic profile isn't necessarily immediately identifiable.
2014-12-11 15:30:35
The ingress (and to some extent, the egress) rate of the tunnel fluctuates quite a lot, sometimes it's quite high and sometimes really quite low.
Whilst a larger sampling is needed, there doesn't appear to be any specific pattern emerging, which if true, means that the current traffic levels (and pattern) should quite effectively mask any modest spikes in usage - some of the time.
The highest ingress rate observed (in the ongoing charting) is around 500 KB/s (note, bytes not bits!) with the media rate somewhere around the 250 KB/s mark.
Therefore, any 'real' traffic which caused the traffic rate to spike above 500 KB/s would likely be immediately identifiable. Although it would be difficult to calculate what proportion of that spike was genuine traffic, subtracting 250 KB/s would give a rough estimate.
That's likely to be an issue in any case, though - any system relying on junk traffic masking the real traffic will only ever be effective to a certain percentage of it's own levels. What might be helpful, though, is a way to make it easier to calculate what this percentage actually is on a per deployment basis.
In the current build (v0.1), it should be possible to calculate roughly, but because of the random nature of the traffic levels it would be incredibly difficult to do so accurately. Creating traffic profiles based on 'real' usage would likely make this a lot easier, especially if the option were available to use multiple profiles per run (so that the generator could simulate streaming video at the same time as someone browses the web for example).
2014-12-11 15:34:24
Hardship that it is, that means doing some web-browsing and video-streaming (at a minimum) whilst running PCAPs so that granular traffic patterns can be analysed.
2014-12-11 17:16:35
Had to double check the script I used to generate the chart, as there's a far higher level of egress than I expected. Script looks fine, but I'm going to double check the test methodology (and have a closer look at the PCAP) as it just doesn't look quite right.