Commits: Websites / Privacy Sensitive Analytics

Remove support for image requests from LUA websites/privacy-sensitive-analytics#21

disable pixel info collection for websites/privacy-sensitive-analytics#21

Sort response times table

Remove defaults that are specific to me

Use FQDN

Make database connection details configurable in reporting script (websites/privacy-sensitive-analytics#20)

Add config var $permitted_events for websites/privacy-sensitive-analytics#19

Implement handling of event submissions for websites/privacy-sensitive-analytics#19

Start building interface for events API (websites/privacy-sensitive-analytics#19)

Don't throw an expception if there are no records returned

Don't pass through an empty path if a user hit's the domains root

Enable writes into the upstream for websites/privacy-sensitive-analytics#18

Move to using rewrite_by_lua rather than header_filter_by_lua (websites/privacy-sensitive-analytics#18)

Have the counter return the expected gif (websites/privacy-sensitive-analytics#18)

Implement support for a hit-count pixel (websites/privacy-sensitive-analytics#18)

Move to a 1:3 chance of a session ID rotating (websites/privacy-sensitive-analytics#17)

Add a counter and rotate sessionID if it's been used too many times websites/privacy-sensitive-analytics#16

NOOP - add comments

Add link to release notes to header

Use a length rather than empty check

Fix syntax

Update historic dashboard to give bad domain stats for websites/privacy-sensitive-analytics#12

Write a list of observed bad domains and associated pages into long term storage for websites/privacy-sensitive-analytics#12

Downsample bad-domains stats ready for reporting in websites/privacy-sensitive-analytics#12

Start tracking dashboards - these are JSON exports from Chronograf

Capture protocol information as suggested in websites/privacy-sensitive-analytics#15

Update enhanced gather to handle protocols/schemes that don't involve a domain name. See websites/privacy-sensitive-analytics#15

Remove the screen capture ability for websites/privacy-sensitive-analytics#14

Capture search terms into a seperate bucket for websites/privacy-sensitive-analytics#13

Add missing column

Downsample 404 stats for websites/privacy-sensitive-analytics#11

Record average number of search results for websites/privacy-sensitive-analytics#13

Downsample search terms into a count of searches over time for websites/privacy-sensitive-analytics#13

Don't attempt to process referrer if it's empty

Include referrer info when reporting errors

Add agent support for reporting 404s for websites/privacy-sensitive-analytics#11

Add server side support for logging 404s for websites/privacy-sensitive-analytics#11

Create utility for viewing bad domains screenshots

Write screenshot to disk rather than trying to push it upstream. See websites/privacy-sensitive-analytics#10

Implement collection of additional information when a bad domain is detected - see websites/privacy-sensitive-analytics#10

Create function to record a search term and (optional) result count - for websites/privacy-sensitive-analytics#9

Oops, missed one

Adjust agent so that the write path is provided when calling submit(). This is in prep for websites/privacy-sensitive-analytics#9

Move the say in scope. Also, it quite inconveniently won't show up on the server unless you actually write it there. Strange eh?

Implement search term submission processing for websites/privacy-sensitive-analytics#9

Write search terms into a different measurement websites/privacy-sensitive-analytics#9

Prepare Nginx config for accepting search term submissions (see websites/privacy-sensitive-analytics#9)

Fix braces

Report on eepsites too

Add support for writing via i2p

Collapse known tor2web domains

Tabulate some of the data that's currently only graphed

Collapse youtube domains

Collapse various domains into descriptors

Also change the window size for longer queries - it was only actually introduced to help reduce memory demands of the query anyway

Use a lower granularity bucket for longer periods - this helps avoid OOMs (and reduce query times etc etc)

Increase timeout and set unbuffered mode

Ensure that case is normalised when checking if domain is authorised

Change base

Differentiate report periods in subject lines

Take number of days to report on from commandline

Dockerise - allows us to roll without worrying about deps, and mail pass can be provided via environment

Implement reporting script for websites/privacy-sensitive-analytics#8

Generate some basic pie charts

Tidy out the defunct code

Capture video playback info for websites/privacy-sensitive-analytics#8

Collect details of top 10 pages for websites/privacy-sensitive-analytics#8

Collect referrer info for websites/privacy-sensitive-analytics#8

Implement extraction of page views for websites/privacy-sensitive-analytics#8

Capture page views and response times for reporting. See websites/privacy-sensitive-analytics#8

Add generic protection against tor2web style domains

Remove view counter from unauth domain writes

Don't break on older data without an action tag

Add handling of the downsample tag

Filter and handle "state"

Add a state field to the payload - this is to allow easier integration with my video embedder

Add a ping against the Tor endpoint - the idea being to get a circuit established

Write to Tor endpoint if we've been accessed via Tor.

Update to use the new endpoint - this abstracts writes out to the CDN, so we never get the user's IP

Whoops, switch regex out for an actual LUA pattern

No-op: whitespace

Default the session identifier to on.

Have the client periodically rotate session identifier

Remove pageview counter - it's not proven as useful/interesting as expected and requires cross-request tracking to work

Add filtering of request info to keep fuzzers etc out

Downsample field values

Add function to handle domready event

Add missing function

Add downsample for referring domain - uses the strings library to collapse google domains down to a single entry

Create the most basic of downsampling - it pulls platform info and downsamples it

Implement the ability to use a session specific psuedo-identifier for websites/privacy-sensitive-analytics#6

Switch to using ms to allow for higher concurrency - websites/privacy-sensitive-analytics#6

Tidy up scopes and prevent caching

Quote domain when writing unauthorised referrers in

Allow the dbname to be specified in config.

Move the influx config out to Nginx config

Pass measurement name through to the unauth function

Add a page view counter for websites/privacy-sensitive-analytics#4

Update the example Nginx config to include the settings for websites/privacy-sensitive-analytics#3

Add support for a list of domains to skip for websites/privacy-sensitive-analytics#3

Only write stats for whitelisted domains for websites/privacy-sensitive-analytics#3

Measure from beginning of DomReady event - if we're running in that event, the end will still be 0

ResponseTime is ms - treat as an integer

Timezone is minute offset from UTC - so force as integer

Switch path to a tag and timezone to a field for websites/privacy-sensitive-analytics#2

Domains and platform should be normalised to lowercase

Add a field for referring domain

Blank the referrer if we detect the user navigated in-site: we're not looking to track them around our site, the referrer is captured only to see what sources of traffic there are

We have a working PoC for websites/privacy-sensitive-analytics#1

Set up repo