websites/privacy-sensitive-analytics#19: Events API

There are some concerns we'd need to address here around cardinality.

The event type is going to need to be a tag. So, obviously we need to be a little careful about the event types that we add.

But, we also need to ensure that a malicious user can't drive up cardinality by spamming nonsense events into the system - the server side should probably maintain a whitelist of accepted events.

As a quick fag packet design though, we probably want to capture

• domain
• page
• Event type
• Sess ID (if enabled)

It might also be prudent to capture platform and timezone - partly to maximise the chances of the series key being unique to that user, but also so it's possible to identify whether a given event is more common amongst a certain portion of a userbase.

I've created a function in the agent, usage is as follows

recordEvent("bentest")
recordEvent("bentest2","this will be recorded as a note against the event")

The first results in the following post request body

{
"domain":"www.bentasker.co.uk",
"page":"/posts/blog/general/695-an-analysis-of-search-terms-used-on-bentasker-co-uk.html",
"timezone":-60,
"platform":"Linux x86_64",
"sess_id":"07ab-8022",
"event":"bentest",
"note":""
}

The Nginx config has been updated to add support (so that we get valid CORS headers back), but there's currently no processing support for these requests

This seems to work, I've tested by adding the following

if (typeof(recordEvent) == "function"){
    recordEvent("searchTermIntercepted", term);
}

And the events get logged as they should

A count of events can be graphed out with

from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "pf_analytics_event")
  |> filter(fn: (r) => r._field == "counter")
  |> filter(fn: (r) => r.event == "searchtermintercepted")
  |> group(columns: ["domain"])
  |> aggregateWindow(every: v.windowPeriod, fn: sum)

For a slightly more complex one, the following query graphs out the number of clicks per social share button

from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r._measurement == "pf_analytics_event")
  |> filter(fn: (r) => r._field == "counter" or r._field == "note")
  |> filter(fn: (r) => r.event == "socialclick")
  |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> rename(columns: {counter: "_value"})
  |> group(columns: ["domain", "note"])
  |> aggregateWindow(every: v.windowPeriod, fn: sum)

Downsampling can be achieved with a script like this

option task = {
    name: "downsample_pfanalytics_events",
    every: 15m,
    offset: 1m,
    concurrency: 1,
}

out_bucket = "websites/analytics"
host="http://192.168.3.84:8086"
token=""

sourcedata = from(bucket: "telegraf/autogen", host: host, token: token)
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "pf_analytics_event")
    |> filter(fn: (r) => r.event != "socialclick")
    |> filter(fn: (r) => r._field == "counter")
    |> drop(columns: ["_start", "_stop", "type", "sess"])
    |> aggregateWindow(every: 15m, fn: sum)
    |> to(bucket: out_bucket, host: host, token: token)

sourcedata2 = from(bucket: "telegraf/autogen", host: host, token: token)
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "pf_analytics_event")
    |> filter(fn: (r) => r.event == "socialclick")
    |> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
    |> rename(columns: {counter: "_value", note : "clicked"})
    |> drop(columns: ["_start", "_stop", "type", "sess", "counter"])
    // Because we pivoted we need to restore the group key so that 
    // the aggregate doesn't strip it
    |> group(columns: ["domain","event","page","platform","timezone","clicked","_measurement"])
    |> aggregateWindow(every: 15m, fn: sum)
    |> set(key: "_field", value: "counter")
    |> to(bucket: out_bucket, host: host, token: token)

The first query handles simple counters, the second handles events where we want the note to become a group key (in this case, which social network share button was clicked).