websites/privacy-sensitive-analytics#18: Image endpoint

One of the reasons that I'm interested in building this is so that I can see what percentage of visitors have the main analytics blocked (and how that ratio varies between clearnet, tor and i2p users).

It'll also help identify whether PFA is an effective means for delivering the protections that I also use it for - for example, would doFakeOnionThing() protect more users if it was delivered via other routes separately from analytics?

To implement this, we probably need to come up with a server-side means of increasing cardinality without impacting upon privacy.

If we only tag with domain and page then our series key will be

$measurement_name,domain=www.bentasker.co.uk,page=/,counter

If there are multiple simultaneous views with the same series key, the writes will effectively upsert one another and we'll end up with a count of 1 rather than 4 (or however many there actually were).

This could be mitigate a little by writing the user-agent (or something derived from it) in as a tag

$measurement_name,domain=www.bentasker.co.uk,page=/,browser=firefox,platform=win32,counter

But there are a couple of issues with this approach

1. By blocking the analytics agent/endpoint, the user has signalled that they don't want data about them collected. It seems wrong to disregard that to solve a technical challenge on our side
2. It only mitigates the issue a little - if there were simultaneous views by users using the same browser/platform we run into the same thing

We could similarly mitigate by using some or part of the user's subnet, but that runs up against point 1 too (and, if anything, more severely).

In #6 we adjusted the agent to generate a UUID for use as a session identifier to address a similar issue relating to high concurrency.

I think that's the best approach to follow - it'll mean extremely high cardinality in the raw data, but that identifier can be stripped when the data is downsampled.

Commit f2762510 implemented the agent-side logic, but we can't use that here - we need the calculation to be entirely server side.

OK, so the server side logic is built, and I've got a copy of it currently active.

There are two methods by which this can be enabled/deployed within a site.

The first is to just direct link it

<img src="https://pfanalytics.bentasker.co.uk/count.gif">

The second is to update a site's config to serve it from a local path and proxy it through

location /count.gif {

    proxy_pass https://pfanalytics.bentasker.co.uk;
    proxy_set_header Referer $http_referer;
    proxy_set_header Host pfanalytics.bentasker.co.uk;
}

And them embed with

<img src="/count.gif">

Whilst more complex, this approach means you can return sensible Refferer-Policy headers without knackering the efficiency of the site's analytics.

The counter will be triggered whether the user uses the JS agent or not, so if we're interested in how many users didn't have JS we need to do some maths.

The following Flux query will subtract the JS agent derived count from that calculated by the hit counter

hc = from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "pf_analytics_test_pixel")
  |> filter(fn: (r) => r.domain == v.domain)
  |> filter(fn: (r) => r._field == "count")
  |> group()
  |> count()
  // Inject a fake timestamp for use when joining
  |> map(fn: (r) => ({r with _time: 1234}))


js = from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "pf_analytics_test")
  |> filter(fn: (r) => r.domain == v.domain)
  |> filter(fn: (r) => r._field == "response_time")
  |> filter(fn: (r) => r.action != "\"ready\"")
  |> group()
  |> count()
  // Inject a fake timestamp for use when joining
  |> map(fn: (r) => ({r with _time: 1234}))


join(tables: {js: js, hc: hc}, on: ["_time"])
  // Calculate the delta
  |> map(fn: (r) => ({_value: r._value_hc - r._value_js}))

The figure tells us how many hit the hitcounter but didn't write stats in.

It's not perfect - if there are no JS derived stats, then no results will be returned instead of a number. We need join.tables for that, but I'm running InfluxDB 1.8.10 and that predates that Flux package.

Most of the time though, there are going to be at least a few users with JS enabled, so it's not an issue I expect to run into regularly.

The data can be downsampled with the following Flux task

option task = {
    name: "downsample_hitcounter",
    every: 15m,
    offset: 1m,
    concurrency: 1,
}


out_bucket = "websites/analytics"
host="http://192.168.3.84:8086"
token=""

sourcedata = from(bucket: "telegraf/autogen", host: host, token: token)
    |> range(start: -task.every)
    |> filter(fn: (r) => r._measurement == "pf_analytics_test_pixel")
    |> drop(columns: ["sess"])
    |> aggregateWindow(every: 15m, fn: sum)
    |> map(fn: (r) => ({ r with
                _field: "hitcount",
        _measurement: "pf_analytics_pixel"
    }))
    |> drop(columns: ["_start", "_stop", "type"])
    |> to(bucket: out_bucket, host: host, token: token)

I've created a Wiki page describing this functionality.