websites/privacy-sensitive-analytics#12: Implement bad-domains reporting and utilities

We now capture stats about the number of bad domain requests.

Number of writes originating from a bad domain:

from(bucket: "telegraf/autogen")
  |> range(start: ''' + START + ''')
  |> filter(fn: (r) => r._measurement == "pf_analytics_test_unauth" and r._field == "domain")
  |> map(fn: (r) => ({ _time: r._time, ref: r._value, _value: 1, host: r.host }))
  |> group()
  |> aggregateWindow(every: 15m, fn: sum)

Number of unique bad domains observed in period

from(bucket: "telegraf/autogen")
  |> range(start: ''' + START + ''')
  |> filter(fn: (r) => r._measurement == "pf_analytics_test_unauth" and r._field == "domain")
  |> map(fn: (r) => ({ _time: r._time, _value: r._value, host: r.host}))
  |> window(every: 15m)
  |> distinct()
  |> count()
  |> map(fn: (r) => ({ _time: r._stop, host: r.host, _value: r._value}))
  |> group()

That ones actually currently probably less useful than it sounds. In most periods it'll be 1 or 2, and there's no reliable way to use it to extrapolate out to later periods (were the 3 in the next hour the same bad-domain or different ones).

If we want to be able to capture that, we'd need to write the actual domain in as a field or tag value. The problem with that is you either risk runaway cardinality, or accept that the data's not really downsampled (because we'd need to write a point per hit).

Will give that one some more thought

We now write a list of the domains in, so I've added a graph to the dashboard that shows the number of unique bad domains in the time period - it uses better Flux than the downsampling version above

from(bucket: "websites/analytics")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "pf_analytics_bad_domains" 
                       and r._field == "bad_domain_domains")
  |> aggregateWindow(every: v.windowPeriod, 
  fn: (column, tables =<-) 
          => tables 
                |> distinct() 
                |> count(),
  createEmpty: true)

We have reporting in place

Commit b1911bf2 implemented a utility for viewing the generated screenshots, but again there's definitely some refinement possible

I've since opted to remove the screenshot functionality (#14) so there's nothing extra to do here.