websites/privacy-sensitive-analytics#13: Downsample Search Information

It probably isn't possible to downsample the search terms themselves

Actually, on reflection, I'd go further. It isn't really desirable to downsample the terms themselves - some users search for some horrible terms, I don't really want those in a long-lived database.

What'd be nice is if we could find a good way to categorise those terms and record a count for each. Of course, that's easier said than done (at least in any kind of reliable manner) so may be better left as an aspiration.

So, yeah, I think we probably want to capture the following (per domain)

• Number of searches
• Avg num results per search

We can pull number of searches per domain with

from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "pf_analytics_test_search_terms")
  |> filter(fn: (r) => r._field == "search_term")
  |> drop(columns: ["sess_id"])
  |> aggregateWindow(every: 15m, fn: count)
  |> map(fn: (r) => ({r with _field: "search_count", action: "search"}))

Result counts are much simpler

from(bucket: "telegraf/autogen")
  |> range(start: v.timeRangeStart)
  |> filter(fn: (r) => r._measurement == "pf_analytics_test_search_terms")
  |> filter(fn: (r) => r._field == "result_count")
  |> drop(columns: ["sess_id"])
  |> aggregateWindow(every: 15m, fn: mean)
  |> group()

We now have count and average result counts - don't think there's really anything else we want to keep long term.

I'll keep this open for now, on the offchance I feel inspired to try some kind of categorisation but I suspect I won't (if nothing else, the categories are likely to be quite site specific).

One option might be to "downsample" them into a dedicated DB whilst I try and decide what to do, that way they can easily be dropped if I decide I don't want to do anything with them

I've implemented capture of search terms into bucket search_terms/autogen.

There's a wiki page detailing how to list them