utilities/file_location_listing#60: Experimental: indexing headings

Extracting headings isn't any particular bother, but it leads us onto the question of how best to index them. Do we:

1. Treat them like tags and have a dedicated index for them
2. Effectively roll them into the main index

Option 1 works best if we assume that (like tags) there's a high level of duplication between pages. That's certainly likely to be true to some extent (with headers like Background and Conclusion being pretty common).

That, though, is a big part of why I'm curious to play around with this - I've no idea whether duplicated headers are the exception or the rule.

If they are the rule, then we probably want to create an ignoretags equivalent so that we don't bother indexing the uninteresting ones (like Conclusion).

To control whether or not headings get indexed, I've added support for a new config env var (HEADINGS_ENABLED).

It defaults to y, setting it to n will disable building of the headings index (or, will once I've implemented that bit)

Commit 227f5620e4a587eb9df304ca7aad900673939a5b implements creation of the index file - headings.

I don't intend to make the search code consume this quite yet. I've implemented creation of the index so that it's easy to look over and identify whether it's unnecessary cardinality of if there's some benefit to it.

OK, so taking a look at heading duplication.

For the sake of time, I'm crawling a relatively small set of sites

https://snippets.bentasker.co.uk
https://shot.83n.uk/
https://recipebook.bentasker.co.uk/
https://www.bentasker.co.uk

Note: importantly, this won't currently include my notes. Should probably set another crawl running to include those (in fact, I'll take a copy of the index and run one now)

There are quite a few

ben@optimus:~/tmp$ zgrep '!#!' headings | awk -F '!#!' '{print $1}' | wc -l
3056

That's partly because I screwed up in 0a7f151a2712d4f936b4a2c2201e4bd8564c829c by forgetting to compare against a lowercased copy of the title. Fixed in 0ca3c7bc47fa05d8b9983a36e826e29193ac36a0

Rather than waiting for a recrawl, though, I'll starting by listing headings to see how many low-quality ones stand out

zgrep '!#!' headings | awk -F '!#!' '{print $1}' | less

I can see

about
aim
aims
background
categories
conclusion
contents
cooking time

I got fed up of making my eyes bleed so stopped in the c's.

zgrep '!#!' headings | python -c '
import sys
import json
items = {}

for line in sys.stdin:
        l_s = line.split("!#!")
        pages = json.loads(l_s[1])
        l = len(pages)
        if l > 1:
            items[l_s[0]] = l

s = {k: v for k, v in sorted(items.items(), key=lambda item: item[1], reverse=True)}

for k in s:
    print(f"{k}: {s[k]}")
' | head -n 20

related snippets: 1638
keywords: 317
conclusion: 227
latest posts: 197
description: 174
snippet: 172
language: 171
latest recipes: 166
categories: 144
method: 143
ingredients: 140
cooking time: 137
usage example: 127
based on: 75
project info: 59
archives: 49
read only service: 49
release notes: 45
license: 40
requires: 40

Which makes sense - the Related Snippets header is in the site template of www.bentasker.co.uk so it's going to show up everywhere.

That's easily excluded, but what happens if I add something to the template?

Automatic heading exclusion

Rather than relying solely on manual excludes (which we should still add support for), maybe we should include logic to exclude an item if it has more than x URLs associated with it?

The problem with that, though, is that headings will start dropping out as the size of the index grows. That's fine if the heading is "Introduction" but not so fine if it's because I've been writing about foo a lot

But then... that's probably not in line with the use-case laid out in the issue description: if I'm writing about foo a lot, you'd expect the titles (or at least tags) to be associated with it.

The aim of this feature was to try and catch instances where I mention bar in passing whilst writing about foo.

So, arguably, the keywords that are most interesting are the ones with the fewest associated URLs?

That does still carry the risk of losing stuff from the index - if I mention bar in passing but later gain an interest in it, we wouldn't want it that first reference to be lost from the index. But for that to happen, I'd have to be using the same title over and over which seems unlikely.

zgrep '!#!' headings | python -c '
import sys
import json


for line in sys.stdin:
        l_s = line.split("!#!")
        pages = json.loads(l_s[1])
        l = len(pages)
        if l == 1:
            print(l_s[0])
' | sort

There's a while bunch of cruft in there because of the title's screw up, but this does look like it should be more useful (capturing things like windscreen drain holes) for example.

So, I think we want 2 config sets for this

• A max threshold: if the number of urls with this heading is over the threshold, it's left out of the index
• A set of ignored headings: This'll allow me to quickly exclude a heading if I notice that a not-useful one has snuck in

I don't think the threshold necessarily needs to be 1, we could probably start with 5 and go from there.

The crawl's not done and there are still far too many entries

$ zcat ~/tmp/search_db/headings | wc -l
15838

There are more than a few domains where skipping based on title matching isn't working because of differences in formatting.

So, I sort of wonder whether the answer might be to have the crawler not look at <h1>.

There is some scope for it to miss stuff if the title and <h1> are wildly different, but that's probably the sort of edge-case that I don't really want to have to worry about

Following the big crawl, that's resulted in a pleasing drop in the size of the index

$ zcat ~/tmp/search_db/headings | wc -l
4675

It looks like that number's actually still a little high, it looks like title exclusion isn't working on some markdown files.

One thing that I notice whilst looking through is there are a bunch of common tokens that I'm unlikely to search for.

For example, there are a ton of headings which start with "what is".

It probably doesn't make sense to include those phrases in the index, it's basically just wasted bytes.

So, we could perhaps have an excluded phrase list, with it's constituents being stripped out of any indexed heading (although we'd probably then need to dedupe because "What is Scunthorpe" and "Where is Scunthorpe" suddenly became the same thing).

Something to think about for later rather than something I want to implement now

As we're not yet implementing support for using the generated index, I'm going to turn headings support off by default.

That way, I can enable it on my systems but anyone crazy enough to be pulling my images won't get hit by an issues it brings with it.

OK, to summarise this issue then

• Setting environment variable HEADINGS_ENABLED to y will enable indexing of page headings
• Only headings with fewer than HEADINGS_MAX_URLS (default 5) associated URLs will be indexed
• Specific headings can be ignored by adding them to config/ignoreheadings.txt
• The resulting index is headings which has index type 2

Before closing, I'm going to update the title of this issue to reflect the fact we're only generating (but do not use the index).

I'll likely implement use of the index in a feature branch so that I can play around with it over an extended time.