utilities/file_location_listing#43: Index Handling Improvements

Re-reads of the index flush the storage cache - if nothing's changed, this does nothing but harm cache performance

This one was actually partially addressed yesterday in d2335032533a89448f008ea32ce8b6a0beb0d172

feat: don't replace index unless reindex has actually led to changes
Storage now generates a checksum for each index file and will only swap in the newly generated index if it includes changes.
This means that the search front-end won't need to reload indexes unless something actually changed

The only issue is, the commit has a bug in it:

if hashIndex(f"{DB_PATH}/{index_type}") != hashIndex(f"{DB_PATH}/{index_type}"):

It's comparing the index to itself (rather than the temporary file) so will never update the index. Fixing that in a moment.

The read in file isn't verified at all, so we can't be sure that it hasn't become corrupted on disk etc

I think what I'd like to do, is to add a "header" line to the index - a single line containing metadata about the index.

Amongst those could be a checksum (it probably doesn't need to be much more than a CRC32) of the rest of the data, perhaps along with a line count.

That way, after loading, we can check whether we've loaded the data that the index claims to contain.

The most recent commit adds stats on checksum mismatches:

    "warnings": {
      "index_checksum_mismatches": 1,
      "tags_checksum_mismatches": 0
    }

We obviously want that to always be 0. We can get a failure rate by dividing by index.loads

OK, obviously needs testing before we can think about cutting a release, but I think we're probably ready to merge that branch.

The temporary file is simply called index.tmp - it's not safe for two processes to try and build an index at the same time

There are two ways to go at this (though not strictly mutually exclusive):

1. Generate a unique name for the temporary file
2. Use locks

Option 1 is much simpler to implement but means we can still get unexpected behaviour - if an index build runs slow (for whatever reason) and another fast run starts, we might ultimately end up overwriting a new up-to-date index with an out of date one (when the slow run completes).

Option 2 prevents that, because the second run simply wouldn't start. It's the same outcome but with less resource wastage but does leave open the possibility of deadlock (we could perhaps expose a stat to show the presence of locks, allowing alerting to be implemented).

I think, for now, making sure the filename is unique will be more than sufficient.

I don't want to over-engineer this either. Whilst we'd give a strong guarantee of uniqueness by importing something like uuid (or shortuuid) adding that to storage would mean the server importing those at runtime for no good reason.

Whilst we could inline the import, for our purposes, it's probably actually enough to call time.time_ns() and use that as a filename suffix.

Talking of locking though, there is potential for a race with the tombstone:

1. indexStorage() runs
2. new index is compiled
3. A delete is triggered for an entry that's already been examined, removing file and updating tombstone
4. new index is written out
5. tombstone is deleted

When the new index is loaded, it'll include an item that no longer exists in storage. Whilst storage should handle this gracefully, it's still less than ideal.

There's a bit of a catch-22 here though:

• If we remove the tombstone first, searches run whilst reindexing will lose the benefit of the tombstone (at least, if the index reloads in the meantime)
• If we leave it until afterwards, we run into the issue above

What we probably need to do is

1. take a copy of the tombstone
2. do the indexing operation
3. Check whether the main tombstone has changed on disk
4. if not, remove it and the copy
5. if so, identify changes and write only those out (or, maybe take the easy route and don't remove the tombstone - it'll likely get rationalised away next reindex)

Either way, I think this probably deserves it's own issue - raised as #44