utilities/file_location_listing#46: Revalidation should take crawler version into account

The crawler will now look at the stored version when revalidating a page. If that's changed, it overrides the following checks

• Has the page been indexed today?
• Has last-modified changed

The first is overridden so that re-crawls can be done after a software update.

The second is overridden because the stored metadata needs to be updated (otherwise, the version check will fail on every crawl). We could perhaps have added the ability to edit files and update the metadata but that adds a load of complexity compared to simply re-fetching.

There is, however, a drawback to both.

Previous version (obviously) didn't store a version number, so any pre-existing stored info will return a "version" of 0.0.0.

This means that, if I were to cut a release now, a full re-crawl would end up being triggered.

Within the use-case for this change, that's what we want - the whole idea is that we're signalling a change in stored attributes.

In this case, that's undesirable - other than adding the version number, we haven't actually made any changes to the stored attributes (and won't), so it's not worth the additional load.

To avoid that, I think we should hardcode the default version value to be the same as is currently returned by config.getEngineVersion() (0.2.6) - that way all existing storage files will, effectively, be grandfathered into this version.

The other thing I want to look at before closing this out, is naming.

At the moment, we refer to the newly added version as the Engine Version (getEngineVersion() etc).

I'm not sure that that's actually what we want though.

In the next release, we might (for example) release improvements to index navigation. Technically, the engine version will have changed, but there will have been no change to stored attributes.

In that scenario, we wouldn't want files to be re-fetched. But, the naming of the version string does imply that it should be updated with each release.

I think, before releasing, we should re-name this to something like Metadata Version (or StoreFile version?) to avoid any confusion in that area

I've taken a slightly different approach to prevent a full recrawl.

I've added logic to the crawler rather than to the storage module:

    if last_mod != 'FFFFFF' and crawler_ver != ENGINE_VERSION:
        # This is a temporary check added when this logica was 
        # introduced (in '0.2.6'). It acts to prevent an unnecessary
        # recrawl of files that don't have the attribute.
        #
        # TODO: remove this when the engine version next changes
        if ENGINE_VERSION != '0.2.6':
            print(f"\tIndexed by different engine version ({crawler_ver}) triggering recrawl")
            force_recrawl = True

If it were added to storage it'd likely need to be there forever, leaving a weird legacy of the default value being some old version.

Doing it this way means that the weirdness can safely be removed in a later release - if 0.2.7 ends up changing storage attributes then that full recrawl will be both necessary & desirable.

In the latest commit, EngineVersion becomes StoreFileVersion fetched by calling config.getStoreFileVersion()

Re-opening - #49 introduces a change to storage headers, so it now makes sense to allow the new release to trigger a recrawl, need to take that conditional back out.

Change reverted. Re-closing.