project Utilities / File Location Listing avatar

utilities/file_location_listing#47: Stats fetch requests should be able to trigger index reloads



Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: v0.2.6
Created: 26-Feb-24 08:40



Description

When searches are run, the following logic is applied

    # Check that the index is up to date
    # and refresh if not
    if time.time() - LAST_INDEX_CHECK > 300:
        LAST_INDEX_CHECK = time.time()
        # Check the index
        if not storage.getIsIndexFresh():
            storage.loadIndex()

So, if the index has not been checked in 5 minutes, it'll be checked for freshness - if there's a fresher index available we'll load it.

However, this logic only runs when searches are submitted - no other request to the HTTP server applies it.

Because searches are only performed periodically, if the index has changed, this has the result of front-loading index-loading latency onto searches themselves (slowing the user visible response time - even if imperceptibly ).

It also leads to stats graphs looking a little odd

Screenshot of Index Age graph

The HTTP server also has a /stats path which can be polled programmatically (there's an example of doing so with Telegraf here)

We should update that path so that it can also trigger an index reload - increasing the chances of a user's search being handled by a fresh copy of the index (probably also want to make the freshness-check period configurable, but will raise a separate ticket for that)



Toggle State Changes

Activity


assigned to @btasker

verified

mentioned in commit b855eb63677242631a08a61f14b14d0f8f22d994

Commit: b855eb63677242631a08a61f14b14d0f8f22d994 
Author: B Tasker                            
                            
Date: 2024-02-26T09:05:06.000+00:00 

Message

fix: Poll index for freshness on the stats path too (utilities/file_location_listing#47)

+21 -9 (30 lines changed)