When searches are run, the following logic is applied
# Check that the index is up to date
# and refresh if not
if time.time() - LAST_INDEX_CHECK > 300:
LAST_INDEX_CHECK = time.time()
# Check the index
if not storage.getIsIndexFresh():
storage.loadIndex()
So, if the index has not been checked in 5 minutes, it'll be checked for freshness - if there's a fresher index available we'll load it.
However, this logic only runs when searches are submitted - no other request to the HTTP server applies it.
Because searches are only performed periodically, if the index has changed, this has the result of front-loading index-loading latency onto searches themselves (slowing the user visible response time - even if imperceptibly ).
It also leads to stats graphs looking a little odd
The HTTP server also has a /stats
path which can be polled programmatically (there's an example of doing so with Telegraf here)
We should update that path so that it can also trigger an index reload - increasing the chances of a user's search being handled by a fresh copy of the index (probably also want to make the freshness-check period configurable, but will raise a separate ticket for that)
Activity
26-Feb-24 08:40
assigned to @btasker
26-Feb-24 09:05
mentioned in commit b855eb63677242631a08a61f14b14d0f8f22d994
Message
fix: Poll index for freshness on the stats path too (utilities/file_location_listing#47)