LOC-6: Periodic message clearance



Issue Information

Issue Type: New Feature
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: LocalChat (LOC)
Resolution: Done (2018-05-19 12:39:15)
Target version: V0.0.2,
Components: Server ,

Created: 2018-05-12 16:15:54
Time Spent Working


Description
One of the core principles of the design is that the server should only store the message payloads for a short time.

So need to have a thread (or something) running to periodically flush out older messages (I originally said 15 minutes, but 5 sounds better).


Toggle State Changes

Activity


This needs doing, but also need to work out the best way to go about it.

The easiest (tm) way of doing it would be to add a new Flask routed path (say /flush) which would trigger the flush. Then just add a cronjob.

But it means an extra step in the deployment.

I'm wondering if it might be better to spin up an additional thread which could periodically place the request (just as a cronjob would) to avoid the install procedure involving editing cronttab's (as if they're forgotten, or, deleted it could lead to leakage/compromise).
So the way I'm leaning on this, is it's probably best to spin up a thread and create an API endpoint.

So that thread would periodically (via urllib2) place a request to the API to trigger the purge. As this is potentially an expensive operation we should protect it with a passphrase to stop arbitrary users from triggering it and blocking the API.

At startup, the server app should generate a passphrase, which can then be passed into both the API request handler and the 'cron' thread.
Repo: LocalChat
Host:Rimmer

commit 74c2eb491a5bede03dd926d20afd73ffe105a3b7
Author: B Tasker <github@<Domain Hidden>>
Date: Sat May 19 12:04:00 2018 +0100

Commit Message: LOC-6 Initial (and very rough) periodic message clearance

At server startup we spin up a seperate scheduler thread, which places a request to the main app every 60 seconds, in order to purge any messages older than the threshold in the system.

It's currently very, very rough, but seems to work.

There's still a bit to be done here though

- Need to validate the submitted password
- Need to actually generate a password rather than using the temporarily hardcoded 1234

And probably quite a lot else too

server/LocalChat.py | 75 ++++++++++++++++++++++++++++++++++++++++++++++++---
1 files changed, 71 insertions(+), 4 deletions(-)


View Commit | View Changes
Repo: LocalChat
Host:Rimmer

commit 60da935cdf68ed97f73f3cdd0012bf008b44f572
Author: B Tasker <github@<Domain Hidden>>
Date: Sat May 19 12:20:47 2018 +0100

Commit Message: LOC-6 Use autogenerated password for scheduler requests

At startup we generate a 64 char password for the scheduler to identify itself with.

Also, move the scheduler thread launch to within the app processing thread. We do this because otherwise we end up with two scheduler threads (because of the way Flask handles launching the 'main' thread). Also means it'll refresh if the processing thread is changed for some reason.

server/LocalChat.py | 40 ++++++++++++++++++++++++++--------------
1 files changed, 26 insertions(+), 14 deletions(-)


View Commit | View Changes
Repo: LocalChat
Host:Rimmer

commit 6aad0d3551218f84dc186b6333323d78d49574f5
Author: B Tasker <github@<Domain Hidden>>
Date: Sat May 19 12:31:58 2018 +0100

Commit Message: LOC-6 Switch to shorter values and insert print for testing

Configures to run the scheduler every minute and expire messages after a minute.

Created a room, joined it, posted some messages and then left.

Scheduler dumped the following to stdout
        1526729206.144380 Request start
        {u'action': u'schedulerTrigger', u'pass': u'GRu5yYY8ngPGtBfbl6wYTdVA7R5CtYPuTjkG4Xh2SfcClhdEaMQmx9JCWhqQDCtg'}
        Tidying
        Start 8 messages
        End 8 messages
        127.0.0.1 - - [19/May/2018 12:26:46] "POST / HTTP/1.1" 200 -
        1526729266.210174 Request start
        {u'action': u'schedulerTrigger', u'pass': u'GRu5yYY8ngPGtBfbl6wYTdVA7R5CtYPuTjkG4Xh2SfcClhdEaMQmx9JCWhqQDCtg'}
        Tidying
        Start 8 messages
        End 0 messages
        127.0.0.1 - - [19/May/2018 12:27:46] "POST / HTTP/1.1" 200 -
        1526729326.238788 Request start
        {u'action': u'schedulerTrigger', u'pass': u'GRu5yYY8ngPGtBfbl6wYTdVA7R5CtYPuTjkG4Xh2SfcClhdEaMQmx9JCWhqQDCtg'}
        Tidying
        Start 0 messages
        End 0 messages
        

As the auto-purge is an important function, should probably make the period configurable so that we can write automated tests to verify the behaviour.

server/LocalChat.py | 12 +++++++++++-
1 files changed, 11 insertions(+), 1 deletions(-)


View Commit | View Changes
Repo: LocalChat
Host:Rimmer

commit ec523c9d6a648bfc037d4d6e0eff25e17016b987
Author: B Tasker <github@<Domain Hidden>>
Date: Sat May 19 12:37:02 2018 +0100

Commit Message: LOC-6 Remove test lines and make purge interval configurable

It's currently hardcoded at 10 minutes (600 seconds)

server/LocalChat.py | 23 ++++++-----------------
1 files changed, 6 insertions(+), 17 deletions(-)


View Commit | View Changes
This is now implemented, and based on testing, seems to be working.

Marking this issue as complete (and we've now unblocked LOC-11 woohoo)
btasker changed status from 'Open' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'