LOC-24: Build Tests

Issue Information

Issue Type: Task
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: LocalChat (LOC)
Resolution: Done (2018-06-08 18:31:10)
Target version: V0.0.2,

Created: 2018-05-19 13:11:09
Time Spent Working


Description
Need to build a test wrapper so that we can test that important functionality in the server is working before release.

Also means we can test for regressions etc more easily in future - will just need to add a test case for whatever bugs get fixed.

The wrapper should import the client and use it's methods rather than re-implementing (at least where possible) so that client breakage also affects the tests.


Issue Links

is blocked by LOC-15: Testing Mode
Toggle State Changes

Activity


2018-05-20 10:46:36
Repo: LocalChat
Host:Rimmer

commit 77224eae90c2f3d8ce0ff1c1a8412c7872abd687
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 10:44:42 2018 +0100

Commit Message: LOC-24 Start building test harness

Creating the test harness - it can currently spin up a background instance of the server (and will kill it once done).

Doesn't currently contain or run any tests, and doesn't currently do anything with the client.

tests/run_tests.py | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 78 insertions(+), 0 deletions(-)


View Commit | View Changes

2018-05-20 22:22:36
Repo: LocalChat
Host:Rimmer

commit c2d0159f3d7352f4d2a970c0a0bb1af8b3ed108c
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 22:18:33 2018 +0100

Commit Message: LOC-24 Build some basic tests

As much to start getting the structure as anything - it could be better/tidier but it's functional.

Uses the client as a module in order to run 3 basic tests

- test_one Create a room
- test_two Attempt to join with invalid credentials
- test_three Join with valid credentials

Test functions simply return a list of the following structure 

    	[
    	{'Test' : 'Join the room','Result' : 'FAIL', 'Notes': '' },
    	isFatal
    	]


Where isFatal dictates whether a failure should abort the test run or not (i.e. if other tests depend on something it does, just give up)

tests/run_tests.py | 225 +++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 222 insertions(+), 3 deletions(-)


View Commit | View Changes

2018-05-20 22:24:36
Repo: LocalChat
Host:Rimmer

commit 0e3dcf4dfe636578e88305d28247ce75bce145dc
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 22:22:51 2018 +0100

Commit Message: LOC-24 Add numbers to tests for easier reference when failures happen

tests/run_tests.py | 6 ++++--
1 files changed, 4 insertions(+), 2 deletions(-)


View Commit | View Changes

2018-05-20 22:38:36
Repo: LocalChat
Host:Rimmer

commit 6bb4b1f7db36dbc3b0840710eea3c71d5d4eb53c
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 22:35:59 2018 +0100

Commit Message: LOC-24 Add test to ensure SYSTEM is using E2E encryption

Test steps

- Check that a message was pushed to the room
- Check it isn't plain text
- Check we can decrypt it with the syskey we got when joining the room
- Check the plaintext is valid JSON
- Check it's a correctly formatted message payload

tests/run_tests.py | 47 ++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 46 insertions(+), 1 deletions(-)


View Commit | View Changes

2018-05-20 22:52:36
Repo: LocalChat
Host:Rimmer

commit 7d46d53ee2e22d1b1e8d9cdf0a3c8a48f17a2ceb
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 22:49:45 2018 +0100

Commit Message: LOC-24 Close and re-open DB connection, and implement E2E test

These are part of one commit because implementing the test revealed a potential race.

API requests can fail because of lock contention with the test script, leading to a failed test (and often then an exception when we try to query the database).

Now, we open a new connection to the database just before querying and then close straight after to ensure locks do not become an issue.

This wouldn't be an issue with a production deployment as the DB is in memory and only ever being accessed by the one thread.

tests/run_tests.py | 77 +++++++++++++++++++++++++++++++++++++++++++++++----
1 files changed, 71 insertions(+), 6 deletions(-)


View Commit | View Changes

2018-05-20 23:04:37
Repo: LocalChat
Host:Rimmer

commit f6539e6c016cb6a51fa92aeb5f0e1366184dd23b
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 22:57:57 2018 +0100

Commit Message: LOC-24 Added a test for user invitation

tests/run_tests.py | 31 ++++++++++++++++++++++++++++++-
1 files changed, 30 insertions(+), 1 deletions(-)


View Commit | View Changes

2018-05-20 23:04:37
Repo: LocalChat
Host:Rimmer

commit 4e9cdea7c681f114bbbd7db12c27322a69d8ae3d
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 23:02:34 2018 +0100

Commit Message: LOC-24 Invited user now joins under a new client instance

This is partly to make things available for later tests, but also to ensure that the invite functionality is actually working in the background

tests/run_tests.py | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 58 insertions(+), 2 deletions(-)


View Commit | View Changes

2018-05-20 23:40:36
Repo: LocalChat
Host:Rimmer

commit bf92471864dcf9c22ae9a6fe3431586746c915fd
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 23:22:09 2018 +0100

Commit Message: LOC-24 Added "Send a message" test

Test does the following

- Poll for messages as admin user to flush the queue
- Send a message as testuser
- Poll again as admin user to (hopefully) fetch the message
- Poll as testuser to flush queue (so it's tidy for any future tests)
- Check admin received a message and that it's well formed
- Check that the reported payload is a match

tests/run_tests.py | 45 ++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 44 insertions(+), 1 deletions(-)


View Commit | View Changes

2018-05-20 23:58:36
Repo: LocalChat
Host:Rimmer

commit e39d36ad48666caae5a9df1cd05b3db7500f786d
Author: B Tasker <github@<Domain Hidden>>
Date: Sun May 20 23:55:45 2018 +0100

Commit Message: LOC-24 Added tests for kick/ban functionality

Test 11 invites a user, has them join and then kicks them (before verifying in the database)
Test 12 has that user rejoin and then bans them (again verifying against the database)

tests/run_tests.py | 170 +++++++++++++++++++++++++++++++++++++++++++++++++++-
1 files changed, 169 insertions(+), 1 deletions(-)


View Commit | View Changes

2018-05-21 00:41:38
A variety of tests have now been added, though the script output is simplistic 
Running test_one 
Running test_two 
Running test_three 
Running test_four 
Running test_five 
Running test_six 
Running test_seven 
Running test_eight 
Running test_nine 
Running test_ten 
Running test_eleven 
Running test_twelve 
Running test_thirteen 
Running test_fourteen 
| No | Test                             | Result | Notes |
|--------------------------------------------------------|
| 1  | Create a Room                    | Pass   |       |
| 2  | Join the room with invalid creds | Pass   |       |
| 3  | Join the room                    | Pass   |       |
| 4  | SYSTEM uses E2E                  | Pass   |       |
| 5  | Ensure payloads are encrypted    | Pass   |       |
| 6  | Invite a user                    | Pass   |       |
| 7  | Join as Invited User             | Pass   |       |
| 8  | Send a Message                   | Pass   |       |
| 9  | LOC-5 Try to invite SYSTEM       | Pass   |       |
| 10 | Send a Direct Message            | Pass   |       |
| 11 | Invite and kick a user           | Pass   |       |
| 12 | Ban a user                       | Pass   |       |
| 13 | Automated message expiration     | Pass   |       |
| 14 | Automated Room Closure           | Pass   |       |
----------------------------------------------------------


The output at the top is largely just to aid in identifying where an exception happened (if there is one) as the backtrace won't tell you. It'll be removed later.

Tests 13 & 14 are quite long running as they each have to wait a number of minutes to allow the server's scheduler to fire.

It could actually do with being moved to using more of a plugin like model (with each test being it's own plugin) but I was in the mood to write the tests so have done it as one monolithic file for now.

There's functionality that isn't currently covered too - for example leaveRoom is not currently tested - so more tests will need to be added, but it's getting quite late so I figured I'd call it a day.

Also need to add a datestamp to the output so that it can be piped to a file to create a record of a test. Along those same lines, I'd also like the test script (if possible) to get the commit reference of the last commit in the repo so that any record of the test includes the revision that was tested.

2018-05-21 00:52:36
Repo: LocalChat
Host:Rimmer

commit 52b178296b78a821e74de19ac9eab8dd61730f8a
Author: B Tasker <github@<Domain Hidden>>
Date: Mon May 21 00:50:13 2018 +0100

Commit Message: LOC-24 Add timestamp and revision to test output

The test script will now output a serial number for the test in the format 
    	[YYYYmmDDHHMMSS]-[Latest commit]


tests/run_tests.py | 7 +++++++
1 files changed, 7 insertions(+), 0 deletions(-)


View Commit | View Changes

2018-05-23 11:42:38
Repo: LocalChat
Host:Rimmer

commit e614010c1b267314f313cd5f3fb67e2d0f8a72de
Author: B Tasker <github@<Domain Hidden>>
Date: Wed May 23 11:40:34 2018 +0100

Commit Message: LOC-24 add test for /leave functionality

Moved it before the earlier tests as otherwise the room it tests against will auto-close because of the delay

tests/run_tests.py | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 files changed, 88 insertions(+), 3 deletions(-)


View Commit | View Changes

2018-06-08 18:31:11
btasker changed status from 'Open' to 'Resolved'

2018-06-08 18:31:11
btasker added 'Done' to resolution

2018-06-08 18:31:14
btasker changed status from 'Resolved' to 'Closed'