utilities/gadgetbridge_to_influxdb#9: Query bounds use incorrect units



Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: v0.4
Created: 26-Aug-23 16:37



Description

This has been driving me crazy for the last few minutes, and then I finally clicked.

When writing into Influx we get a message back that some of the points are older than the retention policy allows

Content-Type: application/json
Request-Id: 035b228a-4417-11ee-be77-0242ac130002
X-Influxdb-Build: OSS
X-Influxdb-Error: partial write: points beyond retention policy dropped=18
X-Influxdb-Version: 1.8.10
X-Request-Id: 035b228a-4417-11ee-be77-0242ac130002
Date: Sat, 26 Aug 2023 13:46:52 GMT
Content-Length: 69

{"error":"partial write: points beyond retention policy dropped=18"}

This is fine... except that it shouldn't be happening.

The telegraf DB has a RP of 7 days, and the script is only supposed to be querying the last 2 days

QUERY_DURATION=7200

This gets turned into a start bound by subtracting it from the current time

    query_start_bound = int(time.time()) - QUERY_DURATION

It's then injected into queries

    stress_data_query = ("SELECT TIMESTAMP, DEVICE_ID, TYPE_NUM, STRESS FROM HUAMI_STRESS_SAMPLE "
        f"WHERE TIMESTAMP >= {query_start_bound} "
        "ORDER BY TIMESTAMP ASC")

So all should be working. Printing the query confirms that the timestamp is being correctly applied

SELECT TIMESTAMP, DEVICE_ID, TYPE_NUM, STRESS FROM HUAMI_STRESS_SAMPLE WHERE TIMESTAMP >= 1693060334 ORDER BY TIMESTAMP ASC

When running the query, though, it all becomes clear

sqlite> SELECT TIMESTAMP, DEVICE_ID, TYPE_NUM, STRESS FROM HUAMI_STRESS_SAMPLE WHERE TIMESTAMP >= 1693060334 ORDER BY TIMESTAMP ASC LIMIT 2;
TIMESTAMP|DEVICE_ID|TYPE_NUM|STRESS
1692079980000|1|1|73
1692084480000|1|1|44

We're getting every record back because the timestamps in this table are millisecond epochs. Some of the other tables use seconds.

So, we need to calculate and use a ms bound for those tables



Toggle State Changes

Activity


assigned to @btasker

verified

mentioned in commit b9b587806015ec1f19f0055f00d5896391feea35

Commit: b9b587806015ec1f19f0055f00d5896391feea35 
Author: B Tasker                            
                            
Date: 2023-08-26T17:40:09.000+01:00 

Message

Calculate a query start bound in milliseconds (utilities/gadgetbridge_to_influxdb#9)

Huami derived tables use a millisecond epoch as their timestamps.

Attempting to force a start time using an epoch in seconds will lead to every record being returned

+7 -6 (13 lines changed)

mentioned in issue #11