project Utilities / Python Influxdb Downsample avatar

utilities/python_influxdb_downsample#9: Record failures to disk



Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: vnext
Created: 05-Feb-23 17:34



Description

With Kapacitor, when a downsample fails, those points are essentially lost.

It'd be nice to have the option to have a failed batch written to disk so that it could be retried later



Toggle State Changes

Activity


assigned to @btasker

I guess the best way to go at this is probably to have a file per write/batch.

That way, things can more easily be removed if they're clearly never going to work.

I also think this mode should be optional (and default to off) - having a YAML representation of failed points being written out sounds like a good way to fill your disk.

verified

mentioned in commit 5634e97adea037443c8d74b71f5a3090a498b185

Commit: 5634e97adea037443c8d74b71f5a3090a498b185 
Author: B Tasker                            
                            
Date: 2023-02-05T23:32:00.000+00:00 

Message

Implement ability to write failures to disk for (possible) later replay (utilities/python_influxdb_downsample#9)

If enabled in config, when an exception is encountered writing to an output, details of the write will be written to a YAML file on disk.

+37 -0 (37 lines changed)
verified

mentioned in commit 944a407dd70e38aa056fe0f701dd32e1a5885341

Commit: 944a407dd70e38aa056fe0f701dd32e1a5885341 
Author: B Tasker                            
                            
Date: 2023-02-05T23:36:00.000+00:00 

Message

docs: update README example to note the ability to dump to disk (utilities/python_influxdb_downsample#9)

+11 -0 (11 lines changed)
verified

mentioned in commit 1717485379a4de6d16e37f1e7c2cbc2525d97e1d

Commit: 1717485379a4de6d16e37f1e7c2cbc2525d97e1d 
Author: B Tasker                            
                            
Date: 2023-02-05T23:44:03.000+00:00 

Message

Gzip the dump YAML, can save a significant amount of disk space (utilities/python_influxdb_downsample#9)

+3 -2 (5 lines changed)

OK, so the dump side is largely implemented.

New config is supported

failures:
    record: True
    path: dump.d

if record is True, then when a write failure occurs the payload will be written out into path (dump.d in the above example).

The dump's filename is of the format f"{op_influx_id}-{bucket_slug}-{dt.strftime('%Y%m%d%H%M%S')}.yaml.gz"

The structure inside the file is as follows

{
    "output_id" : op_influx_id,
    "bucket" : bucket,
    "records" : new_records,
    "failure_time" : dt
}

What we don't currently have is support for reading that data in and replaying it.

I think that can come later though - it's probably better implemented as a standalone(ish) utility.

mentioned in issue #11

verified

mentioned in commit 3248a2c069e4ce4c25b3e6fb511a892a8e2b1386

Commit: 3248a2c069e4ce4c25b3e6fb511a892a8e2b1386 
Author: B Tasker                            
                            
Date: 2023-02-05T23:51:04.000+00:00 

Message

Add logline when dumping a failed write (utilities/python_influxdb_downsample#9)

+3 -0 (3 lines changed)

mentioned in issue #15