I guess the best way to go at this is probably to have a file per write/batch.
That way, things can more easily be removed if they're clearly never going to work.
I also think this mode should be optional (and default to off) - having a YAML representation of failed points being written out sounds like a good way to fill your disk.
Activity
05-Feb-23 17:34
assigned to @btasker
05-Feb-23 17:44
I guess the best way to go at this is probably to have a file per write/batch.
That way, things can more easily be removed if they're clearly never going to work.
I also think this mode should be optional (and default to off) - having a YAML representation of failed points being written out sounds like a good way to fill your disk.
05-Feb-23 23:33
mentioned in commit 5634e97adea037443c8d74b71f5a3090a498b185
Message
Implement ability to write failures to disk for (possible) later replay (utilities/python_influxdb_downsample#9)
If enabled in config, when an exception is encountered writing to an output, details of the write will be written to a YAML file on disk.
05-Feb-23 23:36
mentioned in commit 944a407dd70e38aa056fe0f701dd32e1a5885341
Message
docs: update README example to note the ability to dump to disk (utilities/python_influxdb_downsample#9)
05-Feb-23 23:44
mentioned in commit 1717485379a4de6d16e37f1e7c2cbc2525d97e1d
Message
Gzip the dump YAML, can save a significant amount of disk space (utilities/python_influxdb_downsample#9)
05-Feb-23 23:47
OK, so the dump side is largely implemented.
New config is supported
if
record
is True, then when a write failure occurs the payload will be written out intopath
(dump.d
in the above example).The dump's filename is of the format
f"{op_influx_id}-{bucket_slug}-{dt.strftime('%Y%m%d%H%M%S')}.yaml.gz"
The structure inside the file is as follows
05-Feb-23 23:48
What we don't currently have is support for reading that data in and replaying it.
I think that can come later though - it's probably better implemented as a standalone(ish) utility.
05-Feb-23 23:49
mentioned in issue #11
05-Feb-23 23:53
mentioned in commit 3248a2c069e4ce4c25b3e6fb511a892a8e2b1386
Message
Add logline when dumping a failed write (utilities/python_influxdb_downsample#9)
07-Feb-23 19:10
mentioned in issue #15