Commits: Utilities / Python Influxdb Downsample

Fix: Round run start time down to a fixed precision (utilities/python_influxdb_downsample#27)

Prevent stats error when using custom query

Err on the side of caution, prevent exceptions during stat writes from bubbling up

docs: add document on setting up replication using copy (utilities/python_influxdb_downsample#21)

docs: Add details on copy aggregate (utilities/python_influxdb_downsample#21)

fix: Convert row to a dict when copying (utilities/python_influxdb_downsample#21)

NotWorking: Start to implement copy aggregate utilities/python_influxdb_downsample#21

chore: refactor - move the group-key calculations out of the aggregate loop, it's getting a bit long and hard to read

chore: remove debug print (utilities/python_influxdb_downsample#23)

fix: prevent empty starting window from blatting existing values utilities/python_influxdb_downsample#23

Re-enable dummy creation, located the source of the issue utilities/python_influxdb_downsample#20

Disable dummy creation for now utilities/python_influxdb_downsample#20

feat: Insert a 0 point for empty windows by default (utilities/python_influxdb_downsample#20)

Update aggregates to account for presence of dummies utilities/python_influxdb_downsample#20

feat: Add code to enable the inclusion/population of empty windows utilities/python_influxdb_downsample#20

Add example Grafana dashboard

fix: Prevent failure stats being written when outputs are closing (utilities/python_influxdb_downsample#17)

Add process level stats utilities/python_influxdb_downsample#16

docs: create document detailing statistics utilities/python_influxdb_downsample#16

Implement ability to specify custom tags in config

Push stat showing buffer flush time too

Enable additional stats

Add stats providing times taken to run each stage of the job (measured in nanos) - utilities/python_influxdb_downsample#16

Print when flushing queues, makes it look less like something has hung

Add job-level stats utilities/python_influxdb_downsample#16

Write final stats when closing the object utilities/python_influxdb_downsample#16

Implement ability to push run-time stats to Influx utilities/python_influxdb_downsample#16

docs: update README with stats example utilities/python_influxdb_downsample#16

Implement config support for stats collection (utilities/python_influxdb_downsample#16)

chore: Move code-block closer to the section it relates to

Adjust recordFailures so that we log when a batch write failed. utilities/python_influxdb_downsample#15

Switch upstream writes to batching mode utilities/python_influxdb_downsample#15

Log a line when we've received the query result (helps judge processing times for each stage)

Fix conf.d population in dockerfile

Bugfix: enforce minimum value count for percentile calculation

Chore: fix readability - many comments and whitespace adjustments

Move examples out of the root

Remove unneeded vars

Refuse to continue processing if necessary field(s) are missing

Optimise sooner

Fix aggregate name and enable

Enforce minimum value count

bugfix: fix function name

chore: factor out repeated conditional

Bugfix: don't break on last, the result being returned would be first

Remove debug messages

Add short-circuit if last is the only active aggregator utilities/python_influxdb_downsample#12

Short-circuit iteration if first is the only active aggregator utilities/python_influxdb_downsample#12

feat: Count aggregates when loading config

chore: tidy up

chore: make max/min calculations DRY

chore: Remove repeated conditional

Add support for standard deviation

chore: max average calculation DRY

Add completion message

Add job name to the dump

Add logline when dumping a failed write (utilities/python_influxdb_downsample#9)

Gzip the dump YAML, can save a significant amount of disk space (utilities/python_influxdb_downsample#9)

docs: update README example to note the ability to dump to disk (utilities/python_influxdb_downsample#9)

Bugfix: fix variable for failure path

Implement ability to write failures to disk for (possible) later replay (utilities/python_influxdb_downsample#9)

Update gitignore

docs: update README to detail lint mode

Update entrypoint to accept arbitrary cmdline args

Add support for cmdline args "lint" and "configtest" utilities/python_influxdb_downsample#10

Add doc

Removed unneeded prints

docs: add note on requirement to preserve _stop when using custom functions

Bugfix: When includes are used, some config elements can end up being lists rather than dicts. Add handling

docs: update README for utilities/python_influxdb_downsample#8

Implement ability to use !include within YAML to split config across multiple YAML files utilities/python_influxdb_downsample#8

docs: document examples of porting Flux tasks over

Add support for the imports attribute

Bugfix: measurement is optional, ensure it exists

docs: Update README for utilities/python_influxdb_downsample#3

Implement support for multiple outputs for utilities/python_influxdb_downsample#3

docs: update example at the top

docs: update README to document not_fields utilities/python_influxdb_downsample#7

Add support for excluding fields by name utilities/python_influxdb_downsample#7

docs: update README with examples of using multi-measurment support for utilities/python_influxdb_downsample#2

Allow multiple measurements to be included utilities/python_influxdb_downsample#2

docs: update README to give example with custom flux calls (utilities/python_influxdb_downsample#1)

Implement support for attribute flux_calls for utilities/python_influxdb_downsample#1

docs: update docs for utilities/python_influxdb_downsample#6

Add support for new query attribute in jobs, to define a custom flux query utilities/python_influxdb_downsample#6

chore: Move query generation to dedicated function in prep for utilities/python_influxdb_downsample#6

docs: update README for utilities/python_influxdb_downsample#4

Adjust "as" so that it can also be a dict providing a mapping when multiple fields are being processed utilities/python_influxdb_downsample#4

Implement support for "as" for utilities/python_influxdb_downsample#4

chore: max field name calculations DRY in preparation for utilities/python_influxdb_downsample#4

docs: update README utilities/python_influxdb_downsample#5

Don't custom calculate time windows, have Flux do it for us utilities/python_influxdb_downsample#5

Add support for window() utilities/python_influxdb_downsample#5

Add quiet - it can be omitted safely, but worth including in the example

Add support for a quiet mode

Add timestamp to output

Not sure how I missed that that wasn't part of the class, oops

Add license

Have the docker image ship with an example config

Allow start time to be overridden

Remove unnecessary prints

Create Dockerfile

Start tweaking documentation

Provide control over timeouts

Calculate fixed query start and stop times.

Trap exceptions and start creating human friendly output

Trap query failures

Implement ability to add a name

Add type detection to prevent exceptions/failures when non-numeric fields are present

Try and ensure memory is freed up between tables

Don't gate based on aggregates, the number of functions which use table_vals is now too high for that

Add support for sum and count aggregates

All aggregates are currently calculated within the same function, so remove the conditional and rename the function

Allow output to be written to a different Influx instance

Adjust config loading to have it populate attributes which are called later - this allows them to optionally be omitted from YAML

Implement ability to add arbitrary filter() statements to the flux query

Add support for first and last

Add support for percentiles

Add support for median and mode

Combine mean calculations into function - makes processing more efficient as we only need to iterate each group once

Remove defunct call

Add support for "max" aggregate

Support for aggregate "min"

Prepare to allow multiple aggregates

Strip some more of flux's boilerplate

Break mean calculation out into discrete function

Start a README so I don't forget later

Dynamically build the query

Start moving over to dynamically downsampling based on config

Prepare to move to using a config file

Add yesterday's hacked together script

Setting up repo