Manually deleting data records above certain thresholds. Implications??

[Sven]

Morning All,

I have an Owl energy meter that I'm feeding the data in via a an SDR dongle, rtl_433 and a json log file.

I noticed that when the batteries in the transmitter start to fail, the value of the records spike and it completely ruins the graphs/trends etc.

I'm going to delete the items that are obviously wrong. In my case, values about 10,000W

(Yes... I'm going to alter my template to preprocess and filter out items above 10,000W for the future - we don't use that level here)

Do I need to worry about historical trends? Is there anything else I need to worry about, or can i just get on with:

delete from history where itemid=(xxx) and value > 10000;

[tim.mooney]

How long do you keep history (and trends) for that device?

The trend data has already been poisoned. Whether or not you delete the history data for that item in those periods, the trend data min/max/avg already includes incorrect results.

I'm not sure what I would do if I were in your situation. It might be better to start fresh after fixing the problem, by setting the history and trends values for that item temporarily to something small and then letting the housekeeper clean out all the old data.

If you absolutely don't want to lose the old good data, then you might also consider altering the history values that are out of bounds, rather than deleting them. If you did that, though, you would also want to alter the value_min, value_max, and value_avg for the item during certain periods too.

[Sven]

How long do you keep history (and trends) for that device?

I have it configured to keep 365d of values, 1095d of trends... but I've only been collecting since 2020-12-27, so not a full year yet.

I'd rather not bin the old good data. The last year has been a good base line for the next year coming up (working less from home, newer lower power equipment coming in, etc etc etc)

[Rizzah]

I know this about 4 year later, but for people who stumbled upon this for the same issue as I have:

I have fixed this by logging the value as numeric, with a javascript processing step that changes the value to NULL if above a certain value (in my case above 200).
This makes it so the value is not updated into the graph.