Data Quality Annotation Using Flags
PowerTSM annotates the quality of every data point with a flag. This page explains the concept of flags and how they are inherited, and lists the supported flags and their priorities, including custom flags.
Concept of Flags
Section titled “Concept of Flags”The quality of individual data points, i.e. whether a value is valid, missing, faulty, manually replaced, estimated etc., is annotated in PowerTSM per value in the so-called “Flag” information. Flags can be inherited. A formula time series, for example, inherits the flags of a time series referenced in the formula. Flags are also inherited during interval transformation (for example during interval transformation of hourly data for a daily grid). In all cases the worst flag for a time period overrules all other time series value flags involved.
Supported Flags and Their Priorities
Section titled “Supported Flags and Their Priorities”The following table shows the list of currently supported flags and their priorities:
| ID | Name | Priority |
|---|---|---|
| 0 | Ignore¹ | 0 |
| 20 | Accounted | 10 |
| 5 | Manually Replaced | 20 |
| 9 | Valid | 30 |
| 12 | Schedule | 40 |
| 21 | Estimated | 50 |
| 7 | Faulty | 60 |
| 22 | Interpolated² | 70 |
| 19 | Missing | 80 |
Ignore flag ending an acyclic value
Section titled “ Ignore flag ending an acyclic value”| Time Stamp | Input Time Series | Result | |
|---|---|---|---|
| 2022-01-01 | 10 | Valid | Value “10, Valid” will be applied until the next data point |
| 2022-01-02 | 0 | Missing | Value “0, Missing” will be applied until the next data point |
| Time Stamp | Input Time Series | Result | |
|---|---|---|---|
| 2022-01-01 | 10 | Valid | Value “10, Valid” will be applied until the next data point |
| 2022-01-02 | 0 | Ignore | Value “10, Valid” will now only be applied for 2022-01-01 |
Flags in time series calculations and interval transformation
Section titled “ Flags in time series calculations and interval transformation”Behavior of flags when adding two time series:
| Time Stamp | Input Time Series A | Input Time Series B | Result | |||
|---|---|---|---|---|---|---|
| 00:00:00 | 10 | Valid | 0 | Missing | 10 | Missing |
| 01:00:00 | 20 | Valid | 20 | Accounted | 40 | Valid |
| 02:00:00 | 30 | Valid | 30 | Valid | 60 | Valid |
Behavior of flags during interval transformation (15m to 1h):
| Time Stamp (Original) | Input Time Series | Time Stamp Result | Result | ||
|---|---|---|---|---|---|
| 00:00:00 | 0 | Missing | 00:00:00 | 60 | Missing |
| 00:15:00 | 10 | Valid | |||
| 00:30:00 | 20 | Valid | |||
| 00:45:00 | 30 | Accounted |
Ignore flag with the interval transformation rule Average
Section titled “ Ignore flag with the interval transformation rule Average”When calculating the average of a dataset there is a difference if missing zero values are included. The interval transformation rule Average includes all values, including zero values with Missing flag, since this flag does not necessarily mean no data exists in the given time range), but values with flag “Ignore” (i.e. timestamps that were never set to any value) are excluded from the average result.
| Time Stamp (Original) | Input Time Series | Time Stamp Result | Result | ||
|---|---|---|---|---|---|
| 00:00:00 | 0 | Missing | 00:00:00 | 10 | Missing |
| 00:15:00 | 10 | Valid | |||
| 00:30:00 | 20 | Valid | |||
| 00:45:00 | 0 | Ignore |
Explanation: The value 10 is being calculated with the following logic: (0 (Missing) + 10 (Valid) + 20 (Valid)) / 3, since the 4th value is flagged ‘Ignore’.
Custom Flags
Section titled “Custom Flags”Custom flags are optional: the predefined flags above work without any configuration. If needed, it is possible to define custom flags with custom priorities by specifying them in the HAKOM.xml configuration file. Custom flags will overwrite pre-defined flags if a value already exists. However, it is not possible to overwrite pre-defined flag names. How to configure them is described in TSM Custom Flags Configuration; how to inject them in code is described in Custom Flags of the PowerTSM .NET API guide.