Guarding against future data

Sometimes, data may arrive in your eagle.io data source with timestamps that are newer than the current time; effectively "future" data.

Apart from some forecasting uses, future data is often incorrect and unwanted; an error which could be caused by:

• An incorrectly configured clock on a data logging device
• Sensor, logger or RTU malfunction
• An incorrect timezone in the data source configuration
• Importing data with the wrong timezone configured
• Exporting data with the wrong timezone configured, then re-importing it

As you can see, the errors fall in to two broad categories; either the timestamps were created incorrectly by some fault with the logging hardware/software, or timestamps were incorrectly interpreted while being ingested into the system, due to a timezone mismatch. You can read more about timezones issues, and how to avoid them, in this article.

But assuming for the moment that future data may occur which cannot be easily prevented, how can it be detected and stopped before it is ingested by eagle.io? The answer is to configure a data alarm. This will detect timestamps that occur more than a configurable interval in the future, and prevent them from being ingested, while also optionally triggering an alarm.

In this example, the data alarm has been configured to exclude incoming data more than 3 hours in the future:

Notifications will be sent when data becomes valid or invalid. If the intent is to just exclude the future data and notifications are not required, then they can be disabled:

Finally, although it's not the topic of this particular article, note that the data alarm can also be used for the opposite purpose; to exclude data that is too old from being ingested. But be cautious when excluding old data, as scheduling and caching delays or other issues can occur which may result in data being ingested later than expected. Eagle.io is very good at recovering from problems that can affect the creation and transmission of data, but if you choose to exclude all but the most recent timestamps, then any delays could result in data being missing rather than just late.