Unit 4: Managing and Processing Real-time Data¶
With your devices connected, your data will start to accumulate in EnOS. It could be quite a challenge to manage the vast amount of IoT data. This is where the EnOS Data Asset Management comes in, where it provides you with data storage and streaming data processing services to help you reduce development costs and improve data access efficiency.
Before you dive in, it is recommended that you familarise yourself with the key concepts on stream data processing and data storage.
Real-Time Data Flow¶
To be able to efficiently manage your data, you will first need to understand the Real-time Data Flow.
Configure Storage Policy¶
Note that the raw data uploaded to EnOS and the stream processing results are NOT stored when you do not configure the data storage policy to define which data goes where.
The storage policy gives you the flexibility to determine what data to store, where to store, and how long to store. The design of EnOS data store takes the best practices of the energy/utility domain into consideration by allowing you to store time-series data into different buckets according to the data types and provides different APIs for you to access the data in different buckets to increase data access efficiency. For more information about data storage policy, see Configuring TSDB Storage.
Develop Stream Data Processing Jobs¶
To lower the development barrier of stream data processing, EnOS encapsulates the complexity of the underlying stream processing engine and provides GUI-based data development IDE to help you easily perform stream analytics.
For a more in-depth understanding of stream data processing on EnOS, see Stream Processing.
Get Data through APIs¶
As mentioned above, EnOS provides a data service API for each TSDB storage type.
Subscribe to Data¶
You can also use the Data Subscription Service to get your data more efficiently. The Data Subscription Service improves the efficiency of API calls by actively pushing data to the subscriber applications when new data arrives. The service allows you to subscribe to the following types of data: