Stream Computing

Stream Processing Service provides a set of generic and domain-based operators, to help data developers to quickly compose data processing solutions with just a few clicks. It significantly lowers the entry barrier and improves the efficiency for data development.
 Compared with centralized deployment of cloud computing, Stream Computing not only solves the problems of excessive delay and excessive aggregated traffic, but also provides better support for real-time and bandwidth-intensive services.

../_images/edge_computing_struc.png
  • Low latency, Stream Computing focuses on real-time, short-period data analysis, which can better support the real-time intelligent processing and execution of local services;
  • More secure, the data in Stream Computing is only exchanged between the source data device and the edge device, and no longer uploads all to the cloud computing platform, preventing the risk of data leakage;
  • Relieve the flow pressure, filter and preprocess the data in advance at the edge to reduce the upload flow pressure; For model points, it is often necessary to do some preprocessing, or perform some simple calculations, which can be performed directly using the EnOS Stream Computing module.

Application

EnOS Edge is committed to precipitating common algorithms for stream processing in the IoT field. Developers can complete the development, operation and maintenance of stream data processing tasks through configuration. In addition, the EnOS Stream Computing service also provides a complete set of underlying encapsulated StreamSets operators for developers to develop customized stream data processing tasks to meet complex business needs.

../_images/edge_computing_struc.png

EnOS Stream Computing service can be applied to the following scenarios:

  • Execute data filtering and analysis at the edge more efficiently
  • Reduce data traffic from the site to the cloud
  • Data is only exchanged between device and edge, and no longer needs to be uploaded to Cloud in full -> Prevents the risk of data leakage.

Specific application scenarios such as:

  • Active power calculation of photovoltaic inverters, which supports the calculation of the 5-minute window average value of a single telemetry point of a single device;
  • The 10-minute average active power of the fan supports the calculation of the 10-minute window average value of a single telemetry point of a single device;
  • Equipment status statistics, according to the equipment status (remote communication point) to count the number of equipment in a certain state.

Features

  • Support compatible with pipeline exported from EnOS Cloud
  • Provide rich operators of typical algorithms
  • Optimize computing framework to improve the computing performance of single node by about 50%
  • High-performance operators for wind domain to break bottleneck of single node

Continuous real-time data stream

The data that the streaming data processing engine needs to process is real-time and continuous. The data stream is subscribed and consumed by the stream data processing service in chronological order. Data is generated continuously, so the data stream is continuously integrated into the stream data processing system. Therefore, streaming data is always real-time and continuous.

Continuous and efficient calculation

The computing mode of EnOS Stream Computing service is “event triggered”. The trigger is the continuous stream data mentioned earlier. Whenever new streaming data is sent to the streaming data processing system, the system immediately starts and executes calculation tasks.

Real-time streaming data integration

After stream data triggers stream data processing, the calculation result is continuously recorded in the target data storage according to the pre-configured storage strategy.

Key Concepts

Streaming data processing flow

  • Raw data processing: After the original data of the measuring point is collected by the protocol, it is converted into a model measuring point according to the point mapping template, and then sent to the Stream Computing module. Stream Computing services filter data according to specified thresholds.
  • Data calculation: The data after threshold filtering is aggregated and calculated by the algorithm defined in the data processing strategy.
  • Output calculation result: The data processed by the streaming data processing module will flow into the downstream module, and will be recorded in the time series database (TSDB) or other target storage system according to the pre-configured storage strategy. Users can query the stored data through API.

StreamSets operator

EnOS Stream Computing service provides a complete set of underlying encapsulated StreamSets operators for developers to develop customized stream data processing tasks based on business needs. For more information about the StreamSets operators supported by EnOS Edge, see The list of supported StreamSets operators.