Record Generator


This stage generates new records based on the specified conditions. The functions of this stage include:

  • Generating required records by fixed calculation or querying frequency, or by triggering points.
  • Determining the dataset to be generated for the specific device and point, i.e., Map<assetId,Set<pointId>>, based on the filtering conditions (such as asset ID, model ID, asset tag, asset tree ID, and asset tree tag).


The triggering conditions for generating the records are based on the determined dataset:

  • Get the record data from Redis based on the fixed query frequency and time interval.
  • Get the record data from TSDB based on the fixed query frequency and time range.
  • Simulate the record data of certain types based on the fixed calculation frequency.
  • Simulate triggering point data based on the fixed calculation frequency.

Configuration

The configuration tabs for this stage are General, Basic, Input/Output, and Record Generate.

General

Name Required? Description
Name Yes The name of the stage.
Description No The description of the stage.
Stage Library Yes The streaming operator library to which the stage belongs.
Required Fields No The fields that the data records must contain. If the specified fields are not included, the record will be filtered out.
Preconditions No The conditions that must be satisfied by the data records. Records that do not meet the conditions will be filtered out. For example, ${record:value('/value') > 0}. For the syntax of EL expressions, see Expression Language.
On Record Error Yes

The processing method for error data.

  • Discard: Error data will be discarded and ignored
  • Send to Error: Error messages will be reported
  • Stop Pipeline: The pipeline will be stopped

Basic

Name Required? Description
Trigger Type Yes Select the method of triggering the data record generation. Options are By Fixed Frequency and By Input Point.
Trigger Frequency Yes When you select generating data records by fixed frequency, select the frequency for generating new records.
Trigger Point Yes When you select generating data records by input point, enter the name of the triggering point (by the format of pointId, not modelId::pointId).

Input/Output

Name Required? Description
Generate Type Yes

Select the method to generate new records.

  • By AssetIDs：Match measurement points by asset IDs
  • By ModelIDs：Match measurement points by model IDs
  • By Asset Tags：Match measurement points by asset tags
  • By Asset Tree：Match measurement points by asset tree IDs or asset tree tags
Output Asset No When the By AssetIDs method is selected, specify the asset ID. In the Output Point field, specify the measurement point for the generated data records, using the format {pointId}.
Asset Tag No When the By Asset Tags method is selected, specify the tag key and value. In the Output Point field, specify the measurement point for the generated data records, using the format {modelId}::{pointId} or {pointId}.
Tree Selector No When the By Asset Tree method is selected, specify the method of searching asset trees, either by asset tree IDs or by asset tree tags.
Tree ID No Specify asset tree ID. In the Output Point field, specify the measurement point for the generated data records, using the format {modelId}::{pointId} or {pointId}.
Tree Tag No Specify asset tree tag and value. In the Output Point field, specify the measurement point for the generated data records, using the format {modelId}::{pointId} or {pointId}.
Output Point Yes When the By ModelIDs method is selected, specify the measurement point for the generated data records, using the format {modelId}::{pointId}.

Record Generate

The methods of generating records include: Query From Redis, Query From TSDB, Generate New Point, and Generate Trigger Point. The configuration for each method is as follows.

Query From Redis

Name Required? Description
Use LastUpdate Interval Filter No Select whether to get the latest asset data from Redis using a fixed time interval.
LastUpdate Interval(Minutes) No Enter the time interval for getting the latest asset data.
NotExistHandle Yes

Select the processing method when no data is retrieved.

  • Ignore this Point: Ignore the point data
  • Mock New Point: For all devices of a model, when no latest data is found for a measurement point in Redis, generate mock data for the point.

Query From TSDB

Name Required? Description
Start Time Yes Enter the start time to query the data from TSDB, using the format 2019-08-26T00:00:00+08:00.
Time Interval Yes Enter the time range to query the data.
End Time Yes Enter the end time to query the data from TSDB, using the format 2019-08-26T00:00:00+08:00.
NotExistHandle Yes

Select the processing method when no data is retrieved.

  • Ignore this Point: Ignore the point data
  • Ignore and Send Error: Ignore the point data and report error

Generate New Point

Name Required? Description
Date Format Yes

Select the date format of the generated data. Options are SimpleDateFormat and TimeFunctions.

  • SimpleDateFormat: Generate timestamp by the existing date format.
  • TimeFunctions: Generate timestamp by StreamSets expressions.
Start Time Yes Enter the timestamp for the start time of the new point, using the format 2019-08-26T00:00:00+08:00.
Time Interval Yes Specify the time interval for generating the new point data.
Value Generator Yes

Select the method of generating new point data and the data type.

  • Random Number (Double): Randomly generate the point data.
  • Fixed Incremental as Value: Generate point data by a fixed incremental value. In the Beginning Number field, specify the starting value for the new point data. In the Fixed Incremental field, specify the fixed incremental value.
  • Random Number in Range: Randomly generate the point data within a specified value range. In the MinMax Range field, specify the value range. In the Decimal Scale field, specify the number of decimal places for the point data value.
  • Current System Time: Use the system time when the point is generated as its data value.
  • Input String as Value: Specify a string as the point data value. In the Input field, specify the string. In the Data Type menu, select the data type of the string.

Generate Trigger point

Name Required? Description
Date Format Yes

Select the date format of the generated data. Options are SimpleDateFormat and TimeFunctions.

  • SimpleDateFormat: Generate timestamp by the existing date format.
  • TimeFunctions: Generate timestamp by StreamSets expressions.
Start Time Yes Enter the start time of the generated data, using the format 2019-08-26T00:00:00+08:00, or enter the StreamSets expression for generating the timestamp.
Time Interval Yes Specify the time interval for generating the new point data.
Value Generator Yes

Select the method of generating new point data and the data type.

  • Random Number (Double): Randomly generate the point data.
  • Fixed Incremental as Value: Generate point data by a fixed incremental value. In the Beginning Number field, specify the starting value for the new point data. In the Fixed Incremental field, specify the fixed incremental value.
  • Random Number in Range: Randomly generate the point data within a specified value range. In the MinMax Range field, specify the value range. In the Decimal Scale field, specify the number of decimal places for the point data value.
  • Current System Time: Use the system time when the point is generated as its data value.
  • Input String as Value: Specify a string as the point data value. In the Input field, specify the string. In the Data Type menu, select the data type of the string.

Output Results

The output results of this stage are the generated records, which include the following:

  • Common fields (such as assetId, pointId, modelId, time, value)
  • Stage configuration attributes, which are included in the /attr/recordGenerator field, described in the table below.
Name Data Type Description
assetCfg String The selected method used to determine the dataset to be generated.
generateBy String The selected method used to generate the records.
queryStart/queryEnd String The specified start time and end time to query data from TSDB.
offset String The count of the generated data when the record is simulated.
curQueryBatch String The count of the record batches that triggers the generation of the simulated data.
triggerPoint Boolean Whether the generated data records are triggering points.
generatorKey String When generating triggering points, the value of Generate Type.

Output Example

By Tree ID + Query From Redis

../../../_images/record_generator_result_1.png

By Asset ID + Query From TSDB

../../../_images/record_generator_result_2.png

Generate New Point

../../../_images/record_generator_result.png

Generate Trigger Point

../../../_images/record_generator_result_3.png