Record Filter

Supports filtering of input data based on user-configured expressions.

Configuration

The configuration tabs for this stage are General, Basic and Input/Output.

General

Name Required? Description
Name Yes The name of the stage.
Description No The description of the stage.
Stage Library Yes The streaming operator library to which the stage belongs.
Required Fields No The fields that the data records must contain. If the specified fields are not included, the record will be filtered out.
Preconditions No The conditions that must be satisfied by the data records. Records that do not meet the conditions will be filtered out. For example, ${record:value('/value') > 0}. For the syntax of EL expressions, see Expression Language.
On Record Error Yes

The processing method for error data.

  • Discard: Error data will be discarded and ignored
  • Send to Error: Error messages will be reported
  • Stop Pipeline: The pipeline will be stopped

Basic

Name Required? Description
Quality Filter No Filter the data according to the data quality. Only records that meet the quality conditions will be processed by this stage.
Filter Expression Yes The data is filtered according to the configured expression, see below for the specific expression syntax: Filter Expression Syntax Summary
Output Measurement ID Yes Note: If a data satisfies multiple Filter Expressions, the record will be output multiple times and the mesurementId of the output record will be replaced with the configured Output Measurement ID. replaced accordingly

Filter Expression Syntax Summary

To support more flexible and convenient filtering of data, the Filter Expression configuration supports the use of Aviator Script to filter the input data. Here is a brief description of the syntax and functions for this operator usage scenario, for further details, please refer to Aviator Script Documentation.

Basic Types and Operations


Type Description Sample
Number Can be integer, floating point, etc. 123，3.1415
String Consecutive characters enclosed in single or double quotation marks ‘hello’，”world”
+, -, *, / Add, subtract, multiply and divide, + can also be used to splice strings 1+1.2, 5/2, ‘ab’ + ‘c’
<, <=, >, >=, ==, != Comparison Operators 3 != 1
&&, ||, ! Logical with/or/without operators true && false
Variables

Generally, variables are not created directly in the Filter Expression configuration; the Record Formatter algorithm already converts the input data into an object with the name record for use.

Built-in Objects Description
record The input data is converted into a Map object to be passed into the expression execution context, which can be accessed by record as a variable name. Note: For a batch of input data, record represents only one of the data, and the expression will act on each record of the batch in turn.


Since record is a Map object, Aviator provides a syntax for accessing the contents of the Map object in addition to using functions. Here is a brief example.


Syntax Description
record.a.b Indicates access to the value of the b attribute in the a attribute on the variable record
#record.a[1].b indicates access to the b attribute value of the 2nd (starting from 0) object in the array a on the variable record, note: if the variable in the access path is an element of the array/List, you need to start with the #
#`record.name:a.b` indicates access to the value of the b attribute in the name:a attribute on the variable record. Note: If the name of the accessed attribute/variable contains special characters, it needs to be surrounded by # plus ``
Common Functions

There are many functions built into Aviator Script, here are only the common functions used to configure Filter Expression, for more functions please refer to Aviator Script Documentation.


Functions Description
seq.contains_key(map, key) Get the value of the corresponding element from a list, array or map, where element is the index position of the element (starting from 0) for lists and arrays, and element is the key for maps
seq.get(map, element) Returns true when there is a key in the map (which may be null); for arrays and chained tables, the key can be index, and returns true when index is in the valid range [0..len-1], otherwise returns false
string.startsWith(s1,s2) Returns Boolean if string s1 starts with string s2.
string.endsWith(s1,s2) If or not the string s1 ends with the string s2,return Boolean
Sample

For a record like the following:

{
    "orgId": "o15638787142121",
    "modelId": "Darwen_SGBuilding",
    "modelIdPath": "/Darwen_Building/Darwen_SGBuilding",
    "assetId": "UXQB9oLc",
    "pointId": "temp",
    "time": 1542609276270,
    "value": 23.4,
    "quality": 0,
    "dq": 0,
    "attr": {},
    "measurementId": "Darwen_SGBuilding::temp",
    "assetTags": {
        "DCMModel": {
            "DcmModel:Darwen_Building": null,
            "DcmModel:Darwen_SGBuilding": null
        }
    },
    "measurementTags": {
        "MyHaystack": {
            "MyHaystack:zone": {"floor": 2},
            "MyHaystack:co2": null,
            "MyHaystack:sensor": null
        }
    }
}


Filter Expression Sample Description
seq.contains_key(record.assetTags.DCMModel, 'DcmModel:Darwen_SGBuilding') && #record.measurementTags.MyHaystack.MyHaystack:zone.floor > 1 First, determine whether record.assetTags.DCMModel This Map object contains a Key with the name DcmModel:Darwen_SGBuilding, if true, continue to determine whether record.measurementTags.MyHaystack: zone.floor is greater than 1, both are true, then return true

Output Results

Output Example

Configuration information

../../../_images/record_filter1.png ../../../_images/record_filter2.png