Partitioner¶
This stage partitions the records of each batch by the specified conditions and is usually used for joint operations among different points of the same device, or operations among points of different devices. The functions of this stage include:
- Supporting customized partitioning conditions.
- Supporting path expressions in customized partitioning conditions. For example,
/assetIdindicates partitioning by the value of theassetIdfield in each record.
This stage does not support streaming lineage.
Configuration¶
The configuration tabs for this stage are General and Partitioner.
General¶
| Name | Required? | Description |
|---|---|---|
| Name | Yes | The name of the stage. |
| Description | No | The description of the stage. |
| Stage Library | Yes | The streaming operator library to which the stage belongs. |
| Required Fields | No | The fields that the data records must contain. If the specified fields are not included, the record will be filtered out. |
| Preconditions | No | The conditions that must be satisfied by the data records. Records that do not meet the conditions will be filtered out. For example, ${record:value('/value') > 0}. For the syntax of EL expressions, see Expression Language. |
| On Record Error | Yes | The processing method for error data.
|
Partitioner¶
| Name | Required? | Description |
|---|---|---|
| Parallelism (Standalone Mode Only) | Yes | Specify the number of partitions to create per batch of records in Standalone mode. |
| Application Name (Standalone Mode Only) | Yes | The name of the application that runs this stage in Standalone mode (SDC Spark App by default). |
| Init Method Arguments | Yes | Specify the record fields used for creating partitions. For example, specifying /assetId means assigning records with the same assetId to the same partition. |
