Transforming Pipeline Type


In pipeline designing mode, general stream processing pipelines can be transformed to advanced stream processing pipelines based on business needs.

Prerequisites

  • Ensure that your organization has requested for the Stream Processing - Message Queue resource that is required by advanced stream processing pipelines through the Resource Management page.

Procedure

Take the following steps to transform a general stream processing pipeline to an advanced pipeline:

  1. Click Stream Processing > Pipeline Designer to view the created stream processing pipelines. The pipeline type can be recognized by the icon before the pipeline name.

  2. In the list of stream processing pipelines, right click on the target pipeline, and select Transform to Advanced Pipeline.

    _images/upgrading_pipeline.png

  3. Confirm the operation in the pop-up window to complete pipeline type transformation. The icon before the pipeline type will be changed.

    _images/upgraded_pipeline.png

Seamlessly Migrating Stream Processing Pipelines

When migrating stream processing pipelines, if custom Topic Reader is configured to replace the automatically generation system pipeline (Data Reader), contact EnOS operation team for migrating Kafka Offset to avoid any data loss. The operation process is shown in the following figure.

_images/migrating_pipeline.png

Procedure

  1. Stop the system pipeline Data Reader.
  2. Stop the general pipeline and system pipeline Data Reader when no lag exists.
  3. Change the input and output Kafka topics of the general pipeline based on business needs. Then, start the pipeline.
  4. Contact EnOS operation team for migrating Kafka Offset.
  5. Change the output Kafka topic of the custom Topic Reader pipeline based on business needs. Then, start the pipeline.