Pipelines

A pipeline is a feature that visually and functionally represents the flow of data through a series of connected and customizable steps. It is a customizable workflow builder where you collect data (input connector), refine it (transform connector or custom transform connector), and send it where it needs to go (output connector).

The Pipelines feature is used to manage deployed connectors (e.g., integrating systems, deploying applications, or processing data) Pipelines consist of a sequence of nodes (tasks or processes) connected in a specific order to achieve a desired outcome. Each node performs a specific function, such as data ingestion, transformation, or delivery, and the flow of data or tasks between nodes is managed through connectors.

Key Features

Node Details Panel

When a node (e.g., a connector transform) is clicked, a detailed section opens with:

• Connector Information: Name, category, type, and status (e.g., Teltonika as a Input connector with category as Telematics Device type and status as Running ).

• Kafka Topic Information: Input and output topics configured by the user for the deployment of the connector.

Real-Time Monitoring

• Displays real-time metrics for each node, such as resource utilization (memory and CPU utilization).

• Visual indicators (e.g., colour-coded status for running, stopped, etc ) to quickly identify issues

Logs and Debugging

(add image here)

• Access detailed logs for each node to track activities and troubleshoot issues.

• It supports real-time log streaming with customizable intervals (e.g., 5 seconds by default) and allows users to play, stop, and control the flow of logs for better monitoring and debugging.

Edit Configurations

After a system or pipeline is deployed, issues or bugs may be discovered that require adjustments to ensure smooth operation. The Edit Configurations functionality allows users to resolve these problems by correcting misconfigurations.

Types of Edit Configurations

Edit Configurations are categorized into two distinct types, each serving specific use cases:

Edit Configurations for Custom Transform as a Connector

Step 1: Accessing the Edit Configuration Page

1. Navigate to the Pipeline section where the custom transform connector is deployed.

2. Click on the custom transform connector card that you want to “Edit”.

3. Click on the Edit Configuration tab to make changes.

Step 2: Editing the Configuration Fields

1. All the configurations associated with the custom transform connector are populated.

2. Make changes to the configuration field values as required.

Step 3: Saving or Cancelling Changes

Save Changes:

If you are satisfied with the changes, click the Save Changes button to apply the new configurations. Note that once you save, the previously added configurations will be overwritten.

Cancel

If you do not wish to save the changes, click the Cancel button to exit without making any modifications.

Edit Configurations for Telematics Device Type

Step 1: Accessing the Edit Configuration Page

1. Navigate to the Pipeline section where the telematics device type connector is deployed.

2. Click on the telematics device type connector card that you want to “Edit”.

3. Click on the Edit Configuration tab to make changes.

Step 2: Editing the Configuration Fields

1. All the configurations associated with the telematics device type connector are populated.

2. Make changes to the configuration field values as required.

Step 3: Saving or Cancelling Changes

Save Changes

If you are satisfied with the changes, click the Save Changes button to apply the new configurations. Note that once you save, the previously added configurations will be overwritten.

Cancel

If you do not wish to save the changes, click the Cancel button to exit without making any modifications.

Delete Connector

The Delete Connector functionality allows users to remove a connector that is no longer needed or is causing issues in the pipeline. However, deleting a connector can have significant consequences, including:

1. Disruption of Data Flow: Removing a connector may break the flow of data in the pipeline, affecting downstream processes and applications.

2. System Integrity: Deleting a connector without proper consideration can lead to errors or failures in deployed applications that rely on the connector.

3. Resource Cleanup: Deleting unused or obsolete connectors helps free up system resources, such as memory and CPU, improving overall system performance.

Steps to Delete a Connector

Step 1: Access the Connector Information Navigate to the Pipeline section where the connector that you wish to delete is listed.

Step 2: Confirm the Connector to Delete Click on the Delete Connector option associated with the connector.

Step 3: Review the Warning Message

Carefully read the warning to understand the potential impact of deleting the connector.

Step 4: Proceed or Cancel Delete Connector If you are certain that deleting the connector will not affect your system, click Delete Connector to proceed.

Cancel If you are unsure or want to reconsider, click Cancel to abort the deletion process.