Pipelines
Last updated
Was this helpful?
Last updated
Was this helpful?
A pipeline is a feature that visually and functionally represents the flow of data through a series of connected and customizable steps. It is a customizable workflow builder where you collect data (input connector), refine it (transform connector or custom transform connector), and send it where it needs to go (output connector).
The Pipelines feature is used to manage deployed connectors (e.g., integrating systems, deploying applications, or processing data) Pipelines consist of a sequence of nodes (tasks or processes) connected in a specific order to achieve a desired outcome. Each node performs a specific function, such as data ingestion, transformation, or delivery, and the flow of data or tasks between nodes is managed through connectors.
When a node (e.g., a connector transform) is clicked, a detailed section opens with:
Connector Information: Name, category, type, and status (e.g., Teltonika as a Input connector with category as Telematics Device type and status as Running ).
Kafka Topic Information: Input and output topics configured by the user for the deployment of the connector.
Displays real-time metrics for each node, such as resource utilization (memory and CPU utilization).
Visual indicators (e.g., colour-coded status for running, stopped, etc ) to quickly identify issues
(add image here)
Access detailed logs for each node to track activities and troubleshoot issues.
It supports real-time log streaming with customizable intervals (e.g., 5 seconds by default) and allows users to play, stop, and control the flow of logs for better monitoring and debugging.
After a system or pipeline is deployed, issues or bugs may be discovered that require adjustments to ensure smooth operation. The Edit Configurations functionality allows users to resolve these problems by correcting misconfigurations.
Edit Configurations are categorized into two distinct types, each serving specific use cases:
Step 1: Accessing the Edit Configuration Page
Navigate to the Pipeline section where the custom transform connector is deployed.
Click on the custom transform connector card that you want to “Edit”.
Click on the Edit Configuration tab to make changes.
Step 2: Editing the Configuration Fields
All the configurations associated with the custom transform connector are populated.
Make changes to the configuration field values as required.
Step 3: Saving or Cancelling Changes
Save Changes:
If you are satisfied with the changes, click the Save Changes button to apply the new configurations. Note that once you save, the previously added configurations will be overwritten.
Cancel
If you do not wish to save the changes, click the Cancel button to exit without making any modifications.
Step 1: Accessing the Edit Configuration Page
Navigate to the Pipeline section where the telematics device type connector is deployed.
Click on the telematics device type connector card that you want to “Edit”.
Click on the Edit Configuration tab to make changes.
Step 2: Editing the Configuration Fields
All the configurations associated with the telematics device type connector are populated.
Make changes to the configuration field values as required.
Example
Title
The current title is set to data-validator. If you need to change the title, enter the new title in the Title field.
Output Topic
The current output topic is set to user data. If you need to change the output topic, enter the new output topic in the output topic field.
Step 3: Saving or Cancelling Changes
Save Changes
If you are satisfied with the changes, click the Save Changes button to apply the new configurations. Note that once you save, the previously added configurations will be overwritten.
Cancel
If you do not wish to save the changes, click the Cancel button to exit without making any modifications.
The Delete Connector functionality allows users to remove a connector that is no longer needed or is causing issues in the pipeline. However, deleting a connector can have significant consequences, including:
Disruption of Data Flow: Removing a connector may break the flow of data in the pipeline, affecting downstream processes and applications.
System Integrity: Deleting a connector without proper consideration can lead to errors or failures in deployed applications that rely on the connector.
Resource Cleanup: Deleting unused or obsolete connectors helps free up system resources, such as memory and CPU, improving overall system performance.
Step 1: Access the Connector Information Navigate to the Pipeline section where the connector that you wish to delete is listed.
Step 2: Confirm the Connector to Delete Click on the Delete Connector option associated with the connector.
Step 3: Review the Warning Message
A warning message will appear, stating: "Are you sure you want to delete the connector Title?" "If you delete the Connector title, you might break the flow of data in the pipeline and might cause problems in your deployed applications."
Carefully read the warning to understand the potential impact of deleting the connector.
Step 4: Proceed or Cancel Delete Connector If you are certain that deleting the connector will not affect your system, click Delete Connector to proceed.
Cancel If you are unsure or want to reconsider, click Cancel to abort the deletion process.
