# Build your First Pipeline A **Pipeline** in Condense is the visual and functional representation of data flow between deployed connectors, transforms, and utilities inside a workspace. It is automatically materialized when a connector or transform is deployed. No manual creation needed. {% hint style="info" %} **Before you begin**, make sure you have: * A workspace created and configured * At least one environment linked to the workspace * Workspace Admin or Maintainer role {% endhint %} ## What You'll Build ``` [ Input Connector ] → [ Kafka Topic ] → [ Transform ] → [ Output Connector ] (Data Source) (Stream) (Process) (Destination) ``` By the end of this guide, you will have a live pipeline ingesting data from a source, processing it through a transform, and delivering it to a destination, all inside your Condense workspace. *** ## Step 1 - Deploy an Input Connector An **Input Connector** receives data from an external system and injects it into your pipeline. 1. Navigate to your **Workspace** → click **Connectors** in the left sidebar 2. Click **+ Add Connector** 3. Select **Input** as the connector role 4. Choose your connector type (e.g. Kafka, HTTP, Teltonika, Pub/Sub) 5. Fill in the required configuration fields: * **Title** - a unique name for this connector * **Output Topic** - the Kafka topic this connector will publish data to * **Environment Variables** - credentials, endpoints, or protocol settings
6. Click **Deploy**

Once deployed, your Input Connector will appear as a block on the Pipeline canvas and begin publishing data to the configured output topic.

*** ## Step 2 - Add a Transform A **Transform** processes messages as they flow between your Input and Output connectors. #### Option A - Prebuilt Transform 1. On the **Pipeline canvas**, click **+ Add Transform** 2. Select **Prebuilt** 3. Choose a transform type: * **Geofence** - location-based event triggering * **Alert Utility** - trigger alerts when a condition is met * **Split Utility** - route messages based on rules 4. Configure the required parameters: * **Input Topic** - must match the Output Topic from your Input Connector * **Output Topic** - the topic the transform will publish results to 5. Click **Deploy** #### Option B - Custom Transform 1. Go to **Applications** → create or select an existing application 2. Build, test, and publish the application 3. Back on the Pipeline canvas, click **+ Add Transform** → **Custom** 4. Select your published application and version 5. Configure input/output topics and environment variables 6. Click **Deploy** *** ## Step 3 - Deploy an Output Connector An **Output Connector** takes the processed data and delivers it to an external destination. 1. Navigate to **Connectors** → click **+ Add Connector** 2. Select **Output** as the connector role 3. Choose your connector type (e.g. PostgreSQL, Big Query, S3, Kafka) 4. Fill in the required configuration fields: * **Title** - a unique name for this connector * **Input Topic** - must match the Output Topic from your Transform * **Environment Variables** - destination credentials and settings 5. Click **Deploy**

Condense automatically draws a connecting line between blocks that share matching topics. Your pipeline is now live.

*** ## Step 4 - View Your Pipeline Click **Pipeline** in the left sidebar. You will see all deployed blocks connected by lines:

Pipeline Canvas Showing Connected Blocks

Clicking any block opens its **Component Detail Panel** showing the connector name, category, type, status, and Kafka topic information:

{% hint style="info" %} Pipelines are automatically materialized. They appear automatically as soon as connectors and transforms are deployed. You do not manually create a pipeline. {% endhint %} *** ## Step 5 - Monitor Your Pipeline #### Real-Time Metrics Each block displays live CPU and memory utilization:

Colour-coded status indicators let you instantly identify Running, Stopped, or Error states:

#### Logs and Debugging Access detailed real-time logs for each block with customizable streaming intervals (5 seconds by default):

### Editing Your Pipeline If you need to fix a misconfiguration after deployment, use **Edit Configurations**:

1. Click the block you want to edit on the Pipeline canvas 2. Click **Edit Configuration** 3. Update the required fields 4. Click **Save Changes**

Fields with a lock icon cannot be edited after deployment. To change locked fields, delete and redeploy the component.

### Roles and Permissions | Operation | Admin | Maintainer | Developer | Viewer | | ----------------------------------- | :-------------------------------------------------------------: | :-------------------------------------------------------------: | :-------------------------------------------------------------: | :-------------------------------------------------------------: | | Deploy pre-built connectors |

| | Deploy custom connectors/transforms |

| | Configure deployed components |

| | Delete deployed components |

| | View pipelines and connections |

| | View component logs/configuration |

| *** ## Best Practices

Use **clear topic naming** to make the pipeline canvas self-explanatory

**Group related connectors and transforms** logically to simplify understanding

**Avoid unused topic links** to reduce visual clutter

**Regularly check component logs** to catch issues early

Document the pipeline purpose in the workspace description *** ## Common Pitfalls * **Misaligned Topic Names** - Double-check that Output Topic of one block exactly matches the Input Topic of the next block * **Unused Components Left Running** - Remove or disable unused connectors to prevent unnecessary resource usage * **Overcrowded Canvas** - Use multiple workspaces when a single pipeline becomes too dense --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.zeliot.in/condense/v2.4.0/overview/build-your-first-pipeline.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.