Custom Transforms
Last updated
Was this helpful?
Last updated
Was this helpful?
Condense allows you to create and deploy custom transforms to manipulate data flowing through your connectors.
A Custom Transform is a published version of an Application that has been finalized and is ready to be used in data pipelines. It represents the implementation of the custom logic or transformation developed in the Application
This section explains concepts of Applications and how to create an application from scratch.
The below section explains the end-to-end workflow for building, publishing, and deploying Custom Transform using the Condense platform. The workflow includes version control integration, development container management, code editing and debugging, building and publishing custom transforms, and deploying applications on pipelines.
Integrate your Git repository (e.g., GitHub, GitLab, Bitbucket) with Condense for seamless code management. You can connect your Condense account to your chosen Git provider. Condense offers Github, Gitlab and Bitbucket as version control.
Go to Settings > Version Control.
Connect your Condense account to your chosen Git provider.
Go to Connectors > Your Connectors > Create a New Connector.
Click on Add New Application.
The development environment set-up is initialized and successful.
Post-deployment is successful, select the threshold for development environment set-up expiry. By default, the system will provide 3 days.
Click on Next.
Deploy the Custom Transform on a pipeline and monitor its performance.
Custom Connector Card in Connectors Section To deploy a Custom Transform, the associated application must first be published. Once published, the Custom Transform will be available on the "Connectors" page under "Your Connectors".
Go to Connectors and select the published application.
Click Configure and Deploy to deploy the connector on the pipeline.
Monitor the deployment status and debug any issues using logs and metrics.
Each custom transform listed on the Connectors page has an associated Application Card on the Applications page. However, applications can be manually deleted by the user, or their development environment may expire. If a user wishes to publish a new version of a custom transform by restoring the associated application, the Restore feature serves this purpose.
This feature allows users to:
Restore deleted or expired applications.
Update and publish new versions of custom transforms.
Maintain version control and ensure continuity in data pipelines.
By restoring the application, users can make necessary changes, rebuild, and republish the custom transform, ensuring it remains functional and up-to-date.
Here's how you can restore the version of a Custom Transform
Navigate to the Connectors section.
Click on the Custom Transform card and click on the “Restore” option.
Check User Role
The system verifies if the user has the required role (e.g., Admin, Maintainer, or Developer) to perform the restoration.
If the user lacks the necessary permissions, the process is halted, and an error message is displayed.
Verify Version Control Connection
The system checks if the version control system (e.g., GitLab) is connected and accessible.
If the connection is unavailable, then the process is halted, and an error message is displayed.
Validate Repo and Branch Access
The system verifies that the user has read and write access to the repository and branch associated with the application. Additionally, it ensures that the repository and branch still exist in the version control system (e.g., GitLab, Bitbucket).
If the repository or branch is missing, or if the user lacks the necessary permissions, the process is halted, and the user is notified and must request access or contact the administrator.
A new development environment is initialized and associated with the restoration of the Application. Post-deployment is successful, select the threshold for development environment set-up expiry. By default, the system will provide 3 days.
Click on Next.
The repository and branch associated with the application are auto-populated by the system.
Open the code editor to make necessary changes to the application.
Click Publish Application and provide a name for the image and tag value to create a build.
The system auto-fills the image and tag value from the previous build. If you provide a new tag value, it will be associated with a newer version.
Initialize the Build by checking the build logs and build status. Based on the build status, proceed to the next step by clicking on “Configure Env”.
All environment variables associated with the previous build are auto-populated by the system. Make changes to the environment variables as required. Save the updated environment variables.
Publish the application by providing a new version tag. A Published Application card is created on the Applications page.
The new version of the custom transform is associated with the same custom connector card on the Connectors page.
Go to Connectors and select the published application.
Select the version from the dropdown that you wish to deploy.
Click Configure and Deploy to deploy the application on the pipeline.
Monitor the deployment status and debug any issues using logs and metrics.
Select the Custom Connector:
Navigate to the Connectors section.
Click on the Custom Transform card and click on the “Edit” option.
A confirmation message will appear, asking if you want to edit the custom transform. Custom Transform name, description and icon for display can be edited.
Delete Custom Transform through Pipeline
Go to the Pipelines section that uses the custom transform you want to delete.
Open Pipeline Configuration:
Click on the pipeline to open its configuration.
Locate the Custom Transform:
Find the custom transform within the pipeline configuration.
Delete the Custom Transform:
Click the Delete option by clicking on the custom transform card from the pipeline.
Confirm the deletion when prompted.
Navigate to the Connectors Page:
Go to the Connectors section from the main dashboard.
Locate the Custom Transform:
Find the custom transform you want to delete
Initiate Deletion:
Click on the Delete option for the selected custom transform.
Confirm Deletion:
A confirmation message will appear, asking if you want to delete the custom transform.
Confirm the deletion when prompted.
IMPORTANT!
Deleting a custom transform will remove it from the list of available applications on the Connectors page. If the custom transform is currently deployed in any pipeline, you will need to remove it from those pipelines before deletion.
To create a new Custom Connector, please read:
