Condense
Visit WebsiteRaise a Support TicketBook a Meeting
  • Overview
  • Introduction to Condense
    • What is Condense?
    • Features of Condense
    • Condense Architecture
      • Detailed Component Breakdown
      • Additional Services
      • Components and Services within the Kubernetes Cluster
    • Key Benefits of Condense
    • Why Condense?
    • Condense Use-Cases
    • FAQs
  • Fully Managed kafka
    • Kafka Management
    • Kafka Connect
    • Schema Registry
    • Securing Kafka
    • Kafka Administration
  • Security
  • Condense Deployment
    • Bring Your Own Cloud (BYOC)
      • Deployment from GCP Marketplace
      • Deployment from AWS Marketplace
      • Deployment from Azure Marketplace
  • Condense App - Getting Started
    • Glossary
    • Features of Condense App
    • Video Guide
    • SSO (Single Sign On) - Creating an Account/Logging into the Condense App
    • Workspace in Condense
    • Pre-Built Connectors
    • Custom Transforms
    • Applications
    • Pipelines
    • Settings
    • Role Based Access Control (RBAC)
    • Activity Auditor
    • Campaigns
    • Split Utility
    • Alert Utility
    • KSQL
  • Connectors in Condense
    • Available Connectors
    • Upcoming Connectors
  • Certifications
    • ISO 27001:2013
    • ISO 9001:2015
  • Legal
    • End User License Agreement (EULA)
    • Privacy Policy
    • Usage of Cookies
    • Terms and Conditions
  • Marketing Assets
    • Wallpapers
    • Social Media
Powered by GitBook
On this page
  • Kafka Component Architecture
  • Kafka Component Interactions
  • Kafka Connect
  • Kafka MirrorMaker
  • Kafka Exporter
  • Securing Kafka
  • Encryption
  • Authentication
  • Supported authentication mechanisms
  • Authorization
  • Federal Information Processing Standards (FIPS)
  • Monitoring Kafka
  • Prometheus
  • Kafka Exporter
  • Grafana
  • Kafka management on your cloud
  • Multi-Zone Deployment for High Availability
  • Multi-Region Deployment for Disaster Recovery
  • Monitoring Kafka

Was this helpful?

  1. Fully Managed kafka

Kafka Management

PreviousFully Managed kafkaNextKafka Connect

Last updated 3 months ago

Was this helpful?

Kafka Component Architecture

A KRaft-based Kafka cluster consists of broker nodes responsible for message delivery and controller nodes that manage cluster metadata and coordinate clusters. These roles can be configured using node pools in deployment manifests.

Other Kafka components interact with the Kafka cluster for specific tasks.

Kafka Component Interactions

Kafka Connect

Kafka Connect is an integration toolkit for streaming data between Kafka brokers and other systems using connector plugins. Kafka Connect provides a framework for integrating Kafka with an external data source or target, such as a database, for import or export of data using connectors. Connectors provide the connection configuration needed.

  • A source connector pushes external data into Kafka.

  • A sink connector extracts data out of Kafka

External data is translated and transformed into the appropriate format.

Kafka Connect can be configured to build custom container images with the required connectors.

Kafka MirrorMaker

Kafka MirrorMaker replicates data between two Kafka clusters, either in the same data center or across different locations.

Kafka Exporter

Kafka Exporter extracts data for analysis as Prometheus metrics, primarily data relating to offsets, consumer groups, consumer lag and topics. Consumer lag is the delay between the last message written to a partition and the message currently being picked up from that partition by a consumer

Securing Kafka

Encryption

Kafka in Condense supports Transport Layer Security (TLS), a protocol for encrypted communication.

IMPORTANT! Communication is always encrypted between Kafka components.

Authentication

Kafka listeners use authentication to ensure a secure client connection to the Kafka cluster. Clients can also be configured for mutual authentication. Security credentials are created and managed by the Cluster and User Operator.

Supported authentication mechanisms

  • mTLS authentication (on listeners with TLS-enabled encryption)

  • SASL SCRAM-SHA-512

  • OAuth 2.0 token-based authentication

  • Custom authentication (supported by Kafka)

Authorization

Authorization controls the operations that are permitted on Kafka brokers by specific clients or users.

Supported authorization mechanisms

  • Simple authorization using ACL rules

  • OAuth 2.0 authorization (if you are using OAuth 2.0 token-based authentication)

  • Open Policy Agent (OPA) authorization

  • Custom authorization (supported by Kafka)

Federal Information Processing Standards (FIPS)

Kafka in Condense can run on FIPS-enabled Kubernetes clusters to ensure data security and system interoperability if the native Kubernetes service of the cloud provider supports it.

Monitoring Kafka

Monitoring data allows you to monitor the performance and health of Kafka in Condense. You can configure your deployment to capture metrics data for analysis and notifications.

Metrics data is useful when investigating issues with connectivity and data delivery. For example, metrics data can identify under-replicated partitions or the rate at which messages are consumed. Alerting rules can provide time-critical notifications on such metrics through a specified communications channel. Monitoring visualizations present real-time metrics data to help determine when and how to update the configuration of your deployment.

The following tools are used for metrics and monitoring which is shipped with the Condense deployment

Prometheus

Kafka Exporter

Grafana

Kafka management on your cloud

Multi-Zone Deployment for High Availability

This architecture demonstrates how Condense achieves a highly available, scalable, and resilient Kafka deployment within a Virtual Private Cloud (VPC) on Azure Kubernetes Service (AKS).

Key Components

  • A dedicated node pool is distributed across three availability zones (Zone A, B, C) in Region A to enhance fault tolerance.

  • Each node contains:

    • Kafka Operator – Manages Kafka lifecycle and configurations.

    • Kafka Broker – Handles message storage and distribution, backed by Persistent Volume Claims (PVCs) for data durability.

    • Controller – Manages cluster metadata and leadership election, also using PVCs for persistence.

High Availability & Fault Tolerance

  • Multi-Zone Redundancy

    Kafka components are distributed across three zones, preventing single points of failure and ensuring continuous availability.

  • Node Affinity & Pod Anti-Affinity

    • Node Affinity ensures components are deployed on specific nodes that are running in different regions.

    • Pod Anti-Affinity prevents multiple critical Kafka instances from running on the same node, improving redundancy.

Networking & Load Balancing

  • Internal Access (VNet Peering): Allows internal clients to communicate with Kafka through an internal load balancer, maintaining security and low latency. This makes it easier to connect existing services to Kafka internally.

  • External Access (Internet Load Balancer): Distributes traffic efficiently to external clients, ensuring reliable connectivity.

Scalability & Self-Healing

  • Dynamic Scaling: Kubernetes automatically scales Kafka brokers based on workload demands.

  • Self-Healing Mechanisms: Kubernetes reschedules failed pods and maintains cluster stability without manual intervention.

Multi-Region Deployment for Disaster Recovery

Building on the high availability architecture defined earlier, this approach extends Kafka’s resilience by implementing a multi-region deployment for disaster recovery using Kafka MirrorMaker 2. This ensures seamless data replication and failover across geographically distributed clusters.

Unidirectional

Bidirectional

Unidirectional Replication (Primary-Backup Model)

  • Primary Region (Region A):

    A local producer writes data to the source Kafka cluster.

  • Cross-Region Replication:

    Kafka MirrorMaker 2 replicates data to a secondary Kafka cluster in Region B.

  • Backup Region (Region B):

    The target Kafka cluster serves as a hot backup, ensuring data availability in case of failure in Region A.

  • Use Case: Ensures a disaster recovery mechanism with a standby cluster that can take over operations if the primary region fails.

Bidirectional Replication (Active-Active Model)

  • Multi-Region Active Clusters:

    Clusters in Region A and Region B actively handle data ingestion, processing, and replication.

  • Real-Time Cross-Replication:

    • Kafka MirrorMaker 2 ensures continuous synchronization between both regions.

    • If one cluster fails, the other automatically continues operations, preventing downtime.

Use Case:

Enables an active-active multi-region setup, ensuring real-time failover and load balancing.

Monitoring Kafka

Monitoring data allows you to monitor the performance and health of Kafka in Condense. You can configure your deployment to capture metrics data for analysis and notifications.

Metrics data is useful when investigating issues with connectivity and data delivery. For example, metrics data can identify under-replicated partitions or the rate at which messages are consumed. Alerting rules can provide time-critical notifications on such metrics through a specified communications channel. Monitoring visualizations present real-time metrics data to help determine when and how to update the configuration of your deployment.

The following tools are used for metrics and monitoring which is shipped with the Condense deployment

Prometheus

Kafka Exporter

Grafana

pulls metrics from Kafka, Controllers, and Kafka Connect clusters. The Prometheus Alertmanager plugin handles alerts and routes them to a notification service.

adds additional Prometheus metrics.

provides dashboard visualizations of Prometheus metrics.

pulls metrics from Kafka, Controllers, and Kafka Connect clusters. The Prometheus Alertmanager plugin handles alerts and routes them to a notification service.

adds additional Prometheus metrics.

provides dashboard visualizations of Prometheus metrics.

Prometheus
Kafka Exporter
Grafana Labs
Prometheus
Kafka Exporter
Grafana Labs