The Onibex Kafka Connector App is a Snowflake Native Application that provides a fully integrated framework for managing Snowflake connectors in Confluent Cloud directly from Snowflake. It allows users to create, delete, and update connectors using the official Onibex Kafka integration, simplifying end-to-end data streaming management.
The application is composed of several logical components that work together to ensure secure, reliable, and efficient operation:
Streamlit-based User Interface:
Delivers an intuitive and interactive interface within Snowflake, enabling users to easily create, update, and delete Confluent connectors without leaving the Snowflake environment.
Snowpark Core Layer:
Implements the core business logic of the application, validating configurations and managing the interaction between Snowflake and Confluent Cloud through the Onibex Kafka Connector. This layer ensures operational consistency, reliability, and transparency.
External Access Integrations (EAIs):
Provide secure and controlled communication between Snowflake and Confluent Cloud. These integrations safeguard credentials, enforce strict access controls, and maintain compliance with security standards.
Configuration and Metadata Storage:
Stores all connector-related metadata, configuration parameters, and environment details in Snowflake tables such as core.app_config. This design enables centralized management, persistence, and auditability of all connector operations.
This architecture achieves an optimal balance between automation, security, and maintainability, empowering organizations to efficiently manage Confluent connectors directly within Snowflake while leveraging the robustness of the Onibex platform.
The Onibex Kafka Connector App includes a guided setup wizard designed to simplify onboarding and ensure that all necessary configurations and integrations between Snowflake and Confluent Cloud are properly established.
The process consists of seven intuitive steps, each ensuring that users can quickly and securely begin managing their Snowflake connectors in Confluent.
Terms and Conditions:
Users must review and accept the Onibex licensing terms and responsibilities before proceeding with the installation and configuration process.
Cloud Provider Selection:
Allows users to specify their preferred Confluent Cloud deployment environment (AWS, Azure, or GCP), ensuring compatibility with their existing infrastructure.
Confluent API Access:
Configures authentication by securely registering the user’s Confluent Cloud API key and secret within Snowflake using an External Access Integration (EAI). This step validates access and establishes a secure communication channel with Confluent Cloud.
Environment Selection:
Enables users to select the target Confluent environment in which Snowflake connectors will be created, updated, or deleted.
Kafka Cluster Configuration:
Automatically retrieves the available Kafka clusters from the selected environment, validates user permissions, and verifies connectivity between Snowflake and Confluent Cloud through the Onibex connector.
Schema Registry Setup:
Integrates with the Confluent Schema Registry when required, enabling schema-based data management for compatible connectors and ensuring smooth data serialization workflows.
Connector Plugin Deployment:
Allows users to upload and register custom connector plugins securely within Snowflake stages. These plugins can then be deployed to Confluent Cloud, extending functionality and supporting custom connector configurations.
Once the setup is completed, all configuration details are securely stored within Snowflake, allowing users to access and manage their connectors in future sessions without repeating the setup process.
The Connector Management Interface in the Onibex Kafka Connector App provides complete visibility and control over all Snowflake connectors deployed in Confluent Cloud, allowing users to efficiently monitor, configure, and maintain their streaming integrations directly from Snowflake.
Consolidated View:
Displays all active and inactive connectors in a single, unified interface. Key attributes such as connector name and status are clearly presented for quick reference.
Detailed View:
Offers an in-depth overview of each connector, including configuration parameters and status updates retrieved directly from Confluent Cloud via the Onibex integration layer.
Search and Filtering:
Enables users to quickly locate specific connectors by name, environment, or cluster, streamlining navigation and simplifying management at scale.
Edit and Maintenance:
Provides full lifecycle management capabilities, allowing users to update configurations or permanently delete connectors. All operations are executed securely through Snowpark functions interacting with Confluent APIs.
Real-Time Refresh:
Ensures that connector information displayed in the interface always reflects the latest operational state in Confluent Cloud. Updates are fetched dynamically to maintain accuracy and reliability.
This interface empowers users to manage all aspects of their Snowflake-to-Confluent connectivity from within the Snowflake environment, minimizing context switching while maintaining full administrative control.
The Connector Configuration Interface in the Onibex Kafka Connector App enables the creation and modification of Snowflake connectors in Confluent Cloud with full control over connection parameters. It offers two modes to accommodate different user preferences: Guided and Advanced.
In Guided Mode, users can easily configure a new connector through a structured form without manually editing JSON configurations. Required parameters include:
Connector Name
Kafka Topic
Snowflake Database, Warehouse, and Schema
Snowflake Authentication Method
Endpoints retrieved from SYSTEM$ALLOWLIST()
This mode automatically:
Inserts the Onibex Kafka Connector license, ensuring proper operation.
Includes the plugin ID corresponding to the connector provided by Onibex.
Performs real-time validation to confirm all parameters are compatible with both Snowflake and Confluent Cloud before deployment.
Advanced Mode allows users to paste a complete JSON configuration for full customization and flexibility. This mode is ideal for advanced users who need to fine-tune connector behavior or deploy complex configurations. It also:
Automatically embeds the Onibex license and plugin ID.
Performs syntax and compatibility checks in real time to prevent misconfiguration.
Validates essential parameters to ensure the connector can be successfully deployed in Confluent Cloud.
General Configuration: Define connector name, associated Kafka topics, and number of parallel tasks.
Snowflake Target Parameters: Specify target database, warehouse, schema, and automatically generate the JDBC URL.
Endpoint Configuration: Transform JSON data from SYSTEM$ALLOWLIST() into validated endpoint formats for secure connection setup.
Authentication: Supports both username/password and private key (PEM) authentication.
Advanced Settings: Configure transaction behavior, batch sizes, deletion rules, and table naming conventions.
This dual-mode design allows both novice and expert users to create and manage connectors efficiently while maintaining security, compliance, and reliability.
The Onibex Kafka Connector App includes intelligent schema and format handling capabilities to ensure seamless data integration between Snowflake and Confluent Cloud (Kafka).
Automatic Format Detection:
Automatically detects the message format (Avro, JSON, or String) and applies the appropriate Kafka converter.
Schema Registry Verification:
Validates Schema Registry credentials and network connectivity before schema-based processing is enabled.
Compatibility Checks:
Ensures that incoming data conforms to expected serialization structures, preventing schema drift or misalignment between Snowflake and Kafka.
This system guarantees robust, schema-aware data flow while minimizing manual configuration and reducing the risk of serialization errors.
The Onibex Kafka Connector App enforces a strong security framework to protect both Snowflake and Confluent Cloud environments, ensuring compliance, data privacy, and operational integrity.
Secure Credential Management:
All credentials are managed through Snowflake Secret References, keeping sensitive information encrypted and isolated.
Controlled Network Access:
External Access Integrations (EAIs) strictly limit outbound communication to pre-approved Confluent Cloud endpoints.
Encrypted Data Transfers:
All communication between Snowflake and Confluent Cloud uses TLS encryption to maintain data confidentiality.
License Protection:
The Onibex license is securely stored, encrypted, and validated at runtime to prevent unauthorized use of the application.
Data Isolation:
The entire app operates within the user’s Snowflake account, ensuring complete separation from Onibex infrastructure and maintaining tenant-level data isolation.
This security model ensures trust, compliance, and integrity across all operations, allowing organizations to safely manage Confluent connectors from within Snowflake.
The Onibex Kafka Connector App supports multi-cloud plugin deployment, enabling users to deploy connectors seamlessly across AWS, Azure, and GCP environments.
Each provider utilizes its respective External Access Integration:
AWS: S3_EXTERNAL_ACCESS
Azure: AZURE_EXTERNAL_ACCESS
GCP: GCP_EXTERNAL_ACCESS
Connector plugins are stored in Snowflake stages and registered directly within the application, enabling:
Version control and rollback support for plugin updates.
Secure and auditable access to connector artifacts.
Cross-cloud compatibility, simplifying deployment across heterogeneous infrastructures.
This capability ensures consistent and flexible connector management in any cloud environment, maintaining the same reliability and governance standards across all platforms.
The core.get_configuration(ref_name STRING) procedure provides configuration and access control information for external service endpoints used by the Onibex Kafka Connector App.
It plays a critical role in managing secure network connections between Snowflake and external systems such as Confluent Cloud, Kafka Clusters, Schema Registry, cloud storage services, and the Onibex license server.
Returns a JSON configuration object containing:
host_ports – The list of URLs or endpoints that the application is authorized to access.
allowed_secrets – Indicates whether credentials are permitted for each endpoint (ALL or NONE).
Serves as a configuration callback for secure references such as:CONFLUENT_CLOUD_API_EXTERNAL_ACCESS,AZURE_EXTERNAL_ACCESS,AWS_EXTERNAL_ACCESS,GCP_EXTERNAL_ACCESS,
and other external integrations managed by the Onibex Kafka Connector App.
This design ensures that only approved and validated endpoints are used, maintaining full compliance with Snowflake’s External Access Integration (EAI) security model.
Reference | Description | Host(s) / URL | Allowed Secrets |
CONFLUENT_CLOUD_API_EXTERNAL_ACCESS | Access to Confluent Cloud API | api.confluent.cloud | ALL |
KAFKA_CLUSTER_EXTERNAL_ACCESS | Network access to Kafka Cluster (extracted from core.app_config table) | Variable depending on configured cluster | ALL |
SCHEMA_REGISTRY_EXTERNAL_ACCESS | Access to Confluent Schema Registry | Variable depending on configured Schema registry | ALL |
AZURE_EXTERNAL_ACCESS | Access to presigned URLs in Azure Blob Storage | byocprodcentralus.blob.core.windows.net | ALL |
ONIBEX_CONNECTOR_LICENSE | Access to Onibex license service | apigateway.onibex2.com | ALL |
GCP_EXTERNAL_ACCESS | Access to presigned URLs in Google Cloud Storage | storage.googleapis.com | ALL |
AWS_EXTERNAL_ACCESS | Access to presigned URLs in AWS S3 (extracted from core.app_config) | Variable depending on configured bucket | ALL |
S3_EXTERNAL_ACCESS | Direct access to multiple S3 buckets | Examples: *.s3.dualstack.us-east-1.amazonaws.com, *.s3.dualstack.eu-west-1.amazonaws.com, etc. | NONE |
The following credentials are used to authenticate and authorize secure communication with external Confluent components:
CONFLUENT_CLOUD_API_CREDENTIALS
KAFKA_CLUSTER_API_CREDENTIALS
SCHEMA_REGISTRY_API_CREDENTIALS
Type: PASSWORD
These credentials are securely managed using Snowflake Secret References, ensuring encrypted storage and controlled runtime access.
Fixed URLs restrict network access to approved domains, preventing unauthorized external communication.
Dynamic endpoints are resolved at runtime from the core.app_config table based on the user’s connector configuration.
The procedure returns a standardized JSON object containing:
type – Defines the configuration type.
payload – Contains the detailed endpoint and secret configuration.
This endpoint management model ensures security, flexibility, and compliance, providing a robust foundation for all network interactions within the Onibex Kafka Connector App.