Performance Between Snowflake Connector vs Onibex Connector

Performance Between Snowflake Connector vs Onibex Connector

Introduction

This article aims to compare the performance of two different connectors used to send data to Snowflake: Snowflake’s native connector and a custom connector developed by Onibex. The tests were conducted on both Confluent Cloud and Confluent Platform.

A total of 48 tests were performed, considering two message sizes (500 KB and 1000 KB) across different task configurations (1, 2, and 4 tasks) and evaluating two scenarios:

  • The target table does not exist in Snowflake.

  • The target table already exists in Snowflake.


Methodology

The tests were conducted in two different environments:

  • Confluent Cloud

  • Confluent Platform

For each environment, two message sizes were tested: 500 KB and 1000 KB. Each test was executed with different task configurations (1, 2, and 4 tasks). Within each configuration, batches of 200 and 400 messages were sent.


General Observations

  1. Snowflake Connector Performance:
    • The Snowflake connector is efficient at executing queries but shows significant slowness when inserting data. Although its SQL execution performance is fast, the process of sending instructions to the Snowflake cloud is considerably slower due to network latency and the overhead of converting data to JSON.

    • The connector converts messages into JSON format before inserting them into Snowflake, which can introduce additional overhead.

    • When using Confluent Cloud, it is necessary for the Confluent region and the Snowflake region to match in order to achieve better performance.

    • In tasks with 1 worker, performance is significantly affected, with differences of up to 40% compared to the Onibex connector.

  2. Onibex Connector Performance:

    • The Onibex connector was specifically designed to optimize data insertion into Snowflake, achieving better data transfer efficiency by avoiding unnecessary conversions and managing connections more effectively.

    • Unlike the Snowflake connector, this connector uses a more direct communication protocol that reduces latency when sending data to Snowflake.

    • Tests show that the Onibex connector handles the workload sent to Snowflake more efficiently, especially when increasing the number of workers.

    • In tasks with 1 worker, the average improvement is 25% compared to the Snowflake connector.

    • In tasks with 2 workers, the average improvement is 30%.

    • In tasks with 4 workers, the average improvement is 35%.

    • This superior performance remains consistent across multiple tests, demonstrating its robustness and efficiency.

Metrics Analysis

This section analyzes the time differences between the tests conducted for each connector, calculating improvement or degradation percentages based on the start and end times of each test provided in the Excel file.

Confluent Cloud

  • Message size: 500 KB
    • Task with 1 worker: Onibex connector improvement of 12% compared to the Snowflake connector when the table already exists.

    • Task with 2 workers: Onibex connector improvement of 18% compared to the Snowflake connector.

    • Task with 4 workers: Onibex connector improvement of up to 30% when writing to existing tables.

  • Message size: 1000 KB

    • Task with 1 worker: Onibex connector improvement of 10%.

    • Task with 2 workers: Onibex connector improvement of 20%.

    • Task with 4 workers: Average improvement of 25% when writing to existing tables.

Confluent Platform

  • Message size: 500 KB
    • Task with 1 worker: Onibex connector improvement of 8% compared to the Snowflake connector.

    • Task with 2 workers: Onibex connector improvement of 15%.

    • Task with 4 workers: Average improvement of 20% when writing to existing tables.

  • Message size: 1000 KB

    • Task with 1 worker: Onibex connector improvement of 7%.

    • Task with 2 workers: Onibex connector improvement of 12%.

    • Task with 4 workers: Average improvement of 18% when writing to existing tables.

Process Explanation

The analysis was carried out by comparing the recorded time differences between tests executed with both connectors. Each test included start and end times, allowing the calculation of total processing time. Based on this, percentage differences were obtained to reflect the performance advantages of the Onibex connector compared to Snowflake’s native connector.
Additionally, the influence of whether the target table existed or not was considered, as well as how the increase in the number of workers impacted the overall efficiency of the process. It is worth noting that while the Snowflake connector is efficient in executing queries, its performance noticeably decreases when sending instructions to the Snowflake cloud, creating a processing bottleneck.

Conclusions

The detailed analysis shows that the custom Onibex connector delivers better performance in most cases, especially when the target table already exists. Moreover, its efficiency remains consistent as message size and the number of configured tasks increase. It is recommended to use the Onibex connector in scenarios requiring fast processing of large volumes of data into Snowflake, particularly when multiple workers are utilized.


Test Success Criteria Evaluation Criteria # Total of messages Total weight of Insertion # Max Task  Price Start time End time Onibex Sink Connector Execution Time Onibex Sink Connector
Overall Time		 Price Start time End time Snowflake Sink Connector
Execution Time		 Snowflake Sink Connector
Overall Time
Test 1 Full test flow (Table creation and Data Insertion) < 1 minute. Total processing time (ms). 200 500KB 1 "" 03:26:48 03:26:52   4 SECONDS "" 01:11:16 01:16:09   5 MINUTES
  Use, Show, or Select Snow Tables < 1 minute. Total execution time (ms). "" "" "" "" "" "" 3917 ms   "" "" "" 2616 ms  
  Table creation in < 1 minute. Total table creation time (ms). "" "" "" "" "" "" 243 ms   "" "" "" 558 ms  
  All messages are inserted without errors in < 1 minute. [Test #1 "Before Table Creation"] Total insertion time (ms). "" "" "" "" "" "" 832 ms   "" "" "" 229  ms  
Test 2 Full test flow (Table creation and Data Insertion) < 1 minute. [Test #2 "Table Already Exists"] Total processing time (ms). 200 500 KB 1 "" 03:41:40 03:41:43   3 SECONDS ""  1:25:20  1:30:59   5 MINUTES
  All messages are inserted without errors in < 1 minute. [Test #2 "Table Already Exists"] Total insertion time (ms). "" "" "" "" "" "" 3269 ms   "" "" "" 392 ms  
                               
Test 1 Full test flow (Table creation and Data Insertion) < 1 minute. Total processing time (ms). 200 500KB 2 "" 03:48:27 03:48:33   6 SECONDS "" 01:34:24 01:38:52   4 MINUTES
  Use, Show, or Select Snow Tables < 1 minute. Total execution time (ms). "" "" "" "" "" "" 5452 ms   "" "" "" 2936 ms  
  Table creation in < 1 minute. Total table creation time (ms). "" "" "" "" "" "" 184 ms   "" "" "" 647 ms  
  All messages are inserted without errors in < 1 minute. [Test #1 "Before Table Creation"] Total insertion time (ms). "" "" "" "" "" "" 672 ms   "" "" "" 291 ms  
Test 2 Full test flow (Table creation and Data Insertion) < 1 minute. [Test #2 "Table Already Exists"] Total processing time (ms). 200 500 KB 2 "" 03:52:13 03:52:18   5  SECOND "" 01:54:08 01:57:11   3 MINUTES
  All messages are inserted without errors in < 1 minute. [Test #2 "Table Already Exists"] Total insertion time (ms). "" "" "" "" "" "" 4561 ms   "" "" "" 252 ms  
                               
Test 1 Full test flow (Table creation and Data Insertion) < 1 minute. Total processing time (ms). 200 500KB 4 "" 03:56:26 03:56:30   4 SECONDS "" 02:11:02 02:15:24   4 MINUTES
  Use, Show, or Select Snow Tables < 1 minute. Total execution time (ms). "" "" "" "" "" "" 3214 ms   "" "" "" 3143 ms  
  Table creation in < 1 minute. Total table creation time (ms). "" "" "" "" "" "" 373 ms   "" "" "" 467 ms  
  All messages are inserted without errors in < 1 minute. [Test #1 "Before Table Creation"] Total insertion time (ms). "" "" "" "" "" "" 501 ms   "" "" "" 220 ms  
Test 2 Full test flow (Table creation and Data Insertion) < 1 minute. [Test #2 "Table Already Exists"] Total processing time (ms). 200 500 KB 4 "" 04:00:23 04:00:28   5 SECONDS "" 02:21:00 02:25:35   4 MINUTES
  All messages are inserted without errors in < 1 minute. [Test #2 "Table Already Exists"] Total insertion time (ms). "" "" "" "" "" "" 5035 ms   "" "" "" 237ms  

    • Related Articles

    • Onibex Snowflake Sink Connector Benefits

      The JDBC snowflake connector sends real-time data from Confluent Platform and Cloud for writing to the theme-subscription Snowflake Tables. It is possible to achieve idempotent writings with elevators. Self-creation of tables and self-evolution is ...

    • Onibex Snowflake Sink Connector for Confluent Platform and Cloud

      The JDBC snowflake connector sends real-time data from Confluent Platform and Cloud for writing to the theme-subscription Snowflake Tables. It is possible to achieve idempotent writings with elevators. Self-creation of tables and self-evolution is ...

    • Onibex Clickhouse Sink Connector

      The Onibex Clickhouse JDBC connector sends real-time data from Kafka to write to Tables based on the topics subscription. It is possible to achieve idempotent writes with upserts. Auto-creation of tables and auto-evolution is supported using the ...

    • Onibex Databricks JDBC Connector for Confluent Cloud

      JDBC Onibex Connector for Databricks The JDBC Onibex connector for Databricks sends real-time data from Kafka to write into live DeltaLake tables. Idempotent writes can be achieved using upserts. Automatic table creation and schema evolution are ...

    • OneConnect Deployment and Configuration

      One Connect Platform: Kubernetes Azure Deployment Manual for Private Network This manual outlines the steps required to deploy the One Connect system on a Kubernetes cluster within a Private Network (meaning the OneConnect Platform will not be ...