Prerequisites 

1. Docker and docker-compose with support for 3.8 
2. ClickHouse database. 
3. Download the dbt project, the repository will be provided by Onibex 
   

Configuration of docker-compose services 

Configuration of the docker-compose file with security

1. ...
2. dbt_bi_material:
3.    build:
4.      context: .
5.      dockerfile: Dockerfile
6.    container_name: dbt_bi_material
7.    volumes:
8.      - .:/dbt
9.    environment:
10.      #service config
11.      #Time window to group messages in a single dbt execution. Time in seconds
12.      - MESSAGE_TIME_WINDOW_SECONDS=3
13.      #Waiting time after the last dbt execution. Time in seconds
14.      - REFRESH_TIME_THRESHOLD=15
15.      #Auto refresh time after last dbt execution, set -1 to disable. Time in seconds
16.      - AUTO_REFRESH_TIME=600
17.      #DBT config
18.      # The container will try get ./profiles, otherwise will use default /dbt/profiles
19.      - DBT_PROFILES_RELATIVE_DIR=./profiles
20.      - DBT_PROJECT_NAME=dbt_oneconnect_ecc
21.      - DBT_PROFILE=dbt_oneconnect_ecc
22.      - DBT_TARGET=dev
23.      - DBT_MODELS=BI_SILVER_MM_MATERIAL_ECC
24.      - DBT_FULL_REFRESH=
25.      #Kafka Config
26.      - KAFKAPROPERTIES_DEPLOYMENTTYPE=cloud
27.      - KAFKAPROPERTIES_BOOTSTRAPADDRESS=<ip1:port1>,<ip2:port2>,<ip3:port3>
28.      - KAFKAPROPERTIES_SECURITYPROTOCOL=SASL_PLAINTEXT
29.      - KAFKAPROPERTIES_SASLMECHANISM=PLAIN
30.      - KAFKAPROPERTIES_SASLUSER=<user>
31.      - KAFKAPROPERTIES_SASLPASSWORD=<password>
32.      - KAFKAPROPERTIES_TOPICS=<kafka_topic>
33. ...

Configuration of the archive docker-compose without security

1. ...
2. dbt_bi_material:
3.    build:
4.      context: .
5.      dockerfile: Dockerfile
6.    container_name: dbt_bi_material
7.    volumes:
8.      - .:/dbt
9.    environment:
10.      #service config
11.      #Time window to group messages in a single dbt execution. Time in seconds
12.      - MESSAGE_TIME_WINDOW_SECONDS=3
13.      #Waiting time after the last dbt execution. Time in seconds
14.      - REFRESH_TIME_THRESHOLD=15
15.      #Auto refresh time after last dbt execution, set -1 to disable. Time in seconds
16.      - AUTO_REFRESH_TIME=600
17.      #DBT config
18.      # The container will try get ./profiles, otherwise will use default /dbt/profiles
19.      - DBT_PROFILES_RELATIVE_DIR=./profiles
20.      - DBT_PROJECT_NAME=dbt_oneconnect_ecc
21.      - DBT_PROFILE=dbt_oneconnect_ecc
22.      - DBT_TARGET=dev
23.      - DBT_MODELS=BI_SILVER_MM_MATERIAL_ECC
24.      - DBT_FULL_REFRESH=
25.      #Kafka Config
26.      - KAFKAPROPERTIES_DEPLOYMENTTYPE=onpremise
27.      - KAFKAPROPERTIES_BOOTSTRAPADDRESS=<ip1:port1>,<ip2:port2>,<ip3:port3>
28.      #- KAFKAPROPERTIES_SECURITYPROTOCOL=SASL_PLAINTEXT
29.      #- KAFKAPROPERTIES_SASLMECHANISM=PLAIN
30.      #- KAFKAPROPERTIES_SASLUSER=DAUHNRRCSF5ELA4W
31.      #- KAFKAPROPERTIES_SASLPASSWORD=YVV/7T7fVt9v2XJ9yTWkLTf04U6n0fux1mpsmnm9IgHo/bkd7cvQVGQyqasUc+ZQ
32.      - KAFKAPROPERTIES_TOPICS=<kafka_topic>
33. ...

Details of Environment Variables 

Service Configuration

• MESSAGE_TIME_WINDOW_SECONDS: Defines the time window to group messages in a single dbt execution. Value in seconds. 
• REFRESH_TIME_THRESHOLD: Specifies the waiting time after the last dbt execution before processing again. Value in seconds. 
• AUTO_REFRESH_TIME: Sets an automatic refresh time. Use -1 to disable this functionality. Value in seconds. 

DBT Configuration

• DBT_PROFILES_RELATIVE_DIR: Relative path to search for the profiles configuration file. If not found, the default directory /dbt/profiles will be used.
• DBT_PROJECT_NAME: Name of the dbt project. 
• DBT_PROFILE: dbt profile to use. 
• DBT_TARGET: Target environment for dbt (e.g., dev, prod), where you can also configure these variables in Database Configuration.  
• DBT_MODELS: Specific dbt models to execute. 
• DBT_FULL_REFRESH: Configuration to force a full refresh of the models. 

Kafka Configuration

• KAFKAPROPERTIES_DEPLOYMENTTYPE: Defines the type of Kafka deployment (e.g., cloud). 
• KAFKAPROPERTIES_BOOTSTRAPADDRESS: List of Kafka broker addresses, in the format <ip:port>, separated by commas. 
• KAFKAPROPERTIES_SECURITYPROTOCOL: Security protocol for Kafka (e.g., SASL_PLAINTEXT). 
• KAFKAPROPERTIES_SASLMECHANISM: SASL authentication mechanism (e.g., PLAIN). 
• KAFKAPROPERTIES_SASLUSER: User for SASL authentication. 
• KAFKAPROPERTIES_SASLPASSWORD: Password for SASL authentication. 
  

Database Configuration

1. Go to the project root.
2. Navigate to the profiles folder, which is inside the project directory. 
3. Locate and edit the profiles.yml file, where the database credentials are stored. 

1. dbt_oneconnect_ecc:
2.  #default target
3.  target: oneconnect_oi
4.  outputs:
5.    oneconnect_bi:
6.      type: clickhouse
7.      schema: <database_schema>
8.      host: <ip_host>
9.      port: <port>
10.      user: user
11.      password: 'password'
12.    oneconnect_oi:
13.      type: clickhouse
14.      schema: <database_schema>
15.      host: <ip_host>
16.      port: <port>
17.      user: user
18.      password: 'password'

Index 

INVENTORY_MANAGEMENT 

1. SILVER_MM_MATERIAL_ECC 
   

ORDER_TO_CASH 

1. SILVER_SD_CUSTOMER_ECC 

2. SILVER_SD_INVOICE_ECC 

3. SILVER_SD_OUTBOUND_DELIVERY_ECC 

4. SILVER_SD_SALES_CONTRACT_ECC 

5. SILVER_SD_SALES_ORDER_ECC 

6. SILVER_SD_SALES_QUOTATIONS_ECC 

7. SILVER_SD_SHIPPING_DOCUMENT_ECC 
   

PLAN_TO_PRODUCE 

1. SILVER_PP_PRODUCTION_ORDER_ECC 
   

PROCURE_TO_PAY 

1. SILVER_MM_PURCHASE_ORDER_ECC 
   

DASHBOARD 

1. INVOICE_DASHBOARD 

2. INVENTORY_DASHBOARD 

3. PRODUCCTION_DASHBOARD