Implementation of Service Access Host Service in docker

Directory 1. Scenarios STEP 2 Resolve 3. Summary 4. Reference

1. Scenarios

Use windows and wsl2 for daily development and testing. However, wsl2 often encounters network problems. For example, we are testing a project today, and the core function is to synchronize the data of postgres to clickhouse using the open source component synch.

Test the required components

postgres kafka zookeeper redis synch container

At the beginning of the test, the selected scheme is to arrange the above five services using docker-compose, and network_modules uses hosts mode, because considering the listening security mechanism of kafka, this network mode does not need to specify an exposed port separately.

The file docker-compose. yaml is as follows

version: "3"
 
services:
  postgres:
    image: failymao/postgres:12.7
    container_name: postgres
    restart: unless-stopped
    privileged: true                                                      #  Settings docker-compose env  Documents 
    command: [ "-c", "config_file=/var/lib/postgresql/postgresql.conf", "-c", "hba_file=/var/lib/postgresql/pg_hba.conf" ]
    volumes:
      - ./config/postgresql.conf:/var/lib/postgresql/postgresql.conf
      - ./config/pg_hba.conf:/var/lib/postgresql/pg_hba.conf
    environment:
      POSTGRES_PASSWORD: abc123
      POSTGRES_USER: postgres
      POSTGRES_PORT: 15432
      POSTGRES_HOST: 127.0.0.1
    healthcheck:
      test: sh -c "sleep 5 && PGPASSWORD=abc123 psql -h 127.0.0.1 -U postgres -p 15432 -c '\q';"
      interval: 30s
      timeout: 10s
      retries: 3
    network_mode: "host"
 
  zookeeper:
    image: failymao/zookeeper:1.4.0
    container_name: zookeeper
    restart: always
    network_mode: "host"
 
  kafka:
    image: failymao/kafka:1.4.0
    container_name: kafka
    restart: always
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: localhost:2181
      KAFKA_LISTENERS: PLAINTEXT://127.0.0.1:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_LOG_RETENTION_HOURS: 24
      KAFKA_LOG_DIRS: /data/kafka-data  # Data mount 
    network_mode: "host"
 
  producer:
    depends_on:
      - redis
      - kafka
      - zookeeper
    image: long2ice/synch
    container_name: producer
    command: sh -c "
      sleep 30 &&
      synch --alias pg2ch_test produce"
    volumes:
      - ./synch.yaml:/synch/synch.yaml
    network_mode: "host"
 
  # 1 Individual consumer consumption 1 Databases 
  consumer:
    tty: true
    depends_on:
      - redis
      - kafka
      - zookeeper
    image: long2ice/synch
    container_name: consumer
    command: sh -c
      "sleep 30 &&
      synch --alias pg2ch_test consume --schema pg2ch_test"
    volumes:
      - ./synch.yaml:/synch/synch.yaml
    network_mode: "host"
 
  redis:
    hostname: redis
    container_name: redis
    image: redis:latest
    volumes:
      - redis:/data
    network_mode: "host"
 
volumes:
  redis:
  kafka:
  zookeeper:

During the test, because postgres and wal2json components were used, it was very troublesome to install components separately in the container, and several attempts failed, so postgres service was later installed on the host machine, and synch service in the container used ip and port ports of the host machine.

However, when the service is restarted, synch service 1 cannot be started directly, and the log shows that postgres cannot be connected. The synch configuration file is as follows

core:
  debug: true # when set True, will display sql information.
  insert_num: 20000 # how many num to submit,recommend set 20000 when production
  insert_interval: 60 # how many seconds to submit,recommend set 60 when production
  # enable this will auto create database `synch` in ClickHouse and insert monitor data
  monitoring: true
 
redis:
  host: redis
  port: 6379
  db: 0
  password:
  prefix: synch
  sentinel: false # enable redis sentinel
  sentinel_hosts: # redis sentinel hosts
    - 127.0.0.1:5000
  sentinel_master: master
  queue_max_len: 200000 # stream max len, will delete redundant ones with FIFO
 
source_dbs:
  - db_type: postgres
    alias: pg2ch_test
    broker_type: kafka # current support redis and kafka
    host: 127.0.0.1
    port: 5433
    user: postgres
    password: abc123
    databases:
      - database: pg2ch_test
        auto_create: true
        tables:
          - table: pgbench_accounts
            auto_full_etl: true
            clickhouse_engine: CollapsingMergeTree
            sign_column: sign
            version_column:
            partition_by:
            settings:
 
clickhouse:
  # shard hosts when cluster, will insert by random
  hosts:
    - 127.0.0.1:9000
  user: default
  password: ''
  cluster_name:  # enable cluster mode when not empty, and hosts must be more than one if enable.
  distributed_suffix: _all # distributed tables suffix, available in cluster
 
kafka:
  servers:
    - 127.0.0.1:9092
  topic_prefix: synch

This situation is very strange. First, confirm postgres, start it, and the listening port (here, 5433) is normal. Both localhost and host eth0 network card addresses report errors.

STEP 2 Resolve

google answer, refer to stackoverflow high praise answer, the problem is solved, the original answer is as follows

If you are using Docker-for-mac or Docker-for-Windows 18.03+, just connect to your mysql service using the host host.docker.internal (instead of the 127.0.0.1 in your connection string).

If you are using Docker-for-Linux 20.10.0+, you can also use the host host.docker.internal if you started your Docker

container with the --add-host host.docker.internal:host-gateway option.

Otherwise, read below

Use** --network="host" **in your docker run command, then 127.0.0.1 in your docker container will point to your docker host.

See the source post for more details

In-container service accesses host service in host mode

Modify the postgres listening address as follows: host. docker. internal error reporting solution. View the host/etc/hosts file as follows

root@failymao-NC:/mnt/d/pythonProject/pg_2_ch_demo# cat /etc/hosts
# This file was automatically generated by WSL. To stop automatic generation of this file, add the following entry to /etc/wsl.conf:
# [network]
# generateHosts = false
127.0.0.1       localhost
 
10.111.130.24    host.docker.internal

You can see the mapping between the host ip and the domain name. By accessing the domain name, it resolves to the host ip and accesses the host service.

The final startup synch service configuration is as follows

core:
  debug: true # when set True, will display sql information.
  insert_num: 20000 # how many num to submit,recommend set 20000 when production
  insert_interval: 60 # how many seconds to submit,recommend set 60 when production
  # enable this will auto create database `synch` in ClickHouse and insert monitor data
  monitoring: true
 
redis:
  host: redis
  port: 6379
  db: 0
  password:
  prefix: synch
  sentinel: false # enable redis sentinel
  sentinel_hosts: # redis sentinel hosts
    - 127.0.0.1:5000
  sentinel_master: master
  queue_max_len: 200000 # stream max len, will delete redundant ones with FIFO
 
source_dbs:
  - db_type: postgres
    alias: pg2ch_test
    broker_type: kafka # current support redis and kafka
    host: host.docker.internal
    port: 5433
    user: postgres
    password: abc123
    databases:
      - database: pg2ch_test
        auto_create: true
        tables:
          - table: pgbench_accounts
            auto_full_etl: true
            clickhouse_engine: CollapsingMergeTree
            sign_column: sign
            version_column:
            partition_by:
            settings:
 
clickhouse:
  # shard hosts when cluster, will insert by random
  hosts:
    - 127.0.0.1:9000
  user: default
  password: ''
  cluster_name:  # enable cluster mode when not empty, and hosts must be more than one if enable.
  distributed_suffix: _all # distributed tables suffix, available in cluster
 
kafka:
  servers:
    - 127.0.0.1:9092
  topic_prefix: synch    host: host.docker.internal
core:
  debug: true # when set True, will display sql information.
  insert_num: 20000 # how many num to submit,recommend set 20000 when production
  insert_interval: 60 # how many seconds to submit,recommend set 60 when production
  # enable this will auto create database `synch` in ClickHouse and insert monitor data
  monitoring: true
 
redis:
  host: redis
  port: 6379
  db: 0
  password:
  prefix: synch
  sentinel: false # enable redis sentinel
  sentinel_hosts: # redis sentinel hosts
    - 127.0.0.1:5000
  sentinel_master: master
  queue_max_len: 200000 # stream max len, will delete redundant ones with FIFO
 
source_dbs:
  - db_type: postgres
    alias: pg2ch_test
    broker_type: kafka # current support redis and kafka
    host: 
    port: 5433
    user: postgres
    password: abc123
    databases:
      - database: pg2ch_test
        auto_create: true
        tables:
          - table: pgbench_accounts
            auto_full_etl: true
            clickhouse_engine: CollapsingMergeTree
            sign_column: sign
            version_column:
            partition_by:
            settings:
 
clickhouse:
  # shard hosts when cluster, will insert by random
  hosts:
    - 127.0.0.1:9000
  user: default
  password: ''
  cluster_name:  # enable cluster mode when not empty, and hosts must be more than one if enable.
  distributed_suffix: _all # distributed tables suffix, available in cluster
 
kafka:
  servers:
    - 127.0.0.1:9092
  topic_prefix: synch

3. Summary

When starting the container in--networks= "host" mode, if you want to access services on the host machine within the container, change ip to ` host. docker. internal '

4. Reference

https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach