[Kafka/모니터링] Prometheus에 Kafka JMX Exporter 연결하기(w. Docker compose)

개요

Kafka 데이터를 모니터링 하기 위해, Exporter를 연결하여 Prometheus, Grafana 로 모니터링 대시보드를 구축하기 위해 테스트.

Kafka Exporter, JMX Exporter 두 가지 수집기 중 JMX Exporter를 Prometheus와 연결하는 작업 시행착오 기록.

 

 

환경

EC2 서버 내 DockerCompose를 통한 테스트.
Kafka, Exporter, Prometheus, Grafana 하나의 서버 내에 같이 존재.

 

작업 내용

Docker Compose Yaml 파일 생성

services:
  grafana:
    image: "grafana/grafana:9.4.7"
    ports:
      - "13000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: {비밀번호} #계정명은 admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus

  prometheus:
    image: "bitnami/prometheus:latest"
    ports:
      - "19090:9090"
    volumes:
      - ./etc/prometheus:/etc/prometheus
    command: "--config.file=/etc/prometheus/prometheus.yml"

  jmx-kafka101:
    image: "sscaling/jmx-prometheus-exporter"
    ports:
      - "15556:5556"
    environment:
      CONFIG_YML: "/etc/jmx_exporter/config.yml"
      JVM_OPTS: "-Xmx128m"
    volumes:
      - ./etc/jmx_exporter/config_kafka101.yml:/etc/jmx_exporter/config.yml
    depends_on:
      - kafka101

  kafka102:
    image: "sscaling/jmx-prometheus-exporter"
    ports:
      - "15557:5556"
    environment:
      CONFIG_YML: "/etc/jmx_exporter/config.yml"
      JVM_OPTS: "-Xmx128m"
    volumes:
      - ./etc/jmx_exporter/config_kafka102.yml:/etc/jmx_exporter/config.yml
    depends_on:
      - kafka102

  jmx-kafka103:
    image: "sscaling/jmx-prometheus-exporter"
    ports:
      - "15558:5556"
    environment:
      CONFIG_YML: "/etc/jmx_exporter/config.yml"
      JVM_OPTS: "-Xmx128m"
    volumes:
      - ./etc/jmx_exporter/config_kafka103.yml:/etc/jmx_exporter/config.yml
    depends_on:
      - kafka103
      
  zk1:
    image: confluentinc/cp-zookeeper:7.4.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 12181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
      - "12181:2181"

  kafka101:
    image: confluentinc/cp-kafka:7.4.1
    depends_on:
      - zk1
    ports:
      - "12000:29092"
      - "19092:9092"
      - "19991:9991"
    environment:
      KAFKA_BROKER_ID: 101
      KAFKA_JMX_PORT: 9991
      KAFKA_ZOOKEEPER_CONNECT: zk1:12181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka101:29092,PLAINTEXT_HOST://localhost:19092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka101:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zk1:12181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'false'
      KAFKA_HEAP_OPTS: "-Xmx1G"
    deploy:
      resources:
        limits:
          memory: "1G"

  kafka102:
    image: confluentinc/cp-kafka:7.4.1
    depends_on:
      - zk1
    ports:
      - "12100:29092"
      - "19093:9092"
      - "19992:9992"
    environment:
      KAFKA_BROKER_ID: 102
      KAFKA_JMX_PORT: 9992
      KAFKA_ZOOKEEPER_CONNECT: zk1:12181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka102:29092,PLAINTEXT_HOST://localhost:19093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka102:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zk1:12181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'false'
      KAFKA_HEAP_OPTS: "-Xmx1G"
    deploy:
      resources:
        limits:
          memory: "1G"

  kafka103:
    image: confluentinc/cp-kafka:7.4.1
    depends_on:
      - zk1
    ports:
      - "12200:29092"
      - "19094:9092"
      - "19993:9993"
    environment:
      KAFKA_BROKER_ID: 103
      KAFKA_JMX_PORT: 9993
      KAFKA_ZOOKEEPER_CONNECT: zk1:12181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka103:29092,PLAINTEXT_HOST://localhost:19094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: kafka103:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zk1:12181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'false'
      KAFKA_HEAP_OPTS: "-Xmx1G"
    deploy:
      resources:
        limits:
          memory: "1G"
volumes:
  grafana-data:

 

prometheus.yaml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'jmx'
    static_configs:
      - targets: ['jmx-kafka101:5556', 'jmx-kafka102:5556', 'jmx-kafka103:5556']

 

빌드 후 WEB UI 통해 확인

• Prometheus Web Ui가 잘 빌드 되었고, Target인 JMX Exporter가 잘 연결 되었는지 확인
  • 프로메테우스 - 타겟 서비스 : http://{호스트}:19090/targets?search=
• Grafana가 잘 빌드되었는지 확인
  • 그라파나 : http://{호스트}:13000/login

 

원래 jh-XXX로 서비스명을 설정했었을 때 캡쳐. 위 yaml 파일에선 jh-가 빠졌다.

 

yaml 파일 생성 후 빌드 시 겪은 오류

• kafka101, 102, 103 컨테이너 올라오지 않음 → Heap 설정 오류
  
  • 오류 내용
  • WARN An exception was thrown while closing send thread for session 0x10001f62c910000. (org.apache.zookeeper.ClientCnxn) EndOfStreamException: Unable to read additional data from server sessionid 0x10001f62c910000, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) [2024-06-05 13:53:14,492] INFO Session: 0x10001f62c910000 closed (org.apache.zookeeper.ZooKeeper) [2024-06-05 13:53:14,492] INFO EventThread shut down for session: 0x10001f62c910000 (org.apache.zookeeper.ClientCnxn) Using log4j config /etc/kafka/log4j.properties ===> Launching ... ===> Launching kafka ... Error: Could not find or load main class '-Xmx1G' Caused by: java.lang.ClassNotFoundException: '-Xmx1G'

환경 변수를 인식하지 못해서 발생한 오류

기존 : KAFKA_HEAP_OPTS: "${KAFKA_BROKER_HEAP_OPTS:-'-Xmx1G'}"

환경 변수 없이 직접 설정을 통해 해결

변경 : KAFKA_HEAP_OPTS: “-Xmx1G"

 

• jmx-kafka101, 102, 103 컨테이너 올라오지 않음 → JVM_OPTS 설정 오류
  • 오류 내용
  • docker logs jmx-kafka101
    
    Error: Could not find or load main class '-Xmx128m'

JVM_OPTS 설정 오류

기존 : JVM_OPTS: "${PROMETHEUS_JMX_AGENT_JVM_OPTS:-'-Xmx128m'}"

environment: CONFIG_YML: "/etc/jmx_exporter/config.yml" JVM_OPTS: "${PROMETHEUS_JMX_AGENT_JVM_OPTS:-'-Xmx128m'}"

Error-1과 같이 직접 지정

변경 : JVM_OPTS: "-Xmx128m"