By default Kafka Connect sends its output to stdout, so you’ll see it on the console, Docker logs, or wherever. Sometimes you might want to route it to file, and you can do this by reconfiguring log4j. You can also change the configuration to get more (or less) detail in the logs by changing the log level.

Finding the log configuration file 🔗

The configuration file is called connect-log4j.properties and usually found in etc/kafka/connect-log4j.properties.

To find out the config file location simply inspect the running Kafka Connect process, or the existing stdout, and look for the location of connect-log4j.properties in -Dlog4j.configuration:

[2019-01-15 10:34:22,755] INFO WorkerInfo values:
        jvm.args = -Xms256M, -Xmx2G, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/var/folders/q9/2tg_lt9j6nx29rvr5r5jn_bw0000gp/T/confluent.tcrfo4zH/connect/logs, -Dlog4j.configuration=file:/Users/Robin/cp/confluent-5.1.0/bin/../etc/kafka/connect-log4j.properties

By default the file looks like this:

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

Changing the log level 🔗

In troubleshooting Kafka Connect it can be useful to increase the log level to DEBUG (or even TRACE). To do this simply change the config file’s rootlogger:

#log4j.rootLogger=INFO, stdout
log4j.rootLogger=DEBUG, stdout

Sending logs to file 🔗

To send the output to file instead, change it to this:

log4j.rootLogger=INFO, logFile

log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.logFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.logFile.File=/tmp/connect-worker.log
log4j.appender.logFile.layout=org.apache.log4j.PatternLayout
log4j.appender.logFile.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

This will write it to /tmp/connect-worker.log.

Sending logs to both stdout and file 🔗

You can get fancy and send TRACE to file whilst retaining INFO (and above) on stdout, using the Threshold option for the stdout appender:

log4j.rootLogger=TRACE, traceFile, stdout

log4j.appender.traceFile=org.apache.log4j.DailyRollingFileAppender
log4j.appender.traceFile.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.traceFile.File=/tmp/connect-worker-trace.log
log4j.appender.traceFile.layout=org.apache.log4j.PatternLayout
log4j.appender.traceFile.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Threshold=INFO
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n

log4j.logger.org.apache.zookeeper=ERROR
log4j.logger.org.I0Itec.zkclient=ERROR
log4j.logger.org.reflections=ERROR

See also Configuring Kafka Connect to log REST HTTP messages to a separate file