故障报告

1. 问题（故障）描述

[2021-01-17 13:10:59,696] WARN Stopping serving logs in dir /app/kafka_2.13-2.6.0/data (kafka.log.LogManager)
[2021-01-17 13:10:59,699] ERROR Shutdown broker because all log dirs in /app/kafka_2.13-2.6.0/data have failed (kafka.log.LogManager)

[2021-01-17 13:26:57,450] INFO Completed load of Log(dir=/app/kafka_2.13-2.6.0/data/10006.prd_i_hygeia.ha_i_cus_person.s.2-0, topic=10006.prd_i_hygeia.ha_i_cus_person.s.2, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=3) with 1 segments in 23ms (2707/2707 loaded in /app/kafka_2.13-2.6.0/data) (kafka.log.LogManager)
[2021-01-17 13:26:57,452] INFO Loaded 2707 logs in 165834ms. (kafka.log.LogManager)
[2021-01-17 13:27:23,674] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2021-01-17 13:27:23,675] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2021-01-17 13:27:25,178] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.makePipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:65)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
at java.nio.channels.Selector.open(Selector.java:227)
at kafka.network.Acceptor.(SocketServer.scala:528)
at kafka.network.SocketServer.createAcceptor(SocketServer.scala:277)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:250)
at kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:248)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
at kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:248)
at kafka.network.SocketServer.startup(SocketServer.scala:122)
at kafka.server.KafkaServer.startup(KafkaServer.scala:286)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:82)
at kafka.Kafka.main(Kafka.scala)
[2021-01-17 13:27:25,181] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)

2. 问题（故障）原因

kafka打开文件数不够用导致

3. 解决

手工修改了open files 内核参数，在配置服务前在对系统参数/etc/security/limits.conf，已经做了优化

* soft nproc 65536
* hard nproc 65536
* soft nofile 655360
* hard nofile 655360

重启xshell生效。

然后修改全系统总限制
sudo vim /etc/sysctl.conf
在文件底部添加：fs.file-max=655350
立即生效：sudo sysctl -p
最后重启kafka

4. 参考