Kafka概述

定义

Kafka是一个分布式的基于发布/订阅模式的消息队列,应用于大数据实时处理领域

消息队列的优点

主要是解耦和削峰

  • 解耦
  • 可恢复,如果系统中一部分组件失效,加入队列的消息仍然可以在系统恢复后被处理
  • 削峰
  • 灵活,可动态维护消息队列的集群
  • 异步

消息队列的两种模式

点对点

一对一,消费者主动拉取消息,收到后清除

发布/订阅模式

一对多,消费者消费后,消息不会清除,当然也不是永久保留,

分两种,一个是发布者主动推送,另一个是消费者主动拉取,Kafka就是消费者主动拉取,

推送 拉取
不好照顾多个消费者的接受速度 主动拉取,由消费者决定
消费者要每过一段时间就询问有没有新消息,长轮询

基础架构

Kafka Cluster 中有多个 Broker

Broker中有多个Topic Partion

每个Topic的多个Parttition,放在多个Broker上,可以提高Producer的并发,每个Topic Partition在其他Cluster上存有副本,用于备份,他们存在leader和follower,我们只找leader,不找follower

Topic是分区的,每个分区都是有副本的,分为leader和follower

消费者存在消费者组,一个分区只能被同一个组的某一个消费者消费,我们主要是把一个组当作一个大消费者,消费者组可以提高消费能力,消费者多了整个组的消费能力就高了,消费组中消费者的个数不要比消息多,不然就是浪费资源

Kafka利用Zookeeper来管理配置

0.9前消费者把自己消费的位置信息储存在Zookeeper中

0.9后是Kafka自己储存在某个主题中(减少了消费者和zk的连接)

我偷了个图

Kafka入门

常规安装

官网下载Kafka

brew install kafka

docker pull wurstmeister/kafka

Kafka安装

先安装zookeeper

然后安装kafka

1
2
3
4
5
docker run -d \
-eZK_HOSTS=zookeeper.zk \
--link zookeeper:zookeeper.zk \
--name=kafka \
kafkamanager/kafka-manager

Kafka compose 安装

1
2
3
4
5
mkdir ~/DockerDesktop
mkdir ~/DockerDesktop/Kafka
cd ~/DockerDesktop/Kafka
mkdir node1 node2 node3 node4 node5
vim docker-compose.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
version: '3'

services:
Kafka1:
image: wurstmeister/kafka
hostname: Kafka1
environment:
KAFKA_ADVERTISED_HOST_NAME: Kafka1
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: Zookeeper1:2181,Zookeeper2:2181,Zookeeper3:2181,Zookeeper4:2181,Zookeeper5:2181
volumes:
- ~/DockerDesktop/Kafka/node1:/kafka
external_links:
- Zookeeper1
- Zookeeper2
- Zookeeper3
- Zookeeper4
- Zookeeper5
networks:
default:
ipv4_address: 172.17.2.1

Kafka2:
image: wurstmeister/kafka
hostname: Kafka2
environment:
KAFKA_ADVERTISED_HOST_NAME: Kafka2
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: Zookeeper1:2181,Zookeeper2:2181,Zookeeper3:2181,Zookeeper4:2181,Zookeeper5:2181
volumes:
- ~/DockerDesktop/Kafka/node2:/kafka
external_links:
- Zookeeper1
- Zookeeper2
- Zookeeper3
- Zookeeper4
- Zookeeper5
networks:
default:
ipv4_address: 172.17.2.2

Kafka3:
image: wurstmeister/kafka
hostname: Kafka3
environment:
KAFKA_ADVERTISED_HOST_NAME: Kafka3
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: Zookeeper1:2181,Zookeeper2:2181,Zookeeper3:2181,Zookeeper4:2181,Zookeeper5:2181
volumes:
- ~/DockerDesktop/Kafka/node3:/kafka
external_links:
- Zookeeper1
- Zookeeper2
- Zookeeper3
- Zookeeper4
- Zookeeper5
networks:
default:
ipv4_address: 172.17.2.3

Kafka4:
image: wurstmeister/kafka
hostname: Kafka4
environment:
KAFKA_ADVERTISED_HOST_NAME: Kafka4
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: Zookeeper1:2181,Zookeeper2:2181,Zookeeper3:2181,Zookeeper4:2181,Zookeeper5:2181
volumes:
- ~/DockerDesktop/Kafka/node4:/kafka
external_links:
- Zookeeper1
- Zookeeper2
- Zookeeper3
- Zookeeper4
- Zookeeper5
networks:
default:
ipv4_address: 172.17.2.4


Kafka5:
image: wurstmeister/kafka
hostname: Kafka5
environment:
KAFKA_ADVERTISED_HOST_NAME: Kafka5
KAFKA_ADVERTISED_PORT: 9092
KAFKA_ZOOKEEPER_CONNECT: Zookeeper1:2181,Zookeeper2:2181,Zookeeper3:2181,Zookeeper4:2181,Zookeeper5:2181
volumes:
- ~/DockerDesktop/Kafka/node5:/kafka
external_links:
- Zookeeper1
- Zookeeper2
- Zookeeper3
- Zookeeper4
- Zookeeper5
networks:
default:
ipv4_address: 172.17.2.5


networks:
default:
external:
name: net17

执行下面的指令,Kafka集群开始运行

1
docker-compose up

看到了输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Kafka3_1  | [2020-04-18 10:26:27,441] INFO [Transaction Marker Channel Manager 1002]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
Kafka4_1 | [2020-04-18 10:26:27,451] INFO [ExpirationReaper-1005-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
Kafka5_1 | [2020-04-18 10:26:27,473] INFO [TransactionCoordinator id=1001] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
Kafka5_1 | [2020-04-18 10:26:27,524] INFO [TransactionCoordinator id=1001] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
Kafka5_1 | [2020-04-18 10:26:27,554] INFO [Transaction Marker Channel Manager 1001]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
Kafka1_1 | [2020-04-18 10:26:27,635] INFO [TransactionCoordinator id=1003] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
Kafka1_1 | [2020-04-18 10:26:27,644] INFO [TransactionCoordinator id=1003] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
Kafka1_1 | [2020-04-18 10:26:27,669] INFO [Transaction Marker Channel Manager 1003]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
Kafka2_1 | [2020-04-18 10:26:27,748] INFO [ExpirationReaper-1004-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
Kafka4_1 | [2020-04-18 10:26:27,753] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
Kafka3_1 | [2020-04-18 10:26:27,843] INFO [ExpirationReaper-1002-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
Kafka4_1 | [2020-04-18 10:26:27,882] INFO [SocketServer brokerId=1005] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
Kafka4_1 | [2020-04-18 10:26:27,945] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
Kafka4_1 | [2020-04-18 10:26:27,950] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
Kafka4_1 | [2020-04-18 10:26:27,955] INFO Kafka startTimeMs: 1587205587891 (org.apache.kafka.common.utils.AppInfoParser)
Kafka4_1 | [2020-04-18 10:26:27,976] INFO [KafkaServer id=1005] started (kafka.server.KafkaServer)
Kafka2_1 | [2020-04-18 10:26:27,989] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
Kafka1_1 | [2020-04-18 10:26:28,076] INFO [ExpirationReaper-1003-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
Kafka3_1 | [2020-04-18 10:26:28,095] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
Kafka2_1 | [2020-04-18 10:26:28,190] INFO [SocketServer brokerId=1004] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
Kafka2_1 | [2020-04-18 10:26:28,239] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
Kafka2_1 | [2020-04-18 10:26:28,241] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
Kafka2_1 | [2020-04-18 10:26:28,243] INFO Kafka startTimeMs: 1587205588196 (org.apache.kafka.common.utils.AppInfoParser)
Kafka2_1 | [2020-04-18 10:26:28,244] INFO [KafkaServer id=1004] started (kafka.server.KafkaServer)
Kafka3_1 | [2020-04-18 10:26:28,253] INFO [SocketServer brokerId=1002] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
Kafka3_1 | [2020-04-18 10:26:28,292] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
Kafka3_1 | [2020-04-18 10:26:28,295] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
Kafka3_1 | [2020-04-18 10:26:28,297] INFO Kafka startTimeMs: 1587205588257 (org.apache.kafka.common.utils.AppInfoParser)
Kafka3_1 | [2020-04-18 10:26:28,313] INFO [KafkaServer id=1002] started (kafka.server.KafkaServer)
Kafka1_1 | [2020-04-18 10:26:28,327] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
Kafka5_1 | [2020-04-18 10:26:28,365] INFO [ExpirationReaper-1001-AlterAcls]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
Kafka1_1 | [2020-04-18 10:26:28,533] INFO [SocketServer brokerId=1003] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
Kafka1_1 | [2020-04-18 10:26:28,582] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
Kafka1_1 | [2020-04-18 10:26:28,582] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
Kafka1_1 | [2020-04-18 10:26:28,584] INFO Kafka startTimeMs: 1587205588534 (org.apache.kafka.common.utils.AppInfoParser)
Kafka1_1 | [2020-04-18 10:26:28,607] INFO [KafkaServer id=1003] started (kafka.server.KafkaServer)
Kafka5_1 | [2020-04-18 10:26:28,931] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
Kafka5_1 | [2020-04-18 10:26:29,129] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)
Kafka5_1 | [2020-04-18 10:26:29,218] INFO Kafka version: 2.4.1 (org.apache.kafka.common.utils.AppInfoParser)
Kafka5_1 | [2020-04-18 10:26:29,218] INFO Kafka commitId: c57222ae8cd7866b (org.apache.kafka.common.utils.AppInfoParser)
Kafka5_1 | [2020-04-18 10:26:29,220] INFO Kafka startTimeMs: 1587205589130 (org.apache.kafka.common.utils.AppInfoParser)
Kafka5_1 | [2020-04-18 10:26:29,222] INFO [KafkaServer id=1001] started (kafka.server.KafkaServer)

同时我们在Zookeeper集群也看到了输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
Zookeeper1_1  | 2020-04-18 10:26:09,983 [myid:1] - WARN  [QuorumPeer[myid=1](plain=0.0.0.0:2181)(secure=disabled):Follower@170] - Got zxid 0x500000001 expected 0x1
Zookeeper1_1 | 2020-04-18 10:26:09,990 [myid:1] - INFO [SyncThread:1:FileTxnLog@284] - Creating new log file: log.500000001
Zookeeper5_1 | 2020-04-18 10:26:09,988 [myid:5] - INFO [SyncThread:5:FileTxnLog@284] - Creating new log file: log.500000001
Zookeeper2_1 | 2020-04-18 10:26:10,002 [myid:2] - WARN [QuorumPeer[myid=2](plain=0.0.0.0:2181)(secure=disabled):Follower@170] - Got zxid 0x500000001 expected 0x1
Zookeeper2_1 | 2020-04-18 10:26:10,045 [myid:2] - INFO [SyncThread:2:FileTxnLog@284] - Creating new log file: log.500000001
Zookeeper4_1 | 2020-04-18 10:26:10,059 [myid:4] - WARN [QuorumPeer[myid=4](plain=0.0.0.0:2181)(secure=disabled):Follower@170] - Got zxid 0x500000001 expected 0x1
Zookeeper1_1 | 2020-04-18 10:26:10,087 [myid:1] - INFO [CommitProcessor:1:LearnerSessionTracker@116] - Committing global session 0x500000589e20000
Zookeeper5_1 | 2020-04-18 10:26:10,092 [myid:5] - INFO [CommitProcessor:5:LeaderSessionTracker@104] - Committing global session 0x500000589e20000
Zookeeper2_1 | 2020-04-18 10:26:10,093 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x500000589e20000
Zookeeper3_1 | 2020-04-18 10:26:10,071 [myid:3] - WARN [QuorumPeer[myid=3](plain=0.0.0.0:2181)(secure=disabled):Follower@170] - Got zxid 0x500000001 expected 0x1
Zookeeper4_1 | 2020-04-18 10:26:10,098 [myid:4] - INFO [SyncThread:4:FileTxnLog@284] - Creating new log file: log.500000001
Zookeeper3_1 | 2020-04-18 10:26:10,109 [myid:3] - INFO [SyncThread:3:FileTxnLog@284] - Creating new log file: log.500000001
Zookeeper1_1 | 2020-04-18 10:26:10,113 [myid:1] - INFO [CommitProcessor:1:LearnerSessionTracker@116] - Committing global session 0x100000589b30000
Zookeeper2_1 | 2020-04-18 10:26:10,126 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x100000589b30000
Zookeeper2_1 | 2020-04-18 10:26:10,141 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000589b20000
Zookeeper4_1 | 2020-04-18 10:26:10,144 [myid:4] - INFO [CommitProcessor:4:LearnerSessionTracker@116] - Committing global session 0x500000589e20000
Zookeeper3_1 | 2020-04-18 10:26:10,137 [myid:3] - INFO [CommitProcessor:3:LearnerSessionTracker@116] - Committing global session 0x500000589e20000
Zookeeper1_1 | 2020-04-18 10:26:10,171 [myid:1] - INFO [CommitProcessor:1:LearnerSessionTracker@116] - Committing global session 0x200000589b20000
Zookeeper3_1 | 2020-04-18 10:26:10,199 [myid:3] - INFO [CommitProcessor:3:LearnerSessionTracker@116] - Committing global session 0x100000589b30000
Zookeeper4_1 | 2020-04-18 10:26:10,176 [myid:4] - INFO [CommitProcessor:4:LearnerSessionTracker@116] - Committing global session 0x100000589b30000
Zookeeper4_1 | 2020-04-18 10:26:10,202 [myid:4] - INFO [CommitProcessor:4:LearnerSessionTracker@116] - Committing global session 0x200000589b20000
Zookeeper3_1 | 2020-04-18 10:26:10,203 [myid:3] - INFO [CommitProcessor:3:LearnerSessionTracker@116] - Committing global session 0x200000589b20000
Zookeeper4_1 | 2020-04-18 10:26:10,204 [myid:4] - INFO [CommitProcessor:4:LearnerSessionTracker@116] - Committing global session 0x200000589b20001
Zookeeper4_1 | 2020-04-18 10:26:10,209 [myid:4] - INFO [CommitProcessor:4:LearnerSessionTracker@116] - Committing global session 0x200000589b20002
Zookeeper2_1 | 2020-04-18 10:26:10,224 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000589b20001
Zookeeper3_1 | 2020-04-18 10:26:10,227 [myid:3] - INFO [CommitProcessor:3:LearnerSessionTracker@116] - Committing global session 0x200000589b20001
Zookeeper3_1 | 2020-04-18 10:26:10,241 [myid:3] - INFO [CommitProcessor:3:LearnerSessionTracker@116] - Committing global session 0x200000589b20002
Zookeeper2_1 | 2020-04-18 10:26:10,243 [myid:2] - INFO [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session 0x200000589b20002
Zookeeper5_1 | 2020-04-18 10:26:10,245 [myid:5] - INFO [CommitProcessor:5:LeaderSessionTracker@104] - Committing global session 0x100000589b30000
Zookeeper5_1 | 2020-04-18 10:26:10,260 [myid:5] - INFO [CommitProcessor:5:LeaderSessionTracker@104] - Committing global session 0x200000589b20000
Zookeeper5_1 | 2020-04-18 10:26:10,270 [myid:5] - INFO [CommitProcessor:5:LeaderSessionTracker@104] - Committing global session 0x200000589b20001
Zookeeper5_1 | 2020-04-18 10:26:10,307 [myid:5] - INFO [CommitProcessor:5:LeaderSessionTracker@104] - Committing global session 0x200000589b20002
Zookeeper1_1 | 2020-04-18 10:26:10,403 [myid:1] - INFO [CommitProcessor:1:LearnerSessionTracker@116] - Committing global session 0x200000589b20001
Zookeeper1_1 | 2020-04-18 10:26:10,407 [myid:1] - INFO [CommitProcessor:1:LearnerSessionTracker@116] - Committing global session 0x200000589b20002

Kafka操作

开始操作

1
2
3
docker exec -it kafka_Kafka1_1 bash
cd /opt/kafka/bin
ls

我们可以看到一大堆东西

1
2
3
4
5
6
7
connect-distributed.sh               kafka-console-producer.sh            kafka-log-dirs.sh                    kafka-server-start.sh                windows
connect-mirror-maker.sh kafka-consumer-groups.sh kafka-mirror-maker.sh kafka-server-stop.sh zookeeper-security-migration.sh
connect-standalone.sh kafka-consumer-perf-test.sh kafka-preferred-replica-election.sh kafka-streams-application-reset.sh zookeeper-server-start.sh
kafka-acls.sh kafka-delegation-tokens.sh kafka-producer-perf-test.sh kafka-topics.sh zookeeper-server-stop.sh
kafka-broker-api-versions.sh kafka-delete-records.sh kafka-reassign-partitions.sh kafka-verifiable-consumer.sh zookeeper-shell.sh
kafka-configs.sh kafka-dump-log.sh kafka-replica-verification.sh kafka-verifiable-producer.sh
kafka-console-consumer.sh kafka-leader-election.sh kafka-run-class.sh trogdor.sh

指定Zookeeper1,看看消息,结果啥都没有,因为kafka中没有消息

1
kafka-topics.sh --zookeeper Zookeeper1:2181 --list

创建主题, –topic 定义topic名字,–replication-factor定义副本数量,–partitions定义分区数量, 我们创建3个副本一个分区的主题first

1
kafka-topics.sh --zookeeper Zookeeper1:2181 --create --replication-factor 3 --partitions 1 --topic first

看到输出

1
Created topic first.

然后使用kafka-topics.sh --zookeeper Zookeeper1:2181 --list就可以看到输出了一个first

1
first

现在我们回到docker外面的宿主机的终端

1
2
cd ~/DockerDesktop/Kafka
ls node1/kafka-logs-Kafka1/ node2/kafka-logs-Kafka2 node3/kafka-logs-Kafka3 node4/kafka-logs-Kafka4 node5/kafka-logs-Kafka5

得到了输出,由此可见,我们的node3,node4,node5上分别保留了first的副本,这里还有一个细节,我们现在是在kafka1上执行的命令,这也能说明我们的集群是搭建成功了的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
node1/kafka-logs-Kafka1/:
cleaner-offset-checkpoint log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint

node2/kafka-logs-Kafka2:
cleaner-offset-checkpoint log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint

node3/kafka-logs-Kafka3:
cleaner-offset-checkpoint first-0 log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint

node4/kafka-logs-Kafka4:
cleaner-offset-checkpoint first-0 log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint

node5/kafka-logs-Kafka5:
cleaner-offset-checkpoint first-0 log-start-offset-checkpoint meta.properties recovery-point-offset-checkpoint replication-offset-checkpoint

然后我们回到docker中,多来几次

1
2
3
4
kafka-topics.sh --zookeeper Zookeeper2:2181 --create --replication-factor 3 --partitions 1 --topic second
kafka-topics.sh --zookeeper Zookeeper3:2181 --create --replication-factor 3 --partitions 1 --topic third
kafka-topics.sh --zookeeper Zookeeper4:2181 --create --replication-factor 3 --partitions 1 --topic four
kafka-topics.sh --zookeeper Zookeeper5:2181 --create --replication-factor 3 --partitions 1 --topic five

最后再查看宿主机中的磁盘映射,这里一切正常,并且访问zookeeper集群中的任意一台机器都可行

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
node1/kafka-logs-Kafka1/:
cleaner-offset-checkpoint log-start-offset-checkpoint recovery-point-offset-checkpoint second-0
five-0 meta.properties replication-offset-checkpoint third-0

node2/kafka-logs-Kafka2:
cleaner-offset-checkpoint log-start-offset-checkpoint recovery-point-offset-checkpoint second-0
four-0 meta.properties replication-offset-checkpoint

node3/kafka-logs-Kafka3:
cleaner-offset-checkpoint five-0 log-start-offset-checkpoint recovery-point-offset-checkpoint
first-0 four-0 meta.properties replication-offset-checkpoint

node4/kafka-logs-Kafka4:
cleaner-offset-checkpoint five-0 meta.properties replication-offset-checkpoint third-0
first-0 log-start-offset-checkpoint recovery-point-offset-checkpoint second-0

node5/kafka-logs-Kafka5:
cleaner-offset-checkpoint four-0 meta.properties replication-offset-checkpoint
first-0 log-start-offset-checkpoint recovery-point-offset-checkpoint third-0

全删掉

1
2
3
4
5
kafka-topics.sh --delete --zookeeper Zookeeper1:2181 --topic first
kafka-topics.sh --delete --zookeeper Zookeeper1:2181 --topic second
kafka-topics.sh --delete --zookeeper Zookeeper1:2181 --topic third
kafka-topics.sh --delete --zookeeper Zookeeper1:2181 --topic four
kafka-topics.sh --delete --zookeeper Zookeeper1:2181 --topic five

看到输出,在我的集群中,我发先几秒钟后,就被删干净了

1
2
Topic first is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.

为了后续的操作,我们重新创建一个新的主题

1
kafka-topics.sh --zookeeper Zookeeper5:2181 --create --replication-factor 3 --partitions 2 --topic first

随便起一台Kafka1, 作为生产者, 这里可以用localhost是因为他自己就是集群的一部分

1
kafka-console-producer.sh --topic first --broker-list localhost:9092

再起另外一台Kafka2作为消费者,这台就开始等待了

1
kafka-console-consumer.sh --topic first --bootstrap-server localhost:9092

在生成者中输出>hello I am producer, 我们就能在消费者中看到,那么过时的消费者怎么办呢?我们使用上面的指令再起一台消费者Kafka3, 发现他并不能收到hello那条消息了,在生成者中输入>this is the second msg,发现kafka2和kafka3都可以收到消息,然后我们使用下面的指令再其一台Kafka4,等待片刻,发现kafka4收到了所有的消息

1
kafka-console-consumer.sh --topic first --bootstrap-server localhost:9092 --from-beginning

在宿主机中输入

1
ls node1/kafka-logs-Kafka1/ node2/kafka-logs-Kafka2 node3/kafka-logs-Kafka3 node4/kafka-logs-Kafka4 node5/kafka-logs-Kafka5

得到输出,可以看到offsets是轮流保存的, 因为分区是为了负载均衡,而备份是为了容错

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
node1/kafka-logs-Kafka1/:
__consumer_offsets-14 __consumer_offsets-29 __consumer_offsets-4 __consumer_offsets-9 log-start-offset-checkpoint replication-offset-checkpoint
__consumer_offsets-19 __consumer_offsets-34 __consumer_offsets-44 cleaner-offset-checkpoint meta.properties
__consumer_offsets-24 __consumer_offsets-39 __consumer_offsets-49 first-1 recovery-point-offset-checkpoint

node2/kafka-logs-Kafka2:
__consumer_offsets-0 __consumer_offsets-20 __consumer_offsets-35 __consumer_offsets-5 log-start-offset-checkpoint replication-offset-checkpoint
__consumer_offsets-10 __consumer_offsets-25 __consumer_offsets-40 cleaner-offset-checkpoint meta.properties
__consumer_offsets-15 __consumer_offsets-30 __consumer_offsets-45 first-0 recovery-point-offset-checkpoint

node3/kafka-logs-Kafka3:
__consumer_offsets-13 __consumer_offsets-28 __consumer_offsets-38 __consumer_offsets-8 log-start-offset-checkpoint replication-offset-checkpoint
__consumer_offsets-18 __consumer_offsets-3 __consumer_offsets-43 cleaner-offset-checkpoint meta.properties
__consumer_offsets-23 __consumer_offsets-33 __consumer_offsets-48 first-0 recovery-point-offset-checkpoint

node4/kafka-logs-Kafka4:
__consumer_offsets-1 __consumer_offsets-21 __consumer_offsets-36 __consumer_offsets-6 first-1 recovery-point-offset-checkpoint
__consumer_offsets-11 __consumer_offsets-26 __consumer_offsets-41 cleaner-offset-checkpoint log-start-offset-checkpoint replication-offset-checkpoint
__consumer_offsets-16 __consumer_offsets-31 __consumer_offsets-46 first-0 meta.properties

node5/kafka-logs-Kafka5:
__consumer_offsets-12 __consumer_offsets-22 __consumer_offsets-37 __consumer_offsets-7 log-start-offset-checkpoint replication-offset-checkpoint
__consumer_offsets-17 __consumer_offsets-27 __consumer_offsets-42 cleaner-offset-checkpoint meta.properties
__consumer_offsets-2 __consumer_offsets-32 __consumer_offsets-47 first-1 recovery-point-offset-checkpoint

查看zk中的数据,起一台zk,执行zkCli.sh, 再执行ls /, 其中除了zookeeper文件以外,其他的数据都是Kafka的,部分终端显示如下

1
2
3
4
5
6
7
8
9
10
11
12
Welcome to ZooKeeper!
2020-04-19 07:03:58,554 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1154] - Opening socket connection to server localhost/127.0.0.1:2181.
2020-04-19 07:03:58,557 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1156] - SASL config status: Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2020-04-19 07:03:58,638 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@986] - Socket connection established, initiating session, client: /127.0.0.1:41878, server: localhost/127.0.0.1:2181
2020-04-19 07:03:58,690 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1420] - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x1000223de0e000b, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[admin, brokers, cluster, config, consumers, controller, controller_epoch, isr_change_notification, latest_producer_id_block, log_dir_event_notification, zookeeper]

Kafka架构深入

文件储存

面向主题,消息按照主题分类,生产者生产消息,消费者消费消息

topic是逻辑概念, partition是物理概念,因为文件夹是用topic+parttiton命名的
查看first-0的文件内容, 0000.log实际上存的是数据,不是日志

1
2
bash-4.4# ls
00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex leader-epoch-checkpoint

Kafka的配置文件中有谈到, 即上面的000000.log最多只能保存1G,当他超过1G以后,会创建新的.log

1
2
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

分片和索引

1
2
3
4
5
6
00000000000000000000.index 
00000000000000000000.log
00000000000000170410.index
00000000000000170410.log
00000000000000239430.index
00000000000000239430.log

文件名其实值得是当前片段(segment)中最小的消息的偏移量,log只存数据,index存消息在log中的偏移量

当我们要寻找某个消息的时候,先通过二分消息的编号,找到该消息再哪个index中,由于index中的数据都是等长的,所以可以直接用乘法定位index文件中的偏移量,然后根据这个偏移量定位到log文件中的位置

生产者

分区

方便扩展,提高并发,可以指定分区发送,可以指定key发送(key被hash成分区号), 可以不指定分区不指定key发送(会被随机数轮循)

数据可靠性保证

怎么保证可靠?Kafka需要给我们返回值,但是是leader写成功后返回还是follower成功后返回呢?哪个策略好呢?

副本数据同步策略

方案 优点 缺点
半数以上同步则ack 延迟低 选举新leader的时候,容忍n台节点故障,需要2n+1个副本
完全同步则ack 选举新leader的时候,容忍n台节点故障,需要n+1个副本 延迟高

Kafka选择了完全同步才发送ack,这有一个问题,如果同步的时候,有一台机器宕机了,那么永远都不会发送ack了

ISR

in-sync replica set

leader 动态维护了一个动态的ISR,只要这个集合中的机器同步完成,就发送ack,选举ISR的时候,根据节点的同步速度和信息差异的条数来决定,在高版本中只保留了同步速度,为什么呢?延迟为什么比数据重要?

由于生产者是按照批次生产的,如果我们保留信息差异,当生产者发送大量信息的时候,直接就拉开了leader和follower的信息差异条数,同步快的follower首先拉小了自己和leader信息差异,这时候他被加入ISR,但最一段时间后他会被同步慢但是,最终信息差异小的follower赶出ISR,这就导致了ISR频繁发生变化,意味着ZK中的节点频繁变化,这个选择不可取

acks

ack级别 操作 数据问题
0 leader收到后就返回ack broker故障可能丢失数据
1 leader写入磁盘后ack 在follower同步前的leader故障可能导致丢失数据
-1/all 等待ISR的follower写入磁盘后返回ack 在follower同步后,broker发送ack前,leader故障则导致数据重复

acks=-1也会丢失数据,在ISR中只有leader一个的时候发生

数据一致性问题

HW(High Watermark) 高水位, 集群中所有节点都能提供的最新消息

LEO(Log End Offset) 节点各自能提供的最新消息

为了保证数据的一致性,我们只提供HW的消费,就算消息丢了后,消费者也不知道,他看起来就是一致性的

leader故障

当重新选择leader后,为了保证多个副本之间的数据一致性,会通知follower将各自的log文件高于HW的地方截断,重新同步,这里只能保证数据一致性,不能保证数据不丢失或者不重复

精准一致性(Exactly Once)

ACKS 为 -1 则不会丢失数据,即Least Once

ACKS 为 1 则生产者的每条数据只发送一次, 即At Most Once

他们一个丢数据一个重复数据

幂等性

开启幂等性,将Producer参数中的enable.idompotence设置为true,Producer会被分配一个PID(Producer ID), 发往同一个Partition的消息会附带序列号,而Broker会对PID,Partition,SeqNumber做缓存,当具有相同的主键消息提交的时候,Broker只会持久化一条,但是要注意PID重启会变化,不同的Partition也有不同的主键,所以幂等性无法保证跨分区会话的Exactly Once。

消费者

分区分配策略

一个consumer group中有多个consumer,一个topic中有多个partition,那么怎么分配呢?

RoundRobin策略

1
2
3
4
Topic1: x0,x1,x2
Topic2: y0,y1,y2
-> [x0,y2,y1,y0,x1,x2]
-> [x0,y1,x1],[y2,y0,x2]

把所有主题中的所有partition放到一起,按照hash值排序,然后轮循分配给消费者

这样太混乱了,不太好

Range策略

1
2
3
Topic1: x0,x1,x2
Topic2: y0,y1,y2
-> [x0,x1,y0,y1],[x2,y2]

对于每个主题分开考虑,各自截取一段,分给消费者,

负载不均衡了

重新分配

当消费者的个数发生变化的时候,就会触发重新分配

offset维护

按照消费者组、主题、分区来维护offset,不能按照消费者维护,要是这样就不能让消费者组具有动态性质了
进入zk中

1
2
3
4
ls /brokers # 查看kafka集群
ls /brokers/ids # 查看ids
ls /brokers/topics # 查看主题
ls /consumers # 查看消费者组

消费者会默认生成一个消费者组的编号,其中有offset/mytopic/0

单机高效读写

顺序写磁盘

写磁盘的时候一直使用追加,官方数据表明同样的磁盘,顺序写可以达到600M/s但是随机写只有100K/s,

零拷贝技术

一般情况下,用户读取文件需要操作系统来协助,先读到内核空间,然后读到用户空间,然后写入内核空间,最后写入磁盘,零拷贝技术允许直接将这个拷贝工作交给操作系统完成

Zookeeper

Kafka集群中有一个broker会被选举为Controller,负责管理集群broker的上下线、topic分区副本分配和leader选举等工作

Kafka事务

Producer事务

引入全局唯一的Transaction ID,替代PID,为了管理Transaction,Kafka引入了Transaction Producer和Transaction Coordinator交互获得Transaction ID。

Consumer事务

相对弱一些,用户可以自己修改offset或者跨segment的消费如果出错并且等满7天以后,segment被删除了,这些都导致问题

Kafka API

消息发送流程

Kafka的Producer发送消息是异步消息,两个线程main和sender,

发送消息的时候分三步,先经过拦截器,然后经过序列化器,最后经过分区器,最后才发出去

创建kafka项目

springinit 里面选择kafka

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 指定kafka集群
bootstrap.servers=172.17.1.1:9092
# ack应答级别
acks=all
# 重试次数
retries=3
# 批次大小 16K, 当超过16K就提交
batch.size=16384
# 等待时间 , 当超过1ms就提交
linger.ms=1
# RecordAccmulator缓冲区大小 32M
buffer.memory=33554432
# key value 的序列化类
key.serializer=org.apache.kafka.serialization.StringSerializer
value.serializer=org.apache.kafka.serialization.StringSerializer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
package com.wsx.study.kafka.debug;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.util.Properties;

public class Main {
public static void main(String[] args) {
// 创建Kafka生产者配置信息
try {
Properties properties = new Properties();
FileInputStream in = new FileInputStream("KafkaProducer.properties");
properties.load(in);
in.close();
KafkaProducer<String, String> stringStringKafkaProducer = new KafkaProducer<>(properties);
for (int i = 0; i < 10; i++) {
stringStringKafkaProducer.send(new ProducerRecord<>("first", "javarecord" + i));
}
stringStringKafkaProducer.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}

然后创建消费者

1
kafka-console-consumer.sh --topic first --bootstrap-server localhost:9092
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 指定kafka集群
bootstrap.servers=172.17.2.1:9092 # 日了狗了,这些mac似乎不行了
# ack应答级别
acks=all
# 重试次数
retries=3
# 批次大小 16K, 当超过16K就提交
batch.size=16384
# 等待时间 , 当超过1ms就提交
linger.ms=1
# RecordAccmulator缓冲区大小 32M
buffer.memory=33554432
# key value 的序列化类
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
package com.wsx.study.kafka.debug;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;

public class Main {
public void test() {
// 创建Kafka生产者配置信息
try {
Properties properties = new Properties();
InputStream in = getClass().getClassLoader().getResourceAsStream("KafkaProducer.properties");
properties.load(in);
assert in != null;
in.close();
KafkaProducer<String, String> stringStringKafkaProducer = new KafkaProducer<>(properties);
for (int i = 0; i < 1; i++) {
stringStringKafkaProducer.send(new ProducerRecord<>("first", "javarecord" + i));
}
stringStringKafkaProducer.close();
} catch (IOException e) {
e.printStackTrace();
}
}

public static void main(String[] args) {
new Main().test();
}
}

消费者

1
2
3
4
5
6
7
for (int i = 0; i < 1; i++) {
stringStringKafkaProducer.send(new ProducerRecord<>("first", "javarecord" + i), (recordMetadata, e) -> {
if(e==null){
System.out.println(recordMetadata.offset()+recordMetadata.offset());
}
});
}

自己写分区器

配置文件配置一下就可以了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class MyPartitioner implements Partitioner {

@Override
public int partition(String s, Object o, byte[] bytes, Object o1, byte[] bytes1, Cluster cluster) {
return 0;
}

@Override
public void close() {

}

@Override
public void configure(Map<String, ?> map) {

}
}

生产者

同理,

1
2
3
4
consumer.subscribe(Arrays.asList("first"));
while(true){
ConsumerRecords<String,Strings> consumerRecods = consumer.poll(long timeout); // 延迟
}

如何–beginning

auto.offset.reset 当没有初始offset或者offset被删除了(数据过期)就会启动earliest,从最老的数据开始消费,这个东西不是0,他叫earlist,是最早不是开头

默认值是latest, 因为命令行的创建出来的是新的消费者组,所以启用了earliest

想要重新开始消费,要设earlist且换新的消费者组

offset加速

消费者只会在启动的时候拉取一次offset,如果没有自动提交offset,那么消费者就不会提交,这会导致数据不一致,如果这个时候消费者被强制终止,那么你下一次跑这个代码的时候,还是从之前的offset开始消费,除非你提交

enable.auto.commit

可以按时间提交

手动提交

同步: 当前线程会阻塞直到offset提交成功

异步: 加一个回调函数就可以

问题

自动提交速度快可能丢数据,比如我还没处理完,他就提交了,然后我挂了,数据就丢了
自动提交速度慢可能重复数据,我处理完了,他还没提交,然后我挂了,下次又来消费一次数据
手动提交也有这些问题

自定义offset

由于消息往往对消费者而言,可能存在本地的sql中,所以就可以和数据以前做成一个事务,

这可以解决问题,但是碰到了rebalace问题,即当一个消费者挂了以后消息资源要重新分配,借助ConsumerRebalanceListener,点这里, 自己维护一个消费者组数据、自己写代码,(可怕)

自定义拦截器

configure 读取配置信息

onSend(ProducerRecord) 拦截

onAcknowledgement(RecordMetadata,Exception), 这个和上面的回调函数一样,拦截器也会收到这个东西,

close 拦截

例子

现在有个需求,消息发送前在消息上加上时间挫,消息发送后打印发送成功次数和失败次数
时间拦截器

1
2
3
4
5
6
7
8
9
10
11
@Override
public ProducerRecord<String, String> onSend(ProducerRecord<String, String> producerRecord) {
// 取出数据
String value = producerRecord.value();
// 创建新的
return new ProducerRecord<String, String>(producerRecord.topic(),
producerRecord.partition(), producerRecord.timestamp(),
producerRecord.key(), System.currentTimeMillis()+","+producerRecord.value(),
producerRecord.headers());
}

计数拦截器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
class CountInterceptor implements ProducerInterceptor<String, String>{

int success = 0;
int error = 0;

@Override
public ProducerRecord<String, String> onSend(ProducerRecord<String, String> producerRecord) {
return null;
}

@Override
public void onAcknowledgement(RecordMetadata recordMetadata, Exception e) {
if(e==null){
success++;
}else{
error++;
}
}

@Override
public void close() {
System.out.println("success:"+success);
System.out.println("error:"+error);
}

@Override
public void configure(Map<String, ?> map) {

}
}

注意如果拦截器太多,考虑使用拦截器链

拦截器、序列化器、分区器都是卸载配置文件中的

Kafka监控

Kafka Eagle
修改Kafka的kafka-server-start.sh, 对其进行修改,

1
2
if ["x$KAFKA_HEAP_OPTS" = "x"] then
export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128...."

然后分发这个文件,再上传kafka-eagle-bin-1.3.7.tar.gz到集群的/opt/software中,

配置文件

可以跟踪多个集群

kafka.eagle.zk.cluster.alisa = cluster1,cluster2

cluster1.zk.list=ip:port,ip:port,…

保存的位置

cluster1.kafka.eagle.offset.storage=kafka

监控图表

kafka.eagle.metrics.charts=true

启动

bin/ke.sh start

http://192.168.9.102:8048/ke

有很多信息都能看到,

Kafka面试题

Kafka 的ISR OSR AR

ISR+OSR=AR

HW LEO

高水位,LEO

怎么体现消息的顺序

区内有序

分区器、序列化器、拦截器

生产者的整体结构,几个线程

消费者组中的消费者个数超过了topic就会有消费者收不到数据对吗

对的

提交的是offset还是offset+1

是offset+1

什么时候重复消费

先处理数据后提交

什么时候漏消费

先提交后处理数据

创建topic背后的逻辑

zk中创建新的topic节点,触发controller的监听,controller创建topic然后更新metadata cache

topic分区可以增加吗?

可以,不可以减少

kafka内部有topic吗

有个offset

kafka分区分配的概念

Rodrobin和range

日志目录结构

二分->index->log

kafka controller的作用

相当于老大,他是干活的,他和zk通信,还通知其他人

kafka什么时候选举

选controller,leader,ISR

失效副本是什么

这个问题很奇怪,大概是想说重新选举leader的时候,那个HW变化

为什么kafka高效率

顺序写+0拷贝

架构

压测

有一个***perf-test.sh

消息积压,消费者消费能力不够怎么办

增加topic分区、提高消费者组的消费者数量、提高消费者每次拉取的数量(默认500)

参考资料

Kafka教程

docker安装kafka