kafka简介
TRANSCRIPT
![Page 2: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/2.jpg)
What Is The History?
![Page 3: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/3.jpg)
Daemon Based Messaging
![Page 4: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/4.jpg)
Tibco Rendezvous
![Page 5: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/5.jpg)
Broker-Based Messaging
![Page 6: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/6.jpg)
HOLD ON…
![Page 7: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/7.jpg)
Broker-less Messaging
![Page 8: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/8.jpg)
ZMQBrokerless
![Page 9: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/9.jpg)
本质上,⽀支持多种通信协议和模式的消息框架
![Page 10: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/10.jpg)
Actor Model
![Page 11: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/11.jpg)
![Page 12: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/12.jpg)
Messaging Scope
• Intra-Node
• Inter-Nodes
• Remoting
• Clustering
![Page 13: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/13.jpg)
Broker Based Messaging
• JMS家族
• AMQP家族
![Page 14: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/14.jpg)
JMS家族
![Page 15: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/15.jpg)
消息模型
Point-To-Point Pub-Sub
![Page 16: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/16.jpg)
相关产品
![Page 17: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/17.jpg)
AMQP家族
![Page 18: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/18.jpg)
消息模型
![Page 19: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/19.jpg)
相关产品
![Page 20: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/20.jpg)
Ultra Messaging
Todd L. Montgomery
![Page 21: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/21.jpg)
![Page 22: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/22.jpg)
理论模型
应⽤用模型
![Page 23: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/23.jpg)
![Page 24: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/24.jpg)
FAST 1. DISK FAVORED IO!2. Zero-Copy Transfer
SCALABLE PARTITIONING FOR SCALING READ&WRITE
DURABLE 1. REPLICA IN MIND!2. FLUSH PROPERLY
DISTRIBUTED FAULT-TOLERANCE!
![Page 25: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/25.jpg)
KAFKA的应⽤用场景
![Page 26: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/26.jpg)
KAFKA@Linkedin
![Page 29: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/29.jpg)
KAFKA InternalsKafka集群拓扑结
构, 相关配置,结点之间的协调,消费管理,统统
我来管~
![Page 30: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/30.jpg)
Zookeeper With Kafka
![Page 31: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/31.jpg)
Message Producer
!
"
![Page 32: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/32.jpg)
消息路由
props.put("partitioner.class", "xx.xxx.StringPrefixPartitioner")
class StringPrefixPartitioner(props:VerifiableProperties) extends kafka.producer.Partitioner{ def partition(key:Any, partitionNum:Int) : Int ={ hash(key.asInstanceOf[String].substring(0, 6)) % partitionNum } }
!
"
![Page 33: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/33.jpg)
Message Consumer!
" High Level Consumer API
# Simple Consumer API
![Page 34: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/34.jpg)
ConsumerGroup和Threads的关系
• 同⼀一consumer group中consumer线程的数量应该少于等于topic的partition的数量
• 同⼀一partition的数据会⼀一直分发给同⼀一个消费线程,但同⼀一个消费线程可能接收到来⾃自多个partition的数据
• 消费线程数量⼤大于topic的partition的数量的时候,有的消费线程不会接收到任何消息数据
![Page 35: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/35.jpg)
Ops• Stabilize Zookeeper First
• Higher Scala-Versioned Distribution If Exists
• Bigger Disk, Faster RPM
• Higher file handle number limit
• Raise default Heap Size(1G), make it 2-4G at least
![Page 36: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/36.jpg)
Conclusion
![Page 37: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/37.jpg)
Use Kafka Wisely!
![Page 38: Kafka简介](https://reader030.vdocuments.pub/reader030/viewer/2022012321/55a80b681a28abc4118b45fe/html5/thumbnails/38.jpg)
もんだい?