JAFKA

A fast MQ

overview

● https://github.com/adyliu/jafka● 271KB single jar● 3.5MB with all dependencies and

configurations● So simple, so fast

2jafka mq

Maven Usage

<dependency>

<groupId>com.sohu.jafka</groupId>

<artifactId>jafka</artifactId>

<version>1.0-rc7</version>

</dependency>

3jafka mq

Dependencies

● Zkclient– Zookeeper

● Log4j● Jackson

● Optional dependencies:– Mx4j-tools– jopt-simple

4jafka mq

Future

● A full kafka clone (http://incubator.apache.org/kafka)

● Persisten messaging with O(1)

● Constant time performance even with TB messages

● High-throughput (100k/s)

● Distributed messaging system

– producers/brokers/consumers● Auto load balancing

● Simple message format

5jafka mq

Message format (1)

6jafka mq

Message format (2)

7jafka mq

Log Implementation

8jafka mq

Architech and workflow

9jafka mq

Zookeeper nodes

10jafka mq

Offset length limit

● Long.MAX_VALUE– 9223372036854775807 → 9*1018

● If one message size is 1k– Messages: 9*1015

– If producing 1 billion messages per day● Continuing days: 9*106

● Continuing years: 24657

11jafka mq

About Filesystem

● 6 7200rpm SATA RAID-5 *

– 300MB/s linear writes (300k/s 1KB per message)– 50k/s random writes

http://incubator.apache.org/kafka/design.html 12jafka mq

sendfile() (1)

● Common data transfer

1.OS reads data from disk to pagecache

2.Application reads data from pagecache to buffer

3.Application writes data back into socket buffer

4.OS copies data from socket buffer to NIC buffer

● 4 copies + 2 system_call

13jafka mq

sendfile() (2)

● FileChannel.transferTo(long, long, WritableByteChannel)– Zero copy *

https://www.ibm.com/developerworks/linux/library/j-zerocopy/ 14jafka mq

Batch compression

● GZIP message compression default

15jafka mq

Sync or Async

● Both supported● Setting at connecting● Batch sending while using asynchronous

sending

16jafka mq

Usage

● Simple & Funny

17jafka mq

Simple usage - dependencies

<dependency> <groupId>com.sohu.suc*</groupId> <artifactId>suc-mq</artifactId> </dependency>

●Version >= 2.0-SNAPSHOT

18jafka mq

* our inner project

Simple usage - producer

Producers producers = Producers.buildProducer(); StringMessage data = new StringMessage("demo"); data.add("first message"); data.add("second message"); data.add("third message"); producers.send(data);

19jafka mq

Simple usage - consumer

public static void main(String[] args) {

Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { @Override public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); } });}

20jafka mq

Consumer => Queue

Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); }});

Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); }});

Client A

Client A

21jafka mq

Consumer => Topic

Consumers.buildConsumer("demo", "adyliu", new IMessageListener<String>() { public void onMessage(String message) { System.out.println(message + " ==> " + new Date()); }});

Consumers.buildConsumer("demo", "zhendong", new IMessageListener<String>() { public void onMessage(String message) { System.out.println("[ 震子 ] ==> " + message); }});

Client A

Client B

22jafka mq

Properties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("serializer.class", StringEncoder.class.getName());//ProducerConfig config = new ProducerConfig(props);Producer<String, String> producer = new Producer<String, String>(config);//StringProducerData data = new StringProducerData("demo");for(int i=0;i<100;i++) { data.add("Hello world #"+i);}try { long start = System.currentTimeMillis(); for (int i = 0; i < 100; i++) { producer.send(data); } long cost = System.currentTimeMillis() - start; System.out.println("send 10000 message cost: "+cost+" ms");} finally { producer.close();}

Full Producer code

23jafka mq

Full Consumer CodeProperties props = new Properties();props.put("zk.connect", "localhost:2181");props.put("groupid", "test_group");//ConsumerConfig consumerConfig = new ConsumerConfig(props);ConsumerConnector connector = Consumer.create(consumerConfig);//Map<String, List<MessageStream<String>>> topicMessageStreams = connector.createMessageStreams(ImmutableMap.of("demo", 2), new StringDecoder());List<MessageStream<String>> streams = topicMessageStreams.get("demo");//ExecutorService executor = Executors.newFixedThreadPool(2);final AtomicInteger count = new AtomicInteger();for (final MessageStream<String> stream : streams) { executor.submit(new Runnable() {

public void run() { for (String message : stream) { System.out.println(count.incrementAndGet() + " => " + message); } } });}

24jafka mq

jafka mq 25

More Documents

● https://github.com/adyliu/jafka/wiki

Open sourceFirst Java open source in maven repository at groupId com.sohu.

26jafka mq

About me

● Ady Liu● Email: imxylz@gmail.com● Architech@SOHU.COM

27jafka mq