실시간 인벤트 처리
TRANSCRIPT
실시간 이벤트 처리
솔루션
EsperApache S4Twitter StormHStreaming
비슷함
Event Stream Pro-cessing
Complex Event Processing
Event Stream Process-ing 은 이벤트 스트림의 빠른 처리와 단순한 산술 계산에 초첨을 두고 있다 .
Complex Event Pro-cessing 은 패턴 분석에 주안점을 둔 데이터 분석을 포함하고 있다
Esper
• ESP/CEP with expressive Event Pro-cessing Language: continuous queries, aggregation, joins, causality and missing events, joins to historical data, output flow control...
• High throughput, low latency• Standalone or embedded, open API• Multi platform support
Computing Rates Per Feed
insert into TicksPerSecond select feed, rate(10) as cnt from MarketDataEvent group by feed
Detecting a Fall-off
select feed, avg(cnt) as avgCnt, cnt as feedCnt from TicksPerSecond.win:time(10 seconds) group by feed having cnt < avg(cnt) * 0.75
Storm
Stream processingContinuous computationDistributed RPC
Storm do
• Guaranteed message process-ing• Robust process management• Fault detection and automatic
reassignment• Efficient message passing• Local mode and distributed
mode
TopologyBuilder builder = new TopologyBuilder();builder.setSpout(1, new KestrelSpout("kestrel.backtype.com", 22133, "sentence_queue", new StringScheme()));builder.setBolt(2, new SplitSentence(), 10) .shuffleGrouping(1);builder.setBolt(3, new WordCount(), 20) .fieldsGrouping(2, new Fields("word"));
public class SplitSentence implements IBasicBolt { public void execute(Tuple tuple, BasicOutputCollector collector) { String sentence = tuple.getString(0); for(String word: sentence.split(" ")) { collector.emit(new Values(word)); } }
public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } }
public class WordCount implements IBasicBolt { private Map<String, Integer> _counts = new HashMap<String, Integer>();
public void execute(Tuple tuple, BasicOutputCollector collector) { String word = tuple.getString(0); int count; if(_counts.containsKey(word)) { count = _counts.get(word); } else { count = 0; } count++; _counts.put(word, count); collector.emit(new Values(word, count)); }
public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word", "count")); } }