event driven with libuv and zeromq
TRANSCRIPT
Outline
● Algo Trading Business Flow Review● Order Module Design: Current Flaw● Distributed System: Theory Review● Event Driven System: Review● Event Driven Design In Algo Trading● Project Plan
Algo Trading Flow
I1 I2 I3 …... In
任务(下达 /取消 /停止 )
Instrument (Buy/Sell)
下单
撤单
下单
撤单
步长
步长
.
.
.
...
...
….........
...
...
….........
...
...
….........
End
…............
Current Design Flaw
● Bad Assumptions– Serial Execution Among Instructments
– Batch Job ( 1 Sec Interval)
– I/O Cost/ Latency: Neglectable
– Computation Cost/Latency: Neglectable
– No Uncertainty
Reality
● Parallel execution of Instruments● Market change in Ms, 1 Sec (or More) fixed reponse interval
is 1000X● I/O is SLOW comaring to CPU speed (10^9 order)● Communication Channel will Break● Only God have ALL TIMELY information for decision making.
We people live in UNCERTAINTY● Tradeoff: Best Effort Decision versus Perfect Decision
Distributed System: Our Reality
● Parallel: Everything is Moving● Error Prone (Fault): No 99.9999999....%
Gurantee● Uncertainty: Who/What/Where/When/Why● Concensus/Consistency with HIGH COST● Speed limit: Light Travels at (300,000km/Sec)
CAP: No Perfect World
Case:Two Generals' Problem
Case:Two Generals' Problem
● 1975 . E. A. Akkoyunlu, K. Ekanadham, and R. V. Huber "Some Constraints and Trade-offs in the Design of Network Communications"
● “A pragmatic approach to dealing with the Two Generals' Problem is to use schemes that accept the uncertainty of the communications channel and not attempt to eliminate it, but rather mitigate it to an acceptable degree.”
TCP: Connection Establishment
TCP: Sliding Window
Event Driven: History
● 1960s: Mainframe. Batch Job Oriented● 1970s: Unix. Interactive,Timesharing Terminal● GUI programming:
– Xerox : mouse
– Apple: Apple II
– Microsoft: Windows
– MIT: X windows
– …...
Event Driven: Now
● Libuv: Chrome browser, node.js...● Libev/libevent● Boost ASIO● ACE/ICE● Qt event loop● Glib/Gevent
Event Driven: Now
● Nginx,apache,Twisted,Comet● Node.js● Python Asyncio● Go channel● Scala Actor● C++ 11 Promise/Future,Coroutine,Closure● …........
I/O Modelling
● blocking I/O● nonblocking I/O● I/O multiplexing (select and poll)● signal driven I/O (SIGIO)● asynchronous I/O (the POSIX aio_functions)
Quoted from “UNIX Network Programming Volume 1”
Async/Event Driven Design: Reactor
Proactor
So we are event-driven ready. But...
● Components are mostly event driven● Interface Among components are BLOCKING
INTERFACE! (block/wait/timeout....)● One blocking will ruin the whole infrastructure!● “Public Land Tragic”
class StringQue : public MsgQueBase
StringQueue is everywhere
[luke@rmbp service_multichannel]$ find . -name StringQue.h./orderservice/OMSInfterfaceDemo/StringQue.h./optionhedgeservice/optionhedgeservice/StringQue.h./queryservice/queryservice/StringQue.h./queryservice/queryservice_auto_qt_profitchart/StringQue.h./queryservice/queryservice_auto_qt/StringQue.h./strategyservice_new/hedgefutureservice/StringQue.h./strategyservice_new/algoorderservice/StringQue.h./strategyservice_new/algoorderserviceFuture/StringQue.h./strategyservice/hedgefutureservice/StringQue.h./strategyservice/optionhedgeservice/optionhedgeservice/StringQue.h./strategyservice/algoorderservice/StringQue.h./strategyservice/algoorderserviceFuture/StringQue.h./quoteservice/quoteservicelevel2/StringQue.h./quoteservice/quoteservicelevel2datafeed/StringQue.h./quoteservice/quoteservicelevel2_new/StringQue.h./quoteservice/quoteservicelevel2datafeed_clb/StringQue.h
How we use StringQueue?
Vision
● A fault tolerant, responsive trading platorm– Respond to every marktet tick/heartbeat
● Maintain “fresh” market snapshot● React as every tick comes
– Async, Parallel order fullfillment ● Handle > 1000 orders one server instance without
pressure● Milisec level processing delay per order action
Vision: Unified Event Loop
Libuvevent Loop
ZeroMqsocket
Normal socket
timer
Other Async events
eventfd
fd
timer_t handle
async_t handle
Parallel Tasks req_t
Thread PoolAsync Tasks
Order Fullfill
Market Snapshotupdate
Other Event Loop
Other businessmodules
Async Programming
● Callback● Task(threading)● Future/Promise● Async I/O● Coroutine●
Trading Version 2
I1 I2 I3 …... In
下单
撤单
下单
撤单
timeout
timeout.
.
.
...
...
...
...
….........
...
...
….........
End
…............
tick
….........
EndEndEnd
tick
tick
tick
Project Plan
● Goal: A testable new framwork – Algotrading as typical business test case
– High throughput, low latency
– Fault Tolerant with sensible tradeoff strategy
Side by Side comparison with current order system
Task:
● Project Managment (Wei Song)● Event Loop, ZeroMQ (Tie Gang/Luke)● Market snapshot update (Wei Song/Ze Yu)● Order business module rewrite(Ze Yu/Wei Song)● Testing environment (Wei Song/Tie Gang)
– Simulation data/replay
– OMS simulator
● Testing– All
– + Calvin + other business staff
Schedule?
● What can we accomplish before Spring Festival?