연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - pycon kr 2015
TRANSCRIPT
/ (+ )
lablup.com
,,.
/ (+ )
lablup.com
,,.
▪
▪▪ /
▪▪
/
▪ TNF/ Needlworks
▪ …
▪
▪▪
▪ NBA ▪
80 Gbps
▪
▪ …
http://www.lablup.com .
▪ 21 , 20 , 19
▪
▪ ,
21?
▪
.
▪
.
...
▪
▪
▪ / / /
▪
▪ /
▪ Three keyboardists but…
▪
▪
▪
▪ resource consolidation
▪
▪
!
Python!…
…
Python!▪
▪ :
▪ PEP-8
▪ :
▪
▪
Python T_T▪
▪
▪ PyPy / Pyston
▪ :
▪ C
▪ Cython
▪
Python /▪
▪ CS101
▪ “ ”
▪
▪ indent…
▪
▪ MATLAB numpy
: ▪
▪ MATLAB : matplotlib, ipython notebook
▪ Ultra-complex figures
▪
▪
▪
▪ /
: ▪
▪ NBA
▪
▪ configuration Python
▪ I/Ostdout asyncio
It's time to go Python 3▪ Python 3
▪ Django, numpy/scipy, …
▪ Python 3.3
▪ PEP-393: strunicode codepoint
▪ narrow build UTF-16, wide build UTF-32
▪ UTF-8/16/32
▪ ASCII/Latin-1
▪ / ·
It's time to go Python 3▪ Python 3.4
▪ PEP-3156: asyncio coroutine
▪ I/O
▪ GIL (global interpreter lock)
▪
▪ pip virtualenv
▪
▪ RabbitMQ, Kafka, Celery, ...
▪
▪ (broker) ,
▪ redis …
– ZeroMQ▪
▪ Request-Reply
▪ 1( : HTTP-like server/client)
▪ Push-Pull
▪ N-to-1 ( : logging system, task partitioning)
▪ Publish-Subscribe
▪ 1-to-N / N-to-M ( : broadcast)
– ZeroMQ▪ :
▪ zero broker, zero latency, zero administration, ...
▪ : connection
▪ TCP / UDP / UNIX domain / local "in-process"
▪ zeromq transport header (ZMTP)
▪ payload (64+ )
▪ : persistent queue
▪ TCP reliability
– LogStash
▪ LogStash
▪ input / filter / output
▪ ZeroMQ, S3
▪ ( !)
Django
User Kernel LogstashServer
DatabaseZeroMQ (push-pull)
+ JSONAWS S3
msgpack
On-premise vs. Hosting vs. Cloud▪
▪ " " ...
▪ , ,
▪ AWS
▪
▪
▪ …
▪ 2012 R2 / / .
Docker containers▪ docker container
▪
▪ VM deploy/destroy
▪ AWS EC2 Instance: ( )
▪ pooling
▪ Amazon ECS?
▪ container (run task) latency
▪ …
Docker containers▪
▪ : instance
▪ Microsoft docker
▪ Azure Windows
Polymer library▪ HTML5
▪ HTML imports / Custom elements / Shadow DOM
▪ Web components
▪ “Polyfill”
▪
▪ DOMelements
Polymer library▪ ...
▪ !
▪ Bootstrap + theme?
▪
▪ HTML5 !
HTML imports!
Reusable web components!
Shadow DOM!
Polyfill!
Polymer
▪
▪ 7 ? ?
Electron: Cross-platform Polymer▪ : cross-platform webapp container
▪ Electron
▪ Github Node.js
▪ Node Chromium wrapping
▪ / /
▪ ATOM
▪ ?
▪ Apache Cordova
▪
▪
▪
▪ Django UnitTest + Selenium
▪ Selenium webdriver:
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
▪
▪
▪ sphinx
▪ reST
▪
▪
▪ !
▪
..
, ,
+
,
▪ + .
▪ :
▪ –
▪ , , scaling, ...➞ ++
▪ :
▪ –
▪
▪ .
▪ !
▪
▪
CodeON!//codeonweb.com
!
…
, …
“ .”
▪ React / Flux / AngularJS / Polymer / ...
▪ +UI
▪ Polymer 0.9 API ....
▪ (?) Bootstrap …
▪ I/O 2015 Polymer 1.0...
Polymer: it is too google to be true▪
▪ 0.5 - 0.8rc2 - 0.9 - …▪
▪ ( )
▪ Google I/O - 1.0 :
Polymer: vulcanize▪ Vulcanize
▪ :
▪ IE – +
▪ CSS – +
▪ Crisper javascript map shadow DOMroot - +
Polymer: vulcanize▪ Vulcanize
▪ :
▪ IE –
▪ CSS –
▪ Crisper javascript map shadow DOMroot -
Polymer: polyfill
▪
▪
▪(html import
shadow DOM)polyfill
▪
▪
Polymer + Django▪
▪ {{ … }}
▪ Polymer –
▪ Django –
▪
▪
▪ …
Polymer + Django + Security▪ CORS (cross-origin resource scripting)
▪ Django – CORS header ( )
▪ Polymer – vulcanize Javascript CORS
▪ Crisper CORS Javascript
▪ -
▪ XSS
▪ Django - CSRF token
▪ Polymer – header iron-ajax
▪
Polymer▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Polymer▪ Production ready?!
▪
▪ “Do not swim in sandocean. Swim in the ocean.”
▪ HTML5
▪
▪ X-tag, react, polymer…
▪ ?
Electron + web▪ ... !
▪
▪(Noto )
▪
▪ CSS: .
▪?
: Django + RDS▪ UTF-8 Python ?
▪ (cf., str / bytes / io / codecs)
▪ MySQL SQLite
▪ MySQL
▪ ForeignKey Django
▪ MySQL + Django + Korean = EPIC FAIL
▪ ( ) Postgres ...
▪ postgres ( / ) .
Python 3.4 – asyncio / coroutine
▪ .
▪ yieldfrom ...
▪ AST staticanalysis coroutineyieldfrom
▪ , 100% static analysis
▪ duck typing + dynamic method generation+ method proxy pattern ...
▪ Python 3.5 await/async!
Python 3.4 – asyncio loop
▪ loop.close()..
import asyncioimport asyncio_redisloop = asyncio.get_event_loop()conn = loop.run_until_complete(
asyncio_redis.Connection.create('localhost', 6379))conn.close()loop.close()
Task was destroyed but it is pending!task: <Task pending coro=<_reader_coroutine() running at /.../lib/python3.4/site-packages/asyncio_redis/protocol.py:919> wait_for=<Future pending cb=[Task._wakeup()]>>
Python 3.4 – asyncio loop
▪ asyncio graceful shutdown , 1. loop._run_once()
2. loop.run_until_complete(asyncio.sleep(0))
3. loop.run_until_complete(server.wait_closed())
▪ , wait_closed()coroutine
.
▪ event loop .
Python 3.4 – asyncio loop #! /usr/bin/env python3import asyncioloop = asyncio.get_event_loop()@asyncio.coroutinedef my_timer():
i = 0while True:
yield from asyncio.sleep(1)print(i)i += 1
try:asyncio.async(my_timer(), loop=loop)loop.run_forever()
except KeyboardInterrupt:pass
finally:loop.close()
^CTask was destroyed but it is pending!task: <Task pending coro=<my_timer() done, defined at test.py:4> wait_for=<Future pending cb=[Task._wakeup()]>>
Python 3.4 – asyncio loop #! /usr/bin/env python3import asyncioloop = asyncio.get_event_loop()@asyncio.coroutinedef my_timer():
i = 0while True:
yield from asyncio.sleep(1)print(i)i += 1
try:asyncio.async(my_timer(), loop=loop)loop.run_forever()
except KeyboardInterrupt:for t in asyncio.Task.all_tasks():
t.cancel()try:
loop._run_once()except asyncio.CancelledError:
passfinally:
loop.close()
asyncio_redis▪ Redis asyncio
▪ Heisenbug ! 🚨▪ SCAN key ...▪ while if ➞
▪ https://github.com/jonathanslenders/asyncio-redis/issues/65
▪ Connection pool API ▪ API call connection
stateful API ( : SELECT)▪ (aioredis)
: – ZeroMQ
▪ Asynchronous ? block ??
▪ socket connect . ( ...)
▪ send/recv timeout
▪ aiozmq , blocking callasyncio.wait_for timeout
.
▪ pyzmq , socket.poll(msec)timeout .
: – LogStash
▪ zmq input plugin s3 output pluginzmq recv_string
▪ 1: s3 output plugin “write”, PutObject
DeleteObject .
▪ 2: config , output plugin log level
· .
▪ 3: output plugininput plugin receive timeout .
: Docker▪
▪ docker registry "latest"latest
▪ 2.0 2.0.1 tag
▪
▪
▪ docker 1.8 , CoreOS (rkt)
▪ /API
:
▪ Jupyter / ipython▪ scalable
▪ Jupyter
▪ Unix
▪
▪ :
▪
▪ – Sorna
▪ .
CodeON
▪ Container Resource Consolidation
▪ Kubernetes
▪ paxos raft agreement key-value
store .
▪ Ingen REPL (read-evaluate-print loop)
▪ stdout/stderrstreaming
▪ interactiveplot
▪ image,sound multimedia
▪
▪
▪ User profiler + backtracer + ML with back propagation
▪
▪ d3.js fluid nonlinear navigation UI
▪
▪
▪
▪
▪
▪.
▪ ... + +…
▪ .
▪ ?
Lablup Inc.
http://www.lablup.com