연구자 및 교육자를 위한 계산 및 분석 플랫폼 설계 - pycon kr 2015

/ (+ )

lablup.com

,,.

/ (+ )

lablup.com

,,.

▪

▪▪ /

▪▪

/

▪ TNF/ Needlworks

▪ …

▪

▪▪

▪ NBA ▪

80 Gbps

▪

▪ …

http://www.lablup.com .

▪ 21 , 20 , 19

▪

▪ ,

21?

▪

.

▪

.

...

▪

▪

▪ / / /

▪

▪ /

▪ Three keyboardists but…

▪

▪

▪

▪ resource consolidation

▪

▪

!

Python!…

…

Python!▪

▪ :

▪ PEP-8

▪ :

▪

▪

Python T_T▪

▪

▪ PyPy / Pyston

▪ :

▪ C

▪ Cython

▪

Python /▪

▪ CS101

▪ “ ”

▪

▪ indent…

▪

▪ MATLAB numpy

: ▪

▪ MATLAB : matplotlib, ipython notebook

▪ Ultra-complex figures

▪

▪

▪

▪ /

: ▪

▪ NBA

▪

▪ configuration Python

▪ I/Ostdout asyncio

It's time to go Python 3▪ Python 3

▪ Django, numpy/scipy, …

▪ Python 3.3

▪ PEP-393: strunicode codepoint

▪ narrow build UTF-16, wide build UTF-32

▪ UTF-8/16/32

▪ ASCII/Latin-1

▪ / ·

It's time to go Python 3▪ Python 3.4

▪ PEP-3156: asyncio coroutine

▪ I/O

▪ GIL (global interpreter lock)

▪

▪ pip virtualenv

▪

▪ RabbitMQ, Kafka, Celery, ...

▪

▪ (broker) ,

▪ redis …

– ZeroMQ▪

▪ Request-Reply

▪ 1( : HTTP-like server/client)

▪ Push-Pull

▪ N-to-1 ( : logging system, task partitioning)

▪ Publish-Subscribe

▪ 1-to-N / N-to-M ( : broadcast)

– ZeroMQ▪ :

▪ zero broker, zero latency, zero administration, ...

▪ : connection

▪ TCP / UDP / UNIX domain / local "in-process"

▪ zeromq transport header (ZMTP)

▪ payload (64+ )

▪ : persistent queue

▪ TCP reliability

– LogStash

▪ LogStash

▪ input / filter / output

▪ ZeroMQ, S3

▪ ( !)

Django

User Kernel LogstashServer

DatabaseZeroMQ (push-pull)

+ JSONAWS S3

msgpack

On-premise vs. Hosting vs. Cloud▪

▪ " " ...

▪ , ,

▪ AWS

▪

▪

▪ …

▪ 2012 R2 / / .

Docker containers▪ docker container

▪

▪ VM deploy/destroy

▪ AWS EC2 Instance: ( )

▪ pooling

▪ Amazon ECS?

▪ container (run task) latency

▪ …

Docker containers▪

▪ : instance

▪ Microsoft docker

▪ Azure Windows

Polymer library▪ HTML5

▪ HTML imports / Custom elements / Shadow DOM

▪ Web components

▪ “Polyfill”

▪

▪ DOMelements

Polymer library▪ ...

▪ !

▪ Bootstrap + theme?

▪

▪ HTML5 !

HTML imports!

Reusable web components!

Shadow DOM!

Polyfill!

Polymer

▪

▪ 7 ? ?

Electron: Cross-platform Polymer▪ : cross-platform webapp container

▪ Electron

▪ Github Node.js

▪ Node Chromium wrapping

▪ / /

▪ ATOM

▪ ?

▪ Apache Cordova

▪

▪

▪

▪ Django UnitTest + Selenium

▪ Selenium webdriver:

▪

▪

▪

▪ sphinx

▪ reST

▪

▪

▪ !

▪

▪

▪

▪ sphinx

▪ reST

▪

▪

▪ !

▪

..

, ,

+

,

▪ + .

▪ :

▪ –

▪ , , scaling, ...➞ ++

▪ :

▪ –

▪

▪ .

▪ !

▪

▪

CodeON!//codeonweb.com

!

…

, …

“ .”

▪ React / Flux / AngularJS / Polymer / ...

▪ +UI

▪ Polymer 0.9 API ....

▪ (?) Bootstrap …

▪ I/O 2015 Polymer 1.0...

Polymer: it is too google to be true▪

▪ 0.5 - 0.8rc2 - 0.9 - …▪

▪ ( )

▪ Google I/O - 1.0 :

Polymer: vulcanize▪ Vulcanize

▪ :

▪ IE – +

▪ CSS – +

▪ Crisper javascript map shadow DOMroot - +

Polymer: vulcanize▪ Vulcanize

▪ :

▪ IE –

▪ CSS –

▪ Crisper javascript map shadow DOMroot -

Polymer: polyfill

▪

▪

▪(html import

shadow DOM)polyfill

▪

▪

Polymer + Django▪

▪ {{ … }}

▪ Polymer –

▪ Django –

▪

▪

▪ …

Polymer + Django + Security▪ CORS (cross-origin resource scripting)

▪ Django – CORS header ( )

▪ Polymer – vulcanize Javascript CORS

▪ Crisper CORS Javascript

▪ -

▪ XSS

▪ Django - CSRF token

▪ Polymer – header iron-ajax

▪

Polymer▪ Production ready?!

▪

▪ “Do not swim in sandocean. Swim in the ocean.”

▪ HTML5

▪

▪ X-tag, react, polymer…

▪ ?

Polymer▪ Production ready?!

▪

▪ “Do not swim in sandocean. Swim in the ocean.”

▪ HTML5

▪

▪ X-tag, react, polymer…

▪ ?

Electron + web▪ ... !

▪

▪(Noto )

▪

▪ CSS: .

▪?

: Django + RDS▪ UTF-8 Python ?

▪ (cf., str / bytes / io / codecs)

▪ MySQL SQLite

▪ MySQL

▪ ForeignKey Django

▪ MySQL + Django + Korean = EPIC FAIL

▪ ( ) Postgres ...

▪ postgres ( / ) .

Python 3.4 – asyncio / coroutine

▪ .

▪ yieldfrom ...

▪ AST staticanalysis coroutineyieldfrom

▪ , 100% static analysis

▪ duck typing + dynamic method generation+ method proxy pattern ...

▪ Python 3.5 await/async!

Python 3.4 – asyncio loop

▪ loop.close()..

import asyncioimport asyncio_redisloop = asyncio.get_event_loop()conn = loop.run_until_complete(

asyncio_redis.Connection.create('localhost', 6379))conn.close()loop.close()

Task was destroyed but it is pending!task: <Task pending coro=<_reader_coroutine() running at /.../lib/python3.4/site-packages/asyncio_redis/protocol.py:919> wait_for=<Future pending cb=[Task._wakeup()]>>

Python 3.4 – asyncio loop

▪ asyncio graceful shutdown , 1. loop._run_once()

2. loop.run_until_complete(asyncio.sleep(0))

3. loop.run_until_complete(server.wait_closed())

▪ , wait_closed()coroutine

.

▪ event loop .

Python 3.4 – asyncio loop #! /usr/bin/env python3import asyncioloop = asyncio.get_event_loop()@asyncio.coroutinedef my_timer():

i = 0while True:

yield from asyncio.sleep(1)print(i)i += 1

try:asyncio.async(my_timer(), loop=loop)loop.run_forever()

except KeyboardInterrupt:pass

finally:loop.close()

^CTask was destroyed but it is pending!task: <Task pending coro=<my_timer() done, defined at test.py:4> wait_for=<Future pending cb=[Task._wakeup()]>>

Python 3.4 – asyncio loop #! /usr/bin/env python3import asyncioloop = asyncio.get_event_loop()@asyncio.coroutinedef my_timer():

i = 0while True:

yield from asyncio.sleep(1)print(i)i += 1

try:asyncio.async(my_timer(), loop=loop)loop.run_forever()

except KeyboardInterrupt:for t in asyncio.Task.all_tasks():

t.cancel()try:

loop._run_once()except asyncio.CancelledError:

passfinally:

loop.close()

asyncio_redis▪ Redis asyncio

▪ Heisenbug ! 🚨▪ SCAN key ...▪ while if ➞

▪ https://github.com/jonathanslenders/asyncio-redis/issues/65

▪ Connection pool API ▪ API call connection

stateful API ( : SELECT)▪ (aioredis)

: – ZeroMQ

▪ Asynchronous ? block ??

▪ socket connect . ( ...)

▪ send/recv timeout

▪ aiozmq , blocking callasyncio.wait_for timeout

.

▪ pyzmq , socket.poll(msec)timeout .

: – LogStash

▪ zmq input plugin s3 output pluginzmq recv_string

▪ 1: s3 output plugin “write”, PutObject

DeleteObject .

▪ 2: config , output plugin log level

· .

▪ 3: output plugininput plugin receive timeout .

: Docker▪

▪ docker registry "latest"latest

▪ 2.0 2.0.1 tag

▪

▪

▪ docker 1.8 , CoreOS (rkt)

▪ /API

:

▪ Jupyter / ipython▪ scalable

▪ Jupyter

▪ Unix

▪

▪ :

▪

▪ – Sorna

▪ .

CodeON

▪ Container Resource Consolidation

▪ Kubernetes

▪ paxos raft agreement key-value

store .

▪ Ingen REPL (read-evaluate-print loop)

▪ stdout/stderrstreaming

▪ interactiveplot

▪ image,sound multimedia

▪

▪

▪ User profiler + backtracer + ML with back propagation

▪

▪ d3.js fluid nonlinear navigation UI

▪

▪

▪

▪

▪

▪.

▪ ... + +…

▪ .

▪ ?

Lablup Inc.

http://www.lablup.com