3-way scripts as a base unit for flexible scale-out code

2015/12/15

Marat Zhanikeev [email protected] SE研@博多シティ

PDF: bit.do/151215

3-Way Scripts as a Base Unit

for Flexible Scale-Out Code

Things You Might've Noticed

• most coding today is done in scripting languages -- php, python, ruby, ....

• DiY deploy is becoming more common than platforms -- Docker deploy via filesystem diffs

• smartphone-centric coding is already a reality and reaches further intoIoT, IoV, etc.

• .... a demo?

M.Zhanikeev -- [email protected] -- 3-Way Scripts as a Base Unit for Flexible Scale-Out Code -- bit.do/151215 2/202/20

3-Way Scripts

function callme (…) …

Class NAME

Other code, Manager CLI mode

$a = new NAME () $a ->callme (); HTTP mode $a = new NAME () echo toJSON ( $a ->callme ());

Operator Remote code

Use in Object mode

01 M.Zhanikeev "Github Public Repository for the 3-Way Scripting Project" https://github.com/maratishe/3wayscripting (current)


The Need (also for Speed)• 2 ways wanted 06 詳細 : high-thru local vs low-thru remote calls to A

A

B

C Many Many

HTTP REQ HTTP REP (small data)

new, call

CLI

High volume

Big Data

Manager

06 M.Zhanikeev "Streaming Algorithms for Big Data Processing on Multicore" CRC (2015)


Objectives1. heterogeneity

◦ in clouds known as federated and fog clouds 03◦ in coding, ambient 18 and p2p networking between instances/agents 15◦ ultimately VNE 詳細 for distributed software agents

2. volatility◦ again, fog clouds 03 ambient programming 18◦ hierarchical structures -- like Fractal 14

3. autonomy◦ not much in programming but more actively discussed in clouds, networkmanagement, etc. 09

◦ distant goal is Truly Autonomous 詳細

03 M.Zhanikeev "A Cloud Visitation Platform to Facilitate Cloud Federation and Fog Computing" IEEE Computer (2015)

18 J.Dedecker+4 "Ambient-Oriented Programming in AmbientTalk" 20th ECOOP (2006)

15 M.Albano+2 "Hierarchical P2P Overlays for DVE: An Additively Weighted Voronoi Based Approach" ICUMT (2009)

14 F.Baude+2 "From Distributed Objects to Hierarchical Grid Components" CoopIS, DOA, and ODBASE, Springer LNCS (2003)

09 M.Zhanikeev "Population Management in Clouds is a Do-It-Yourself Technology" IETF94/NMRG (2015)


Theory of Distributed Computing

A

Platform Platform

B c C

Run

A. CORBA model

A B

B. Current (traditional) Model

Platform

C. Distributed Code (proposed)

A

Platform Platform

B

C Run

3-way code Distribution borders (network, etc.)

Code (objects, scripts) Data, messages Runtime Invocation routes

XYZ

• CORBA used tostringify andsend objects

• modern platformsmostly send data

• 3-way code isunique◦ platformless◦ autonomous◦ volatility OK =adaptible

◦ ...


Class/Function Call-by-Variable

• most scripting languages (PHP) allow for some nifty referencing techniques

• key to using the same code in 3+ ways

$a = new NAME();$b = ‘callme’;$a->$b( $param, $defaultparam = 10);


Basic Security for Web API

A B

Many Many

REQ( md5p , …)

Make secure

Operator, Manager

mp5p

mp5 (file) Attacker

Manager


More Specific on MD5 Keys

B A

Each Each

REQ( key , …)

keyfile Operator/ automation (once) c

web.php make()

verify

same dir.

Attacker

Manager (key exchange)


web.php key management

• see source code at 01

• key made by make() is mapped to tag, then to place• load() is internal, used for de-mapping from place/tag

make( $libdir,$stuff,$length=10){ // returns $keytag( $key,$tag,$cldir=‘.’){ // outgoingplace( $name,$iport,$tag,$cldir=‘.’){ // outgoingprivate load( $place,$tag,$cldir=‘.’){ // outgoing

01 M.Zhanikeev "Github Public Repository for the 3-Way Scripting Project" https://github.com/maratishe/3wayscripting (current)


web.php mid-way interface

• remote calls can be reduced to remote procedure and RESTful calls• rrun() : communicate to remote web.php and have it run something on localcommandline

• rcall() : send a RESTful REQ directly to place and return results as JSON

server( $port) { // use own IPrrun( $place,...) { // $place maps to $tag, then $keyrcall( $place,...) {


web.php mid-way interface (2)

A

Each Each

REQ( key , …)

C keyfile

Operator/ automation

b

make()

Attacker

D keyfile

E keyfile

keyfile (global)


Wrapup : Feature Comparison

• note the learning curve -- very important for modern programming and edgeboxes 詳細

• traffic efficiency 詳細 is not mentioned, but is part of current/future work 02

• destined for wireless fog boxes which need platformless agents 詳細

(Web)Sockets

RESTfulcalls

Hierarchicalstructure

Heterogeneous environments

Learningcurve

(simplicity)Needs a platform?

Traffic encoding optional?

Traditional(Corba, ESB, MPI,…) YES NO NO NO NO YES NO

Advanced/modern(SOC, Ibis, …) YES YES YES YES NO YES NO

Distributed code (proposed) YES YES YES YES YES NO YES

02 M.Zhanikeev "DiffHub: An Efficient Cloud Sync Technology based on Binary Diffs" IPSJ JIP (2015)


That’s all, thank you ...


Application: Hadoop InterfaceHadoop Agent

... is software that runs locally as an agent of theremote client

Storage Node (shard)

Time-Aware Sub-Store(s)

Manager

Client Machine

Client

Your Sketcher

You

Start Use

Schedule

Multicore Replay

Replay Node

many

• Hadoop interface is veryintensive, cannot be doneremotely

• there are also alternativesto Hadoop 06

• one of applications for 3-wayscripts 戻る

06 M.Zhanikeev "Streaming Algorithms for Big Data Processing on Multicore" CRC (2015)


VNE: Virtual Network EmbeddingVNE

... is the best way to formulate optimization ofmultiple overlayed virtual (service) networks

Physical layer

Virtual Layer 1

VN Request

2 Make VN

3 VN Reply

4 Virtual

Virtual

Physical

• still active area of research,good heuristics wanted 07

• best for service networks inclouds 戻る

07 M.Zhanikeev "A New VNE Method for More Responsive Networking in Many-to-Many Groups" 7th ICUFN (2015)


Truly Autonomous AgentsTrue Autonomy

... is alternative to VNE in that optimization isdone autonomously at service side

Network Provider’s (NOC)

Agent

Service Traditional agents True Autonomy

• part of current standardsdiscussion on futurenetwork management 09

• traditional: NOC + agentsbut NOC is interface =bottleneck

• true autonomy: agents areopen for directcoordination with removeservices 戻る

09

09 M.Zhanikeev "Population Management in Clouds is a Do-It-Yourself Technology" IETF94/NMRG (2015)


Wireless Network EdgeReachable Devices

... are boxes that talk to your smartphone usingP2P WiFi (WiFi Direct is best today)

AP

User

Internet

User

Internet WiFiCongestion

User

Internet 3G

Smartphone

Device

Device

Device

AP Design

WLAN Design

Reachable Devices Design

AP

Device WiFi User Internet AP Design (2)

Congestion

WiFiP2P WiFi

Cannot use at the same time

Reachable Platform

• modern boxes are mostly based onthe AP Design -- connectivity problem

• even if conenctivity problem isresolved, you have congestionproblem

• with Reachable Technology,smartphone is the realtime mid-wayinterface between clouds anddevices

• link to code: software agent isrunning at (1) smartphone and (2)the box 戻る


Example Edge Box for 3-Way Scripts• currently trying to build this box based on conventional Xen• LHAP is Local Hardware Awareness Platform 03

• software agents are running at the box, those are Beacons, Wireless DataHubs, etc. 戻る

WiFi

Wireless users

Wireless AP

Physical Device

LHAP

VM VM Con. Con. Con. Storage

Sensors

…

Beacon

WiFi AP WiFi Client

P2P WiFi

03 M.Zhanikeev "A Cloud Visitation Platform to Facilitate Cloud Federation and Fog Computing" IEEE Computer (2015)


Binary Diffs and DiffHub IdeaBinary Diffs

... are the best way to achieve trafficefficiency in a datatype-indiferent way

Cloud Drive

User A

User B

New

Old

Old

Diff Diff

Old version New version Binary diff

DiffHub

• exampe: Docker uses filesystemdiffs

• recent study: binary diffs are betterthan filesystem diffs

• connection: efficient deploy/maintain of scale-out code 戻る
