"erlang, webmail and hibari" at rakuten tech talk

People, Software, WebMail, BigDataPowered by

Erlang & Functional

Programming

Gemini Mobile Technologies, Inc.

August 24, 2010

GMT Erlang August 2010 - Rakuten Tech Talk 1/55

Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?

• WebMail Case Study

• Hibari Case Study

• What’s Next?


IntroductionWho is Gemini Mobile Technologies?

• Founded: July, 2001• Offices: San Mateo, CA; Shibuya, Tokyo; Star City, Beijing• Milestones:

• 2003: Multimedia messaging service (MMS), Vodafone Japan• 2005: MMSC, Nextel International• 2006: MMSC, eMobile Japan• 2006: S!Town, Softbank Mobile• 2007: ExCast enterprise mail gateway, NTT docomo• 2008: eXplo(tm) service, China Unicom• 2008: Fax satellite gateway, NTT docomo• 2009: International MMS gateway, NTT docomo• 2010: WebMail, Japanese Mobile Carrier & Internet Provider• 2010: Hibari/BigData, Open Source Community

• Investors: Goldman Sachs, Ignite, Mizuho Capital, TokyoMUFJ, Nomura, Access, Aplix

• Erlang: Deployed in Japan, China, & Europeantelecoms use for 3 years


Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?



• What’s Next?


Erlang/OTPErlang is . . .

• General purpose programming language, runtime environment

• Originally written in Prolog, now self-hosting in ownenvironment

• Functional language, with strict evaluation, single assignment,and dynamic typing

• Support for concurrency, multi-core CPU, networkdistribution, and fault tolerance

• Designed for soft-real-time, non-stop applications


Erlang/OTPOpen Telecom Platform is . . .

• Collection of libraries to support Erlang applications

• Standard support libraries: lists, trees, dictionaries, sets, files,queues, network sockets, time manipulation, generic servers &FSMs, basic mathematical funcs, string handling, timedevents,

• Error handling & logging, alarms, hot codeupgrade/downgrade, process tree manipulation & supervision,. . .

• Protocol stacks for ASN.1, CORBA, HTTP, SSL,Megaco/H.248, . . .

• Mnesia distributed database

• Foreign language interfaces for C/C++, Java, TCP-basedservers


Why Erlang?

• Software for Ericsson’s products (telecomm switches, radiogear) was getting too complex: C, C++, Pascal, EriPascal,assembler, PLEX, . . . over 20 different languages used inproduction and research labs.

• Ad hoc mechanisms for field maintenance, bugfixes, upgrades.

• ”There must be a better way” . . .• Must be high-level to provide productivity gains.• Must support concurrency, error recovery. Soft-realtime

requires no back-tracking, very cheap thread model.• Hot code upgrades very desirable.

• Best-known product: AXD301 ATM switch with now 2 MLoCErlang, plus another 1+ MLoC C and C++ (proprietary h/wdrivers, third-party firmward/drivers and protocol stacks)


Erlang Timelinehttp://www.erlang.org/course/history.html

• 1982-85: Language surveys

• 1985-86: Experiments with LISP, Prolog, Parlog.

• 1988: First Ericsson PBX product to use Erlang (in Prolog)

• 1989: Experimental rewrite of switch code, Plex -¿ Erlang,10x programmer efficiency. First non-Prolog-based interpreter.

• 1990: Conference papers, Erlang spreads to Bellcore & others.

• 1992: Ports to VxWorks, PC, Macintosh. First two ”real”Ericsson products start using Erlang.

• 1993: Network distribution added. Spinoff organization tosupport Erlang development.


Erlang Timelinecontinued . . .

• 1995: Ericsson AXE-N product collapses (non-Erlang). Thereplacement ADX starts with Erlang.

• 1998: Erlang banned for new products: it wasn’t C++ :(

• 1998: Erlang open-sourced, new companies spin off

• Today . . .• Erlang still used in Ericsson (despite ban): productivity is too

high• AXD301 has 11% of world market (market leader), runs

British Telecom’s country-wide ATM network, handles 30-40million calls/week (avg 49-66 calls/sec), has experienced 31milliseconds of downtime per year (9 ”nines” reliability)

• Active and Growing Open Source Community


Erlang Overview

• Concurrency: User-space thread model (extremely cheap tocreate, switch contexts, destroy), now support for multipleCPUs and multi-core CPUs. Such threads are really”processes”.

• Distribution: All inter-process communication by messagepassing. Multiple Erlang VMs (virtual machines)communicate transparently via TCP. Same syntax used formessage passing for intra- and inter-node communication.

• Robustness: All processes are isolated, no data sharing.Reliable detection of crashed processes, even on remote nodes.

• Hot code upgrade: old and new code can run simultaneouslyduring code upgrade. Support for data structure changes,module dependencies, etc.


Erlang Overviewcontinued . . .

• External interfaces: via Erlang message passing over TCP,”standard” TCP & UDP protocols, UNIX pipes, shared libraryAPI interface.

• Portable: Same VM runs on Linux & UNIX, Windows,Macintosh, VxWorks. Message passing between heterogenoussystems not a problem.

• Many programming errors avoided by: garbage collected datastructures, single-assignment variables, robust exceptionhandling and inter-node communication


Currency Oriented Programming

• Utterly independent processes: imagine they’re on differentmachines!

• Process semantics: No data sharing, copy-everything messagepassing

→ Sharing means: inefficient (distribution is Hard), complicated(mutexes, condition variables, write barriers, etc.)

• No penalty for massive parallelism (e.g. tens of thousands ofprocesses)

• Each process has an unforgeable name

• To send a message, the recipient’s process name is required

• Message passing semantics are unreliable, ”send and pray”

• Reliable monitoring of remote processes: when and why

• No unavoidable penalty for distribution

• Same behavior on any hosted OS


Why use a Concurrency Oriented Programminglanguage?

• The world is parallel. And distributed.

• Things fail.

• The biggest challenge is using the proper degree of parallelismin a COP program . . . but it’s difficult to err when processesare cheap.

• Programs are automatically scalable: if it works on 1 CPU, itworks on many.

• Programs are automatically robust when a process fails, nomatter where the process is located.

See Appendix for additional information.


Erlang in 11 Examples

”One minute per example” text courtesy of Joe Armstrong

• Sequential Erlang: 5 examples

• Concurrent Erlang: 2 examples

• Distribute Erlang: 1 example

• Fault-tolerant Erlang: 2 examples

• Bit syntax: 1 example

See Appendix for additional information.


Sequential: Factorial

-module(math).

-export([fac/1]).

fac(N) when N > 0 -> N * fac(N-1);

fac(0) -> 1.

> math:fac(25).

15511210043330985984000000


Sequential: Binary Tree Search

lookup(Key, {Key, Val, _, _}) ->

{ok, Val};

lookup(Key, {Key1, Val, Left, Right}) when Key < Key1 ->

lookup(Key, Left);

lookup(Key, {Key1, Val, Left, Right}) ->

lookup(Key, Right);

lookup(Key, nil) ->

not_found.


Sequential: Append, Sort, Adder

%% append

append([H | T], L) -> [H | append(T, L)];

append([], L) -> L.

%% sort

sort([Pivot | T]) ->

sort([X || X <- T, X < Pivot]) ++

[Pivot] ++

sort([X || X <- T, X >= Pivot]);

sort([]) -> [].

%% adder

> Adder = fun(N) -> fun(X) -> X + N end end.

#Fun

> G = Adder(10).

#Fun

> G(5).

15


Concurrent: Spawn, Send and Receive

%% spawn

Pid = spawn(fun() -> loop(0) end)

%% send

Pid ! Message,

...

%% receive

receive

Message1 ->

Actions1;

Message2 ->

Actions2;

...

after Time ->

TimeOutActions

end


Distributed Erlang

...

true = net_kernel:connect_node(NodeName),

Pid1 = spawn(NodeName, Fun),

Pid2 = spawn(NodeName, Module, Func, ArgList),

true = is_process_alive(Pid1),

...


Fault Tolerance: catch/throw

...

case (catch foo(A, B)) of

{abnormal_case1, Y} ->

...

{’EXIT’, Opps} ->

...

Val ->

...

end,

...

foo(A, B) ->

...

throw({abnormal_case1, ...})


Fault Tolerance: monitor a process

...

process_flag(trap_exit, true),

Pid = spawn_link(fun() -> ... end),

receive

{’EXIT’, Pid, Why} ->

...

end


Parsing an IP datagram

-define(IP_VERSION, 4).

-define(IP_MIN_HDR_LEN,5).

DgramSize = size(Dgram),

case Dgram of

<<?IP_VERSION:4, HLen:4,

SrvcType:8, TotLen:16, ID:16, Flgs:3,

FragOff:13, TTL:8, Proto:8, HdrChkSum:16,

SrcIP:32, DestIP:32, Body/binary>> when

HLen >= 5, 4*HLen =< DgramSize ->

OptsLen = 4*(HLen - ?IP_MIN_HDR_LEN),

<<Opts:OptsLen/binary,Data/binary>> = Body,

...


Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?



• What’s Next?


Why Erlang/OTP?A Killer App

In 2008, Gemini deployed it’s first commercial Erlang-basedproduct . . . a high-performance “User Profile” storage server aspart of a larger system.

What wasn’t selected?

• LDAP - persistent, fast . . . but no transactions

• RDBMS - persistent, transactions . . . but too slow

Why was Erlang selected?

• Mnesia - persistent, fast, and transactions

• plus many other benefits (programmable, high quality, andopen source!)

and we haven’t looked back since . . . no regrets!


Why Erlang/OTP?What have we learned?

Erlang and functional programming has taught us some goodpractices and lessons:

• lots of processes and messaging passing can be cheap

• shared and mutable data can be (are) evil

• side-effects can be (are) evil

• let it crash! . . . defensive programming is evil

• don’t (over) optimize too soon . . . the bottlenecks aren’talways where you expect

• keep it simple . . . less is more

• don’t be afraid to re-factor . . . when you have the right tools

• distributed systems can still be (are) difficult and complex


Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?



• What’s Next?


WebMail: Multi-Tier Architecture20K Meter View

MOBILE

HTTPSMTP/POP/IMAP

PCISPMTA

LDAP

O&M

FRONT API

DIRECTORY STORE DATA STORE

AUTH API

CLIENT API

BACK API


WebMail: Multi-Tier Architecture10K Meter View

MOBILE

HTTP SMTP/POP/IMAP

PC ISP MTA

LDAP

O&M

M2FE I/F

MNESIAMNESIAHIBARI

AUTH I/F

M2CI I/F

M2FE AUTH I/F

M2FE JOBQ I/F

M2BE I/F


WebMail: ErlangWhat’s It Doing?

• All core processing for the “webmail” application

• JSON-RPC with the Web browser-based UI (based on UBF)

• HTTP and LDAP with authentication and proxy to full-textindexing services

• UBF for most inter-application communication

• Interface with C++ components for speed, legacy protocolsupport, and code re-use

• Application/Transaction logging and message tracing

• Hibari distributed, scalable key-value store for all persistentdata

• Mnesia for job queuing and multi-indexed profile data


WebMail: HibariKey-Value Storage for (Almost) Everything

• Profile Store• User• Mail• Mail Incoming & Outgoing Filters• User Interface• External ISP

• Address Book Store• vCards - Singletons & Packs• Labels - Folders, Flags, and User-Defined

• Mail Store• Messages - Singletons & Packs• Message Summaries - Singletons & Packs• Meta Data - Next Uid, Quotas, . . .• Labels - Folders, Flags, and User-Defined

• Quota Policy Store


WebMail: MnesiaStorage for Everything Else

• Subset of Profile Store• Indexing & retrieval by various attributes• The WebMail application keeps Mnesia and Hibari

synchronized for provisioning, updates, and deprovisioning• The WebMail application uses Hibari as the master copy

• Job Queue• Outgoing mail, bounce messages, vacation messages, . . .• Notifications to external text indexer• Asynchronous mail deletion• Asynchronous user deprovisioning• . . .

Possible with Hibari-based storage, but Mnesia was easier (at theproject start).


WebMail: Post (almost) MortemStuff We’ll Repeat

• Erlang, the secret sauce

→ Ericsson’s support of Erlang/OTP is wonderful

• UBF, QuickCheck, & UBF+QuickCheck

→ Auto-compilation of QuickCheck generators from UBFcontracts

• Test in various environments:

→ Exactly the same hardware as customer, on really old & slowhardware, and on a single box/laptop

• Automate everything possible: regression tests, performancetests, cluster setups, post-mortem log file gathering, . . .

• Document everything possible (with good tools): Git,AsciiDoc, Graphviz, “mscgen”


WebMail: Post (almost) MortemStuff We Would Probably Do Differently

• Negotiate “less aggressive” schedule

• More hardware

• Always double check “X & Y” before customer triesdoing “X & Y”

• Always revisit and cleanup “initial” prototypes

• Better and “practical” code review by peers

• Better traffic models (for finding bottlenecks, garbagecollection issues, . . . )

• 100% automated unit test and code coverage analysis


WebMail: Summary

• Technically, Erlang was a great fit for this large system.

→ Used another language (C++) whenever convenient.

• UBF is a very good tool for design, implementation, andtesting phases of a large project.

• Combining UBF and QuickCheck was invaluable in findingbugs that otherwise would’ve been discovered later.

• It’s feasible to develop real-time apps on top of a distributedkey-value database.

→ Hibari’s “strong consistency” support is a large advantage.


Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?



• What’s Next?


HibariWhat is Hibari?

• Hibari is a production-ready, distributed, key-value, big datastore.

→ China Mobile and China Unicom - SNS→ Japanese internet provider - GB mailbox webmail→ Japanese mobile carrier - GB mailbox webmail

• Hibari uses chain replication for strong consistency,high-availability, and durability.

• Hibari has excellent performance especially for read and largevalue operations.

• Hibari is open-source software under the Apache 2.0 license.


HibariEnvironments

• Hibari runs on commodity, heterogeneous servers.

• Hibari supports Red Hat, CentOS, and Fedora Linuxdistributions.

→ Debian, Ubuntu, Gentoo, Mac OS X, and Free BSD arecoming soon.

• Hibari supports Erlang/OTP R13B04.

→ R14A is coming soon.

• Hibari supports Amazon S3, JSON-RPC-RFC4627,UBF/EBF/JSF and native Erlang client APIs.

→ Thrift is coming soon.


HibariWhy Another NonSQL?

Durable updates Every update is written and flushed to stablestorage (fsync() system call) before sendingacknowledgments to the client.

Consistent updates After an update is acknowledged, no clientcan see an older version.

High Availability Each key can be replicated multiple times. Aslong as one copy of the key survives, all operationson that key are permitted.


HibariWhy Another NonSQL?

Lockless API Locks are not required for all client operations.Optionally, Hibari supports “test-and-set” of eachkey-value pair via an increasing (enforced by theserver) timestamp value.

Micro-transactions Under limited circumstances, operations onmultiple keys can be given transactionalcommit/abort semantics.


HibariOverview - Chain Replication


HibariMisc - Chain Balancing


HibariNetwork Partition - Admin Server


HibariNetwork Partition - Chains


HibariNetwork Partition - Clients


HibariWhy Erlang/OTP?

• Functional

• Concurrency and Distribution

• Robustness

• Hot code and incremental upgrade

• Tools

→ Development, analysis, production support, . . .

• Efficiency and Productivity

→ Small teams make big impact.

• Ericsson’s support of Erlang/OTP is wonderful

Everything you need to build robust, high performance distributedsystems!


Agenda

• Introduction

• Erlang/OTP

• Why Erlang/OTP?



• What’s Next?


What’s Next?

• WebMail• improving the end-user’s experience• expanding the system’s capacity• adding new and valueable features and services

• Hibari• Benchmarking - YCSB performance test• Thrift and Cassandra API• Hadoop map/reduce integration• . . .

• Community Building• Erlang and Functional Programming

→ UBF hands-on workshop(s)

• Hibari and BigData

→ Hibari hands-on workshop(s)→ Application developer workshop(s)


Work Hard, Work Smarter, Have Fun

Thank You

http://www.erlang.org/

http://www.geminimobile.com/http://www.geminimobile.jp/http://hibari.sourceforge.net/

http://github.com/norton/ubfhttp://github.com/norton/ubf-jsonrpchttp://github.com/norton/ubf-bertrpc

Feedback, Contributors Wanted: [email protected]


AppendixAdditional Slides

Concurrency OrientedProgramming


Java And COP

”The only safe way to execute multiple applications,written in the Java programming language, on the samecomputer is to use a separate JVM for each of them, andto execute each JVM in a separate OS process. Thisintroduces various inefficiencies in resource utilization,which downgrades performance, scalability, andapplication startup time.”

– Czajkowski & Daynes, Sun Microsystems


JSR-000121, Application Isolation API

JSR-000121, Application Isolation API, appears (?) to implementsuch process separation and inter-object communication. It defines:

• 11 classes, 78 methods (not including constructors), and 3exceptions.

• Does not directly address inter-machine communication.

• Does not directly address debugging and profiling issues.

• ”Links” are used for communication between ”isolates”.However . . .

”To maintain isolation, Links provide only ”data” passingfacilities; normal Java Objects cannot be shared bypassing them. However, a limited number of object typesmay be passed, including byte arrays, strings, isolates,and links themselves.”


C/C++ and COP

• No, neither are even close to being a COP.

• No processes, no memory isolation, non-portable, no GC, . . .

• Pipes, files, FIFOs, UNIX domain sockets, TCP/UDP sockets,. . .

• Advantage: You have complete freedom to create the idealsolution.

• Disadvantage: You have complete freedom to create the idealsolution.


Is Erlang a COP?

• Mostly.

• It’s possible to ”forge” an Erlang process name• The ”E” language uses crypto for provably-difficult-to-forge

process naming. (Ask Google. . . )• Very useful for debugging, almost never used by any

production system.• Would be possible to remove feature from local VM, but would

be very difficult to discriminate between ”legit” vs. ”forged”PIDs received from remote nodes.

• Better security policies are needed for WAN-scale distribution.

• Robust failure handling is almost all there, but programmerinput is slight more than the COP ideal.


AppendixAdditional Slides

ErlangExamples


Behaviors: A generic server

A universal client/server, with hot code swapping.

rpc(A, B) ->

Tag = new_ref(),

A ! {rpc, self(), Tag, B},

receive

{Tag, Val} -> Val

end.

server(Fun, Data) ->

receive

{new_fun, Fun1} ->

server(Fun1, Data);

{rpc, From, ReplyAs, Q} ->

{Reply, Data1} = Fun(Q, Data),

From ! {ReplyAs, Reply},

server(Fun, Data1)

end.


"erlang, webmail and hibari" at rakuten tech talk

Technology