Ruby,NoSQL and TokyoCabinet

盛大创新院 庄表伟http://www.zhuangbiaowei.com/blog/

From RDBMS to NoSQL (1)

Long long ago …Network Database (GE IDS 1961)Hierarchical Database (IBM IMS 1968)

Relational database eraEdgar Frank Codd -- Since 1969Relational modelSQLManagement systemOver $ 20 billion (2008)

Now is Web2.0 eraHave huge storage(TB/PB data size)Need high performanceNeed high scalabilityNeed high availability

NoSQL now comingNew theory supportNew API styleNew buzzword

From RDBMS to NoSQL (2)

Projects or papers…Google BigTable (2004)Tokyo Cabinet (2006)Amazon Dynamo (2007)MongoDB,CouchDB,Cassandra,Voldemort

(2008)Redis,Riak,HBase… (2009)More and more…

From RDBMS to NoSQL (3)

CAP (Eric Brewer 2000)ConsistencyAvailabilityPartition tolerance

BASEBasically AvailableSoft-stateEventual Consistency

MapReduceFrom functional programmingDistributed computing framework

Important NoSQL theory

Loss of data integrityAny data is allowed

Can not ensure data consistencyEventual Consistency

No transaction Stored procedure convert to map/reduce

script Lack support of management system System refactoring or rewrite

Choose NoSQL means…

by Mikio Hirabayashi ( 平林幹雄 )http://1978th.net/

TokyoCabinet -- lightweight database library TokyoTyrant -- lightweight database server Key/Value Database Database types

Hash / B+ Tree / Fixed-length / Table High performance

Insert: 0.4sec/1M records(2,500,000 qps)Search: 0.33sec/1M records(3,000,000 qps)

High concurrency High Scalability

TokyoCabinet and TokyoTyrant

Everyone table need a port Loss table struct and data type Can not access by column Reduction of performance

Read 400,000 record/sec(TCT)Read 1,800,000 record/sec(TCH)Read 1,000,000 record/sec(TCB)

When the data more than 100 million, will performance reduction

Network protocol is not perfect

TC/TT table engine insuffi ciency

One port,more table Row access and column access B+ tree data store New binary api protocol New data storage mode

Config TCHIndex TCBData TCB

About TCDatabase

ActiveTokyoCabinet (1)

Config

Model

ActiveTokyoCabinet (2)

database.ymldevelopment:

adapter: tcdb

path: ./db

database: blog

modelclass Post < ActiveRecord::Base validates_presence_of :body, :title has_many :commentsend

About tcdb-adapter (1)

Migrationclass CreatePosts < ActiveRecord::Migration def self.up create_table :posts do |t| t.string :title t.text :body t.timestamps end end def self.down drop_table :posts endend

About tcdb-adapter (2)

Demo

SQLite MySQL OracleMoney

MySQL MySQL MySQLDBA

RDBMS NoSQLRisk

RDBMS tcdb-adapter NoSQLSQL/NoSQL Bridge

Storage architecture migration

Rails

ActiveRecord

MysqlAdapter

MySQL

Architecture diagram (today)

Rails

ActiveRecord(modify)

TokyoCabinetAdapter

TokyoCabinet/TokyoTyrant

Classic ActiveTokyoCabinet

Rails

ActiveRecord

TcdbAdapter

tcdb.rb

tcdb-adapter

TCDatabase Cloud DatabaseTransparent distributedRunning like mysql proxyAccess to TC / MongoDB / MySQL / moreConfiguration and management support

Tcdb-adapter Cloud ActiveRecordAlmost support all of the SQLRails3 support (based arel)Intergration thinking sphinxOne Interface, One architecture!

You can imagine more …

The future

Architecture diagram (future - 1)

Rails

ActiveRecord

TcdbAdapter

TCDatabase

Architecture diagram (future - 2)

Rails

ActiveRecord

TcdbAdapter

TCDatabaseTCData

base

TCDatabase

TCDatabase

Rails

ActiveRecord

CloudDBAdapter

CloudDatabase

TokyoCabinet MongoDB …… Sphinx MySQL

Architecture diagram (future - 3)

It will be open source!

THANKS