techtalk #14 grokking: couchbase - nosql + memcached + real-time + offline!
TRANSCRIPT
ScalabilityofNoSQL.FlexibilityofJSON.PowerofSQL.
VuNguyen
WhatisCouchbase?
Couchbase Server CouchbaseMobile
• DocumentDatabase
• Key/ValueStore• DistributedCache
• EmbeddedDatabase
• Offline&Real-timesync
DatainCouchbase
• Key:“DocumentID”
• Anystringupto250bytes• Uniqueinsideabucket
• Value• Anyvalueupto20MB
• Anything:JSON,XML,serializedobject,string,etc.
• Metadata
• CAS(check-and-set)• TTL(time-to-live)
• Optionalflags
Architecture
• NodeACouchbase instance.
Architecture
• NodeACouchbase instance.
• ClusterAscalable,networkedsetofnodes.
Architecture
• NodeACouchbase instance.
• ClusterAscalable,networkedsetofnodes.
• BucketAlogicalcollectionofdocumentswithuniquekeys.
Bucketsareevenlydistributedacrossacluster.
[key1:value]
[key2:value]
[key3:value]
[key4:value]
[key5:value]
[key6:value]
Architecture
• NodeACouchbase instance
• ClusterAscalable,networkedsetofnodes
• BucketAlogicalcollectionofdocumentswithuniquekeys.
Bucketsareevenlydistributedacrossacluster.
• Document
Akey/valuepairwithinabucket.Keysareunique
withinabucket.
[key1:value]
[key2:value]
[key1:value]
[key2:value]
[key3:value]
[key4:value]
[key3:value]
[key4:value]
[key5:value]
[key6:value]
[key5:value]
[key6:value]
Howtoaccessdata?
1. Read/writedocumentsbytheirkeys
• Extremelyfastduetocache
2. MapReduceviews
• Distributedsecondaryindexes,builtusingmap-reduce
3. N1QL(“nickel”)query
• SQLsupersetforindexingandqueryingJSONstructure4. Fulltextsearch
• Couchbase FTS
Single-NodeArchitecture
ClusterManager
DataService
IndexService
QueryService
ManagedCache
Storage
Couchbase Servernodesareidentical.
Twocorecomponents
- ClusterManager
- NodeManagers
Threeindependentservices
- DataService
- IndexService
- QueryService
ClusterArchitecture
ClusterManager
DataService
IndexService
QueryService
ManagedCache
Storage
ClusterManager
DataService
IndexService
QueryService
ManagedCache
Storage
ClusterManager
DataService
IndexService
QueryService
ManagedCache
Storage
ClusterManager
DataService
IndexService
QueryService
ManagedCache
Storage
Cluster
Howdoesaget operate?
DataService
ManagedCache
Storage
doc2 doc3
doc1 doc2 doc3 doc4
doc2 doc4
Replication
queue
Persistence
queue
Getisextremelyfast:
- Commondocs
arecached
- BinaryTCPconnection
Twoconfigs:
- Value-onlyejection
- Fullmeta-dataejection
Howdoesaset operate?
DataService
ManagedCache
Storage
doc4
doc1 doc2 doc3 doc4
doc4
Replication
queue
Persistence
queue
Acknowledgetype:
- WheninRAM(default)
- Wheninstorage
- Whenreplicated
Replication
- Veryfast,RAMtoRAM
WhyCouchbase serverfast?
• Getandsetareservedprimarilyfromcache
• Diskwritesareappendonly
doc1 doc2 doc3
WhyCouchbase serverfast?
• Getandsetareservedprimarilyfromcache
• Diskwritesareappendonly
doc1 doc2 doc3 doc1
WhyCouchbase serverfast?
• Getandsetareservedprimarilyfromcache
• Diskwritesareappendonly
doc1 doc2 doc3 doc1 doc3
WhyCouchbase serverfast?
• Getandsetareservedprimarilyfromcache
• Diskwritesareappendonly• Datafilesareperiodicallycompressed
doc2 doc1 doc3doc1 doc3
WhyCouchbase serverfast?
• Getandsetareservedprimarilyfromcache
• Diskwritesareappendonly• Datafilesareperiodicallycompressed
doc2 doc1 doc3
MapReduce
Viewsaresecondaryindexes
• WritteninJavaScript
• ExecutedbyV8engine
function (doc, meta) {if (doc.age > 30) {
emit(doc.department, null)}
}
N1QLQueries
SQLforJSONdata
• SELECT, INSERT, UPDATE, DELETE
• JOIN, WHERE, GROUP BY
• CREATE INDEX, DROP INDEX
• MIN, MAX, COUNT
• UNION, INTERSECT, EXCEPT, ...
SELECT count(*), city FROM customer
WHERE customer.type = “vip”
GROUP BY customer.city ORDER BY customer.city
LIMIT 5 OFFSET 5
Bucket&vBucket
• Eachbucketisdividedinto1024vBucket.• vBuckets aredistributedevenlyacrossallnodesinthecluster.
• Asnodesjoin/leavecluster,vBuckets adjustautomatically.
Active
vBuckets
Replicated
vBuckets
Active
vBuckets
Replicated
vBuckets
Active
vBuckets
Replicated
vBuckets
CouchbaseMobile
• EmbeddeddatabaseforiOS&Android
• UnofficialJavaScriptdatabase(browser&NodeJs)
• Real-timesynchronization
• CouchDB replicationprotocol
Architecture
CouchDB protocol
GET/source/_changes?feed=normal&style=all_docs&heartbeat=10000{
"results":[{"seq":14,"id":"f957f41e","changes":[{"rev":"3-46a3"}],"deleted":true},{"seq":29,"id":"ddf339dd","changes":[{"rev":"10-304b"}],
],"last_req": 78,
}
GET/source/_changes?feed=continuous&style=all_docs&heartbeat=10000
{"seq":14,"id":"f957f41e","changes":[{"rev":"3-46a3"}],"deleted":true}{"seq":29,"id":"ddf339dd","changes":[{"rev":"10-304b"}]
SyncFunctionfunction (doc, oldDoc) {
if (doc._deleted) {requireRole("role:editor");requireUser(oldDoc.writers);return;
}
if (!doc.title || !doc.creator || !doc.channels || !doc.writers) {throw({forbidden: "Missing required properties"});
} else if (doc.writers.length == 0) {throw({forbidden: "No writers"});
}if (oldDoc == null) {
requireRole("role:editor");requireUser(doc.creator)
} else {requireUser(oldDoc.writers);if (doc.creator != oldDoc.creator) {
throw({forbidden: "Can't change creator"});}
}
channel(doc.channels);}
ScalabilityofNoSQL.FlexibilityofJSON.PowerofSQL.
VuNguyen