realm: building a mobile database
TRANSCRIPT
Realm Building a mobile database
Christian Melchior @chrmelchior
Why a new database?
Once upon a time
Once upon a time
BCNF
Once upon a time
Once upon a time
m:n?
Once upon a time
Once upon a time
SELECT owner.name, dog.name, city.name FROM owner
INNER JOIN dog ON owner.dog_id = dog.id
INNER JOIN city ON owner.city_id = city.id
WHERE owner.name = 'Frank'
Once upon a time
String query = "SELECT " + Owner.NAME + ", " + Dog.NAME + ", " + City.NAME + " FROM " + Owner.TABLE_NAME
+ " INNER JOIN " + Dog.TABLE_NAME + " ON " + Owner.DOG_ID + " = " + Dog.ID
+ " INNER JOIN " + City.TABLE_NAME + " ON " + Owner.CITY_ID + " = " + City.ID
+ " WHERE " + Owner.NAME = "'" + escape(queryName) + "'";
Once upon a time
“Lets use an ORM”
Abstract the problem away
–Joel Spolsky
“All non-trivial abstractions, to some degree, are leaky.”
Why a new database?• ActiveRecord
• Androrm
• Blurb
• Cupboard
• DBFlow
• DBQuery
• DBTools
• EasyliteOrm
• GreenDAO
• Ollie
• Orman
• OrmLite
• Persistence
• RushORM
• Shillelagh
• Sprinkles
• SquiDB
• SugarORM
What is Realm?
Zero-copy object storeDesigned for mobile devices
Object Store
A
B C
D E F
G
VS
A
B C
D E F
G
Realm.getA().getC().getF()
Object Store references
x
z
y
x y z
SELECT table1.x, table2.y, table3.z
FROM table1
INNER JOIN table2 ON table1.table2_id = table1.id
INNER JOIN table3 ON table1.table3_id = table3.id
References in SQL
What is zero-copy?
• CursorAdapter
• FlatBuffers
public class MyCursorAdapter extends CursorAdapter { // ... @Override public void bindView(View view, Context context, Cursor cursor) { ViewHolder holder = (ViewHolder) view.getTag(); String foo = cursor.getString(cursor.getColumnIndexOrThrow("foo")); int bar = cursor.getInt(cursor.getColumnIndexOrThrow("bar")); holder.fooView.setText(foo); holder.barView.setText(Integer.toString(bar)); } }
What is zero-copy?
• CursorAdapter
• FlatBuffers
public long id() { int o = __offset(4); return o != 0 ? bb.getLong(o + bb_pos) : 0; }
public String name() { int o = __offset(6); return o != 0 ? __string(o + bb_pos) : null; }
ByteBuffer bb = ByteBuffer.wrap(bytes); MyObject obj = MyObject.getRootAsMyObject(bb); long id = obj.id(); String name = obj.name();
Zero-copy in Realm
Person { • name = Tommy • age = 8 • dog = {
• name = Lassie } }
Person { • name = Tommy • age = 8 • dog = {
• name = Lassie } }
Person { • name = Tommy • age = 8 • dog = {
• name = Lassie } }
PersonProxy { • name • age • dog
}
PersonProxy { • name • age • dog
}
Person { • name = Tommy • age = 8 • dog = {
• name = Lassie } }
Realm ORM Realm SQLite
Zero-copy in Realm// ObjectsPerson javaPerson = new Person(); // Standard Java POJO Person proxyPerson = realm.copyToRealm(javaPerson); // Convert POJO to Proxy
Person proxyPerson = realm.createObject(Person.class); // Proxy
// Query results are lazyRealmResults<Person> queryResults = realm.allObjects(Person.class);
queryResults.get(0) != queryResults.get(0); // Different Java objects
queryResults.get(0).equals(queryResults.get(0)); // Same data
Designed for mobile
• Memory-mapped files
• C++ core
• B+ trees
• Column-oriented
• Bit-packing
• Minimize I/O overhead
• Cross-platform
• Faster reads and writes
• Queries > writes
• Smaller file size
Benchmarks
http://static.realm.io/downloads/java/android-benchmark.zip
1.09%2.26%
3.79%4.55%
22.85%
13.37%
1% 1% 1% 1% 1% 1%
0%
5%
10%
15%
20%
25%
BatchWrite% SimpleWrite% SimpleQuery% FullScan% Sum% Count%
Speedu
p&vs.&SQLite&
Tests&/&1000&objects&
Realm%
SQLite%
Implementing Realm Android
Architecture
JNI
Realm Internal API
Realm Core
Realm Object Store API
Zero copypublic class Pojo { private String foo; private int bar; public Pojo() { } public String getFoo() { return foo; } public void setFoo(String foo) { this.foo = foo; } public int getBar() { return bar; } public void setBar(int bar) { this.bar = bar; } }
Annotation processorpublic class Pojo extends RealmObject { private String foo; private int bar; public Pojo(String foo, int bar) { this.foo = foo; this.bar = bar; } public String getFoo() { return foo; } public void setFoo(String foo) { this.foo = foo; } public int getBar() { return bar; } public void setBar(int bar) { this.bar = bar; } }
@RealmClasspublic abstract class RealmObject { protected Row row; protected Realm realm; // ...}
+ =
Proxy classespublic class PojoRealmProxy extends Pojo implements RealmObjectProxy { private static long INDEX_FOO; private static long INDEX_BAR; @Override public String getFoo() { realm.checkIfValid(); return (java.lang.String) row.getString(INDEX_FOO); } @Override public void setFoo(String value) { realm.checkIfValid(); row.setString(INDEX_FOO, (String) value); } @Override public int getBar() { realm.checkIfValid(); return (int) row.getLong(INDEX_BAR); } @Override public void setBar(int value) { realm.checkIfValid(); row.setLong(INDEX_BAR, (long) value); }
@Override public String toString() { // .. } @Override public int hashCode() { // ... } @Override public boolean equals(Object o) { // ... }
// Static helper methods ... }
public class NormalObject extends RealmObject { public String foo; public int bar; @Override public String toString() { return "[" + foo + ";" + bar + "]"; } } NormalObject obj = new NormalObject(); obj.foo = "DroidCon";
Standard Java objects
Byte code manipulation
• Custom Gradle plugin
• Morpheus
• JavaAssist
Compile *.java
Create proxy classes
Bytecode: Replace field access with accessors
Succes!
Dex
public class Pojo extends RealmObject { public String foo; public int bar; @Override public String toString() { return "[" + foo + ";" + bar + "]"; } public String realm$$getFoo() { return foo; } public void realm$$setFoo(String s) { foo = s; } public int realm$$getBar() { return bar; } public int realm$$setBar(int i) { return bar = i; } }
Byte code manipulation
public class Pojo extends RealmObject { public String foo; public int bar; @Override public String toString() { return "[" + realm$$getFoo() + ";" + realm$$getBar() + "]"; } public String realm$$getFoo() { return foo; } public void realm$$setFoo(String s) { foo = s; } public int realm$$getBar() { return bar; } public int realm$$setBar(int i) { return bar = i; }}
Byte code manipulationPojo pojo = new Pojo(); pojo.realm$$setFoo(“DroidCon”);
Join points
Method count
• Realm: ~1750
• Model class: +2x no. of fields
• Proxy class: + 2x no. of fields + 11
• Total method count: “it depends” (~2000)
Picasso: ~600 GSON: ~950
OrmLite: ~2400 RxJava: ~3500
Support library: ~12300
What does consistency mean?
• Database/ACID: The consistency property ensures that any transaction will bring the database from one valid state to another
• UI consistency: Visible data should represent one valid state.
UI Consistency
Fragment 1
Fragment 2
Fragment 1
Fragment 2
☺ ☹
Consistency today
Fragment 1
Fragment 2
• Loader
• Content URI
• ContentObserver
• ContentProvider
Consistency today
Fragment 1
Fragment 2
ContentProvider1
2
2
3
• Multiversion concurrency control
• Multiple read transactions
• Data is versioned
• One write transaction
R
A B
C D E
MVCC
R
A B
X D E
v1
v2
Consistency in RealmBackground threadUI thread
queue.next();
queue.next();
thread.start();
Consistency in RealmBackground threadUI thread
queue.next();
queue.next();
thread.start();
Realm realm = Realm.getDefaultInstance();
Realm realm = Realm.getDefaultInstance();
Consistency in RealmBackground threadUI thread
queue.next();
queue.next();
thread.start();
realm.close();
realm.close();
realm.executeTransaction(new Realm.Transaction() { @Override public void execute(Realm realm) { //... } });
Consistency in RealmBackground threadUI thread
queue.next();
queue.next();
thread.start();
Consistency in RealmBackground threadUI thread
queue.next();
queue.next();
thread.start();
@OnClickpublic void buttonClicked() { Person p = realm.where(Person.class).findFirst(); int ager = p.getAge();
String name = p.getName(); } ☹
Consistency in RealmBackground threadUI thread
queue.next(); thread.start();
new RealmChangeListener() { @Override public void onChange() { //... } }
handler.sendEmptyMessage(REALM_CHANGED);
Consistency with Realm
Fragment 1
Fragment 2
Fragment 1
Fragment 2
new RealmChangeListener() { @Override public void onChange() { invalidateViews(); } }
Current status
• Moving towards 1.0
• Used in well more than 500M installations.
• Still need to solve hard problems: Interprocess cursors, move objects across threads, Parcelable.
Realm today
