mysql replication outside the box
DESCRIPTION
MySQL Replication using multiple master, parallel replication, seamless fail over, and conflict prevention using Tungsten ReplicatorTRANSCRIPT
MySQL replication outside the box
Giuseppe MaxiaQA DirectorContinuent, Inc@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. 1Tuesday, October 25, 11
about me -Giuseppe Maxia§ a.k.a. The Data Charmer§ QA Director, Continuent, Inc§ Long time hacking with database features§ Formerly, database consultant, designer, coder.§ A passion for QA§ An even greater passion for open source§ ... and community§ Passionate blogger§ http://datacharmer.blogspot.com
2Tuesday, October 25, 11
Why Tungsten?
3Tuesday, October 25, 11
BINARY LOG
REPLICATION
MySQLDBMS
trans
actio
n
trans
actio
n
trans
actio
n
trans
actio
n
trans
actio
ntra
nsac
tion
trans
actio
ntra
nsac
tion
trans
actio
ntra
nsac
tion
trans
actio
n
MySQLDBMS
transactiontransaction
transactiontransaction
transactiontransaction
transactiontransaction
transactiontransaction
transactiontransaction
MySQL replicationis single threaded
4Tuesday, October 25, 11
MySQLDBMS
MySQLDBMS
MySQLDBMS
MySQLDBMS
master
slave slaveslave
MySQLDBMS
MySQLDBMS
MySQLDBMS
MySQLDBMS
master master master
slave
single source multi source (fan-in)
multiple sources?5Tuesday, October 25, 11
MySQLDBMS
MySQLDBMS
MySQLDBMS
MySQLDBMS
mastermaster
master master
from this to this
multiple masters?
6Tuesday, October 25, 11
MySQLDBMS
MySQLDBMS
MySQLDBMS
MySQLDBMS
mastermaster
master master
Avoiding conflicts?
INSERT RECORD
A
INSERT RECORD
A
MODIFY RECORD
B
MODIFY RECORD
B
7Tuesday, October 25, 11
MySQLDBMS
MySQLDBMS
MySQLDBMS
master
slaveslave
MySQLDBMS
MySQLDBMS
MySQLDBMS
master
slaveslave
MySQLDBMS
MySQLDBMS
MySQLDBMS
master
slaveslave
Seamless failover?
8Tuesday, October 25, 11
MySQLDBMS
MySQLDBMS
mysql master
postgresqlmysql
Replicating to something else?
oracle mongodb
9Tuesday, October 25, 11
All these examples tell us:
Nice dream, but MySQL can’t do it
10Tuesday, October 25, 11
Enter Tungsten Replicator
11Tuesday, October 25, 11
http://code.google.com/p/tungsten-replicator
12Tuesday, October 25, 11
http://code.google.com/p/tungsten-replicator
Open Source
12Tuesday, October 25, 11
http://code.google.com/p/tungsten-replicator
100% GPL v2Open Source
12Tuesday, October 25, 11
What can it do?
• Easy failover
• Multiple masters
• Multiple sources to a single slave
• Conflict prevention
• Parallel replication
• Replicate to Oracle and PostgreSQL database
13Tuesday, October 25, 11
© Continuent 2010
MySQL to foreign services
BinLogs
Master DB
(Binlogs enabled) Slave DB
MySQL setup to run as
MySQL master
Data is applied to PostgreSQL or
Oracle or to NoSQL entities
14Tuesday, October 25, 11
Tungsten Replicator in a nutshell
binlog THL
slavemaster
host1 host2
THL
trep_commit_seqnotrep_commit_seqnotrep_commit_seqnoorigin seqno eventid
trep_commit_seqnotrep_commit_seqnotrep_commit_seqnoorigin seqno eventid
global transaction ID
15Tuesday, October 25, 11
Main components
• Transaction History Logs (THL)
• roughly corresponding to MySQL relay logs
• have a lot of metadata
• Service database
• contains metadata for latest transactions
• Metadata is committed together with data
• Makes slaves crash proof
16Tuesday, October 25, 11
A failover scenario1: MySQL native replication
17Tuesday, October 25, 11
© Continuent 2011
1. one Master, two slaves
/ Loading the “employees” test database
18Tuesday, October 25, 11
© Continuent 2011
2. Master goes away
/ Stop replication
/ Slaves are updated at different levels
# 2select count(*) from titles 333,145
# 3
select count(*) from titles 443,308
19Tuesday, October 25, 11
© Continuent 2011
3. Look into Slave #2 binary logs
/ find the last transaction
20Tuesday, October 25, 11
© Continuent 2011
4. Look into Slave #3 binary logs
1. find the transaction that was last in slave #22. Recognize that last transaction in the log of slave #3
(This can actually take you a LOOOONG TIME)3. Get the position immediately after this transaction 4. (e.g. 134000 in file mysql-bin.000018)
21Tuesday, October 25, 11
© Continuent 2011
5. promote Slave #3 to master
/ in slave #2
CHANGE MASTER TO master_host=‘slave_3_IP’,
master_user=‘slavename’,master_password=‘slavepassword’,master_log_file=‘mysql-bin.000018’,master_log_pos=134000;
22Tuesday, October 25, 11
A failover scenario1: Tungsten Replicator
23Tuesday, October 25, 11
© Continuent 2011
1. one master, two slaves
/ loading the ‘employees’ test database
24Tuesday, October 25, 11
© Continuent 2011
2. Master goes away
/ Stop replication
/ Slaves are updated at different levels
# 2select count(*) from titles 333,145
# 3
select count(*) from titles 443,308
25Tuesday, October 25, 11
© Continuent 2011
3. no need to find the last transaction
/ simply change roles
trepctl -host slave3 setrole -role mastertrepctl -host slave2 setrole -role slave \
-uri thl://slave3
trepctl -host slave3 online State: ONLINE
trepctl -host slave2 online State: GOING-ONLINE:SYNCHRONIZING
26Tuesday, October 25, 11
© Continuent 2011
4. Check that the slave has synchronized
# new masterselect seqno from tungsten.trep_commit_seqno;78
# new slaveselect seqno from tungsten.trep_commit_seqno;64
27Tuesday, October 25, 11
© Continuent 2011
4. Tell the replicator to hurry up
# new mastertrepctl -node slave3 flushMaster log is synchronized with database at log sequence number: 78
# new slavetrepctl host slave2 wait -applied 78ONLINE
select seqno from tungsten.trep_commit_seqno;78
28Tuesday, October 25, 11
© Continuent 2011
4. ... and we’re done
# new masterselect count(*) from employees.titlescount(*)443308
# new slave: count(*)443308
29Tuesday, October 25, 11
Parallel replication
30Tuesday, October 25, 11
parallel replication
From here ...
31Tuesday, October 25, 11
parallel replication
To here.
32Tuesday, October 25, 11
Parallel replication facts
✓Sharded by database
✓Good choice for slave lag problems
❖Bad choice for single database projects
33Tuesday, October 25, 11
Benchmarking parallel replication
34Tuesday, October 25, 11
sysbench
sysbench
sysbench
sysbench
sysbench
sysbench
sysbench
sysbench
sysbench
sysbench
db1
db2
db3
db4
db5
db6
db7
db8
db9
preparation (1)db0
35Tuesday, October 25, 11
before the test (2)
binary logs
MySQL slave
Tungsten slave
OFFLINE
STOPPED
replicator alpha
direct: alpha(slave)
36Tuesday, October 25, 11
starting the test
binary logs
MySQL slave
Tungsten slave
OFFLINE
STOPPED
replicator alpha
direct: alpha(slave)
Concurrent sysbenchon 30 databasesrunning for 1 hour
TOTAL DATA: 130 GBRAM per server: 20GB
Slaves will have 1 hour lag37Tuesday, October 25, 11
measuring results
binary logs
MySQL slave
Tungsten slave
ONLINE
START
replicator alpha
direct: alpha(slave)
Recording catch-up time
38Tuesday, October 25, 11
MySQL native replication
slave catch up in 04:29:30
39Tuesday, October 25, 11
Tungsten parallel replication
slave catch up in 00:55:40
40Tuesday, October 25, 11
Parallel replication made simpler
FROM HERE ....41Tuesday, October 25, 11
Parallel replication made simpler
TO HERE42Tuesday, October 25, 11
Parallel replication made simpler
43Tuesday, October 25, 11
parallel replicationdirect slave facts
44Tuesday, October 25, 11
parallel replicationdirect slave facts
✓No need to install Tungsten on the master
44Tuesday, October 25, 11
parallel replicationdirect slave facts
✓No need to install Tungsten on the master
✓Tungsten runs only on the slave
44Tuesday, October 25, 11
parallel replicationdirect slave facts
✓No need to install Tungsten on the master
✓Tungsten runs only on the slave
✓Replication can revert to native slave with two commands (trepctl offline; start slave)
44Tuesday, October 25, 11
parallel replicationdirect slave facts
✓No need to install Tungsten on the master
✓Tungsten runs only on the slave
✓Replication can revert to native slave with two commands (trepctl offline; start slave)
✓Native replication can continue on other slaves
44Tuesday, October 25, 11
parallel replicationdirect slave facts
✓No need to install Tungsten on the master
✓Tungsten runs only on the slave
✓Replication can revert to native slave with two commands (trepctl offline; start slave)
✓Native replication can continue on other slaves
❖Failover (either native or Tungsten) becomes a manual task
44Tuesday, October 25, 11
Multiple masters
45Tuesday, October 25, 11
multiple masters
• Tungsten Replicator recipe: use more services
46Tuesday, October 25, 11
Bi-directional replication
47Tuesday, October 25, 11
Bi-directional replication with slaves
48Tuesday, October 25, 11
Multiple sources
49Tuesday, October 25, 11
multiple sources
• Tungsten Replicator recipe is still valid: use more services
50Tuesday, October 25, 11
Multiple source replication
51Tuesday, October 25, 11
Multiple masters replication: 3 nodes
52Tuesday, October 25, 11
Multiple masters replication: 4 nodes
53Tuesday, October 25, 11
Conflict prevention
54Tuesday, October 25, 11
Conflict prevention facts
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
• Applied either at the master or at the slave
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
• Applied either at the master or at the slave
• methods:
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
• Applied either at the master or at the slave
• methods:
• make replication fail
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
• Applied either at the master or at the slave
• methods:
• make replication fail
• drop silently
55Tuesday, October 25, 11
Conflict prevention facts
• Sharded by database
• Defined dynamically
• Applied either at the master or at the slave
• methods:
• make replication fail
• drop silently
• drop with warning55Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Tungsten Conflict prevention points
56Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Tungsten Conflict prevention points
56Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Tungsten Conflict prevention points
56Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on master
57Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on masterINSERT A x,y
INSERT Ax,z
57Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on masterINSERT A x,y
INSERT Ax,z
57Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on slave
58Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on slaveINSERT A x,y
INSERT Ax,z
58Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: Fail on slaveINSERT A x,y
INSERT Ax,z
58Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: Fail on slave(Multiple sources)
59Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: Fail on slaveINSERT A x,y
INSERT Ax,z
(Multiple sources)
59Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: Fail on slaveINSERT A x,y
INSERT Ax,z
(Multiple sources)
59Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: DROP on master
60Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: DROP on masterINSERT A x,y
60Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: DROP on masterINSERT A x,y
INSERT Ax,z
60Tuesday, October 25, 11
host1host3
master
host2
M
MS
SS
host4
S
Prevention methods: DROP on master
INSERT Ax,z
60Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: DROP on slave(Multiple sources)
61Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: DROP on slaveINSERT A x,y
INSERT Ax,z
(Multiple sources)
61Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: DROP on slave
INSERT Ax,z
(Multiple sources)
INSERT A x,y
62Tuesday, October 25, 11
host1
master
host2
M
M
host3
S
Prevention methods: DROP on slave
INSERT Ax,z
(Multiple sources)
62Tuesday, October 25, 11
Tungsten in practiceInstallation
63Tuesday, October 25, 11
Installation
• Check the requirements
• Get the binaries
• Expand the tarball
• Run ./tools/tungsten-installer
64Tuesday, October 25, 11
REQUIREMENTS
• Java JRE or JDK (Sun/Oracle or Open-jdk)
• Ruby 1.8 (only during installation)
• ssh access to the same user in all nodes
• MySQL user with all privileges
65Tuesday, October 25, 11
Installation types
• master / slave
• slave - direct
66Tuesday, October 25, 11
master/slave
binlog
THL
THL
slave
slave
master
host1host2
host3
THL
67Tuesday, October 25, 11
slave direct
binlogTHL
slavemaster
relay log
host1host2
host3
THL
slave
relay log
68Tuesday, October 25, 11
sample installation
• 4 servers
• qa.r1.continuent.com r1
• qa.r2.continuent.com r2
• qa.r3.continuent.com r3
• qa.r4.continuent.com r4
• MySQL 5.1
• tungsten-replicator-2.0.4
69Tuesday, October 25, 11
Installation (1)# starting at node 4, but any would do
for N in 1 2 3 4do ssh r$N mkdir tinstalldone
cd tinstalltar -xzf /path/to/tungsten-replicator-2.0.4.tar.gzcd tungsten-replicator-2.0.4
70Tuesday, October 25, 11
Installation (2)export TUNGSTEN_BASE=$HOME/tinstall
./tools/tungsten-installer \ --master-slave \ --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$TUNGSTEN_BASE \ --cluster-hosts=r1,r2,r3,r4 \ --start
1 2 3 4 5 6 7 8 9
71Tuesday, October 25, 11
Installation (2)export TUNGSTEN_BASE=$HOME/tinstall
./tools/tungsten-installer \ --master-slave \ # installation mode --master-host=r1 \ # who’s the master --datasource-user=tungsten \ # mysql username --datasource-password=secret \ # mysql password --service-name=dragon \ # name of the service --home-directory=$TUNGSTEN_BASE \ # where we install --cluster-hosts=r1,r2,r3,r4 \ # hosts in cluster --start # start replicator after installing
72Tuesday, October 25, 11
What does the installation do
host4
1: Validate all servers
✔✗✔
✔
host1
✔✗✔
✔
host2
✔✗✔
✔
host3
✔✗✔
✔
Report all errors73Tuesday, October 25, 11
What does the installation do
host4
1 (again): Validate all servers
✔
✔
✔
host1
✔
✔
✔
host2
✔
✔
✔
host3
✔
✔
✔
✔✔ ✔ ✔
74Tuesday, October 25, 11
What does the installation do
2: install Tungsten in all servers
host3
$HOME/ tinstall/ config/ releases/ relay/ logs/ tungsten/
host4host1
host2
75Tuesday, October 25, 11
examplessh r2 chmod 444 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start
ERROR >> qa.r2.continuent.com >> /home/tungsten/tinstall is not writeable
76Tuesday, October 25, 11
examplessh r2 chmod 755 $HOME/tinstall./tools/tungsten-installer \ --master-slave --master-host=r1 \ --datasource-user=tungsten \ --datasource-password=secret \ --service-name=dragon \ --home-directory=$HOME/tinstall \ --thl-directory=$HOME/tinstall/logs \ --relay-directory=$HOME/tinstall/relay \ --cluster-hosts=r1,r2,r3,r4 --start
# no errors
77Tuesday, October 25, 11
Installation - direct./tools/tungsten-installer \ --direct \ --master-host=r1 \ --slave-host=r4 \ --master-user=tungsten \ --slave-user=tungsten \ --master-password=secret \ --slave-password=secret \ --service-name=Castor \ --channels=1 \ --home-directory=$TUNGSTEN_BASE \ --start-and-report
78Tuesday, October 25, 11
Tungsten in practiceAdmin
79Tuesday, October 25, 11
Main components
• THL (Tungsten generated Transaction History Log)
• service database
80Tuesday, October 25, 11
Tools
• replicator
• trepctl
• thl
81Tuesday, October 25, 11
replicator
• It’s the service provider
• You launch it once when you start
• You may restart it when you change config
82Tuesday, October 25, 11
trepctl
• Tungsten Replicator ConTroLler
• It’s the driving seat for your replication
• You can start, update, and stop services
• You can get specific info
83Tuesday, October 25, 11
thl
• Transaction History List
• Gives you access to the Tungsten relay logs
84Tuesday, October 25, 11
http://bit.ly/tr20_cookbook
85Tuesday, October 25, 11
© Continuent 2011© Continuent 2011
Tungsten Replicator vs. Enterprise
Feature Replicator EnterpriseSimple admin with trepctl X XMySQL Row/Statement 5.0/5.1/5.5 X XMulti-master replication X XParallel apply (2.0) X XDistributed management with cctrl XAutomated failover and recovery XZero-downtime upgrade XSQL routing and load-balancing XFloating IP support X
86Tuesday, October 25, 11
Conclusion and Q&A
87Tuesday, October 25, 11
© Continuent 2011
Worldwide560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel (866) 998-3642 Fax (408) 668-1009e-mail: [email protected]
Contact Information
Continuent Web Site:http://www.continuent.com
Tungsten Projecthttp://tungsten-replicator.org
Getting started with Tungsten Replicator:http://datacharmer.blogspot.com
@datacharmer@continuent
88Tuesday, October 25, 11
89Tuesday, October 25, 11