percona live europe 2016 use proxysql to improve …percona live europe 2016 use proxysql to improve...

Percona Live Europe 2016Use ProxySQL to Improve

Your MySQL High Availability Solution

Marco Tusa Manager ConsultingAmsterdam, Netherlands | October 3 – 5, 2016

Percona Live 2017Use ProxySQL to Improve

Your MySQL High Availability Solution

Marco tusaSanta Clara, California | April 24th – 27th, 2017

3

� Marco “The Grinch”• Open source enthusiast

About me

4

•Brief ProxySQL introduction•How it interact Master-slave •How interact with PXC•Extending with scheduler•ProxySQL HA•Time cost

Agenda

5

What is HA9 0. 0 0 0 % (36 days) MySQL Replication9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication9 9 .9 9 5 % (26 minutes) Multi Master (galera - percona cluster) 3 node minimum9 9. 9 9 9 % (5 minutes) MySQL Cluster

6

What is HA

7

What is HA

8

Who will save the Day?

9

Who will save the Day?

10

Objectively What we have is:

MySQL Group Replication

11

The telephone game

12

All talking

13

Black box:

14

Use the Right Tool for the Job…

15

What we need

16

MANAGEBLEFAST

SCALABLEFREE

Solution should be

17

… and do the right thing!My Motto

18

ProxySQL

19

ProxySQL

20

Understand the basics - what it does ...

Write / Read (Master/Slave)

Multiple WritersSharding &

Geographic Distribution

Traffic Load dispatch (aka Mirroring)

Query re-write

21

Define UsersServers group by function

Host GroupsRules define what, where and how

Query RulesSpecial Grouping for Replication

Replication Host Group

Understand the basics – how this is done?

22

HostGroup (HG) & Replication HG

Understand the basics

23

Query rules

Understand the basics

24

Scheduler

Understand the basics

25

Basic scenario

HG: 10U: app_RW

HG: 11U: app_R

HG: 12U: ETL_R

26

insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');

insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');

insert into mysql_users

(username,password,active,default_hostgroup,default_schema) values ('etl_R','test',1,12,’etl_schema');LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;

Basic scenario - Users

27

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.5',10,3306,1000);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.6',11,3306,10000,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.7',11,3306,10000,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight, max_replication_lag) VALUES ('192.168.1.5',11,3306,1,2);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,10);INSERT INTO mysql_replication_hostgroups VALUES (10,11);LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Basic scenario – Servers / HG

28

insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1); insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1); insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1);insert into mysql_query_rules (username,destination_hostgroup,active) values('etl_R',12,1);

LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

Basic scenario – Query rules

29

Basic scenario – with ProxySQL

Writes

ReadsReads

30

READ_ONLY = 0 WRITE HGREAD_ONLY = 1 READ HG

ProxySQL – Replication HG – danger!

READ_ONLY=0 READ_ONLY=1 READ_ONLY=1

HG 10 & HG:11HG 11 HG 11

READ_ONLY=0

HG 10 & HG 11

31

ProxySQL + MHA (or any controller)

Writes

ReadsReads

32

No movable part like:

VIPDNS (Route 53 or similar)

No Crazy scripts to manage other proxy (ie HAProxy)

ProxySQL + MHA (or any controller)

33

MHA discovery time (ping x 3 times) -> ~9 secMHA recovery (binlog) variable time

MHA Slave shift + Master READ-ONLY < 2 sec

ProxySQL will react as soon as READ_ONLY=0

What timing with MHA?

34

Nothing all defaults

master_ip_failover_script=master_ip_failovermaster_ip_online_change_script=master_ip_online_change

Test it:masterha_master_switch --master_state=alive --conf=/etc/mha.cnf --orig_master_is_new_slave --interactive=0 --running_updates_limit=0

What is needed with MHA?

35

Given no VIP or centralize element.

Consistent distributed behavior for HA?

36

Integration with PXC\Galera

37

Integration with PXC\Galera

38

Integration with PXC\Galera

39

Integration with PXC\Galera read_onlywsrep_statuswsrep_rejectquerieswsrep_donorrejectquerieswsrep_connectedwsrep_desinccountwsrep_readywsrep_providerwsrep_segmentNumber of nodes in by segmentRetry loop

Online Offline_softOffline_hardShunned

But we also have HostgroupsReplication HG

40

Integration with PXC\Galera

41

Integration with PXC\Galera

HG: 10

HG: 10

HG: 12HG: 11

HG: 11HG: 12

42

Multiple writer vs Single writerMulti writer based on weight and Hostgroups

Need Replication HG and READ_ONLYFailover is done without the need of performing actions on the nodes

Failover is done modifying the READ_ONLY valueSingle WRITER node isolation not guarantee

Single WRITER node isolation is guarantee

Integration with PXC\Galera

43

insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_RW','test',1,10,'test_schema');

insert into mysql_users(username,password,active,default_hostgroup,default_schema) values ('app_R','test',1,11,'test_schema');

insert into mysql_users

(username,password,active,default_hostgroup,default_schema) values (‘appDR_R','test',1,12,’etl_schema');LOAD MYSQL USERS TO RUNTIME;SAVE MYSQL USERS TO DISK;

Integration with PXC\Galera - Users

44

insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',10,1,3,'^SELECT.*FOR UPDATE',1);

insert into mysql_query_rules(username,destination_hostgroup,active,retries,match_digest,apply) values(’app_RW',11,1,3,'^SELECT ',1);

insert into mysql_query_rules (username,destination_hostgroup,active) values(’app_R',11,1);

insert into mysql_query_rules (username,destination_hostgroup,active) values(‘appDR_R',12,1);LOAD MYSQL QUERY RULES TO RUNTIME;SAVE MYSQL QUERY RULES TO DISK;

Integration with PXC\Galera – QR

45

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)

VALUES ('192.168.1.5',10,3306,50000);192.168.1.6 … 10000; 192.168.1.7 … 10

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight)

VALUES ('192.168.1.5',11,3306,1);192.168.1.6 … 30000; 192.168.1.7 … 30000

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100);192.168.1.9 … 100;LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Integration with PXC\Galera – HG MW

46

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.5',10,3306,100);

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.5',11,3306,1);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES

('192.168.1.6',11,3306,100);INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.7',11,3306,100);

INSERT INTO mysql_servers (hostname,hostgroup_id,port,weight) VALUES ('192.168.1.8',12,3306,100);192.168.1.9 … 100;INSERT INTO mysql_replication_hostgroups VALUES (10,11);LOAD MYSQL SERVERS TO RUNTIME; SAVE MYSQL SERVERS TO DISK;

Integration with PXC\Galera HG SW

47

ProxySQL is configured Now what?

Example scripthttps://github.com/Tusamarco/proxy_sql_toolsFull guide: https://goo.gl/vcDFZx

Integration with PXC

48

Set the script:INSERT INTO scheduler (id,active,interval_ms,filename,arg1) values (10,0,2000,"/var/lib/proxysql/galera_check.pl","-u=admin -p=admin -h=192.168.1.50 -H=500:W,501:R -P=3310 --execution_time=1 --retry_down=2 --retry_up=1 --main_segment=1 --debug=0 --log=/var/lib/proxysql/galeraLog");LOAD SCHEDULER TO RUNTIME;SAVE SCHEDULER TO DISK;

update scheduler set active=1 where id=10;LOAD SCHEDULER TO RUNTIME;

Integration with PXC

49

+--------------+-------------+--------+------------+---------+

| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+

| 10 | 192.168.1.5 | ONLINE | 50000 | || 10 | 192.168.1.6 | ONLINE | 10000 | |

| 10 | 192.168.1.7 | ONLINE | 10 | |

| 11 | 192.168.1.5 | ONLINE | 1 | || 11 | 192.168.1.6 | ONLINE | 10000 | |

| 11 | 192.168.1.7 | ONLINE | 10000 | || 12 | 192.168.1.8 | ONLINE | 100 | |

| 12 | 192.168.1.9 | ONLINE | 100 | |+--------------+-------------+--------+------------+---------+

Integration with PXC MW

50

+--------------+-------------+--------+------------+---------+

| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+

| 10 | 192.168.1.5 |OFFLINE_SOFT | 50000 | 10_W_11_R_retry_up=0;10_W_11_R_retry_down=0; | 10 | 192.168.1.6 | ONLINE | 10000 | |

| 10 | 192.168.1.7 | ONLINE | 10 | || 11 | 192.168.1.5 | OFFLINE_SOFT | 1 | ... |

| 11 | 192.168.1.6 | ONLINE | 10000 | || 11 | 192.168.1.7 | ONLINE | 10000 | |

| 12 | 192.168.1.8 | ONLINE | 100 | |

| 12 | 192.168.1.9 | ONLINE | 100 | |+--------------+-------------+--------+------------+---------+

Integration with PXC MW

51

+---------------------------------+-------+| Variable_name | Value |+---------------------------------+-------+| wsrep_sst_donor_rejects_queries | ON |+---------------------------------+-------+

ProxySQL Think all is ok

Application writes will fail

Integration with PXC MW

52

+--------------+-------------+--------+------------+---------+

| hostgroup_id | hostname | status | weight | comment |+--------------+-------------+--------+------------+---------+

| 10 | 192.168.1.6 | ONLINE | 10000 | || 10 | 192.168.1.7 | ONLINE | 10 | |

| 11 | 192.168.1.6 | ONLINE | 100000 | |

| 11 | 192.168.1.7 | ONLINE | 100000 | || 12 | 192.168.1.8 | ONLINE | 100 | |

| 12 | 192.168.1.9 | ONLINE | 100 | || 9010 | 192.168.1.5 | ONLINE | 50000 | || 9011 | 192.168.1.5 | ONLINE | 1 | |+--------------+-------------+--------+------------+---------+

Integration with PXC MW

53

WRITE Fail-over is automatically perform by ProxySQL

following the weightUntil an active node exists the HG

Integration with PXC MW

54

WRITE Fail-over will not happen

Until READ_ONLY variableis modified on a reader HG node.

WRITER if DONOR will never be OFFLINE_SOFT

Integration with PXC SW

55

Integration with PXC\Galera SW manual

56

Integration with PXC\Galera SW Auto

57

Change state to:OFFLINE_SOFT = ~2 secHostGroup 9000 = ~2 secProxySQL Shunned = ~1.2 sec (5 fails in 1sec)Failover (auto) = ~3 sec

Integration with PXC - Timing

58

Avoid SPOF

59

Avoid SPOF using Keepalived

60

Avoid SPOF using Keepalived

61

Cost in time

62

Cost in time

63

So what.. 9 0. 0 0 0 % (36 days) MySQL Replication9 9. 9 0 0 % (8 hours) Linux Heartbeat with DRBD9 9. 9 0 0 % (8 hours) RHCS with Shared Storage (Active/Passive)9 9. 9 9 0 % (52 minutes) MHA with at least 3 nodes9 9. 9 9 0 % (52 minutes) Linux Heartbeat with DRBD and Replication9 9. 9 9 0 % (52 minutes) Linux Heartbeat with Shared Storage and Replication9 9 .9 9 5 % (26 minutes) Multi Master (galera - perconacluster) 3 node minimum9 9. 9 9 9 % (5 minutes) MySQL Cluster

64

Cost in $$$

65

Improve service flexibility with r/w split;Allow simple way to manage access to specific servers by

usage (like DW-ETL);Implement sharding;

Allow Query revrite/query fix/blocking without need to modify code;

Improve failover time;Reduce master downtime impact on reads;

Not SPOF (as vip)

Brief summary

66

Integration with MHA

67

68

69

ContactsTo contact Me

Marco.tusa@percona.com

marcotusa@tusacentral.net

To follow me

http://www.tusacentral.net/

http://www.percona.com/blog/

https://www.facebook.com/marco.tusa.94

@marcotusa

http://it.linkedin.com/in/marcotusa/

“Consulting = No mission refused!”