scaling mysql writes through partitioning - ipc spring edition
TRANSCRIPT
The ProblemThe Tests
Breakthroughs
Scaling MySQL writes through partitioning
Philip Tellis / [email protected]
IPC Spring – 2010-06-01 – Berlin
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
$ finger philip
Philip Tellisgeekyahoo@bluesmoonhttp://bluesmoon.info/[email protected]
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Web requests
Millions of beacons from a web pageNo response requiredCan be batch processedVery small amounts of data loss is acceptable
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Large volume
2000 requests/second on most daysup to 8000 requests/second on some days200MM requests/daySome data is fake or abusive
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Access patterns
Lots of writes throughout the dayOne huge read at the end of the daySummarise data and throw out the detailsMany reads of summary data over several months
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Why not use a data warehouse?
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
I like to get the most out of my hardware
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Hardware setup
MySQL 5.1Multi-master replication in two colos, 1 remote slave permasterOnly one master writable at any point of time4GB RAM (later 16GB), Big disk with RAID 10RAID 10 7200rpm 6x1TB
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
DB config
innodb_buffer_pool_size = 2078Minnodb_flush_log_at_trx_commit = 1innodb_log_buffer_size = 8Minnodb_max_dirty_pages_pct = 90innodb_doublewrite = 1, innodb_support_xa = 1sync_binlog = 0key_buffer_size = 32M, myisam_sort_buffer_size = 512ktransaction_isolation = REPEATABLE-READ
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Data setup
Each row 120bytes+ InnoDB overheadinnodb_file_per_table so we can see how the table growsNo Autoincrement fieldsPRIMARY KEY derived from data + one other index
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Schema
page identifier - INTtimestamp - INTone-way hash of IP - CHAR(32)page performance informationrequest countryuseragent
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test requirements
Insert records until the system breaks downFind out why it broke downFind out how to make it not break downFind out how fast we can insert records (must be >2000 i/s)
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
How I tested
Insertion script measured insertion speed v/s number ofrecordsNumber of records roughly translates to table sizeOn DB box we measure disk performance and table size
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Our dataDB infrastructurePerformance
Test 1
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 2 - Drop the secondary index
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 3 - innodb_buffer_pool_size=1000
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Realisation
Max table size directly proportional toinnodb_buffer_pool_size
Extra index reduces insertion rateExtra index reduces max table sizePossible solution: increase RAM andinnodb_buffer_pool_size
But this only postpones the problem
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Realisation
Max table size directly proportional toinnodb_buffer_pool_size
Extra index reduces insertion rateExtra index reduces max table sizePossible solution: increase RAM andinnodb_buffer_pool_size
But this only postpones the problem
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 4 - innodb_flush_log_at_trx_commit=2
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 5 - innodb_max_dirty_pages_pct=60
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 6 - Let’s try MyISAM
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Test 7 - Inserts in a transaction
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Other stuff we tried
innodb_doublewrite=0 – no effectServer side prepared statements – no effecttransaction_isolation=READ-COMMITTED – no effectinnodb_support_xa=0 – 12% increase in insertion rateCombination of the best options – negligible effect
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
What we knew at this point
Sticking with InnoDBWe need a large buffer poolWe need to drop extra indicesflush_log_at_trx_commit = 2 is good enoughTransactions are good
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Basic testsGoing crazyInsights
Our big problem
Insert rate was barely reaching the rate of incoming data!Still breaks down before getting a day’s worth of data
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 8 - Single bulk insert
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Bulk insert specifications
40,000 records in one insert statementUse INSERT IGNORE4-6 seconds per statementPRIMARY KEY drops duplicatesWe still have a breakdown when we cross the buffer pool
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Handling insert failures
Typically if one record is bad, the entire insert failsINSERT IGNORE solves this problemBonus is easy recovery from hardware/network failures
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 9 - bulk inserts + partitioning
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
What happened?
Split the table into partitionsEach partition < 0.5× innodb_buffer_pool_sizecurrent and next partition fit in memory at any timePartition key is based on incoming data and not onSELECTs
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Schema
CREATE TABLE (...
) PARTITION BY RANGE( ( time DIV 3600 ) MOD 24 ) (Partition p0 values less than (2),Partition p1 values less than (4),...Partition p10 values less than (22),Partition p11 values less than (24)
);
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Test 10 - Ran for 7 days
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
Still running
Terabytes of dataaround 8500 inserts per secondPotentially 700+ MM inserts per day
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Bulk insertsPartitioningLong running test
A Note on read performance
Data volume before and after is very differentCurrent read rate is approx 8000 records per second
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
What about a key-value store?
Some summarisation queries are complex but notimpossible with a k/v storeSummary data is still relational in natureAvoid sharing resources (RAM/CPU) between twoseparate data storesWe have not yet discarded this idea
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Lots of critics
The title should be "how to get poor performanceby using a completely inappropriate tool"
Dude, that’s a really impressive bit of engineering.However, you failed the interview.
it’s just another case of RDBMS blinding peoplefor all other possible solutions
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Summary
Bulk inserts push up your insert ratePartitioning lets you insert more recordsPartition based on incoming data key for fast insertshttp://tech.bluesmoon.info/2009/09/scaling-writes-in-mysql.html
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
Photo credits
Disused warehouse on Huddersfield Broad Canal / by TDR1http://www.flickr.com/photos/tdr1/3578203727/
Hardware store dog / by sstrudeauhttp://www.flickr.com/photos/sstrudeau/330379020/
North Dakota, Broken Down Van / by mattdentehttp://www.flickr.com/photos/mattdente/46944898/
One red tree / by EssjayNZhttp://www.flickr.com/photos/essjay/155223631/
The Leaning Tree / by stage88http://www.flickr.com/photos/stage88/3179612722/
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning
The ProblemThe Tests
Breakthroughs
contact me
Philip Tellisyahoogeek@bluesmoonhttp://bluesmoon.info/slideshare.net/[email protected]
IPC Spring – 2010-06-01 – Berlin Scaling MySQL writes through partitioning