244 pgconf12 oltp benchmarks
Post on 05-Apr-2018
226 Views
Preview:
TRANSCRIPT
-
7/31/2019 244 PGConf12 OLTP Benchmarks
1/34
2010 VMware Inc. All rights reserved
OLTP Performance Benchmark Review
Jignesh Shah
Product Manager, vFabric Postgres
VMware, Inc
-
7/31/2019 244 PGConf12 OLTP Benchmarks
2/34
2 2012 Copyright VMware Inc
About Me
Currently Product Manager of vFabric Postgres Performance Engineer , vFabric Data Director Previously with Sun Microsystems (2000-2010) Team Member that delivered the first published mainstream
benchmark with PostgreSQL
Blog at : http://jkshah.blogspot.com
-
7/31/2019 244 PGConf12 OLTP Benchmarks
3/34
3 2012 Copyright VMware Inc
Agenda
Introduction pgbench Sysbench Dbt2 BenchmarkSQL
DVDStore A New benchmark
-
7/31/2019 244 PGConf12 OLTP Benchmarks
4/34
4 2012 Copyright VMware Inc
Introduction
Why do we need benchmarks? Reference data points Stress Test for Too Big to Fail scenarios
Uses of Benchmark Improve Product Quality
Understand code path usage
Performance Characteristics Baseline metrics (Reference points)
Release to releaseAgainst other technologies to do same business operation
Abuses of Benchmark Benchmarketing Fixated only on ones that are favorable
-
7/31/2019 244 PGConf12 OLTP Benchmarks
5/34
5 2012 Copyright VMware Inc
pgbenchPostgreSQL
-
7/31/2019 244 PGConf12 OLTP Benchmarks
6/34
6 2012 Copyright VMware Inc
Pgbench
Based on TPC-B workload (circa 1990) Not an OLTP but stress benchmark for database Ratio is Branches: 10 Tellers: 100,000 Accounts Default TPC-B sort-of
Account transactions also impact teller and branch balances Branch table becomes the biggest bottleneck
Branches
Tellers Accounts
History
-
7/31/2019 244 PGConf12 OLTP Benchmarks
7/34
7 2012 Copyright VMware Inc
Pgbench
Hints PGSSLMODE disable (Unless you want to factor SSL communication
overhead. Depending on your distribution )
-M prepared (unless you want to measure overhead of parsing) Various modes of benchmark Default TPC-B sort-of
Account transactions also impact teller and branch balances Branch table becomes the biggest bottleneck
-N Simple Update (with select, insert)Account Update, Select balance. History insertAccount table update becomes the biggest bottleneck
-S read only testAccessShareLock on Accounts table and primary index becomes the
bottleneck
Fixed in 9.2 (Thanks Robert Haas)
-
7/31/2019 244 PGConf12 OLTP Benchmarks
8/34
8 2012 Copyright VMware Inc
PGBench Select Test
0
10000
20000
30000
40000
50000
60000
70000
80000
0 20 40 60 80 100 120
Transactions
Pe
rSecond
Number of Clients
PGBench - Select Test
PG9.1 PGBench Select
9.2 PGBench Select
-
7/31/2019 244 PGConf12 OLTP Benchmarks
9/34
9 2012 Copyright VMware Inc
PGBench TPC-B Like Test
0
1000
2000
3000
4000
5000
6000
7000
8000
0 20 40 60 80 100 120
Transactions
Pe
rSecond
Number of Clients
PGBench (TPC-B Like)
PG9.1 TPC-B Like
9.2 TPC-B Like
-
7/31/2019 244 PGConf12 OLTP Benchmarks
10/34
10 2012 Copyright VMware Inc
SysbenchMySQL
-
7/31/2019 244 PGConf12 OLTP Benchmarks
11/34
11 2012 Copyright VMware Inc
sysbench
Originally developed to test systems Has an OLTP component which was based on MySQL Creates a table sbtest with a pimary key. Various Modes of OLTP operation Simple Read Only (web site primary key lookup)
Complex Read Only Complex Read Write test
-
7/31/2019 244 PGConf12 OLTP Benchmarks
12/34
12 2012 Copyright VMware Inc
Sysbench OLTP Simple Read
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
0 20 40 60 80 100 120
Transactions
PerSecond
Number of Clients
Sysbench Simple Read
OLTP Simple Read
9.2 OLTP Simple Read
-
7/31/2019 244 PGConf12 OLTP Benchmarks
13/34
13 2012 Copyright VMware Inc
Sysbench OLTP Complex Read
0
1000
2000
3000
4000
5000
6000
0 20 40 60 80 100 120
TransactionsP
erSecond
Number of Clients
Sysbench Complex Read
OLTP Complex Read
9.2 OLTP Complex Read
-
7/31/2019 244 PGConf12 OLTP Benchmarks
14/34
14 2012 Copyright VMware Inc
Sysbench OLTP Complex Read/Write
0
500
1000
1500
2000
2500
3000
0 20 40 60 80 100 120
Transactions
Pe
rSecond
Number of Clients
Sysbench OLTP Complex R/W
OLTP Complex Write
9.2 OLTP Complex Write
-
7/31/2019 244 PGConf12 OLTP Benchmarks
15/34
15 2012 Copyright VMware Inc
Sysbench Complex R/W Note
In 9.0 it was impossible to run sysbench complex r/w withouthitting error- ERROR: duplicate key value violates unique constraint "sbtest_pkey"
In 9.1 SSI was introduced and occurrence went down drastically In 9.2 havent encountered the occurence
-
7/31/2019 244 PGConf12 OLTP Benchmarks
16/34
16 2012 Copyright VMware Inc
dbt2
-
7/31/2019 244 PGConf12 OLTP Benchmarks
17/34
17 2012 Copyright VMware Inc
Dbt2 -
Fair Use implementation of TPC-C Implemented using C stored procedures using
driver->client->database server architecture
Nine Tables Five Transactions
New-Order (NOTPM) 45% Payment 43% Delivery 4% Order Status 4% Stock Level 4%
Warehouses Stock
ItemDistrict
Customer
History
Orders
Order Lines
New Orders
-
7/31/2019 244 PGConf12 OLTP Benchmarks
18/34
18 2012 Copyright VMware Inc
Dbt2 -
Why is it not TPC-C compliant? Not audited by TPC No Terminal emulator Official kit requires commercial Transaction Manager Doesnt cover ACID tests
Two versions Available Libpq ODBC
One potential problem is 3 network roundtrips per transactionwhich causes Idle in Transaction at high load BEGIN, SELECT StoredProcedure() , END pattern of transactions
-
7/31/2019 244 PGConf12 OLTP Benchmarks
19/34
19 2012 Copyright VMware Inc
Dbt2 Postgres 9.1
Cached Runs (data in bufferpool)
Short runs (limited checkpoint
and vacuum impacts)
NOTPM = 45% of all DB TransDB Trans rate about 3000 TPS
-
7/31/2019 244 PGConf12 OLTP Benchmarks
20/34
20 2012 Copyright VMware Inc
BenchmarkSQL
-
7/31/2019 244 PGConf12 OLTP Benchmarks
21/34
21 2012 Copyright VMware Inc
BenchmarkSQL-
Another implementation using TPC-C schema Implemented using JDBC Nine Tables Five Transactions
New-Order (NOTPM) 45% Payment 43% Delivery 4% Order Status 4% Stock Level 4%
Surprisingly can do better than dbt2 implementation but still hasidle in transactions which means bottlenecked at network/clientlevel
Warehouses Stock
ItemDistrict
Customer
History
Orders
Order Lines
New Orders
-
7/31/2019 244 PGConf12 OLTP Benchmarks
22/34
22 2012 Copyright VMware Inc
DVDStore
-
7/31/2019 244 PGConf12 OLTP Benchmarks
23/34
23 2012 Copyright VMware Inc
DVDStore
Implementation of Online DVD Store Postgres support contributed by VMware Implemented using various stacks
JSP/Java/JDBC (supports Postgres) Linux/Apache/PHP/MySQL (supports Postgres)ASP.NET (not yet implemented for Postgres) Stored Procedures (supports Postgres via Npgsql)
Eight Tables Main Transactions
New-Customers 0-10% (configurable) Customer Login DVD Browse (By category, by actor, by title) Purchase Order (Metric Orders Per Minute) Stock ReOrder (via Triggers)
Customers Cust_hist
InventoryProduct
Categories
Reorder
Orders
Order Lines
-
7/31/2019 244 PGConf12 OLTP Benchmarks
24/34
24 2012 Copyright VMware Inc
DVDStore
JSP/Java JDBC Implementation Tomcat may need tuning
PHP-Postgres Implementation Suffers from one connection per SQL command Needs pg_bouncer (on same server as web server) and configure local
connections to pg_bouncer which does connection caching to actual Postgres
server
Stored Procedure Implementation Fastest Implementation (> 100,000 orders per minute) Idle in transactions can still occur.
Metric is Orders Per Minute DB Transactions = (6-7 * OPM/60) ~ 10K 11K TPS
-
7/31/2019 244 PGConf12 OLTP Benchmarks
25/34
25 2012 Copyright VMware Inc
TPC-E/V
-
7/31/2019 244 PGConf12 OLTP Benchmarks
26/34
26 2012 Copyright VMware Inc
Genesis of TPC-V
Users are demanding benchmarks to measure performance ofdatabases in a virtual environment
Existing virtualization benchmarks model consolidation: Many VMs Small VMs Non-database workloads
TPC is developing a benchmark to satisfy that demand: TPC-VAn OLTP workload typical of TPC benchmarks Fewer, larger VMs Cloud characteristics:
Variability: mix of small and large VMs Elasticity: load driven to each VM varies by 10X
-
7/31/2019 244 PGConf12 OLTP Benchmarks
27/34
27 2012 Copyright VMware Inc
Benchmark requirements
Satisfies the industry need for a benchmark that: Has a database-centric workloadStresses virtualization layer
Moderate # of VMs, exercising enterprise applications Healthy storage and networking I/O content; emphasizes I/O in a virtualized
environment
NOTmany app environments in an app consolidation scenario Timely development cycle (1-2 years)
Based on the TPC-E benchmark and borrows a lot from it
-
7/31/2019 244 PGConf12 OLTP Benchmarks
28/34
28 2012 Copyright VMware Inc
What is TPC-E
TPC-E is theTPCs latest OLTP benchmark More complex than TPC-C Less I/O than TPC-CA lot of the code is TPC-supplied
Models a brokerage firmCustomers Brokers Market
READ-WRITE
Market-Feed
Trade-Order
Trade-Result
Trade-Update
Security-Detail
Trade-Lookup
Trade-Status
READ-ONLY
Broker-Volume
Customer-Position
Market-Watch
Invoke the following transactions
against the following data
CustomerData Brokerage Data Market Data
Customers Brokers Market
READ-WRITE
Market-Feed
Trade-Order
Trade-Result
Trade-Update
Security-Detail
Trade-Lookup
Trade-Status
READ-ONLY
Broker-Volume
Customer-Position
Market-Watch
READ-WRITE
Market-Feed
Trade-Order
Trade-Result
Trade-Update
Security-Detail
Trade-Lookup
Trade-Status
READ-ONLY
Broker-Volume
Customer-Position
Market-Watch
Invoke the following transactions
against the following data
CustomerData Brokerage Data Market Data
Abstraction of the Functional Components in an OLTP
-
7/31/2019 244 PGConf12 OLTP Benchmarks
29/34
29 2012 Copyright VMware Inc
Abstraction of the Functional Components in an OLTP
Environment
User
InterfacesNetwork
Network
Database
Services
Modeled Business
Application
And
Business Logic
Services
Presentation
Services
Market
Exchange Legend
Customer
Sponsor ProvidedStock Market
User
InterfacesNetwork
Network
Database
Services
Modeled Business
Application
And
Business Logic
Services
Presentation
Services
Market
Exchange
User
InterfacesNetwork
Network
Database
Services
Modeled Business
Application
And
Business Logic
Services
Presentation
Services
Market
Exchange Legend
Customer
Sponsor ProvidedStock Market
LegendLegend
Customer
Sponsor ProvidedStock Market
Customer
Sponsor ProvidedStock Market
-
7/31/2019 244 PGConf12 OLTP Benchmarks
30/34
30 2012 Copyright VMware Inc
Functional Components ofTPC-E Test Configuration
Sponsor
Provided
Frame Implementation
Database Logic
EGenTxnHarness Connector
TPC-E Logic and Frame Calls
EGenDriver Connector
EGenDriverCE
CE
CE
EGenDriverMEE
MEE
MEE
EGenDriverDM
DM
DM
Driving and Reporting
EGenDriver
EGenTxnHarness
Commercial
Product
Sponsor
Provided
Sponsor
Provided
Sponsor
Provided
TPC Defined
Interfaces
TPC Defined
Interface
DBMS
Network
Commercial Product
TPC Provided
Sponsor Provided
TPC Defined
Interface
Legend
Database Interface
Sponsor
Provided
Frame Implementation
Database Logic
EGenTxnHarness Connector
TPC-E Logic and Frame Calls
EGenDriver Connector
EGenDriverCE
CE
CE
EGenDriverMEE
MEE
MEE
EGenDriverDM
DM
DM
Driving and Reporting
EGenDriver
EGenTxnHarness
Commercial
Product
Sponsor
Provided
Sponsor
Provided
Sponsor
Provided
TPC Defined
Interfaces
TPC Defined
Interface
DBMS
Network
Commercial Product
TPC Provided
Sponsor Provided
TPC Defined
Interface
Legend
Commercial Product
TPC Provided
Sponsor Provided
TPC Defined
Interface
Legend
Database Interface
-
7/31/2019 244 PGConf12 OLTP Benchmarks
31/34
31 2012 Copyright VMware Inc
How does this all matter to the PostgreSQL community?
TPC is developing a benchmarking kitfor TPC-V First time TPC has gone beyond publishing a functional specificationFull, end-to-end functionality Publicly available kit
Produces the variability and load elasticity properties of the benchmark Users need not worry about complexities of simulating cloud characteristics
Runs against an open source databaseA reference kit; companies are allowed to develop their own kit
Anyone can install the kit and pound on the server with a clouddatabase workload
Removes the high cost of entry typical to TPC benchmarks The reference kit will run on PostgreSQL
ODBC interface allows running the workload against other databases Tentative plans to also release a TPC-E kit
We started out with a kit to run TPC-E; now adding the TPC-V properties
-
7/31/2019 244 PGConf12 OLTP Benchmarks
32/34
32 2012 Copyright VMware Inc
Our dependence on PostgreSQL
This reference kit will be a very successful new benchmark But only if its performance on the open source database is at least decent
compared to commercial databases
PostgreSQL can benefit a lot from being the reference database for amajor new benchmark
But only its performance is decent! Running the TPC-Eprototype on PGSQL 8.4 on RHEL 6.1, we are at
~20% of published TPC-E results
Very early results Current testbed is memory challenged Good news: Query plans for the 10 queries implemented look good Long, mostly-read queries => issue is the basic execution path, not redo log,
latch contention, etc.
B h k D l t St t
-
7/31/2019 244 PGConf12 OLTP Benchmarks
33/34
33 2012 Copyright VMware Inc
Benchmark Development Status
TPC-V Development Subcommittee 9 member companies 3-4 engineers working actively on the reference kit On version 0.12 of the draft spec
Worked through a lot of thorny issues Betting the farm on the reference kit
But if we produce a good kit, TPC-V will be an immediate success We expect to make a kit available to member companies in Q3 or
Q4
Bottom line: Cooperating to make the TPC-E/TPC-V reference kits
run well on PostgreSQL will greatly benefit all of us
-
7/31/2019 244 PGConf12 OLTP Benchmarks
34/34
34 2012 Copyright VMware Inc
Thank You
top related