#idug monitoring the essentials in db2 10.5 philip k. gunning gunning technology solutions, llc...

#IDUG

Monitoring the Essentials in DB2 10.5

Philip K. GunningGunning Technology Solutions, LLCSession Code: C03May 13, 2014 3:15 PM – 4:15 PM| Platform: DB2 LUW

#IDUG

DISCLAIMER

• DB2™ is a registered trademark of IBM Corporation. • DB2 BLU, BLU Acceleration are registered trademarks or

copyrights of IBM Corporation• Other logos and product/trade names are registered

trademarks or copyrights of their respective companies

2

#IDUG

The Warp Speed Achieved with BLU Acceleration

3

“For data analytics the volumes involved are usually dramatically larger than cache sizes, which highly limits exploitation of CPU caches. The secret to the warp speed-up achieved in BLU Acceleration is that IBM has designed every algorithm to minimize access to RAM, and maximize processing time in L3 and L2 caches. For the processor shown that provides a whopping 15x-41x speed-up for memoryaccess” (Sam Lightstone, IBM, www.Softwaretradecraft.com)

#IDUG

Overview

• Overview of BLU Acceleration• Concepts• Terminology

• DB2 in the New Galaxy• When RAM is too slow

• Learning to speak and think “Column organized” in BLU Acceleration

• Rules of Thumb for column organized data in terms workload, concurrency, bufferpools, sort heap threshold, sortheap

4

#IDUG

Overview

• Important role of UTIL_HEAP_SZ• New “Column Organized” monitoring metrics• How efficient are my column-organized tables

• Are you getting what you think?• Using MONREPORT reporting module• Tuning Recommendations• Best Practices

5

#IDUG

Quick Look at Prerequisites

• DB2 Workgroup Advanced Enterprise Server Edition• DB2 Advanced Enterprise Server Edtion• OS and Hardware prerequisites

6

#IDUG

7

New Registry Variable Setting for Analytical Workload

• Before creating a new database, set the DB2_WORKLOAD registry variable to ANAYLTICS

• This will result in the following:• Makes column organized tables the default table type

• DFT_TABLE_ORG=COLUMN • Page and extent size configured for analytical processing

• PAGESIZE=32• DFT_EXTENT_SZ=4• DFT_DEGREE=ANY

• Memory for sorting, caching and hashing will be set based on the amount of RAM and server size

• Page and extent size configured for analytical processing• Enables automatic space reclamation• Compression is always on

#IDUG

8

Setting the DB2_WORKLOAD Registry VariableContinued

• Memory for sorting, caching and hashing will be set based on the amount of RAM and server size• CATALOGCACHE_SZ, SHEAPTHRES_SHR and SORTHEAP set to a value

that is optimized for the hardware you are using• Intraquery Parallelism enabled for any workload even if DBM

CFG parameter INTRA_PARALLEL=OFF• The default concurrency threshold on the

SYSDEFAULTMANAGEDSUBCLASS service subclass is enabled to ensure maximum efficiency and utilization of the server

• Automatic Space Reclamation is enabled• AUTO_MAINT=ON• AUTON_REORG=ON

• Page and extent size configured for analytical processing• Enables automatic space reclamation• Compression is always on

#IDUG

9

Setting the DB2_WORKLOAD Registry VariableContinued

• Automatic Space Reclamation is enabled• AUTO_MAINT=ON• AUTO_REORG=ON

• Space reclamation performed for column-organized tables by default

#IDUG

10

Use of AUTOCONFIGURE

• For an existing database that you want to configure for BLU Acceleration • Set the DB2_WORKLOAD=ANALYTICS registry variable and restart the

instance• Connect to the database you want to run autoconfigure on

• Run “db2 autoconfigure apply dbname”• After running autoconfigure, review the bufferpool.

SHEAPTHRES_SHR, SORTHEAP , UTIL_HEAP_SZ and adjust as necessary

#IDUG

11

Use of AUTOCONFIGURE

• In an instance with multiple databases some of which are not using BLU, set the registry variable before you create the databases that will be using BLU, then unset the registry variable after creating the databases that will use BLU

#IDUG

12

Setting SHEAPTHRES_SHR and SORTHEAP Parameters

• When BLU Acceleration is enabled, the self-tuning memory manager (STMM) does not adjust the values of the SHEAPTHRES_SHR and SORTHEAP database configuration parameters automatically.

• Set with static values by the AUTOCONFIGURE command when DB2_WORKLOAD=ANALYTICS registry setting is enabled

#IDUG

13

Setting SHEAPTHRES_SHR and SORTHEAP

• Analytical workloads require larger sort memory areas due to the nature of the workload

• Optimal setting provides degree of concurrency required and associated SORTHEAP sizing

• Setting SHEAPTHRES_SHR to AUTOMATIC not supported for databases using column organized tables

• Tuning SHEAPTHRES_SHR and SORTHEAP as important as tuning BUFFERPOOLS • Aim to prevent sort overflows to disk which can significantly slow down

queries• SET SEHAPTHRES_SHR to 40-50% of the memory available to

the database• DATABASE_MEMORY DB CFG Parameter

#IDUG

14

Setting SHEAPTHRES_SHR and SORTHEAP

• The SORTHEAP DB CFG parameter controls the amount of memory an application can use for SORTS

• SORTHEAP memory comes out of total SHEAPTHRES_SHR memory

• Size the SHEAPTHRES_SHR DB CFG parameter and SORTHEAP DB CFG parameter to provide the amount of concurrency desired and to prevent sort overflows

• Rule of thumb for sizing SORTHEAP is set it to 5-20% of SHEAPTHRES_SHR

• Monitor for sort overflows and post threshold sorts via MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table functions

#IDUG

15

Monitoring SHEAPTHRES_SHR and SORTHEAP

• Monitor for sort overflows and post threshold sorts via MON_GET_DATABASE and MON_GET_UNIT_OF_WORK table functions

• Adjust SHEAPTHRES_SHR upward and SORTHEAP upward gradually to eliminate sort overflows• Don’t increase SORTHEAP without concomitant adjustment to

SORTHEAP• Can decrease SORTHEAP independently

• Mine the DYNAMIC SQL and target queries with high number of overflows• Good way to train a DBA in tuning SQL or identifying proper indexes to

eliminate SORTS

#IDUG

16

Setting the STMTHEAP DB CFG Parameter

• Since analytic queries are more complex than OLTP type queries, they can benefit from an increased STMTHEAP DB CFG size

• The STMTHEAP DB CFG parameter specifies the size of the workspace used by the SQL compiler during compilation of an SQL statement

#IDUG

17

Importance of Setting UTIL_HEAP_SZ Correctly

• For LOAD operations and for building compression dictionaries UTIL_HEAP_SZ must be sufficiently sized

• Set to AUTOMATIC if DB2_WORKLOAD=ANALYTICS • If DATABASE_MEMORY set to AUTOMATIC should be sufficient

as it will grow as necessary• If not sized sufficiently, can result in less than optimal

compression

#IDUG

18

Monitoring Adequacy of UTIL_HEAP_SZ setting

• Check the new PCTENCODED column in the SYSCAT.COLUMNS catalog view which is the the percentage of values that are encoded as a result of compression for a column in a column-organized table.

• If the overall compression ratio for you’re the table in is too low, check this statistic to see if values in specific columns were left uncompressed. • Many columns with a very low value (or even 0) can be an indicator that

utility heap might have been too small when the column compression dictionaries were created

• You might also see very low values for columns that were incrementally loaded with data that was outside of the scope of the column compression dictionary

#IDUG

19

So, How are your Column-organized Tables Performing?

• Use monitor elements returned by the MON_GET_TABLE table function to determine the efficiency of column-organized tables and to identify candidate tables suitable to be organized by column

#IDUG

20

New Column-organized Metrics from MON_GET_TABLE table function• Use monitor elements returned by the MON_GET_TABLE table

function to determine the efficiency of column-organized tables and to identify candidate tables suitable to be organized by column

• By dividing the NUM_COLUMNS_REFERENCED monitor element by the SECTION_EXEC_WITH_COL_REFERENCES monitor element, you can determine the average number of columns that a particular query accesses. If this average is much less than the number of columns in the table, the workload favors column organization.

• Controlled by the mon_obj_metrics DB CFG parameter setting• Extended is the default and is required to return column-organized

metrics

#IDUG

Monitor Collection DB CFG Parameters (Review)• Mon_act_metrics – controls collection of activity level monitor elements on the entire

database (DEFAULT – BASE)• MON_GET_ACTIVITY_DETAILS • MON_GET_PKG_CACHE_STMT • Activity event monitor (DETAILS_XML monitor element in the event_activity logical data

groups) • Mon_deadlock – controls generation of deadlock events on the entire database (DEFAULT-

WITHOUT_HIST)• Mon_locktimeout – controls generation of lock timeout events on the entire database

(DEFAULT – NONE)• Mon_lockwait – controls generation of lock wait events for the lock event monitor (DEFAULT –

NONE)• Mon_lw_thresh – the amount of time spent in lock wait before an event for mon_lockwait is

generated (DEFAULT - 5000000)• Mon_obj_metrics – controls collection of data object monitor elements on the entire database

(DEFAULT- EXTENDED *formerly BASE) • MON_GET_TABLE

#IDUG

22

Monitor Collection DB CFG Parameters (Review)

• Mon_req_metrics – controls the collection of request monitor elements on the entire database (DEFAULT – BASE)• MON_GET_UNIT_OF_WORK • MON_GET_UNIT_OF_WORK_DETAILS • MON_GET_CONNECTION • MON_GET_CONNECTION_DETAILS • MON_GET_SERVICE_SUBCLASS • MON_GET_SERVICE_SUBCLASS_DETAILS • MON_GET_WORKLOAD • MON_GET_WORKLOAD_DETAILS • Statistics event monitor (DETAILS_XML monitor element in the event_wlstats

and event_scstats logical data groups) • Unit of work event monitor

• Mon_uow_data – controls the generation of UOW events at the database level for the UOW event monitor (DEFAULT – NONE)

#IDUG

23

Monitoring and MONREPORT Reporting Enhancements

• MONREPORT.DBSUMMARY and MONREPORT.CONNECTION reporting modules enhanced to support on column processing

• New monitor elements to assess bufferpool efficiency• Monitor elements contain COL in the monitoring element name• New monitor elements to monitor prefetch performance for data in column

organized tables• New monitor elements to measure column data size

• ADMIN_GET_TAB_INFO

• New monitor elements to measure time spent• New monitor elements for the HASHED GROUP BY operator• New monitor interfaces to get request metrics that are aggregated along different

dimensions• New monitor interfaces to get activity metrics• Monitor interfaces to get database object metrics• Refer to Database Monitoring Guide and Reference for complete details

#IDUG

24

New CTQ Optimization Operator

• Use to denote operations on column organized tables• Most processing should be below the CTQ operator in the

access path (db2exfmt, db2explain)• Optim Performance Manager• Optim Query Tuner

• If many operations above this operator indicator of inefficient use of column organized table

• Re-look how table is being used

#IDUG

25

Reviewing time spent in column processing versus row processing with new monitor elements

• New monitoring elements returned from MON_GET_ACTIVITY_DETAILS tabe function to show where time is being spent

• Total_app_section_excutions• Total_col_executions• Total_col_proc_time• Total_col_time• Total_cpu_time

#IDUG

Understanding Automatic Space Reclamation

• Space reclamation for column-organized tables• Space reclamation after deletions

• After a delete operation the row is logically deleted• Space that was used by the deleted row is not available to subsequent

transactions and remains unusable until space reclamation occurs• REORG TABLE is required to reclaim the space

#IDUG

Understanding Automatic Space Reclamation

• Space reclamation after updates• After a delete operation the row is first deleted and a new copy of the

row inserted at the end of the table• Updated row uses space in proportion to the number of times that the

row is updated until space reclamation occurs• All rows in the extent where the update took place must be deleted

before any space reclamation can occur• REORG TABLE is required to reclaim the space

#IDUG

Understanding Automatic Space Reclamation

• If DB2_WORKLOAD=ANALYTICS then AUTO_REORG enabled• Define an automatic reorg online and/or offline policy using the

AUTOMAINT_SET_POLICY or AUTOMAINT_SET_POLICYFILE stored procedures with the appropriate parameters

• Offline reorg considerations• Need appropriate window• Reclaims space• Faster than online REORG

• Sample POLICYFILE in SAMPLE DB2 directory

#IDUG

Automatic Space Reclamation

• Online REORG considerations• Still need appropriate window• Schedule ONLINE REORG window during period of low activity• Reclaims space by specifying

• Specify RECLAIM EXTENTS option of REORG command to reclaim space

• To determine when a table REORG is necessary, use the ADMIN_GET_TAB_INFO table function to review an estimate of the amount of reclaimable space to reclaim

• Monitor REORG status with db2pd or SNAPTAB_REORG administrative view

#IDUG

Summary• DB2 BLU New Column-organized capabilities • Use of AUTOCONFIGURE• Review of DB2_WORKLOAD=ANALYTICS registry setting• SHEAPTHRES_SHR and SORTHEAP setting discussion and

recommendations• Table monitoring metrics• Monitoring Enhancements and some review• Activity detail monitoring elements• New Monitoring Elements• Importance of adequate UTIL_HEAP_SZ• REORG Considerations• Space Usage and Reclamation

#IDUG

References

• Tuning and Monitoring System Performance, located at https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Tuning%20and%20Monitoring%20Database%20System%20Performance

• DB2 10.5 Database Monitor Guide and Reference, http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2Monitoring-db2f0e1050.pdf

• Best Practices, Optimizing Analytical Workloads using DB2 10.5 with BLU Acceleration, https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Wc9a068d7f6a6_4434_aece_0d297ea80ab1/page/Optimizing%20analytic%20workloads%20using%20DB2%2010.5%20with%20BLU%20Acceleration/comment/1dc59ee4-8c70-43f9-9d36-fba334b95891

#IDUG

References

• DB2 BLU Acceleration, Rapid Adoption Guide, http://www.ibm.com/developerworks/data/library/techarticle/dm-1309db2bluaccel/

• When RAM is too Slow, http://softwaretradecraft.com/when-ram-is-too-slow-how-dynamic-in-memory-processing-changes-the-game-for-data-analytics/

• DB2 10.5, What’s New, http://public.dhe.ibm.com/ps/products/db2/info/vr105/pdf/en_US/DB2WhatsNew-db2q0e1050.pdf

#IDUG

Philip K. GunningGunning Technology Solutions, LLCpgunning@gts1consulting.comMonitoring the Essentials in DB2 10.5DB2 LUW Please fill out your session

evaluation before leaving!