pitfalls& surpriseswithdbms stats: howtosolvethem · a short history of dbms_stats 5 06/13/2018...

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

@dani_schnider danischnider.wordpress.com

Pitfalls & Surprises with DBMS_STATS:How to Solve ThemDani Schnider, Trivadis AG

Dani Schnider

3

Working for Trivadis in Zurich/Switzerland– Senior Principal Consultant– Data Warehouse Lead Architect– Trainer of several CoursesCo-Author of the books– Data Warehousing mit Oracle– Data Warehouse BlueprintsCertified Data Vault Data ModelerOracle ACE

06/13/2018 Pitfalls and Surprises with DBMS_STATS


Pitfalls and Surprises with DBMS_STATS4 06/13/2018

DBMS_STATS

A Short History of DBMS_STATS


Oracle 8i Gather / Delete Object StatisticsSet / Get Statistic ValuesExport / Import Statistics

Oracle 9i Gather / Delete System Statistics

Oracle 10g Automatic Statistics Gathering JobDictionary / Fixed Objects StatisticsLocking StatisticsStatistics HistorySet / Get Parameters

Oracle 11g Statistics PreferencesPending StatisticsExtended StatisticsIncremental Statistics

Oracle 12c Additional PreferencesNew Synopsis Calculation (12.2)Optimizer Statistics Advisor (12.2)


Histograms

Histograms: Statistics about Data Distribution


Useful for columns with skewed data distribution

Histogram contains multiple buckets

– Until Oracle 11g: ≤ 254

– Since Oracle 12c: ≤ 2048

Different histogram types

– Frequency histograms

– Height-balanced histograms

– Top-frequency histograms

– Hybrid histograms

2

200472

1

10

100

1000

10000

100000

1000000

AD BG CH DK GB HT JP MA NC PE RW SY UY

Country

Rows

Histograms: Parameter METHOD_OPT


dbms_stats.gather_table_stats( ownname => USER, tabname => 'ADDRESSES', method_opt => 'FOR ALL COLUMNS SIZE SKEWONLY');

Parameter to define histograms

SIZE: number of buckets– 1..254 resp. 1..2048

– REPEAT

– AUTO

– SKEWONLY

Example:

Histograms: Real-Life Example


DBMS_STATS.gather_schema_stats( ownname => 'APPL_USER', estimate_percent => 5, method_opt => 'FOR ALL INDEXED COLUMNS SIZE 254', cascade => TRUE, degree => DBMS_STATS.default_degree);

What is wrong here?

Histograms: Real-Life Example


DBMS_STATS.gather_schema_stats( ownname => 'APPL_USER', estimate_percent => 5, method_opt => 'FOR ALL INDEXED COLUMNS SIZE 254', cascade => TRUE, degree => DBMS_STATS.default_degree);

What is wrong here?

DBMS_STATS.gather_schema_stats( ownname => 'APPL_USER', estimate_percent => 5 DBMS_STATS.auto_sample_size, method_opt => 'FOR ALL INDEXED COLUMNS SIZE 254 SKEWONLY', cascade => TRUE, degree => DBMS_STATS.default_degree);

Histograms: Lessons Learned


Avoid FOR ALL INDEXED COLUMNS(or column statistics will be missing)

Number of buckets can be less then SIZE number for frequency histograms

SIZE AUTO (Default) works only on “used” columns

Use SIZE SKEWONLY if possible

Oracle 12c:

Use AUTO_SAMPLE_SIZE for new histogram types

SIZE AUTO / SKEWONLY still create maximum of 254 buckets


Gathering Statistics in ETL Jobs

Statistics Gathering in ETL Job


DBMS_STATS.gather_table_stats(ownname => 'DWH',tabname => 'T1');T1

T2

T3 DBMS_STATS.gather_table_stats(ownname => 'DWH',tabname => 'T2');

| 1 | INSERT STATEMENT | | 1500 || 2 | INSERT | T3 | 1500 || 3 | HASH JOIN | | 1500 || 4 | TABLE ACCESS FULL| T1 | 2000 || 5 | TABLE ACCESS FULL| T2 | 3000 |

ETL Job

T1

2000

T2

3000

T3

1500

Statistics Gathering in ETL Job – The Next Day...



T2


| 1 | INSERT STATEMENT | | 2 || 2 | INSERT | T3 | 2 || 3 | NESTED LOOP | | 2 || 4 | TABLE ACCESS FULL| T1 | 2 || 5 | TABLE ACCESS FULL| T2 | 3500 |

ETL Job

T1

2

T2

3500

T3

Statistics Gathering in ETL Job – Rerun



T2


| 1 | INSERT STATEMENT | | 2 || 2 | INSERT | T3 | 2 || 3 | NESTED LOOP | | 2 || 4 | TABLE ACCESS FULL| T1 | 2 || 5 | TABLE ACCESS FULL| T2 | 3500 |

ETL Job

T1

2500

T2

3500

Why is the Old Execution Plan Used?


no_invalidate Does not invalidate the dependent cursors if set to TRUE. The procedure invalidates the dependent cursors immediately if set to FALSE. UseDBMS_STATS.AUTO_INVALIDATE to have Oracle decide when to invalidatedependent cursors. This is the default.

DBMS_STATS.gather_table_stats(ownname => 'STG',tabname => 'T1',no_invalidate => FALSE);

Oracle® Database PL/SQL Packages and Types Reference 11g Release 2 (11.2)

DBMS_STATS.gather_table_stats(ownname => 'STG',tabname => 'T2',no_invalidate => FALSE);

Gathering Statistics in ETL Jobs: Lessons Learned


Object statistics should be gathered as part of every ETL job

– Loaded table can be used as input for next ETL jobs

– Disable automatic gathering job when ETL runs in the night

Default behavior of NO_INVALIDATE not suitable for ETL jobs– Recommendation: no_invalidate => FALSE

Oracle 12c:

Online Statistics Gathering used for

– CREATE TABLE AS SELECT

– Direct-Path INSERT into empty table (after TRUNCATE)


Incremental Statistics

Global Statistics

Global statistics are essential for good execution plans

– num_distinct, low_value, high_value, density, histogramsGathering global statistics is time-consuming

– All partitions must be scanned

Jan 18 Feb 18 Mar 18 Apr 18 Mai 18 Jun 18 Jul 18 Aug 18 Oct 18Sep 18

gather statisticsfor current partition

gather global statistics Data Dictionary

06/13/2018 Pitfalls and Surprises with DBMS_STATS19

Incremental Global Statistics

Synopsis-based gathering of statistics (since Oracle 11g)

For each partition a synopsis is stored in SYSAUX tablespace

– Statistics metadata for partition and columns of partition

Global statistics by aggregating the synopses from each partition

Jan 18 Feb 18 Mar 18 Apr 18 Mai 18 Jun 18 Jul 18 Aug 18 Oct 18Sep 18

gather statisticsfor current partition

gatherincremental

globalstatistics

synopsis

06/13/2018 Pitfalls and Surprises with DBMS_STATS20

Synopsis Tables


Two tables in SYSAUX tablespace:

WRI$_OPTSTAT_SYNOPSIS_HEAD$

– One row per table/partition/column

WRI$_OPTSTAT_SYNOPSIS$

– One row per distinct value

– Since Oracle 12.1, this table is LIST-HASH partitioned per table/partition

– In Oracle 12.2, this table is only used for backward compatibility (see later)

Some Key Figures of a Customer Data Warehouse


Oracle Database 12.1.0.2 Enterprise Edition on Exadata

Data Warehouse with around 24 TB of data

891 partitioned tables, most of them with INTERVAL daily partitions

Total of 325851 partitions (state end of November 2017)

Synopsis Table Rows Size

WRI$_OPTSTAT_SYNOPSIS_HEAD$ 13315001 718 MB

WRI$_OPTSTAT_SYNOPSIS$ 9998263744 320 GB

Gathering Incremental Statistics


SELECT DISTINCT BO#, GROUP#FROM SYS.WRI$_OPTSTAT_SYNOPSIS_HEAD$ HWHERE GROUP# <> 0AND GROUP# NOT IN

(SELECT T.OBJ# * 2 FROM SYS.TABPART$ TUNION ALLSELECT T.OBJ# * 2 FROM SYS.TABCOMPART$ T)

Gathering statistics with DBMS_STATS is very slow

This query was always on the Top Activity list of SQL statments

Bugs, Bugs, Bugs


Examples:23100700 (13-OCT-2017) Patch

23100700: PERFORMANCE ISSUE

WITH RECLAIM_SYNOPSIS_SPACE

19450139 (01-NOV-2017) Patch

19450139: KN:LNX:PERFORMANCE

ISSUE WHEN RUNNING GATHER

TABLE STATS WITH INCREMENTAL

STATS

20602794 (10-OCT-2017) Patch

20602794: INCREMENTAL STATS

GATHER REPEATEDLY GATHERING

ON SAME PARTITIONS

Our Solution on Oracle 12.1


Disable Incremental Statistics

Weekly job every Sunday to gather global statistics

Waiting for Oracle 12.2 or Oracle 18c

Synopsis Calculation (≤ Oracle 12.1)


Synopsis calculation with Adaptive SamplingNumber of distinct values (NDV) stored in WRI$_OPTSTAT_SYNOPSIS$

Size of synopsis tables:

– WRI$_OPTSTAT_SYNOPSIS_HEAD$

𝑟𝑜𝑤𝑠 = 𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑠 ∗ 𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓𝑐𝑜𝑙𝑢𝑚𝑛𝑠

– WRI$_OPTSTAT_SYNOPSIS$

𝑟𝑜𝑤𝑠 = ∑ 𝑁𝐷𝑉(𝑐𝑜𝑙);<=>?@ABCAD<=;ECADFG

Synopsis Calculation (≥ Oracle 12.2)


Synopsis calculation with HyperLogLog (HLL) algorithm

Approximate NDV stored in WRI$_OPTSTAT_SYNOPSIS_HEAD$

Size of synopsis tables:

– WRI$_OPTSTAT_SYNOPSIS_HEAD$

𝑟𝑜𝑤𝑠 = 𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛𝑠 ∗ 𝑛𝑢𝑚𝑏𝑒𝑟𝑜𝑓𝑐𝑜𝑙𝑢𝑚𝑛𝑠

– WRI$_OPTSTAT_SYNOPSIS$

𝑟𝑜𝑤𝑠 = 0

Configuration of Synopsis Calculation in Oracle 12.2


Values for APPROXIMATE_NDV_ALGORITHM:

DBMS_STATS.set_database_prefs('APPROXIMATE_NDV_ALGORITHM', '<value>')

DBMS_STATS.set_global_prefs('APPROXIMATE_NDV_ALGORITHM', '<value>')

DBMS_STATS.set_schema_prefs('<schema>', 'APPROXIMATE_NDV_ALGORITHM', '<value>')

DBMS_STATS.set_table_prefs('<schema>','<table>','APPROXIMATE_NDV_ALGORITHM','<value>')

Set Optimizer Statistics Preferences in DBMS_STATS

'REPEAT OR HYPERLOGLOG'Preserve format for existing tables, create new tables with HyperLogLog(Default)

'ADAPTIVE SAMPLING' Adaptive sampling is used for all synopses

'HYPERLOGLOG' HyperLogLog is used for all new and stale synopses

Incremental Statistics: Lessons Learned


Incremental Statistics can be problematic for tables withvery high number of partitions

Oracle 12.2:

Reduced space requirements in tablespace SYSAUX

– WRI$_OPTSTAT_SYNOPSIS$ not needed anymore

– WRI$_OPTSTAT_SYNOPSIS_HEAD$ grows (blob column SPARE2)

Better performance for gathering incremental statistics

– Problems of Oracle 12.1 seem to be solved

– HyperLogLog algorithm is very efficient


Bonus Demo

Bonus Demo


What do you expect after this DBMS_STATS call?

BEGINDBMS_STATS.gather_table_stats

( ownname => USER, tabname => 'ADDRESSES', estimate_percent => 0.01, method_opt => 'FOR ALL COLUMNS SIZE 2', cascade => FALSE);

END;

DBMS_STATS Preferences

32 06/13/2018

For all new objects in the database

DBMS_STATS.set_database_prefs('<preference>', '<value>')

The behavior of DBMS_STATS and many parameters can be set with preferences:

For all existing tables in the database

For all existing tables of a schema

For a particular table

DBMS_STATS.set_schema_prefs('<schema>', '<preference>', '<value>')

DBMS_STATS.set_table_prefs('<schema>','<table>','<preference>','<value>')

DBMS_STATS.set_global_prefs('<preference>', '<value>')

Pitfalls and Surprises with DBMS_STATS

DBMS_STATS Preferences


APPROXIMATE_NDV_ALGORITHM REPEAT OR HYPERLOGLOG Oracle 12.2AUTO_STAT_EXTENSIONS OFF Oracle 12.2CASCADE DBMS_STATS.AUTO_CASCADE Oracle 10.1CONCURRENT OFF Oracle 12.1DEGREE NULL Oracle 10.1ESTIMATE_PERCENT DBMS_STATS.AUTO_SAMPLE_SIZE Oracle 10.1GLOBAL_TEMP_TABLE_STATS SESSION Oracle 12.1GRANULARITY AUTO Oracle 10.2INCREMENTAL FALSE Oracle 11.1INCREMENTAL_STALENESS ALLOW_MIXED_FORMAT Oracle 12.1INCREMENTAL_LEVEL PARTITION Oracle 12.1METHOD_OPT FOR ALL COLUMNS SIZE AUTO Oracle 10.1NO_INVALIDATE DBMS_STATS.AUTO_INVALIDATE Oracle 10.1OPTIONS GATHER Oracle 12.1PREFERENCE_OVERRIDES_PARAMETER FALSE Oracle 12.2PUBLISH TRUE Oracle 11.1STALE_PERCENT 10 Oracle 11.1STAT_CATEGORY OBJECT_STATS Oracle 12.2TABLE_CACHED_BLOCKS 1 Oracle 12.1WAIT_TIME_TO_UPDATE_STATS 15 Oracle 12.2





APPROXIMATE_NDV_ALGORITHM REPEAT OR HYPERLOGLOG Oracle 12.2AUTO_STAT_EXTENSIONS OFF Oracle 12.2CASCADE TRUE Oracle 10.1CONCURRENT OFF Oracle 12.1DEGREE NULL Oracle 10.1ESTIMATE_PERCENT DBMS_STATS.AUTO_SAMPLE_SIZE Oracle 10.1GLOBAL_TEMP_TABLE_STATS SESSION Oracle 12.1GRANULARITY AUTO Oracle 10.2INCREMENTAL FALSE Oracle 11.1INCREMENTAL_STALENESS ALLOW_MIXED_FORMAT Oracle 12.1INCREMENTAL_LEVEL PARTITION Oracle 12.1METHOD_OPT FOR ALL COLUMNS SIZE SKEWONLY Oracle 10.1NO_INVALIDATE FALSE Oracle 10.1OPTIONS GATHER Oracle 12.1PREFERENCE_OVERRIDES_PARAMETER TRUE Oracle 12.2PUBLISH TRUE Oracle 11.1STALE_PERCENT 10 Oracle 11.1STAT_CATEGORY OBJECT_STATS Oracle 12.2TABLE_CACHED_BLOCKS 1 Oracle 12.1WAIT_TIME_TO_UPDATE_STATS 15 Oracle 12.2

Further Information


Incremental Statistics – A Real World Scenariohttps://danischnider.wordpress.com/2017/12/30/incremental-statistics-a-real-world-scenario/

Online Statistics Gathering in Oracle 12chttps://danischnider.wordpress.com/2015/12/23/online-statistics-gathering-in-oracle-12c/

Avoid dbms_stats.auto_invalidate in ETL jobshttps://danischnider.wordpress.com/2015/01/06/avoid-dbms_stats-auto_invalidate-in-etl-jobs/

Incremental Statistics Collection in Oracle 12.1.0.2 (Mike Dietrich)https://mikedietrichde.com/2016/04/28/incremental-statistics-collection-in-oracle-12-1-0-2-a-true-story/

Efficient Statistics Maintenance for Partitioned Tables Using Incremental Statistics (Nigel Bayliss)https://blogs.oracle.com/optimizer/efficient-statistics-maintenance-for-partitioned-tables-using-incremental-statistics-part-1https://blogs.oracle.com/optimizer/efficient-statistics-maintenance-for-partitioned-tables-using-incremental-statistics-part-3

Overriding DBMS_STATS Parameter Settings (Maria Colgan)https://sqlmaria.com/2017/07/11/overriding-dbms_stats-parameter-settings/

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH


Thank You.

Dani Schnider, Trivadis AG

pitfalls& surpriseswithdbms stats: howtosolvethem · a short history of dbms_stats 5 06/13/2018...

Documents