deep dive into postgresql statistics

Deep dive intoPostgreSQLStatistics.

Алексей Лесовский

О чем доклад

Статистика активности postgresql.

Как эффективно использовать статистику.

Решение проблем на основе данных статистики.

https://goo.gl/d0pSYK

Чего НЕ будет

Статистика планировщика запросов.

Обзор инструментов.

Итоговая цель

Умение использовать статистику - это полезно.

Статистика в постгресе - это не страшно.

Какую статистику выбрать для решения частной задачи.

Черный ящик

$ ps hf -u postgres -o cmd

/usr/pgsql-9.4/bin/postgres -D /var/lib/pgsql/9.4/data

\_ postgres: logger process

\_ postgres: checkpointer process

\_ postgres: writer process

\_ postgres: wal writer process

\_ postgres: autovacuum launcher process

\_ postgres: stats collector process

\_ postgres: postgres pgbench [local] idle in transaction

\_ postgres: postgres pgbench [local] idle

\_ postgres: postgres pgbench [local] UPDATE

\_ postgres: postgres pgbench [local] UPDATE waiting

\_ postgres: postgres pgbench [local] UPDATE

Как тратится время

Write Ahead Log

SharedBuffers

Buffers IO Autovacuum Workers

Autovacuum Launcher

Background Workers

Indexes IO

Query Execution

Query Planning

Client Backends Postmaster

Relations IO

Logger Process Stats Collector

LogicalReplication

WAL SenderProcess

ArchiverProcess

BackgroundWriter

CheckpointerProcess

Network Storage

Recovery Process

WAL Receiver ProcessTables/Indexes Data Files

Проблемы

Информации много (~109 метрик для 9.4).

Статистика представлена online счетчиками.

Нет истории (но есть функции сброса статистики).

Отсутствие native инструмента.

Множество сторонних инструментов.

Проблемы

Информации много (~109 метрик для 9.4).

Статистика представлена online счетчиками.

Нет истории (но есть функции сброса статистики).

Отсутствие native инструмента.

Множество сторонних инструментов.

Важно уметь брать статистику напрямую.

Нужны базовые знания SQL.

Что предлагают?

События, происходящие в базе.

Свойства объектов базы (баз, таблиц, индексов).

Время, затраченное событием.

Источники статы

Счетчики в shared memory.

Функции.

Встроенные представления (VIEWs).

Официальный пакет contribs.

Неофициальные contrib'ы.

Источники статы

pg_stat_all_tables

pg_current_xlog_location()

pg_replication_slots

pg_stat_replication

pg_stat_archiverpg_stat_database_conflicts

pg_buffercache

pg_stat_database

pg_stat_user_tables

pg_stat_database

pg_stat_bgwrier

pgstattuple

pg_table_size()

pg_database_size

pg_index_size()

pg_stat_activity

EXPLAIN

pg_locks

pg_stat_user_functions

pg_stat_all_indexes

pg_statio_all_indexes

pg_statio_all_tablespg_xlog_location_diff()

pg_stat_kcache

pg_stat_statements

pg_roles

pg_database

pg_class

pg_relation_size()

pg_total_relation_size()

current_setting()

pg_current_xlog_insert_location()

pg_last_xact_replay_timestamp()

pg_last_xlog_replay_location()pg_tablespace_size()

pg_ls_dir()

Write Ahead Log

SharedBuffers


Autovacuum Launcher

Background Workers

Indexes IO

Query Execution

Query Planning


Tables IO


LogicalReplication

WAL SenderProcess

ArchiverProcess

BackgroundWriter

CheckpointerProcess

Network Storage

Recovery Process


pg_stat_all_tables

pg_current_xlog_location


pg_stat_replication

pg_stat_archiver

pg_stat_database_conflicts

pg_buffercachepg_stat_database

pg_stat_activity

pg_stat_user_tables

pg_stat_database

pg_stat_bgwrier

pgstattuplepg_table_size

pg_database_size

pg_index_size

pg_stat_activity

EXPLAIN

pg_stat_statements

pg_locks


pg_stat_all_indexes

pg_stat_database


pg_statio_all_tablespg_xlog_location_diff

nicstat* iostat*

pg_stat_kcache

pg_stat_statements

Cache hit ratio$ select * from pg_stat_database;...blks_read | 7978770895blks_hit | 9683551077519...

$ select sum(blks_hit)*100/sum(blks_hit+blks_read) as hit_ratio from pg_stat_database;

Больше = лучше, но не меньше 90%

Аномалии$ select * from pg_stat_database;...xact_commit | 51167155051xact_rollback | 44781878826conflicts | 0temp_files | 377675temp_bytes | 4783712399875deadlocks | 151...

$ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;

Аномалии$ select * from pg_stat_database;...xact_commit | 51167155051 < 95 %xact_rollback | 44781878826 > 5 %conflicts | 0 > 10temp_files | 377675 > 100temp_bytes | 4783712399875 > 10 GBdeadlocks | 151 > 10...

$ select datname, (xact_commit*100)/(xact_commit+xact_rollback) as c_ratio, deadlocks, conflicts, temp_file, pg_size_pretty(temp_bytes) as temp_size from pg_stat_database;

Checkpoint's$ select * from pg_stat_bgwriter;...checkpoints_timed | 3267checkpoints_req | 6...

checkpoints_req > checkpoints_timed = плохо

postgresql.conf● checkpoint_segments

● checkpoint_timeout

● checkpoint_completion_target

Background Writer$ select * from pg_stat_bgwriter;...buffers_clean | 99602799maxwritten_clean | 157 > 0buffers_backend_fsync | 0 > 0...

Обеспечивает бэкенды чистыми страницами.

Разгружает checkpointer.

postgresql.conf● bgwriter_delay

● bgwriter_lru_maxpages

● bgwriter_lru_multiplier

Лаг репликации$ select * from pg_stat_replication;...sent_location | 1691/EEE65900write_location | 1691/EEE65900flush_location | 1691/EEE65900replay_location | 1691/EEE658D0...

1692/EEE65900 — позиция в журнале транзакций.

Все значения равны = идеально.

Лаг репликацииПричины отставания: Networking, Storage, CPU.

сколько записано xlog в байтах$ select pg_xlog_location_diff(pg_current_xlog_location(),'0/00000000');

лаг репликации в байтах$ select client_addr, pg_xlog_location_diff(pg_current_xlog_location(), replay_location) from pg_stat_replication;

лаг репликации в секундах$ select extract(epoch from now() - pg_last_xact_replay_timestamp());

Sequential scans$ select * from pg_stat_all_tables;...seq_scan | 192 необязательно плохоseq_tup_read | 364544695 > 1000 (seq_tup_avg)...

$ select relname, pg_size_pretty(pg_relation_size(relname::regclass)) as size, seq_scan, seq_tup_read, seq_scan / seq_tup_read as seq_tup_avg from pg_stat_user_tables where seq_tup_read > 0 order by 3,4 desc limit 5;

Размеры таблиц$ select relname, pg_size_pretty(pg_total_relation_size(relname::regclass)) as full_size, pg_size_pretty(pg_relation_size(relname::regclass)) as table_size, pg_size_pretty(pg_total_relation_size(relname::regclass) - pg_relation_size(relname::regclass)) as index_size from pg_stat_user_tables order by pg_total_relation_size(relname::regclass) desc limit 10;

Также есть метакоманды psql: \dt+ и \di+

Write activity$ select * from pg_stat_all_tables;... n_tup_upd | 736119030n_tup_hot_upd | 497688344...

UPDATE = DELETE + INSERT

UPDATE = TABLE UPDATE + INDEX UPDATE

Write activity$ select s.relname, pg_size_pretty(pg_relation_size(relid)), coalesce(n_tup_ins,0) + 2 * coalesce(n_tup_upd,0) - coalesce(n_tup_hot_upd,0) + coalesce(n_tup_del,0) AS total_writes, (coalesce(n_tup_hot_upd,0)::float * 100 / (case when n_tup_upd > 0 then n_tup_upd else 1 end)::float)::numeric(10,2) AS hot_rate, (select v[1] FROM regexp_matches(reloptions::text,E'fillfactor=(\\d+)') as r(v) limit 1) AS fillfactor from pg_stat_all_tables s join pg_class c ON c.oid=relid order by total_writes desc limit 50;

Что такое Heap-Only Tuples?HOT не вызывает перестроения индекса.HOT только для тех значений которые не участвуют в индексе

Большое значение n_tup_hot_upd = хорошо.Как добиться большого n_tup_hot_upd?

Write activity | pg_size_pretty | total_writes | hot_rate | fillfactor -+----------------+--------------+----------+------------ | 9418 MB | 391459091 | 5.03 | | 417 MB | 285948760 | 0.00 | | 76 MB | 232031972 | 87.31 | 70 | 7123 MB | 124281107 | 99.36 | 70

Fillfactor определяет резерв свободного места в странице.

Накладные расходы на размер таблиц (индексов).

ALTER TABLE table_name SET (fillfactor = 70);

Autovacuum queue$ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples));

| av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------| 50 | 0.01 | 60302 | 85445| 50 | 0.01 | 50 | 1349| 50 | 0.01 | 55 | 12479| 50 | 0.01 | 103 | 3134| 50 | 0.01 | 52716 | 173100

Autovacuum queue$ select c.relname, current_setting('autovacuum_vacuum_threshold') as av_base_thresh, current_setting('autovacuum_vacuum_scale_factor') as av_scale_factor, (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples)) as av_thresh, s.n_dead_tup from pg_stat_user_tables s join pg_class c ON s.relname = c.relname where s.n_dead_tup > (current_setting('autovacuum_vacuum_threshold')::int + (current_setting('autovacuum_vacuum_scale_factor')::float * c.reltuples));

| av_base_thresh | av_scale_factor | av_thresh | n_dead_tup +----------------+-----------------+-----------+------------| 50 | 0.01 | 60302 | 60545| 50 | 0.01 | 50 | 1349| 50 | 0.01 | 55 | 12479| 50 | 0.01 | 103 | 3134| 50 | 0.01 | 52716 | 173100

Avito Experiencehttps://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue

https://github.com/eshkinkot/avito/blob/master/munin/vacuum_queue

Лишние индексы$ select * from pg_stat_all_indexes where idx_scan = 0;...indexrelname | products_special2_idxidx_scan | 0 0 = плохо...

https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/low_used_indexes.sql

http://www.databasesoup.com/2014/05/new-finding-unused-indexes-query.html





Лишние индексы$ select * from pg_stat_all_indexes where idx_scan = 0;...indexrelname | products_special2_idxidx_scan | 0 0 = плохо...



Неиспользуемые индексы - плохо.

Занимают место.

Замедляют UPDATE, DELETE, INSERT операции.

Лишняя работа для VACUUM.





Общая активность$ select * from pg_stat_activity;...datname | tsusename | tsagentclient_addr | 172.17.0.37...

общий процент подключений$ select count(*)*100/(select current_setting('max_connections')::int) from pg_stat_activity;

Общая активность$ select * from pg_stat_activity;...datname | tsusename | tsagentclient_addr | 172.17.0.37...

$ select client_addr, usename, datname, count(*) from pg_stat_activity group by 1,2,3 order by 4 desc;

client_addr | usename | datname | count -------------+-----------+---------+------- 127.0.0.1 | cron_role | ts | 508 127.0.0.1 | tsagent | ts | 60 127.0.0.1 | postgres | ts | 1 127.0.0.1 | pupkin | ts | 1

Долгие запросы$ select * from pg_stat_activity;...backend_start | 2015-10-14 15:18:03.01039+00xact_start | 2015-10-14 15:21:15.336325+00query_start | 2015-10-14 15:21:30.336325+00state_change | 2015-10-14 15:21:30.33635+00...


$ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start;


$ select client_addr, usename, datname, clock_timestamp() - xact_start as xact_age, clock_timestamp() - query_start as query_age, query from pg_stat_activity order by xact_start, query_start;

clock_timestamp() для вычисления времени работы.

Запросы запоминаем, отстреливаем, оптимизируем (EXPLAIN).

Плохие транзакции$ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)';...xact_start | 2015-10-14 15:21:21.128192+00query_start | 2015-10-14 15:21:30.336325+00state_change | 2015-10-14 15:21:30.33635+00state | idle in transaction...

Плохие транзакции$ select * from pg_stat_activity where state in ('idle in transaction', 'idle in transaction (aborted)';...xact_start | 2015-10-14 15:21:21.128192+00query_start | 2015-10-14 15:21:30.336325+00state_change | 2015-10-14 15:21:30.33635+00state | idle in transaction...

idle in transaction, idle in transaction (aborted) = плохо

Повод для беспокойств: > 5


Транзакции отстреливаем, приложение оптимизируем.

Блокировки$ select * from pg_stat_activity where waiting;...xact_start | 2015-10-14 15:21:21.128192+00query_start | 2015-10-14 15:21:30.336325+00state_change | 2015-10-14 15:21:30.33635+00waiting | t...

Блокировки$ select * from pg_stat_activity where waiting;...xact_start | 2015-10-14 15:21:21.128192+00query_start | 2015-10-14 15:21:30.336325+00state_change | 2015-10-14 15:21:30.33635+00waiting | t...

waiting = true = плохо.


pg_locks для поиска источника блокировки.

Транзакции отстреливаем, приложение оптимизируем.

Блокировкиhttps://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_show_locked_queries.sql

http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/

locked_item | productswaiting | twaiting_query | update products set avail = false where id = 154;waiting_mode | RowExclusiveLockwaiting_xact_age | 00:00:07.103685waiting_query_age | 00:00:01.813508waiting_pid | 5895waiting_user | app_rolewaiting_state | activewaiting | flocking_query | alter table products drop column avail;locking_mode | AccessExclusiveLocklocking_xact_age | 00:00:16.416441locking_query_age | 00:00:14.416441locking_pid | 5844locking_user | postgreslocking_state | idle in transaction

https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_show_locked_queries.sql

https://github.com/lesovsky/uber-scripts/blob/master/postgresql/sql/c4_06_show_locked_queries.sql

http://big-elephants.com/2013-09/exploring-query-locks-in-postgres/

pg_stat_statements$ select * from pg_stat_statements;...query | SELECT "id" FROM run_plan_xact(?)calls | 11165832total_time | 11743325.6880088rows | 11165832blk_read_time | 495425.535999976blk_write_time | 0

Cреднее время запроса в милисекундах$ select (sum(total_time) / sum(calls))::numeric(6,3) from pg_stat_statements;

Самые активно пишущие (в shared_buffers) запросы$ select query, shared_blks_dirtied from pg_stat_statements where shared_blks_dirtied > 0 order by 2 desc;

Отчетыquery total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%)сalls: 476 (0.00%) rows: 476,000avg_time: 118881.54ms (IO: 21.2%)user: app_user db: ustatsquery: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4

https://github.com/PostgreSQL-Consulting/pg-utils/blob/master/sql/global_reports/query_stat_total.sql



Отчетыquery total time: 15:43:07 (14.9%, CPU: 18.2%, IO: 9.0%)сalls: 476 (0.00%) rows: 476,000avg_time: 118881.54ms (IO: 21.2%)user: app_user db: ustatsquery: select filepath, type, deviceuid from imv5event where state = ?::eventstate and servertime between $1 and $2 order by servertime desc LIMIT $3 OFFSET $4

Используем sum() для подсчет общей статистики.

Вычисляем «вклад» запроса в общую статистику.

Использование ресурсов (CPU, IO).

За кадром

pg_statio_all_tables, pg_statio_all_indexes


Функции определения размеров - \df *size*

pgstattuple (contrib)● точное определение bloat для индексов и таблиц● чем больше размер таблицы, тем больше времени уйдет на

оценку

pg_buffercache (contrib)● инспекция shared buffers● большие накладые расходы (buffers lock)

За кадром

pgfincore● низкоуровневые операции с таблицами через mincore().● инспекция OS page cache.

pg_stat_kcache● использование getrusage() до и после запроса.● cpu usage и дисковый ввод-вывод.● требует pg_stat_statements и postgresql-9.4.● почти не влияет на производительность.

Резюме

● Умение пользоваться статистикой полезно.

● Статистика в постгресе - это не сложно.

● Статистика поможет ответить на вопросы.

● Экспериментируйте.

Годные ссылкиАвтор, пиши ещё www.blog.postgresql-consulting.com (eng) www.thislinux.org (rus)

The Statistics Collector http://www.postgresql.org/docs/current/static/monitoring-stats.html

System Administration Functions http://www.postgresql.org/docs/current/static/functions-admin.html

Contrib modules http://www.postgresql.org/docs/current/static/pgstatstatements.html http://www.postgresql.org/docs/current/static/pgstattuple.html http://www.postgresql.org/docs/current/static/pgbuffercache.html https://github.com/klando/pgfincore https://github.com/dalibo/pg_stat_kcache

SQL utils and sql code examples https://github.com/PostgreSQL-Consulting/pg-utils

http://www.blog.postgresql-consulting.com/

http://www.thislinux.org/

http://www.postgresql.org/docs/current/static/monitoring-stats.html

http://www.postgresql.org/docs/current/static/functions-admin.html

http://www.postgresql.org/docs/current/static/pgstatstatements.html

http://www.postgresql.org/docs/current/static/pgstattuple.html

http://www.postgresql.org/docs/current/static/pgbuffercache.html

https://github.com/klando/pgfincore

https://github.com/dalibo/pg_stat_kcache

https://github.com/PostgreSQL-Consulting/pg-utils

Write Ahead Log

SharedBuffers


Autovacuum Launcher

Background Workers

Indexes IO

Query Execution

Query Planning


Tables IO


LogicalReplication

WAL SenderProcess

ArchiverProcess

BackgroundWriter

CheckpointerProcess

Network Storage

Recovery Process


pg_stat_all_tables

pg_current_xlog_location


pg_stat_replication

pg_stat_archiver

pg_stat_database_conflicts

pg_buffercachepg_stat_database

pg_stat_activity

pg_stat_user_tables

pg_stat_database

pg_stat_bgwrier

pgstattuplepg_table_size

pg_database_size

pg_index_size

pg_stat_activity

EXPLAIN

pg_stat_statements

pg_locks


pg_stat_all_indexes

pg_stat_database


pg_statio_all_tablespg_xlog_location_diff

nicstat* iostat*

pg_stat_kcache

pg_stat_statements

Спасибо. Вопросы?

deep dive into postgresql statistics

Education