postgresql @alibaba cloud / xianming dou (alibaba cloud)

PostgreSQL @ Alibaba Cloud

Xianming Dou 2016.11.05

How to create a PostgreSQL instance from ZERO?

Create PostgreSQL Instance From ZERO

Create PostgreSQL Instance From ZERO

Create PostgreSQL Instance From ZERO in 10 minutes

HA

• Transparent• Replication

Lag limit

After Created

Monitoring

• Performance• Exception

and Alarm• Auto-Repair

Backup And Restore• Backup

Scheduling • Time

Specified Restore

PostgreSQL

• OOM Handling• IO Hung• Privilege

Management• Transparent

Switch-Over• Transparent

Connection Pool

High Availability

Meta Data of instances

HA Worker PG

Master

PG SlaveHA Worker

…

HA Worker

HA Worker

…

PG Master

PG Slave

PG Master

PG Slave

replication

HA Master

HA Master

Detect

Promote

Monitoring

Host Collector

Host Collector

Host Collector

Host Collector

Database StormingAPI

Rules And Check Databas

e

Performance

Alarm Auto-Repairing

Instant Detect

Backup and Restore

Backup

Restore

Instance 1

Instance 2

Instance …

Instance N

Backup Scheduler

Persistent Storage

Meta DB

Control Flow

Data Streaming

Data Streaming

OOM Handling• Standalone Handler• Signal Process

IO Hung• ext4• IO sync

Privilege Management• No super-user• Normal user is not enough

PostgreSQL Kernel Enhancement

Transparent Connection• Connection Pool• Connection Switch

PG

OOM Handling

OOM Handler

PG Process 1

Instance Resource Container

Public Resource Container

OOM Signal

PG Process N…Most

Memory Usage

PG Process 2PG Process 2

OOM Handling

OOM Handler

PG Process 1

Instance Resource Container

PG Process 2

Public Resource Container

Specified signal for statement canceling

PG Process N…

PG Process 2Received cancel and report OOM to frontend.

PG Process 2

MovePG Process 2

OOM Handling

OOM Handling• Standalone Handler• Signal Process

PG

Privilege Management

• Super-user is dangerous• User need lots of privileges• Compatibility

rolname sanityrolsuper frolinherit trolcreaterole trolcreatedb trolcatupdate trolcanlogin trolreplication trolconnlimit -1rolpassword ****rolvaliduntil

OOM Handling• Resource Limit• Process Container Control

Privilege Management• No super-user• Normal user is not enough

PostgreSQL Kernel

PG

IO Hung

ext4 journal

lock write

Data Pages

Could be a lot, and IO consuming

The problem of fsync during checkpoint

BLOCK File System

write

unlock

More Page WriteIO limit

GET WORSE

IO Hung

ext4 journal

lock write

Data Pages

Sync data before fsync during checkpoint

write

unlock

OOM Handling• Resource Limit• Process Container Control

IO Hung• ext4• IO sync

Privilege Management• No super-user• Normal user is not enough

IO Hung

PG

Transparent Connection Control

Master

Slave

Net Proxy HA

User Switch

Keep Connected

Shutdown

Switch

Promote

Transparent Connection Control

Master

Slave

Net Proxy

Keep Connected

Statement Clean

HA

• Transparent• Replication

Lag limit

After Created

Monitoring

• Performance• Exception

and Alarm• Auto-Repair

Backup And Restore• Backup

Scheduling • Time

Specified Restore

PostgreSQL

• OOM Handling• IO Hung• Privilege

Management• Transparent

Connection Control