guarding your data

Andrew Gilmartin senior developer agilmartin@crossref.org

don’t loose the data

•  Oracle holds the data in Boston

Oracle × 2 + tape + off-site tape

•  Oracle holds the data in Denver

Oracle × 2

•  Denver less than 20 minutes out of sync with Boston

syncing Boston & Denver

•  Oracle's

Oracle Streams …

•  Other's

Tungsten Replicator …

•  Our's

the tool’s context

•  use existing tooling

•  use a small set of dependent parts

•  use already monitored parts

•  Oracle

•  file systems

•  networks

•  …

the tool’s necessities

•  reliable

•  recoverable

•  chatty

•  alarming

Oracle’s archive-log

1  archive-log represent the changes to the database since the last archive-log

2  discover created archive-log via Boston Oracle’s tracing-log

3  transfer archive-log to Denver

4  register archive-log with Denver’s Oracle,

5  discover archive-log applied via Denver Oracle’s tracing-log

a bash script

•  script & configuration are short at ~200 & ~10 lines respectively

•  same script & configuration in Boston & in Denver

•  Boston based script relies on Denver based script and vise versa

•  a known set of functions

significant event markers

•  events triggered by tracing-log files

•  events are recorded & coordinated via marker files

•  events are created, transferred, registered, & applied

•  markers are distinctively named, empty files in Boston

arch_1_640475_754538743.arc.created

archive-log’s name! marker!

were what happens

•  Boston continuously watches for archive-log creations

invokes “transfer & register” process in Denver

•  Denver continuously watches for archive-log applications

invokes “applied” process in Boston

•  Boston periodically investigate process’s status

•  Boston & Denver periodically clean up

some implementation notes

•  humans are are good at recovery

•  use the operating system ... ssh keys ... directory as database

•  use configuration files rather than command line options

•  Use command timeouts

•  check command exit status

•  retry commands at lengthening intervals … 5s 30s 300s …

•  use bash's -E option

some resources

•  spark.sh & spark.conf

•  ask Chuck for our tool, ckosher@crossref.org

•  Giuseppe Maxia’s “Script it! Make professional DBA tools out of nothing” https://www.percona.com/live/mysql-conference-2013/sessions/script-it-make-professional-dba-tools-out-nothing