© 2009 convio, inc. end-to-end metrics for troubleshooting and monitoring dave abercrombie...

© 2009 Convio, Inc.

End-to-End Metrics for Troubleshooting and Monitoring

Dave AbercrombiePrincipal Database Architect, ConvioNoCOUG Spring Conference 2009, May 21


Agenda

■ Benefits

■ Brief description

■ Tag Details

■ Setting tags (PL/SQL and Java)

■ Real-time monitoring

■ Historical troubleshooting

■ Overhead

■ Other ideas


Reasons to use End-to-End Metrics

■ Real-time application details

■ Provide meaning to SQL_IDs

■ Application context

■ Customer level

■ Improve staff communication


More reasons

■ Similar, but not identical SQL texts

■ Across sessions

■ Enhance historical diagnostics

■ Focus tuning during development

■ Flexible, so customize!


Overview

■ Session-level tags (four)

■ Application sets the tags

■ Tags visible in V$ ASH, AWRTrace files

■ Initiate tracing

■ Aggregate statistics

■ Flexible, incremental


Simple exampleSQL> -- set tagsSQL> BEGIN 2 DBMS_APPLICATION_INFO.SET_MODULE ( 3 module_name => 'simple', 4 action_name => 'example'); 5 END; 6 /

PL/SQL procedure successfully completed.

SQL> -- retrieve tags for current sessionSQL> SELECT module, action 2 FROM v$session 3 WHERE sid = SYS_CONTEXT('userenv', 'SID');

MODULE ACTION---------- ----------simple example


TagsName Length

Module 48

Action 32

Client_Identifier 64

Client_Info 64


Tables that use these tags

ACTION MODULECLIENT_IDENTIFIER

(CLIENT_ID) CLIENT_INFO

V$SESSION 32 48 64 64

V$ACTIVE_SESSION_HISTORY 32 48 64

DBA_HIST_ACTIVE_SESS_HISTORY 32 48 64

V$SQL 64 64

DBA_HIST_SQLSTAT 64 64

▶ This is not a complete list! ▶ Length varies (use v$session)

▶ Name varies (v$session.client_identifier, others use client_id)


Agenda

■ Benefits


■ Tag Details




■ Overhead

■ Other ideas


Design Challenges

■ Level of detail

■ Availability of tag data

■ Tag quirks:Various lengths,Availability varies,Funny names

■ Short length


More Design Challenges

■ Ability to aggregate

■ Uniqueness

■ Hierarchy

■ Restoring to nulls

■ Session management (connection pools)

■ When do called methods override tags?


Setting tags with PL/SQL

DBMS_APPLICATION_INFO.SET_MODULE( module_name IN VARCHAR2, action_name IN VARCHAR2);

DBMS_APPLICATION_INFO.SET_ACTION( action_name IN VARCHAR2);

DBMS_APPLICATION_INFO.SET_CLIENT_INFO( client_info IN VARCHAR2);


Setting tags with JDBC

String metrics[] = new String[OracleConnection.END_TO_END_STATE_INDEX_MAX];

metrics[END_TO_END_ACTION_INDEX] = "Spike";metrics[END_TO_END_MODULE_INDEX] = "Buffy";

// Set these metricsconn.setEndToEndMetrics(metrics,(short)0);

// Do some work

// Update a metric metrics[END_TO_END_MODULE_INDEX] = "Faith";


Potential tag values

■ PL/SQL package, method■ Java object, method■ JSP file■ Web page URL■ Task ID, current step■ Customer ID■ App server■ IP:port■ Version■ Debugging■ Progress indicator


Ideas for setting tags

■ Don’t forget to set back to NULL■ Human readable vs. encoded■ Overloaded, delimited■ Database API wrapper■ Hierarchical■ Consistency■ Mix and match styles■ Some utilities already set tags■ Integrate with other logging


BUG 8524840 - Do not use multibyte tags

■ Multibyte module tags

■ Length? Bytes vs. characters

■ ORA-07445 core dump

■ _MEMSET()+320 KEWE_SET_NEWMODULE

■ New bug

■ Versions 10.2.0.2, 10.2.0.3 (at least)


Agenda

■ Benefits


■ Tag Details




■ Overhead

■ Other ideas


Real-time monitoring

■ V$SESSION■ Progress indicator

Cannot aggregateCLIENT_INFO

■ Lock pile-ups■ Inter-session communication■ Custom probes■ ASH■ dbms_monitor (trace)


DBMS_MONITOR - tracing

■ Optional■ Enable/disable traces on specific:

ModuleActionModule and ActionClient_Identifier

■ Service nameLook at v$session.service_namePerhaps dba_services.network_name?

■ Listed in dba_enabled_traces■ Examples:

Norton Debes, “Secret Oracle”James Koopmann, “DBAsupport.com”


Example: custom probe

■ Delayed commit from app server

■ Challenges:Idle wait event (waiting for message)Common, frequent, widespread DMLNeither ASH nor SQL_ID helped

■ Probe design:cron scriptJoin v$transaction and v$sessionFilter for idle wait eventTransaction started > 5 seconds agoTags identified culprit


Agenda

■ Benefits


■ Tag Details




■ Overhead

■ Other ideas


Historical Troubleshooting

■ dba_hist_active_sess_history

■ Aggregate wait counts, durations

■ Complementary, not replacement

■ Supported by OEM, etc.

■ Trace files, deadlocks, etc

■ V$SQL, AWR: Session-level, first parse

■ dbms_monitor (stat)


DBMS_MONITOR – aggregate stats

■ Optional■ Enable/disable aggregations■ Module, action, client_id, service_name■ Views

v$serv_mod_act_statsv$client_statsv$service_stats

■ Listed in dba_enabled_aggregations■ Examples:

Norton Debes, “Secret Oracle”James Koopmann, “DBAsupport.com”


Stats gathered, by class User SQL

user callsuser I/O wait timeDB timeapplication wait timeconcurrency wait timecluster wait timeuser rollbacksuser commitssession logical readsopened cursors cumulative

parse time elapsedexecute countsession cursor cache hitsworkarea executions – multipassworkarea executions – onepassparse count (total)workarea executions - optimal

Redo, Cache RAC cache

redo sizephysical readsphysical writesdb block changes

gc current block receive timegc current blocks receivedgc cr blocks receivedgc cr block receive time


Example: prioritize tuning efforts

■ Large, expensive background tasks

■ Challenges:Many executionsMany flavorsBusiness critical Complicated, frequent revisions

■ Study design:Count and sum read wait eventsGroup by tagsPrioritize


Agenda

■ Benefits


■ Tag Details




■ Overhead

■ Other ideas


Overhead

■ About 5 microsecondsPL/SQL loopNeed to defeat compiler optimization

■ No wait events

■ No extra network roundtrips (Norton Debes)

■ OCI no extra DB calls (Cary Milsap)OCI_ATTR_MODULEOCI_ATTR_ACTIONOCI_ATTR_CLIENT_INFO


Loop elapsed time

for loop_counter in 1 .. 5000000Loop

dbms_application_info.set_module ( 'fourtyEight fourtyEight fourtyEight', 'thirtyTwo thirtyTwo thirtyTwo' );

dbms_application_info.set_module ( 'module module module module', 'action action action' );

end loop;

dbms_application_info.set_module ( null, null);

0E+00 1E+07 2E+07 3E+07 4E+07 5E+070

50

100

150

200

250

DBMS_APPLICATION_INFOSET_MODULE()

"DB time" as function of number of calls

4.8 microseconds per call

number of API calls

DB

tim

e in

se

co

nd

s


Agenda

■ Benefits


■ Tag Details




■ Overhead

■ Other ideas


Other Ideas

■ Dynamic Monitoring Service (DMS)Oracle Application ServerControls tag setting

■ Wrap PL/SQL APICustom stat gathering codeRedefine public synonym

■ Prevent Routines from Running SimultaneouslyAmar Kumar Padhi, Database Journal, 3/10/2004Simpler than explicit locking


Reasons to use End-to-End Metrics

■ Real-time application details■ Provide meaning to SQL_IDs■ Application context■ Customer level■ Improve staff communication■ Similar, but not identical SQL texts■ Across sessions■ Enhance historical diagnostics■ Focus tuning during development■ Flexible, so customize!

© 2009 convio, inc. end-to-end metrics for troubleshooting and monitoring dave abercrombie...

Documents