Bridging the Observability Gap for Java and Scripting Applic

2009 sun tech days

DTrace: Bridging the Observability Gap for Java and Scripting ApplicationsAlex Peng Technology Evangelist Sun Microsystems1

2009 sun tech days

Agenda Introduction to DTrace The DVM provider The hotspot provider Scripting language providers DTrace and MySQL Summary and Resources

2009 sun tech days

Traditional Analysis: System Observability My application doesn't work or doesn't perform

as well as it needs to

What tools do I have? What information can I get? How can I relate this to the problem?

2009 sun tech days

What is instrumentation?? Question: How do you typically do

instrumentation?? Answer:

#ifdef DEBUG printf(“Value of Q:%d”, Q);

Question: What do you need to do to add new

instrumentation? Answer: edit, compile, link, deploy ........ the system and your own code??

What if you could do this dynamically, both on4

2009 sun tech days

DTrace Approach My application doesn't work or doesn't perform as

well as it needs to

hypothesis→ instrumentation→ data gathering→ analysis→ hypothesis ........

2009 sun tech days

DTrace Basics DTrace is a dynamic instrumentation framework

that was introduced in Solaris 10 Allows for dynamic instrumentation of the OS and applications (including Java-based applications as well as some scripting languages, as we will see later) Available out of the box; a typical system has more than 50,000 probes> Other systems also instrumented

Includes D, a dynamically interpreted language,

that allows for arbitrary actions and predicates

2009 sun tech days

DTrace Basics Designed explicitly for use on production

systems Zero performance impact when not in use Completely safe; no way to cause panics, crashes, data corruption or pathological performance degradation Powerful data management primitives eliminate need for most postprocessing

2009 sun tech days

DTrace Terminology Probe> A place of interest in the system where we can make observations Provider> Instruments a particular area of the system> Makes probes available> Transfers control into the DTrace framework when an enabled probe is hit.

2009 sun tech days

D Language - Format. A D script consist of one or more statements A statement has the following structure:

probe description/ predicate/{ action statements}

#!/usr/sbin/dtrace -s syscall:::entry/execname==”bash”/{ printf(“%s called\n”, probefunc);}

When a probe fires the actions are executed if

the predicate evaluates true Example,“Print all the system calls executed by bash”

2009 sun tech days

DTrace Terminology Aggregation> Often, patterns are more interesting than individual datum> aggregate data together to look for trends> result of an aggregation function (count(), sum(), max(), avg(), quantize()) associated with arbitrary key in an array

syscall::write:entry{@writes[execname, pid]= sum(arg2);}10

2009 sun tech days

jstack() Action

jstack action prints mixed mode stack trace Both Java frames and native (C/C++) frames are shown Only JVM versions 5.0_01 and ab

ove are supported> jstack shows hex numbers for JVM versions before

5.0_01

First

#!/usr/sbin/dtrace -s syscall::pollsys:entry/ pid==$1/ argument limits number of frames{

shown

jstack(50, string Second argument changes 8192);size} jstackstrsize pragma/-x to increase buffer for all jstack

calls

2009 sun tech days

The dvm Provider project to add DTrace support> 1.4.2 (libdvmpi.so)> 1.5 (libdvmti.so)> https:/// Download shared libs Add location of libs to LD_LIBRARY_PATH

variable Set JAVA_TOOL_OPTIONS to -Xrundvmti:all

2009 sun tech days

The dvm Provider: Probes dvm probes and their signatures:vm-init(), vm-death() thread-start(char *thread_name), thread-end() class-load(char *class_name) class-unload(char *class_name) gc-start(), gc-finish() gc-stats(long used_objects, long used_object_space) object-alloc(char *class_name, long size) object-free(char *class_name) method__entry(char *class_name, char *method_name, char *method_signature) method__return(char *class_name, char *method_name, char *method_signature)13

2009 sun tech days

The dvm Provider: Methods Count methods called

#!/usr/sbin/dtrace -s dvm$target:::method-entry{@[copyinstr(arg0),copyinstr(arg1)]= count();}

2009 sun tech days

DTrace in JDK 6 JDK 6 supports DTrace“out of the box” hotspot provider implements all dvm probes plus

extensions:

> Method compilation (method-compile-begin/end)> Compiled method load/unload(compiled-method-load/

unload)> JNI method probes.> DTrace probes as entry and return from each JNI method. Strings are now unterminated UTF-8 data.

Always use associated length value with copyinstr().

2009 sun tech days

JDK 6 DTrace Usage Certain probes are expensive Turned off by default> object-alloc, method-entry, method-return> All monitor probes: monitor-wait, etc. Use command line flag to enable if required> -XX:+ExtendedDTraceProbes (all expensive probes) Selective probe enablement (less impact)> -XX:+DTraceAllocProbes> -XX:+DTraceMethodProbes> -XX:+DTraceMonitorProbes

2009 sun tech days

JDK6 hotspot_jni Provider Probes for Java Native Interface (JNI) Located at entry/return points of all JNI

functions Probe arguments are same as corresponding JNI function arguments (for _entry probes) For XXX_return probes, probe argument is return value Examples:

hotspot_jni$1:::GetPrimitiveArrayCritical_entry hotspot_jni$1:::GetPrimitiveArrayCritical_return17

2009 sun tech days

Hotspot Probe Demo

2009 sun tech days

DTrace and Scripting Language DTrace providers are being developed for

scripting languages Examples>>>>>

Python PHP JavaScript in Firefox 3.0 Ruby More to come

2009 sun tech days

DTrace and Python Two probes, simple function entry/exit> function-entry> function-return Three arguments available> arg1: filename> arg2: subroutine name> arg3: line number

2009 sun tech days

DTrace and Python ustack() works well on these probes These probes are available in OpenSolaris

2008.05 and later DTrace toolkit

has a set of scripts for Python