Non-native languages

As DTrace became popular, many language interpreters got USDT probes. Some of them adopted them in upstream, some probes are provided by the binaries custom packages shipped with operating system. The basic pair of probes provided by most language interpreters are function entry and exit probes which provide name of the function, line number and file name. For example, Perl can be traced that way:

# echo 'use Data::Dumper; 
      map { Dumper($_, ", ") }  ("Hello", "world");' | 
    dtrace -n '
        perl$target:::sub-entry {
            trace(copyinstr(arg0));  trace(copyinstr(arg1)); trace(arg2);
        }  ' -c 'perl -- -'

# stap -e '
        probe process("/usr/lib64/perl5/CORE/libperl.so").mark("sub__entry") {
            printdln(" : ", user_string($arg1), user_string($arg2), $arg3);
        }
    ' -c $'perl -e \'use Data::Dumper; 
                     map { Dumper($_, ", ") }  ("Hello", "world");\''

Note that we had to use stdin as a script in DTrace example. That happened because DTrace cannot parse -c option value in shell-like manner.

Language interpreters provide not only function entry probes, here are other examples of supplied probes:

• Function entry and exit probes.
  • In PHP and Python –- function-entry/function-return.
  • In Perl –- sub-entry/sub-return.
  • In Ruby –- method-entry/method-return.
  • In Tcl –- proc-entry/proc-return.
• Probes that fire inside function: line in Python which corresponds to a interpreted line and execute-entry/execute-return, which fire per each Zend interpreter VM operation.
• Probes of file execution and compilation: such as compile-file-entry/compile-file-return in PHP and loading-file/loaded-file in Perl
• Error and exception probes: raise in Ruby and exception-thrown/exception-caught/error in PHP
• Object creation probes: obj-create/obj-free in Tcl, instance-new-*/instance-delete-* in Python, object-create-start/object-create-done/object-free in Ruby
• Garbage collector probes: gc-start/gc-done in Python, gc-*-begin/gc-*-end in Ruby 2.0 or gc-begin/gc-end in Ruby 1.8

Here are list of availability of that probes in various interpreters shipped as binary packages. If you lack them, you may want to rebuild interpreters with some configure option like --enable-dtrace.

Interpreter	CentOS	Solaris
Python 2	Python has never accepted DTrace patches into upstream. However, it was implemented by Solaris developers for Python 2.4, and being ported to Fedora's and CentOS python. Only function-related probes are supplied: function-entry and function-return.
Python 3	Like Python 2, Python 3 in CentOS (if installed from EPEL) supports function__entry and function__return probes. In addition to that, SystemTap supplies example python3 tapset.	Python 3 is supplied as FOSS (unsupported) package in Solaris 11 and has line probe, instance creation and garbage-collector related probes.
	Starting with Python 3.6, DTrace probes function entry and exit probes, garbage collector probes and line are supported by vanilla interpreter
PHP5	Doesn't support USDT tracing but can it be enabled via --enable-dtrace switch when it is built from source.	PHP supports tracing functions, exceptions and errors, VM opcodes execution and file compiling from scratch. Its probes will be discussed in the following section, Web applications.
Ruby 2	Supports multiple probes including object creation, method entry and exit points and garbage collector probes in Ruby 2.0 in CentOS or Ruby 2.1 as FOSS package in Solaris 11.
Perl 5	Supports subroutine probes sub-entry and sub-return (see examples above).
Go	Go is pretty close to native languages in Linux, so you can attach probes directly to its functions while backtraces show correct function names. Differences in type system between C-based languages and Go, however prevents SystemTap from accessing arguments.	Go has experimental support for Solaris so it is not considered as a target for DTrace.
Erlang	Neither EPEL nor Solaris package feature USDT probes in BEAM virtual machine, but they are supported in sources, so building with --with-dynamic-trace option enables various probes including function-boundary probes.
Node.JS	Node.JS is not supplied as OS packages, while binaries from official site doesn't have USDT enabled in Linux or simply not working in Oracle Solaris (only in Illumos derivatives). Building from sources, however adds global network-related probes like http-server-request. Function boundary tracing is not supported.

Most interpreted language virtual machines still rely on libc to access basic OS facilities like memory allocation, but some may use their own: i.e. PyMalloc in Python, Go runtime is OS-independent in Go language. For example let's see how malloc calls may be cross-referenced with Python code in yum and or pkg package managers using SystemTap or DTrace. We will need to attach to function entry and exit points to track "virtual" python backtrace and malloc call to track amount of allocated bytes. This approach is implemented in the following couple of scripts:

#!/usr/sbin/dtrace -qCZs

BEGIN {
    self->depth = 0;
}

foo$target::: {
    /* This probe is just a workaround for -xlazyload */
}

python$target:::function-entry {
    func_stack[self->depth] = arg1;
    file_stack[self->depth] = arg0;
    
    self->depth++;
}
python$target:::function-return {
    self->depth--;
}

pid$target::malloc:entry 
/ func_stack[self->depth] != 0 / {
    @mallocs[copyinstr(func_stack[self->depth]),
             copyinstr(file_stack[self->depth])] = sum(arg0);
}

#!/usr/bin/env stap

@define libc %( "/lib64/libc.so.6" %)

global file_stack, func_stack, mallocs, thread_depth

probe python.function.entry {
    thread_depth[tid()]++;
    
    depth = thread_depth[tid()];
    file_stack[tid(), depth] = filename;
    func_stack[tid(), depth] = funcname;
}

probe python.function.return {
    thread_depth[tid()]--;
}

probe process(@libc).function("_int_malloc") {
    depth = thread_depth[tid()];
    mallocs[file_stack[tid(), depth],
            func_stack[tid(), depth]] <<< $bytes;
}

Arguments and local variables are also inaccessible directly by SystemTap or DTrace when program in non-native language is traced. That happens because they are executed within virtual machine which has its own representation of function frame which is different from CPU representation: languages with dynamic typing are more likely to keep local variables in a dict-like object than in a stack. These frame and dict-like objects, however, are usually implemented in C and available for dynamic tracing. All that you have to do is to provide their layout.

Let's see how this can be done for Python 3 in Solaris and Linux. If you try to get backtrace of program interpreted by Python 3, you will probably see function named PyEval_EvalCodeEx which is responsible for evaluation of code object. Code object itself has type PyCodeObject and passed as first argument of that function. That structure has fields like co_firstlineno, co_name and co_filename. Last two fields not just C-style strings but kind of PyUnicodeObject –- an object which represents strings in Python 3. It have multiple layouts, but we rely on the simplest one: compacted ASCII strings. That may not be true for all string objects in the program, but that works fine for objects produced by the interpreter itself like code objects.

DTrace cannot recognize type information from Python libraries, but it supports struct definitions in the code. We will use it to provide PyCodeObject and PyUnicodeObject layouts in a separate file pycode.h. DTrace syntax is pretty much like C syntax, so these definitions are almost copy-and-paste from Python sources. Here is an example of DTrace scripts which trace python program execution:

#ifndef PY_CODE_H
#define PY_CODE_H

/**
 * This is forward definitions taken from Include/object.h and Include/code.h 
 * to support extraction of Python 3.4 interpreter state
 */

typedef long        ssize_t;

typedef struct _object {
    /* _PyObject_HEAD_EXTRA */
    ssize_t ob_refcnt;
    struct PyObject *ob_type;
} PyObject;

/* Bytecode object */
typedef struct _code {
    PyObject base;
    int co_argcount;        /* #arguments, except *args */
    int co_kwonlyargcount;  /* #keyword only arguments */
    int co_nlocals;         /* #local variables */
    int co_stacksize;       /* #entries needed for evaluation stack */
    int co_flags;           /* CO_..., see below */
    PyObject *co_code;      /* instruction opcodes */
    PyObject *co_consts;    /* list (constants used) */
    PyObject *co_names;     /* list of strings (names used) */
    PyObject *co_varnames;  /* tuple of strings (local variable names) */
    PyObject *co_freevars;  /* tuple of strings (free variable names) */
    PyObject *co_cellvars;      /* tuple of strings (cell variable names) */
    /* The rest doesn't count for hash or comparisons */
    unsigned char *co_cell2arg; /* Maps cell vars which are arguments. */
    PyObject *co_filename;  /* unicode (where it was loaded from) */
    PyObject *co_name;      /* unicode (name, for reference) */
    int co_firstlineno;     /* first source line number */
    PyObject *co_lnotab;    /* string (encoding addr<->lineno mapping) See
                   Objects/lnotab_notes.txt for details. */
    void *co_zombieframe;     /* for optimization only (see frameobject.c) */
    PyObject *co_weakreflist;   /* to support weakrefs to code objects */
} PyCodeObject;

/**
 * Compact ASCII object from Python3 -- data starts after PyUnicodeObject -- only if compact, ascii
 * and ready flags are set
 */
typedef struct _unicode {
    PyObject  base;
    ssize_t   length;
    ssize_t   hash;
    char      flags[4];
    void*     wstr;
} PyUnicodeObject;

#endif

#!/usr/sbin/dtrace -Cs

#include "pycode.h"

#define GET_Py3_STRING(obj) (((PyUnicodeObject*) copyin((uintptr_t) obj,                     \
                                           sizeof(PyUnicodeObject)))->flags[0] & 0xE0)       \
            ? copyinstr(((uintptr_t) obj) + sizeof(PyUnicodeObject)) : ""   

foo$target::: {}

pid$target::PyEval_EvalCodeEx:entry {
    self->co = (PyCodeObject*) copyin(arg0, sizeof(PyCodeObject));

    trace(GET_Py3_STRING(self->co->co_filename));
    trace(GET_Py3_STRING(self->co->co_name));
    trace(self->co->co_firstlineno);
}

SystemTap can extract type information directly from DWARF section of shared libraries so all we need to do to achieve same effect in it is to use @cast expression:

#!/usr/bin/env stap

@define PYTHON3_LIBRARY %( "/usr/lib64/libpython3.4m.so.1.0" %)

function get_py3_string:string(uo: long) {
    flags = user_uint32(&@cast(uo, "PyASCIIObject", @PYTHON3_LIBRARY)->state);
    if(flags & 0xE0) {
        size = &@cast(0, "PyASCIIObject", @PYTHON3_LIBRARY)[1]
        return user_string(uo + size);
    }
    
    return "???";
}

probe process(@PYTHON3_LIBRARY).function("PyEval_EvalCodeEx") {
    code = $_co;
    if(code) {
        printf("%s %s:%d\n",
               get_py3_string(@cast(code, "PyCodeObject", @PYTHON3_LIBRARY)->co_name),
               get_py3_string(@cast(code, "PyCodeObject", @PYTHON3_LIBRARY)->co_filename),
               @cast(code, "PyCodeObject", @PYTHON3_LIBRARY)->co_firstlineno);
    }
}

References

• Python: Bugs #4111, #13405 and #21590
• Perl: perldtrace
• PHP: Using PHP and DTrace
• Ruby: DTrace Probes
• Erlang: DTrace and Erlang/OTP