lttng-ust(3) (v2.7)

NAME

lttng-ust —  Linux Trace Toolkit Next Generation User-Space Tracer 2.x

SYNOPSIS

Link liblttng-ust.so with applications, following this manpage.

DESCRIPTION

LTTng-UST, the Linux Trace Toolkit Next Generation Userspace Tracer, is a port of the low-overhead tracing capabilities of the LTTng kernel tracer to user-space. The library "liblttng-ust" enables tracing of applications and libraries.

USAGE WITH TRACEF

The simplest way to add instrumentation to your code is by far the tracef() API. To do it, in a nutshell:

1) #include <lttng/tracef.h>

2) /* in your code, use like a printf */ tracef("my message, this integer %d", 1234);

3) Link your program against liblttng-ust.so.

4) Enable UST events when tracing with the following sequence of commands from lttng-tools:

lttng create lttng enable-event -u -a lttng start [... run your program ...] lttng stop lttng view

That's it!

If you want to have more flexibility and control on the event names, payload typing, etc, you can continue reading on and use the tracepoints below. "tracef()" is there for quick and dirty ad hoc instrumentation, whereas tracepoint.h is meant for thorough instrumentation of a code base to be integrated with an upstream project.

USAGE WITH TRACELOG

If you want to migrate existing logging (info, errors, ...) to LTTng UST, you can use the tracelog() interface. To do it, in a nutshell:

1) #include <lttng/tracelog.h>

2) /* in your code, use like a printf, with extra loglevel info. */ tracelog(TRACE_INFO, "Message with integer %d", 1234);

3) Link your program against liblttng-ust.so.

4) Enable UST events when tracing with the following sequence of commands from lttng-tools:

lttng create lttng enable-event -u "lttng_ust_tracelog:*" lttng start [... run your program ...] lttng stop lttng view

That's it!

You can replace the enable-event line above with a selection of loglevels, e.g.:

lttng enable-event -u -a --loglevel TRACE_INFO

Which will gather all events from TRACE_INFO and more important loglevels.

USAGE WITH TRACEPOINT

The simple way to generate the lttng-ust tracepoint probes is to use the lttng-gen-tp(1) tool. See the lttng-gen-tp(1) manpage for explanation.

Here is the way to do it manually, without the lttng-gen-tp(1) helper script, through an example:

CREATION OF TRACEPOINT PROVIDER


To create a tracepoint provider, within a build tree similar to
examples/easy-ust installed with lttng-ust documentation, see
sample_component_provider.h for the general layout. You will need to
define TRACEPOINT_CREATE_PROBES before including your tracepoint
provider probe in one source file of your application. See tp.c from
easy-ust for an example of a tracepoint probe source file. This manpage
will focus on the various types that can be recorded into a trace
event:

TRACEPOINT_EVENT(
	/*
	 * provider name, not a variable but a string starting with a
	 * letter and containing either letters, numbers or underscores.
	 * Needs to be the same as TRACEPOINT_PROVIDER. Needs to
	 * follow the namespacing guide-lines in lttng/tracepoint.h:
	 *
	 * Must be included before include tracepoint provider
	 * ex.: project_event
	 * ex.: project_component_event
	 *
	 * Optional company name goes here
	 *  ex.: com_efficios_project_component_event
	 *
	 * In this example, "sample" is the project, and "component" is the
	 * component.
	 */
	sample_component,

	/*
	 * tracepoint name, characters permitted follow the same
	 * constraints as the provider name. The name of this example
	 * event is "sample_event".
	 */
	sample_event,

	/*
	 * TP_ARGS macro contains the arguments passed for the tracepoint
	 * it is in the following format
	 *	      TP_ARGS(type1, name1, type2, name2, ... type10,
				 name10)
	 * where there can be from zero to ten elements.
	 * typeN is the datatype, such as int, struct or double **.
	 * name is the variable name (in "int myInt" the name would be
	 * myint)
	 *	      TP_ARGS() is valid to mean no arguments
	 *	      TP_ARGS(void) is valid too
	 */
	TP_ARGS(int, anint, int, netint, long *, values,
		 char *, text, size_t, textlen,
		 double, doublearg, float, floatarg),

	/*
	 * TP_FIELDS describes how to write the fields of the trace event.
	 * You can put expressions in the "argument expression" area,
	 * typically using the input arguments from TP_ARGS.
	 */
	TP_FIELDS(
		/*
		 * ctf_integer: standard integer field.
		 * args: (type, field name, argument expression)
		 */
		ctf_integer(int, intfield, anint)
		ctf_integer(long, longfield, anint)

		/*
		 * ctf_integer_hex: integer field printed as hexadecimal.
		 * args: (type, field name, argument expression)
		 */
		ctf_integer_hex(int, intfield2, anint)

		/*
		 * ctf_integer_network: integer field in network byte
		 * order. (_hex: printed as hexadecimal too)
		 * args: (type, field name, argument expression)
		 */
		ctf_integer_network(int, netintfield, netint)
		ctf_integer_network_hex(int, netintfieldhex, netint)

		/*
		 * ctf_array: a statically-sized array.
		 * args: (type, field name, argument expression, value)
		 */
		ctf_array(long, arrfield1, values, 3)

		/*
		 * ctf_array_text: a statically-sized array, printed as
		 * a string. No need to be terminated by a null
		 * character.
		 * Behavior is undefined if "text" argument is NULL.
		 */
		ctf_array_text(char, arrfield2, text, 10)

		/*
		 * ctf_sequence: a dynamically-sized array.
		 * args: (type, field name, argument expression,
		 *	type of length expression, length expression)
		 * The "type of length expression" needs to be an
		 * unsigned type. As a reminder, "unsigned char" should
		 * be preferred to "char", since the signedness of
		 * "char" is implementation-defined.
		 * Behavior is undefined if "text" argument is NULL.
		 */
		ctf_sequence(char, seqfield1, text,
			     size_t, textlen)

		/*
		 * ctf_sequence_text: a dynamically-sized array, printed
		 * as string. No need to be null-terminated.
		 * Behavior is undefined if "text" argument is NULL.
		 */
		ctf_sequence_text(char, seqfield2, text,
			     size_t, textlen)

		/*
		 * ctf_string: null-terminated string.
		 * args: (field name, argument expression)
		 * Behavior is undefined if "text" argument is NULL.
		 */
		ctf_string(stringfield, text)

		/*
		 * ctf_float: floating-point number.
		 * args: (type, field name, argument expression)
		 */
		ctf_float(float, floatfield, floatarg)
		ctf_float(double, doublefield, doublearg)
	)
)

There can be an arbitrary number of tracepoint providers within an
application, but they must each have their own provider name. Duplicate
provider names are not allowed.

ASSIGNING LOGLEVEL TO EVENTS


Optionally, a loglevel can be assigned to a TRACEPOINT_EVENT using the
following construct:

	TRACEPOINT_LOGLEVEL(< [com_company_]project[_component] >,
		< event >, < loglevel_name >)

The first field is the provider name, the second field is the name of
the tracepoint, and the third field is the loglevel name.  A
TRACEPOINT_EVENT should be declared prior to the the TRACEPOINT_LOGLEVEL
for a given tracepoint name. The TRACEPOINT_PROVIDER must be already
declared before declaring a TRACEPOINT_LOGLEVEL.

The loglevels go from 0 to 14. Higher numbers imply the most verbosity
(higher event throughput expected.

Loglevels 0 through 6, and loglevel 14, match syslog(3) loglevels
semantic. Loglevels 7 through 13 offer more fine-grained selection of
debug information.

   TRACE_EMERG           0
   system is unusable

   TRACE_ALERT           1
   action must be taken immediately

   TRACE_CRIT            2
   critical conditions

   TRACE_ERR             3
   error conditions

   TRACE_WARNING         4
   warning conditions

   TRACE_NOTICE          5
   normal, but significant, condition

   TRACE_INFO            6
   informational message

   TRACE_DEBUG_SYSTEM    7
   debug information with system-level scope (set of programs)

   TRACE_DEBUG_PROGRAM   8
   debug information with program-level scope (set of processes)

   TRACE_DEBUG_PROCESS   9
   debug information with process-level scope (set of modules)

   TRACE_DEBUG_MODULE    10
   debug information with module (executable/library) scope (set of
   units)

   TRACE_DEBUG_UNIT      11
   debug information with compilation unit scope (set of functions)

   TRACE_DEBUG_FUNCTION  12
   debug information with function-level scope

   TRACE_DEBUG_LINE      13
   debug information with line-level scope (TRACEPOINT_EVENT default)

   TRACE_DEBUG           14
   debug-level message

See lttng(1) for information on how to use LTTng-UST loglevels.

ADDING TRACEPOINTS TO YOUR CODE


Include the provider header in each C files you plan to instrument,
following the building/linking directives in the next section.

For instance, add within a function:

		tracepoint(ust_tests_hello, tptest, i, netint, values,
			text, strlen(text), dbl, flt);

As a call to the tracepoint. It will only be activated when requested by
lttng(1) through lttng-sessiond(8).

Even though LTTng-UST supports tracepoint() call site duplicates having
the same provider and event name, it is recommended to use a
provider event name pair only once within the source code to help
map events back to their call sites when analyzing the trace.

Sometimes arguments to the probe are expensive to compute (e.g.
take call stack). To avoid the computation when the tracepoint is
disabled one can use more 'low level' tracepoint_enabled() and
do_tracepoint() macros as following:

	if (tracepoint_enabled(ust_tests_hello, tptest)) {
		/* prepare arguments */
		do_tracepoint(ust_tests_hello, tptest, i, netint, values,
			text, strlen(text), dbl, flt);
	}

Here do_tracepoint() doesn't contain check if the tracepoint is enabled.
Using tracepoint() in such scenario is dangerous since it also contains
enabled check and thus race condition is possible in the following code
if the tracepoint has been enabled after check in tracepoint_enabled()
but before tracepoint():

	if (tracepoint_enabled(provider, name)) { /* tracepoint is disabled */
		prepare(args);
	}
	/* tracepoint is enabled by 'lttng' tool */
	tracepoint(provider, name, args); /* args wasn't prepared properly */

Note also that neither tracepoint_enabled() nor do_tracepoint() have
STAP_PROBEV() call so if you need it you should emit this call yourself.

BUILDING/LINKING THE TRACEPOINT PROVIDER

There are 2 ways to compile the Tracepoint Provider with the
application: either statically or dynamically. Please follow
carefully:

  1) Compile the Tracepoint Provider with the application, either
     directly or through a static library (.a):
    - Into exactly one object of your application, define
      "TRACEPOINT_DEFINE" and include the tracepoint provider.
    - Use "-I." for the compilation unit containing the tracepoint
      provider include (e.g., tp.c).
    - Link the application with "-llttng-ust" and "-ldl".
    - Include the tracepoint provider header into all C files using
      the provider.
    - Examples:
      - doc/examples/easy-ust/   sample.c sample_component_provider.h tp.c
        Makefile
      - doc/examples/hello-static-lib/   hello.c tp.c ust_test_hello.h Makefile

  2) Compile the Tracepoint Provider separately from the application,
     using dynamic linking:
    - Into exactly one object of your application: define
      "TRACEPOINT_DEFINE" _and_ also define
      "TRACEPOINT_PROBE_DYNAMIC_LINKAGE", then include the tracepoint
      provider header.
    - Include the tracepoint provider header into all instrumented C
      files that use the provider.
    - Compile the tracepoint provider with "-I.".
    - Link the tracepoint provider with "-llttng-ust".
    - Link application with "-ldl".
    - Set a LD_PRELOAD environment to preload the tracepoint provider
      shared object before starting the application when tracing is
      needed. Another way is to dlopen the tracepoint probe when needed
      by the application.
    - Example:
      - doc/examples/demo   demo.c  tp*.c ust_tests_demo*.h demo-trace Makefile

  - Note about dlclose() usage: it is not safe to use dlclose on a
    provider shared object that is being actively used for tracing due
    to a lack of reference counting from lttng-ust to the used shared
    object.
  - Enable instrumentation and control tracing with the "lttng" command
    from lttng-tools. See lttng-tools doc/quickstart.txt.
  - Note for C++ support: although an application instrumented with
    tracepoints can be compiled with g++, tracepoint probes should be
    compiled with gcc (only tested with gcc so far).

USING LTTNG UST WITH DAEMONS

Some extra care is needed when using liblttng-ust with daemon
applications that call fork(), clone(), or BSD rfork() without a
following exec() family system call. The library "liblttng-ust-fork.so"
needs to be preloaded for the application (launch with e.g.
LD_PRELOAD=liblttng-ust-fork.so appname).

CONTEXT

Context information can be prepended by the tracer before each, or some, events. The following context information is supported by LTTng-UST:

vtid

Virtual thread ID: thread ID as seen from the point of view of the process namespace.

vpid

Virtual process ID: process ID as seen from the point of view of the process namespace.

ip

Instruction pointer: Enables recording of the exact location where a tracepoint was emitted. Can be used to reverse-lookup the source location that caused the event to be emitted.

procname

Thread name, as set by exec() or prctl(). It is recommended that programs set their thread name with prctl() before hitting the first tracepoint for that thread.

pthread_id

Pthread identifier. Can be used on architectures where pthread_t maps nicely to an unsigned long type.

BASE ADDRESS STATEDUMP

If an application that uses liblttng-ust.so becomes part of a session, information about its currently loaded shared objects will be traced to the session at session-enable time. To record this information, the following event needs to be enabled:

ust_baddr_statedump:soinfo

This event is used to trace a currently loaded shared object. The base address (where the dynamic linker has placed the shared object) is recorded in the "baddr" field. The path to the shared object gets recorded in the "sopath" field (as string). The file size of the loaded object (in bytes) is recorded to the "size" field and its time of last modification (in seconds since Epoch) is recorded in the "mtime" field.

If the event above is enabled, a series of "ust_baddr_statedump:soinfo" events is recorded at session-enable time. It represents the state of currently loaded shared objects for the traced process. If this information gets combined with the lttng-ust-dl(3) instrumentation, all aspects of dynamic loading that are relevant for symbol and line number lookup are traced by LTTng.

ENVIRONMENT VARIABLES

LTTNG_UST_DEBUG

Activate liblttng-ust debug and error output.

LTTNG_UST_REGISTER_TIMEOUT

The environment variable "LTTNG_UST_REGISTER_TIMEOUT" can be used to specify how long the applications should wait for sessiond "registration done" command before proceeding to execute the main program. The default is 3000ms (3 seconds). The timeout value is specified in milliseconds. The value 0 means "don't wait". The value -1 means "wait forever". Setting this environment variable to 0 is recommended for applications with time constraints on the process startup time.

LTTNG_UST_WITHOUT_BADDR_STATEDUMP

Prevent liblttng-ust to perform a base-address statedump on session-enable.

LTTNG_UST_GETCPU_PLUGIN

Used by the getcpu override plugin system. The environment variable provides the path to the shared object which will act as the getcpu override plugin. An example can be found in the lttng-ust documentation under examples/getcpu-override .

LTTNG_UST_CLOCK_PLUGIN

Used by the clock override plugin system. The environment variable provides the path to the shared object which will act as the clock override plugin. An example can be found in the lttng-ust documentation under doc/examples/clock-override .

SEE ALSO

COMPATIBILITY

Older lttng-ust libraries reject more recent, and incompatible, probe providers. Newer lttng-ust libraries accept older probe providers, even though some newer features might not be available with those providers.

BUGS

LTTng-UST 2.0 and 2.1 lttng-ust libraries do not check for probe provider version compatibility. This can lead to out-of-bound accesses when using a more recent probe provider with an older lttng-ust library. These error only trigger when tracing is active. This issue has been fixed in LTTng-UST 2.2.

If you encounter any issues or usability problem, please report it on our mailing list <lttng-dev@lists.lttng.org> to help improve this project.

CREDITS

liblttng-ust is distributed under the GNU Lesser General Public License version 2.1. The headers are distributed under the MIT license.

See http://lttng.org for more information on the LTTng project.

Mailing list for support and development: <lttng-dev@lists.lttng.org>.

You can find us on IRC server irc.oftc.net (OFTC) in #lttng.

THANKS

Thanks to Ericsson for funding this work, providing real-life use-cases, and testing.

Special thanks to Michel Dagenais and the DORSAL laboratory at Polytechnique de Montreal for the LTTng journey.

AUTHORS

liblttng-ust was originally written by Mathieu Desnoyers, with additional contributions from various other people. It is currently maintained by Mathieu Desnoyers <mathieu.desnoyers@efficios.com>.