Overview

The LTTng project aims at providing highly efficient tracing tools for Linux. Its tracers help tracking down performance issues and debugging problems involving multiple concurrent processes and threads. Tracing across multiple systems is also possible.

Apart from LTTng's kernel tracer and userspace tracer, viewing and analysis tools are part of the project. The LTTV viewer permits to analyze and show traces, both in text format and graphically.

LTTng's performance relies on techniques such as Userspace RCU, lockless algorithms, per-cpu data structures and cache impact minimization.

Getting started with LTTng 2.x

Start by looking at the LTTng 2.x page to find out how to deploy LTTng 2.x on your system or you can go directly to the Download section.

Project Updates

LTTNG TOOLCHAIN 2.5.0 IS OUT!

Hi all!

The LTTng 2.5 stable release, codenamed Fumisterie, is finally ready!

LTTng-tools 2.5.0

As a reminder, this release includes a save and load session feature, UST perf counter support, daemon configuration file and the ability to load user defined kernel probes using --kmod-probes with lttng-sessiond.

LTTng-UST 2.5.0

  • New tracef() instrumentation facility.
  • Excerpt from lttng-ust(3) man page:

    USAGE WITH TRACEF

    The simplest way to add instrumentation to your code is by far the
    tracef() API. To do it, in a nutshell:

    1) #include

    2) /* in your code, use like a printf */
    tracef("my message, this integer %d", 1234);

    3) Link your program against liblttng-ust.so.

    4) Enable UST events when tracing with the following sequence of commands
    from lttng-tools:

    lttng create
    lttng enable-event -u -a
    lttng start
    [... run your program ...]
    lttng stop
    lttng view

    That's it!

    If you want to have more flexibility and control on the event names,
    payload typing, etc, you can continue reading on and use the trace‐
    points below. "tracef()" is there for quick and dirty ad hoc instrumen‐
    tation, whereas tracepoint.h is meant for thorough instrumentation of a
    code base to be integrated with an upstream project.

  • Perf PMU counters support from user-space on x86.
  • Library base address dump is now stable (thanks to Mentor for their contribution)
  • LTTng-modules 2.5.0
    State dump of block devices,
    - State dump of file descriptor flags and modes,
    - 3.15 Linux kernel support,
    - v4lv2 instrumentation support,
    - MIPS32 system call tracing,
    - New /proc/lttng-logger ABI.

    How to use the new /proc/lttng-logger ABI:

    From the command line (for instance):

    Start lttng-sessiond as root,

    # lttng-sessiond

    Start tracing from root or from a user belonging to the
    "tracing" group:

    % lttng create
    % lttng enable-event -k lttng-logger

    Write event payload data to /proc/lttng-logger (from any user,
    does not need to be in any group):

    % echo -n "Some message" > /proc/lttng-logger

    % Stop tracing, view trace

    % lttng stop
    % lttng view

    You should see:

    [10:44:24.810273367] (+?.?????????) thinkos lttng_logger: { cpu_id = 3 }, { _msg_length = 12, msg = "Some message" }

    It is a quick and easy way to trace some events from user-space through the kernel tracer. It is much more basic than UST: lttng-logger is slower (involves system call round-trip to the kernel) and only supports logging strings. But it's very quick and easy to use, especially from scripts.

    As always you can find the different packages in the Download section.

    LTTNG TOOLCHAIN 2.5.0 RC1 NOW AVAILABLE

    A new release candidate of the LTTng toolchain is now available.

    Theses release adds multiple fixes and new features detailed below.

    LTTng-tools

    Save and Load sessions

    This introduce two new commands along with API's functions allowing the user to save/load a session to/from an XML file.

    $ lttng save [OPTIONS] [SESSION]
    (lttng_save_session_* in lttng/save.h)

    $ lttng load [OPTIONS] [SESSION]
    (lttng_load_session* in lttng/load.h)

    The XML format of the session file is described in the tarball in ./src/common/config/session.xsd. Note that this adds a new dependency to libxml2.

    Userspace Perf counter support

    With the lttng add-context command you can now use perf context "perf:thread:*" along with the -u/--userspace option. See --help for an full list.

    Daemon configuration files

    A configuration file can be used for each daemon to control their options. Three configuration files are available:

  • lttng-sessiond.conf
  • lttng-relayd.conf
  • lttng-consumerd.conf
  • These are simple INI format and can be placed in the system wide directory /etc/lttng, ~/$HOME/.lttng/ or user defined with:

    -f --config Load daemon configuration file

    User defined list of kmod-probes

    Thanks to Jan Glauber, it's now possible to load a user defined list of lttng modules probes with the new option of the session daemon or by using an environment variable. For instance:

    --kmod-probes ext4,btrfs,kvm
    or
    LTTNG_KMOD_PROBES=ext4,btrfs,kvm

    These are the major features added for this release. A couple of things also have changed that are worth mentionning.

    The lttng.h header file has been split into several components like event.h, session.h and so on. If you are using lttng.h directly in your application, this should not affect you at all because including lttng.h also includes every other components.

    The deprecated lttng_enable_consumer(), lttng_disable_consumer() and lttng_health_check() API functions have been removed for good.

    The list command now prints the loglevel of JUL events.

    Debug logs now have a timestamp (YAMAN!)

    LTTng-UST

    New tracef() instrumentation facility

    Excerpt from lttng-ust(3) man page:

    USAGE WITH TRACEF

    The simplest way to add instrumentation to your code is by far the
    tracef() API. To do it, in a nutshell:

    1) #include

    2) /* in your code, use like a printf */
    tracef("my message, this integer %d", 1234);

    3) Link your program against liblttng-ust.so.

    4) Enable UST events when tracing with the following sequence of commands
    from lttng-tools:

    lttng create
    lttng enable-event -u -a
    lttng start
    [... run your program ...]
    lttng stop
    lttng view

    If you want to have more flexibility and control on the event names, payload typing, etc, you can continue reading on and use the tracepoints below. "tracef()" is there for quick and dirty ad hoc instrumentation, whereas tracepoint.h is meant for thorough instrumentation of a code base to be integrated with an upstream project.

    LTTng-modules

    New and noteworthy:

  • State dump of block devices
  • State dump of file descriptor flags and modes
  • 3.15 Linux kernel support
  • v4lv2 instrumentation support
  • MIPS32 system call tracing
  • New /proc/lttng-logger ABI
  • How to use the new /proc/lttng-logger ABI:

    From the command line (for instance):

    Start lttng-sessiond as root,

    # lttng-sessiond

    Start tracing from root or from a user belonging to the
    "tracing" group:

    % lttng create
    % lttng enable-event -k lttng-logger

    Write event payload data to /proc/lttng-logger (from any user,
    does not need to be in any group):

    % echo -n "Some message" > /proc/lttng-logger

    % Stop tracing, view trace

    % lttng stop
    % lttng view

    You should see:

    [10:44:24.810273367] (+?.?????????) thinkos lttng_logger: { cpu_id = 3 }, { _msg_length = 12, msg = "Some message" }

    It is a quick and easy way to trace some events from user-space through the kernel tracer. It is much more basic than UST: lttng-logger is slower (involves system call round-trip to the kernel) and only supports logging strings. But it's very quick and easy to use, especially from scripts.

    2014-05-28 lttng-tools 2.5.0-rc1
    Changelog

    2014-05-28 lttng-ust 2.5.0-rc1
    Changelog

    2014-05-28 lttng-modules 2.5.0-rc1
    Changelog

    Available in the Download section.

    LTTng toolchain 2.4.1 minor releases

    This version is bug fixes mostly aiming at stability.

    LTTng Tools 2.4.1
    2014-04-08 lttng-tools 2.4.1 (OpenSSL heartbleed day)

  • Fix: don't delete stream from connection recv list
  • Fix: use after free of a relayd stream
  • Fix: don't print stream name in error message
  • Fix: take session list lock when listing tp
  • Fix: add consumer wake up pipe to avoid race
  • Fix: don't spawn relayd if URL is provided
  • Fix: don't ask data pending if session was not started
  • Fix: missing test file in EXTRA dist
  • Fix: allow empty URL for live session creation
  • Fix: missing valid return code when adding an URI to consumer
  • Fix: syntax error in lttng.1
  • Fix: check relayd fd leak in lttng cmdline
  • Fix: remove unused tp in high-throughput test
  • Use autoconf AM_MAINTAINER_MODE.
  • Fix: clang 'constant-out-of-range-compare' warning
  • Fix: Unchecked session pointer when destroying a connection in relayd
  • LTTng-UST 2.4.1
    2014-04-08 lttng-ust 2.4.1

  • Revert "Fix: disable liblttng-ust-dl if dlinfo is not available in C library"
  • Fix: .split() the CC environment variable in lttng-gen-tp
  • Fix: disable liblttng-ust-dl if dlinfo is not available in C library
  • Fix: python invocation through env
  • Fix: Override AM_PATH_PYTHON's default action-if-not-found
  • Fix: don't accept configure --disable-shared
  • Fix: configure.ac: add missing result to alignment req. check
  • Fix: malloc wrapper: infinite recursion with compat TLS
  • Fix: liblttng-ust-libc-wrapper recursive use of calloc
  • Fix: mismatch between code and comments
  • Fix: incorrect urcu git URL in README
  • LTTng-Modules 2.4.1
    2014-04-08 LTTng modules 2.4.1

  • Fix: rcu instrumentation: add const qualifier to char pointers
  • Fix: add missing module version information
  • Fix: block instrumentation: < 3.14 don't have bi_iter
  • Fix: update btrfs instrumentation to kernel 3.14
  • Fix: update block layer instrumentation to kernel 3.14
  • Fix file permissions for lttng-statedump-impl.c
  • As always you can find the different packages in the Download section.

    Help us improve the usability of the TMF viewer

    École Polytechnique de Montréal is welcoming an intern in software
    ergonomy. Ludovic Lefebvre will contribute in identifying problems with
    TMF, the Eclipse viewer used with traces from LTTng, and make updates which will make it more
    user-friendly.

    The following questionnaire will help understand the usage one does
    with TMF and how it is perceived. It takes approximately 10 minutes to
    complete and is anonymous. Here is a link to the questionnaire:
    http://secretaire.dorsal.polymtl.ca/~llefebvre/

    Your feedback is welcome, whether you are a long time user of TMF or
    just starting. Ludovic can be contacted directly at
    ludovic.lefebvre@polymtl.ca

    Thanks you for your help.

    Babeltrace 1.2.0

    After almost a year in the making, Babeltrace v1.2 is out and packs a host of great features!

    The first of which is the CTF-Writer API which provides an easy way for applications to produce CTF traces. This API is not meant as a tracing solution. It provides an easy way to write spec-compliant trace converters, merge existing traces and, perhaps, add CTF logging capabilities in low event throughput applications where integrating LTTng would prove impractical.

    We have also overhauled and integrated the Python bindings branch into this release. These bindings now provide an easy way for Python application developers to read and produce CTF traces. Have a look at the example scripts to learn how you can roll out your own trace analysis and conversion tools!

    Finally, we added an lttng-live plugin which leverages LTTng 2.4’s brand new live tracing feature. See the LTTng 2.4 release notes for more details on this feature.

    Babeltrace 1.2.0 Changelog

    Available in the Download section.

    LTTng Toolchain 2.4.0 is out!

    Hi all!

    After five release candidates, the LTTng 2.4 stable release, codenamed Époque Opaque, is finally ready!

    LTTng Tools 2.4.0

    The most important feature of this 2.4 release is the addition of live tracing support. A trace reading client may now connect to the relay daemon to read a trace as it is produced by both kernel and userspace tracers.

    Here is a short guide demonstrating this feature using Babeltrace v1.2.0:

    $ lttng create my_session --live
    $ lttng enable-event -k -a
    $ lttng start

    Active sessions can then be listed using Babeltrace

    $ babeltrace -i lttng-live net://localhost
    net://localhost/host/Hostname/my_session (timer = 1000000, 5 stream(s), 0 client(s) connected)

    And a connection can be established

    $ babeltrace -i lttng-live net://localhost/host/Hostname/my_session

    See the "live reading how-to" and the "create session" section in lttng's man page for more information on this feature.

    Another major feature added as part of this release is Java Util Logging support. It is now possible to enable events in the new JUL domain. These events are produced using lttng-ust's new JUL backend. More information is available in lttng's man page.

    Finally, userspace event exclusion has been added to lttng’s enable-event command which allows users to exclude events even if wildcards are used. This feature is also usable with the --all option.

    LTTng-Tools changelog

    LTTng-UST 2.4.0

    This release brings Java Util Logging tracing to lttng-ust. A new library, liblttng-ust-jul, provides a JUL LogHandler implementation along with the necessary shim to trace events from Java applications. More information is available here, along with an example.

    Also of note, shared object base-address tracing has been added as an experimental feature. This feature allows trace readers to perform address to symbol mappings on dynamically loaded libraries by generating the ust_baddr:push and ust_baddr:pop events on calls to dlopen() and dlclose(). See the lttng-ust-dl man page for more details.

    Noteworthy changes:

  • Added tracing instrumentation for pthread mutex lock functions
  • Added event exclusion-list support
  • LTTng-UST changelog

    LTTng-Modules 2.4.0

    Noteworthy changes:

  • Addition of a new tracepoint in KVM (kvm_write_tsc_offset)
  • Blacklist ARM gcc 4.8.0, 4.8.1, 4.8.2
  • LTTng-Modules changelog

    Please feel free to email the list about any questions/comments concerning this release at lttng-dev@lists.lttng.org. For bugs or missing documentation, use our bug tracker at bugs.lttng.org.

    As always you can find the different packages in the Download section.

    Userspace RCU 0.8.3 and 0.7.11

    Sorry about the quick releases, but I though I should save our
    packagers some time by releasing this fix, so they can skip over
    yesterday's version. It fixes a higher-than-normal CPU usage in
    presence of long RCU read-side critical sections. It does not affect
    correctness of RCU, just the amount of CPU usage in very specific
    scenarios. Since there is only one new commit, I'm putting the full
    commit changelog (from the stable-0.8 branch) below.

    Changelog:
    commit e198fb6a2ebc22ceac8b10d953103b59452f24d4
    Author: Mathieu Desnoyers
    Date: Sat Mar 1 11:33:25 2014 -0500

    Fix: high cpu usage in synchronize_rcu with long RCU read-side C.S.

    We noticed that with this kind of scenario:
    - application using urcu-mb, urcu-membarrier, urcu-signal, or urcu-bp,
    - long RCU read-side critical sections, caused by e.g. long network I/O
    system calls,
    - other short lived RCU critical sections running in other threads,
    - very frequent invocation of call_rcu to enqueue callbacks,
    lead to abnormally high CPU usage within synchronize_rcu() in the
    call_rcu worker threads.

    Inspection of the code gives us the answer: in urcu.c, we expect that if
    we need to wait on a futex (wait_gp()), we expect to be able to end the
    grace period within the next loop, having been notified by a
    rcu_read_unlock(). However, this is not always the case: we can very
    well be awakened by a rcu_read_unlock() executed on a thread running
    short-lived RCU read-side critical sections, while the long-running RCU
    read-side C.S. is still active. We end up in a situation where we
    busy-wait for a very long time, because the counter is !=
    RCU_QS_ACTIVE_ATTEMPTS until a 32-bit overflow happens (or more likely,
    until we complete the grace period). We need to change the wait_loops ==
    RCU_QS_ACTIVE_ATTEMPTS check into an inequality to use wait_gp() for
    every attempts beyond RCU_QS_ACTIVE_ATTEMPTS loops.

    urcu-bp.c also has this issue. Moreover, it uses usleep() rather than
    poll() when dealing with long-running RCU read-side critical sections.
    Turn the usleep 1000us (1ms) into a poll of 10ms. One of the advantage
    of using poll() rather than usleep() is that it does not interact with
    SIGALRM.

    urcu-qsbr.c already checks for wait_loops >= RCU_QS_ACTIVE_ATTEMPTS, so
    it is not affected by this issue.

    Looking into these loops, however, shows that overflow of the loop
    counter, although unlikely, would bring us back to a situation of high
    cpu usage (a negative value well below RCU_QS_ACTIVE_ATTEMPTS).
    Therefore, change the counter behavior so it stops incrementing when it
    reaches RCU_QS_ACTIVE_ATTEMPTS, to eliminate overflow.

    Signed-off-by: Mathieu Desnoyers

    Available in the Download section.

    Another round of releases for 2.4 release candidate cycle

    Theses releases provides fixes for the 2.4 release candidate cycle of the LTTng toolchain.

    2014-01-29 lttng-modules 2.4.0-rc3
    Changelog

    2014-01-29 lttng-ust 2.4.0-rc3
    Changelog

    2014-01-29 lttng-tools 2.4.0-rc4
    Changelog

    Available in the Download section.

    LTTng-Tools 2.4 RC3

    This release provides fixes for the lttng-tools 2.4 release candidate cycle.

    2014-01-14 lttng-tools 2.4.0-rc3

  • Fix: metadata stream should be always flagged as ready
  • Fix: wrong check before destroying the viewer metadata stream
  • Fix: race with the viewer and readiness of streams
  • Fix: missing reset when listing UST fields for multiple PIDs
  • Fix: filter: check binary op nesting
  • Fix: relayd cmd line option for live port
  • Fix: remove break in epoll loop of apps. thread
  • Fix: wrong comment in snapshot public API
  • Fix: clear the CTF traces when all the streams are closed
  • Available in the Download section.

    LTTng toolchain 2.4.0 RC2 now available

    Theses releases provides fixes for the 2.4 release candidate cycle of the LTTng toolchain.

    2013-12-10 lttng-modules 2.4.0-rc2
    Changelog

    2013-12-10 lttng-ust 2.4.0-rc2
    Changelog

    2013-12-10 lttng-tools 2.4.0-rc2
    Changelog

    Available in the Download section.

    Syndicate content