Tracer

• Efficient synchronized User-space and Kernel tracing using the same time-source provides view of correlated kernel and user-space events.
• LTTng 2.0 kernel modules build against a vanilla or distribution kernel, without need for additional patches.
• Tracepoints, detailed syscall tracing (fast strace replacement), CPU Performance Monitoring Unit (PMU) counters, dynamic address/symbol probing (kprobes) and function call/return tracing (kretprobes support).
• Have the ability to attach "context" information to events in the trace (e.g. any PMU counter, pid, ppid, tid, comm name, etc). All the extra information fields to be collected with events are optional, specified on a per-tracing-session basis (except for timestamp and event id, which are mandatory).
• Allows non-root users part of the "tracing" group to perform kernel and userspace tracing.
• Allows multiple tracing sessions to be active concurrently, each with its own instrumentation set.
• Integrated interface for both kernel and userspace tracing.
• Produces CTF (Common Trace Format) natively.
• Userspace tracing supports trace loglevels and event wildcards.
• Support for x86-32, x86-64, PowerPC 32/64, ARMv7 OMAP3, MIPS, sh, sparc64, s390, and others.
• Static tracing (highly optimized).
• Dynamic tracing support using kprobes.
• Self-describing binary trace format (CTF) highly optimized for compactness.
• Efficient probes with no trap and no system call.
• Almost zero performance impact with instrumentation points disabled (kernel: uses code modification, userspace: uses fast boolean evaluation).
• Active instrumentation points have low performance impact (a fast C language function call).
• Kernel tracer scalable to high core numbers (uses RCU mechanism, per-cpu buffers, non-blocking atomic operations).
• Support C/C++ and any language that can call C code.
• Reentrant: kernel tracer NMI safe, userspace tracer thread and signal safe.
• Zero copy from event generation to disk write.
• Generate events with arbitrary number of arguments (variable event sizes).
• Support for conditional tracing through custom probes.
• Accurate timestamps.
• Accurate event ordering, even across cores/cpus (except if hardware makes it impossible).
• Supports simultaneous recording of multiple traces (tracing sessions).
• Support for arbitrary architecture endianness.
• Tracing modes: write-to-disk, flight recorder (currently in development for LTTng 2.x), stream to remote disk (under development for LTTng 2.x), live monitoring (local or remote) (under development for LTTng 2.x).
• Traces composed of several channels that have distinct buffers with their own size.
• Flight recorder mode activation is per-channel.

Trace Analysis

• Analysis of extremely large traces (>100GB) with LTTV.
• New Eclipse-based analyzer program (http://www.eclipse.org/linuxtools/projectPages/lttng/).
• Analysis views: resource view, per thread execution state (control flow view), event rate histogram, detailed event list, filtering, view synchronization, IRQ latency.

Features in progress

• Online monitoring and trace streaming
• Trace correlation
• Multi-level, multi-core distributed trace synchronization