Comparison with SystemTap and DTrace

Feature Comparison

LTTng Systemtap DTrace
Project
license Kernel tracer: GPLv2
Userspace tracer: LGPLv2.1 (library) and GPLv2 (tools)
GPL CDDL
operating system support Linux Linux Solaris, Mac OS X, BSD, QNX
processor support Kernel tracer: x86-32, x86-64, SPARC, SPARC64, ppc, ppc64, sh, sh64, ia64, s390, MIPS 32/64, ARM, (arch-agnostic core)
Userspace tracer: x86-32, x86-64, ppc, ppc64, S390, S390x, ARM, Alpha, ia64, Sparcv9 32/64
as per kprobes: x86-32, x86-64, ppc64, ia64, s390, arm, sparc? x86-32, x86-64, SPARC, ppc, ppc64
overhead low high high
core developers open community open community open community
development began January 2005 January 2005 October 2001
development status ongoing ongoing stable with continuing development
Collaborators Red Hat, IBM, Hitachi, and Oracle Oracle (owner of Sun Microsystems)
target audience application (userspace) developers, kernel developers, users, sysadmins developers, users, sysadmins developers, users, sysadmins
target usage debugging, tracing, profiling, monitoring debugging, tracing, profiling debugging, tracing, profiling
Language
style C scripting scripting
speculative tracing work in progress yes (from first principles via auxiliary data and control structures) yes
binary tracing yes yes ?
early boot tracing not "as early as it could" yet not yet [[http://sourceware.org/PR2035] yes
probe execution optimized native code optimized native code interpreted bytecodes
Probing Capability
number of available symbolic probe points in the kernel thousands (functions, tracepoints, markers) millions (statements, markers) thousands (functions, markers)
type of instrumentation available in user-space Markers inserted by the developer. GDB tracepoints are used for statement and function instrumentation. statements, functions functions, developer-inserted markers
high-speed breakpoint-less userspace data extraction yes no no
statically inserted probe points, user side yes, fast, breakpoint-less and system call-less in progress http://sourceware.org/PR1918 (slower, breakpoint based) yes (USDT) (slower, breakpoint based)
probe arbitrary statements in code symbolically (function entry, exit, interior, source code co-ordinates) Kernel tracer: within kprobes capabilities
Userspace tracer: with gdb tracepoints
yes (using debugging information) limited to ABI boundaries (function entry/exit, markers)
statically identify any arbritrary data to extract yes (with tracepoints and markers). UST markers integrated with GDB, usable from Eclipse GDB GUI along with GDB dynamic tracepoints. yes (with help of markers) yes (with markers)
non-symbolic (manually addressed) probe points / data Kernel tracer: yes, via kprobes
Userspace tracer: with gdb tracepoints
yes yes
probe dynamically loaded kernel objects yes yes yes
concurrent probes on multiprocessors yes yes yes
context pointer type punning/casting yes not yet http://sourceware.org/PR5634 yes
statically inserted probe points, kernel side yes (tracepoints, markers) yes (markers) yes (SDT)
end-user extendable probe library yes yes (script based tapsets) no
trace user-space stack backtraces had this feature previously, not currently soon http://sourceware.org/PR5635 yes
trace Java programs yes, via debugfs special file (highly efficient java tracing in progress), or via userspace tracer soon (special case of user-space) yes
trace Java stack backtraces no no yes
statically inserted probe points, Java in progress no yes
trace script language programs yes, if the script language can call C not yet yes: Ruby, JavaScript, Perl, Python, PHP, APL, Bourne shell, ksh, zsh, Tcl
timer-based probing no yes (profiling interrupt, software timers) yes
hardware performance counter based probing soon soon http://sourceware.org/PR909 soon
Safety
non-blocking, atomic probe handlers yes yes yes
space-limited execution yes (bounded to buffer size + probe size + memory allocated by probe) yes (static allocation of all data) yes
Analysis performed offline (post-mortem) online online
Trace streaming support yes, traces sent efficiently through TCF (http://git.dorsal.polymtl.ca/?p=lttng-agent.git) ? ?
Built-into the system yes (mainlining in progress) no yes
Lockless tracing yes no yes
Use by unprivileged users work in progress (tracing group) developer vs. user distinction graduated access by privilege level
Efficiency
Zero-copy trace data extraction yes ? no (designed to allow for concise traces)
Analysis Tools
Provides analysis tools to navigate in large multi-GB traces no no (designed to allow for concise traces)
Trace synchronization for correlated analysis and statistics across multiple systems no no

This table is based upon the one on the Systemtap site.

Userspace Tracing Performance Comparison of LTTng / DTrace

See the paper Combined Tracing of the Kernel and Applications with LTTng presented at the Linux Symposium 20