LTTng 2.6 Has Been Released!
At long last, LTTng 2.6 Gaïa has been released!
On top of several bug fixes and improvements, this new release introduces three new major features: Java log4j support, per-syscall tracing, and client machine interface (MI).
Introducing Java log4j support
Building upon the infrastructure developed for the
java.util.logging framework support
introduced in LTTng 2.4, this new release adds support for the
log4j logging framework. This was
accomplished by supplying an LTTng
Appender
implementation as part of the user space tracer.
This means that LTTng can now be used as an Appender in applications
instrumented with the log4j logging framework. By enabling events in
the newly introduced log4j domain, log4j events can be recorded as part
of LTTng traces and be automatically correlated with other user space or kernel events.
log4j events are enabled in the same way as events of any other type. The only
difference being that the log4j domain must be selected using the --log4j/-l
command option.
lttng enable-event --log4j my_log4j_event
A blog post specifically dedicated to Java tracing is coming soon… In the meantime, here's a quick demo!
Per-syscall tracing
While LTTng has supported system call tracing for a while, this release introduces per-syscall events which makes it possible to trace only a subset of syscalls. This has the benefit of significantly reducing the tracing bandwidth in scenarios where only a few system calls prove to be of interest.
This feature is provided on top of LTTng's existing kernel tracing facilities which makes the control of syscall tracing behave in the same way as regular kernel tracing.
For instance, a list of available syscalls can be obtained using:
lttng list --kernel --syscall
Individual system calls can then be enabled using:
lttng enable-event --kernel --syscall open
What's more, this new version of the kernel tracer records a subset of syscall payloads which makes the resulting traces much clearer and more useful. Let's have a look at two examples.
In previous versions of the LTTng kernel tracer, the syscall information was limited to the syscall's number along with
a list of unnamed arguments. However, looking at the connect() syscall
in a trace produced by LTTng 2.6, we immediately notice that arguments are now named and, in some cases,
complete structures are serialized to the trace.
[14:04:55.255727726] (+0.000000599) XThink syscall_entry_connect: { cpu_id = 3 }, { fd = 253, uservaddr = 0x7F224E251660, addrlen = 0x10, family = 2, dport = 53, _v4addr_length = 4, v4addr = [ [0] = 172, [1] = 18, [2] = 0, [3] = 12 ], _v6addr_length = 0, v6addr = [ ] }
[14:04:55.255733547] (+0.000005821) XThink syscall_exit_connect: { cpu_id = 3 }, { ret = 0 }
Also, while previous versions of the kernel tracer have always provided the syscalls' return codes, this new version also
serializes output parameters. accept() is a good
example of such a syscall which provides an optional output parameter in addition to a file descriptor.
[16:51:03.465062024] (+20.740838903) XThink syscall_entry_accept: { cpu_id = 1 }, { fd = 17, upeer_sockaddr = 0x7F8674000AAC, upeer_addrlen = 16 }
[16:51:03.465067460] (+0.000005436) XThink syscall_exit_accept: { cpu_id = 1 }, { ret = 41, upeer_addrlen = 16, family = 2, sport = 52530, _v4addr_length = 4, v4addr = [ [0] = 192, [1] = 222, [2] = 138, [3] = 66 ], _v6addr_length = 0, v6addr = [ ] }
Here is a quick demo of the syscall tracing feature in action:
This payload collection capability was also instrumental in the development of the LTTng-Analyses project's trace analysis scripts.
Client machine interface (MI)
This feature introduces a new lttng tool option, --mi TYPE/-m TYPE,
which can be used in conjunction with all lttng commands to produce
a machine output instead of the regular human-readable output emitted by the tool.
As of this release, the only available MI format is xml. A schema
definition (XSD) is made available
to ease the integration with external tools as much as possible.
In essence, this interface has been added to streamline the development of client usability improvements since external tools can now depend on a stable and versioned interface. Therefore, it will now be easier to improve the client's UI in future release without fear of breaking third-party tools compatibility.
To use the machine interface, simply use the --mi TYPE/-m TYPE
global command option.
lttng --mi xml list my_session
Which turns the regular pretty-printed output into the following machine readable XML.
<?xml version="1.0" encoding="UTF-8"?>
<command>
<name>list</name>
<output>
<sessions>
<session>
<name>my_session</name>
<path>/home/jgalar/lttng-traces/my_session-20150223-184101</path>
<enabled>false</enabled>
<snapshot_mode>0</snapshot_mode>
<live_timer_interval>0</live_timer_interval>
<domains>
<domain>
<type>KERNEL</type>
<buffer_type>PER_PID</buffer_type>
<channels>
<channel>
<name>channel0</name>
<enabled>true</enabled>
<attributes>
<overwrite_mode>DISCARD</overwrite_mode>
<subbuffer_size>262144</subbuffer_size>
<subbuffer_count>4</subbuffer_count>
<switch_timer_interval>0</switch_timer_interval>
<read_timer_interval>200000</read_timer_interval>
<output_type>SPLICE</output_type>
<tracefile_size>0</tracefile_size>
<tracefile_count>0</tracefile_count>
<live_timer_interval>0</live_timer_interval>
</attributes>
<events>
<event>
<name>sched_switch</name>
<type>TRACEPOINT</type>
<enabled>true</enabled>
<filter>false</filter>
<loglevel>TRACE_EMERG</loglevel>
<loglevel_type>ALL</loglevel_type>
<exclusion>false</exclusion>
</event>
</events>
</channel>
</channels>
</domain>
<domain>
<type>UST</type>
<buffer_type>PER_UID</buffer_type>
<channels>
<channel>
<name>lttng_log4j_channel</name>
<enabled>true</enabled>
<attributes>
<overwrite_mode>DISCARD</overwrite_mode>
<subbuffer_size>131072</subbuffer_size>
<subbuffer_count>4</subbuffer_count>
<switch_timer_interval>0</switch_timer_interval>
<read_timer_interval>0</read_timer_interval>
<output_type>MMAP</output_type>
<tracefile_size>0</tracefile_size>
<tracefile_count>0</tracefile_count>
<live_timer_interval>0</live_timer_interval>
</attributes>
<events>
<event>
<name>lttng_log4j:sys*</name>
<type>TRACEPOINT</type>
<enabled>true</enabled>
<filter>false</filter>
<exclusion>false</exclusion>
</event>
</events>
</channel>
</channels>
</domain>
<domain>
<type>LOG4J</type>
<buffer_type>PER_UID</buffer_type>
<events>
<event>
<name>*</name>
<type>TRACEPOINT</type>
<enabled>true</enabled>
<filter>false</filter>
<loglevel>LOG4J_ALL</loglevel>
<loglevel_type>ALL</loglevel_type>
<exclusion>false</exclusion>
</event>
</events>
</domain>
</domains>
</session>
</sessions>
</output>
</command>
Amazingly, Trace Compass and the LTTng Bash completion script have already started using this new feature!
Conclusion
This new release should be rolling out into your distribution of choice's repositories soon enough. In the meantime, feel free to give it a try.
The documentation section has also been updated to cover all changes introduced in LTTng 2.6.
Thanks to everyone who has contributed to this release!