yactfr

yactfr (yac·tif·er) is yet another CTF reader.

The yactfr library is written in C++17 and offers a (🥁…) C++17 API.

While the CTF reading libraries that I know about focus on decoding and providing you with complete, ordered event record objects, the yactfr API offers a lower level of CTF processing: you iterate individual element sequences to obtain elements: beginning/end of packet, beginning/end of event record, beginning/end of structure, individual data stream scalar values like fixed-length integers, fixed-length floating point numbers, and null-terminated strings, specific clock value update, known data stream ID, and the rest. This means the sizes of event records and packets don’t influence the performance or memory usage of yactfr.

Some use cases where this library can prove useful:

Trace inspection programs, for example when you need to debug a custom CTF producer.
Foundation of an API with a higher level of abstraction.
Targeted offline analyses with low memory usage.

The name yactfr takes its inspiration from yajl, a JSON parser with a similar approach.

Warning

yactfr is not mature yet! Its API is still experimental: the interface could change at any time.

Table of Contents

Notable features
Limitations
Performance
Build and install yactfr
Use yactfr in a CMake project
Run the tests
Usage examples
Contribute and report bugs
Community

Notable features

Full CTF 1.8.3 and CTF 2 support.

Only depends on Boost at build time, and nothing at run time (except for your C++ runtime library):

$ ldd libyactfr.so
        linux-vdso.so.1 (0x00007ffe98fa5000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fb8337d2000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fb83343d000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fb833225000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fb832e69000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb833e51000)

Simple, documented C++17 API with a focus on immutability.
Efficient data stream decoding: zero memory allocation, zero smart pointer reference count change, zero virtual method call, and zero copy during an element sequence iteration.
- yactfr decodes a data stream fixed-length bit array in place: the result is available as an element member.
- A data stream string/BLOB is available as one or more raw data parts in consecutive elements. The actual raw data is a pointer to the current data block of the data source (for example, from a memory mapped region), which means zero string/BLOB copy.
  
  This is possible because, as per CTF 1.8.3 and CTF2-SPEC-2.0, data stream strings and BLOBs have an 8-bit alignment requirement.
Decodes CTF packets of any size and event records of any size with steady memory usage and performance.
Offers a memory mapped file view data source, but you can also implement your own data source.
It’s safe to use two different iterators on the same element sequence in two different threads.
Full CTF metadata text inspection and validation: a parsing error exception contains a list of locations (offset in bytes, line number, and column number) with associated context messages.
Full data stream packet decoding information with the input iterator interface, for example:
- The decoded packet magic number.
- The decoded metadata stream UUID.
- The expected total length of the current packet.
- The expected content length of the current packet.
- The default clock is updated.
- The next data stream type to use.
- The next event record type to use.
- The data stream (instance) ID.
“Seek packet” iterator method to seek the beginning of a packet known to be located at a specific offset, in bytes, within the same element sequence.

You can use this method with non-CTF packet index information, for example LTTng's packet index files (not directly supported by yactfr).
Decoding error reporting with precise message, offset in element sequence (bits) where this message applies within the element sequence, and other relevant properties.
Performance (nop iteration loop) is about 1.3 times faster than Babeltrace 1.5’s (-o dummy) with a release build.

See Performance for actual numbers.

Limitations

Note that after a few years of working with all sorts of real life CTF traces, I never caught one which yactfr would refuse to decode, but I’m still honest:

Only builds on a Linux/Unix platform.

The non-portable part is the memory mapped file source which uses system functions such as open(), close(), fstat(), mmap(), and madvise().
Decodes up to, and including, 64-bit signed/unsigned CTF fixed-length bit arrays and integers (no “big integers”).
Decodes up to, and including, 63-bit (effective, which means 72 total) signed/unsigned CTF variable-length integers.
Only decodes 16-bit (CTF 2 only), 32-bit, and 64-bit CTF fixed-length floating point numbers (IEEE 754 binary16, binary32, and binary64).
Packet lengths must be multiples of 8 bits (I’m still not sure, but this could be enforced by the specification anyway), and be at least 8 bits.
API and ABI backward compatibility is not guaranteed at this point.

Please rebuild your project if you change the yactfr version.

Performance

I measured the throughput of a nop iteration loop, that is, a loop which only counts the elements and the event record beginning elements:

yactfr::MemoryMappedFileViewFactory factory {
    path, std::nullopt,
    yactfr::MemoryMappedFileViewFactory::AccessPattern::Sequential
};

yactfr::ElementSequence seq {traceType, factory};

for (auto& elem : seq) {
    ++elemCount;

    if (elem.isEventRecordBeginningElement()) {
        ++erCount;
    }
}

The data stream file is a 509-MiB CTF 1.8 LTTng kernel data stream file (16,143,024 event records, 385,128,330 elements, about 33 bytes and 24 elements per event record).

Measure	Throughput (median)
Elements/second	154 M
Event records/second	6.45 M
Data stream bytes/second	203 MiB

Test conditions:

Release build (GCC 16.2, -O3 -DNDEBUG).
Single thread, pinned to one core of an AMD Ryzen 7 PRO 8840U laptop CPU (amd-pstate-epp driver, powersave governor, boost enabled, on AC power).
The data stream file is in the page cache: no disk I/O.
Three untimed warm-up iterations, then 15 timed iterations of the whole data stream file, repeated in three distinct processes.

The standard deviation within a process is about 0.6 % and the medians of the three processes are within about 1 % of each other.

Your numbers will vary depending on your CPU and on the trace: data streams with more, smaller fields produce more elements per byte.

Build and install yactfr

Make sure you have the build time requirements:

Linux/Unix platform
CMake ≥ 3.30.0
C++17 compiler
Boost ≥ 1.70
If you build the API documentation: Doxygen

Build and install yactfr from source

$ git clone https://github.com/eepp/yactfr
$ cd yactfr
$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make
# make install

You can specify your favorite C and C++ compilers with the usual CC and CXX environment variables when you run cmake, and additional options with CFLAGS and CXXFLAGS.

Specify -DOPT_BUILD_DOC=YES to cmake to enable the HTML API documentation build (requires Doxygen). The documentation is available in BUILD/doc/api/output/html, where BUILD is your build directory.

Specify -DCMAKE_INSTALL_PREFIX=PREFIX to cmake to install yactfr to the PREFIX directory instead of the default /usr/local directory.

For example, this is how I run cmake for development:

$ CC=clang CXX=clang++ CXXFLAGS='-Wextra -Wall -pedantic' \
  cmake .. -DCMAKE_BUILD_TYPE=Debug -DOPT_BUILD_DOC=ON

For production, you should make a release build:

$ CC=clang CXX=clang++ \
  cmake .. -DCMAKE_BUILD_TYPE=Release -DOPT_BUILD_DOC=ON

Other cmake options:

YACTFR_BUILD_SHARED: Build yactfr as a shared library instead of a static one.

Default: ON when yactfr is the top-level project, OFF otherwise.
YACTFR_INSTALL: Generate the installation rules (library, headers, and CMake package).

Default: ON when yactfr is the top-level project, OFF otherwise.
YACTFR_BUILD_TESTS: Add the test targets (tests and check).

Default: ON when yactfr is the top-level project, OFF otherwise.

Use yactfr in a CMake project

Both methods below give you the yactfr::yactfr target which carries the include directory and the other usage requirements.

Use an installed yactfr.

find_package (yactfr REQUIRED)
target_link_libraries (my-program yactfr::yactfr)

If yactfr isn’t installed in a standard location, set CMAKE_PREFIX_PATH to its installation prefix when you run cmake.

Fetch and build yactfr as part of your project.

include (FetchContent)
FetchContent_Declare (
    yactfr
    GIT_REPOSITORY https://github.com/eepp/yactfr.git
    GIT_TAG COMMITID
)
FetchContent_MakeAvailable (yactfr)
target_link_libraries (my-program yactfr::yactfr)

As a subproject, yactfr builds a static library by default, without tests or installation rules: see the options above.

Run the tests

Once you have built the project in the build directory, you can run the tests. You need Python 3 and pytest.

Run the yactfr tests from the build directory.

$ make check

If you’re in a hurry and you have the pytest-xdist package, you can parallelize the testing process. You need to set the YACTFR_BINARY_DIR environment variable to the build directory (absolute path), for example:

Run the yactfr tests in parallel from the build directory.

$ make tests
$ YACTFR_BINARY_DIR=$(pwd) pytest -n logical ../tests

Usage examples

In the examples below, the program accepts two arguments:

The path to the metadata stream file of the trace (required).
The path to a data stream file of the same trace (required).

Build the API documentation for a thorough reference.

Note	The examples are not necessarily optimal: their purpose is to show what the yactfr API looks like.

Example 1. Print all the data stream’s event record names.

#include <cassert>
#include <fstream>
#include <iostream>
#include <yactfr/yactfr.hpp>

int main(const int argc, const char * const argv[])
{
    assert(argc == 3);

    // open metadata stream file
    std::ifstream metadataFile {argv[1], std::ios::binary};

    // create metadata stream object
    const auto metadataStream = yactfr::createMetadataStream(metadataFile);

    // we have the metadata text at this point: safe to close the file
    metadataFile.close();

    // get a trace type from the metadata text
    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream->text());

    // create a memory mapped file view factory to read the data stream file
    yactfr::MemoryMappedFileViewFactory factory {argv[2]};

    // create an element sequence from the trace type and data source factory
    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};

    // print all the event record names
    for (auto& elem : seq) {
        if (elem.isEventRecordInfoElement()) {
            auto& erInfo = elem.asEventRecordInfoElement();

            // the name of an event record type is optional
            if (erInfo.type()->name()) {
                std::cout << *erInfo.type()->name() << std::endl;
            }
        }
    }
}

Example 2. Print all the fixed-length signed integers of the sched_switch event records and their offset.

#include <cassert>
#include <fstream>
#include <iostream>
#include <yactfr/yactfr.hpp>

int main(const int argc, const char * const argv[])
{
    assert(argc == 3);

    // open metadata stream file
    std::ifstream metadataFile {argv[1], std::ios::binary};

    // create metadata stream object
    const auto metadataStream = yactfr::createMetadataStream(metadataFile);

    // we have the metadata text at this point: safe to close the file
    metadataFile.close();

    // get a trace type from the metadata text
    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream->text());

    // create a memory mapped file view factory to read the data stream file
    yactfr::MemoryMappedFileViewFactory factory {argv[2]};

    // create an element sequence from the trace type and data source factory
    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};

    // print all the fixed-length signed integers of the `sched_switch` ERs
    const auto endIt = seq.end();
    bool inSchedSwitchEventRecord = false;

    for (auto it = seq.begin(); it != endIt; ++it) {
        if (it->isEventRecordInfoElement()) {
            auto& ertElem = it->asEventRecordInfoElement();

            // the name of an event record type is optional
            if (ertElem.type()->name() && *ertElem.type()->name() == "sched_switch") {
                std::cout << "---" << std::endl;
                inSchedSwitchEventRecord = true;
            } else {
                inSchedSwitchEventRecord = false;
            }

            continue;
        } else if (it->isEventRecordEndElement()) {
            // leaving the event record: clear the flag
            inSchedSwitchEventRecord = false;
            continue;
        }

        if (inSchedSwitchEventRecord && it->isFixedLengthSignedIntegerElement()) {
            std::cout << it.offset() << ": ";

            auto& intElem = it->asFixedLengthSignedIntegerElement();

            if (intElem.structureMemberType()) {
                std::cout << intElem.structureMemberType()->displayName() << ": ";
            }

            std::cout << intElem.value() << std::endl;
        }
    }
}

Example 3. Print all the packet offsets and lengths (both in bits): slow version.

In this example, we iterate all the elements of the data stream. The next example shows how to do the same faster.

#include <cassert>
#include <fstream>
#include <iostream>
#include <iomanip>
#include <yactfr/yactfr.hpp>

int main(const int argc, const char * const argv[])
{
    assert(argc == 3);

    // open metadata stream file
    std::ifstream metadataFile {argv[1], std::ios::binary};

    // create metadata stream object
    const auto metadataStream = yactfr::createMetadataStream(metadataFile);

    // we have the metadata text at this point: safe to close the file
    metadataFile.close();

    // get a trace type from the metadata text
    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream->text());

    // create a memory mapped file view factory to read the data stream file
    yactfr::MemoryMappedFileViewFactory factory {argv[2]};

    // create an element sequence from the trace type and data source factory
    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};

    // print all the packet offsets and lengths (both in bits)
    const auto endIt = seq.end();
    yactfr::Index curPktOffset = 0;
    unsigned long curPktNumber = 0;

    for (auto it = seq.begin(); it != endIt; ++it) {
        if (it->isPacketBeginningElement()) {
            // save packet beginning offset
            curPktOffset = it.offset();
        } else if (it->isPacketEndElement()) {
            // back to first level: end of packet
            const auto pktLen = it.offset() - curPktOffset;

            std::cout << "Packet #" << curPktNumber << ":    " <<
                         "Offset: " << std::setw(10) << curPktOffset << "    " <<
                         "Length: " << std::setw(10) << pktLen <<
                         std::endl;
            ++curPktNumber;
        }
    }
}

Example 4. Print all the packet offsets and lengths (both in bits): fast version.

This is a faster version of the previous example.

Instead of decoding the whole packet to find its length, we use the expected packet total length property of the “packet info” element. This element is available after the decoder reads the packet context. Then, we make the iterator seek the next packet directly.

Note that this example doesn’t work if the packet context type doesn’t contain an expected packet total length fixed-length unsigned integer, in which case the data stream must contain a single packet. This could be detected by inspecting the metadata (trace type) and using the size of the whole data stream file as the unique packet total length.

#include <cassert>
#include <fstream>
#include <iostream>
#include <iomanip>
#include <yactfr/yactfr.hpp>

int main(const int argc, const char * const argv[])
{
    assert(argc == 3);

    // open metadata stream file
    std::ifstream metadataFile {argv[1], std::ios::binary};

    // create metadata stream object
    const auto metadataStream = yactfr::createMetadataStream(metadataFile);

    // we have the metadata text at this point: safe to close the file
    metadataFile.close();

    // get a trace type from the metadata text
    auto traceTypeMsUuidPair = yactfr::fromMetadataText(metadataStream->text());

    // create a memory mapped file view factory to read the data stream file
    yactfr::MemoryMappedFileViewFactory factory {argv[2]};

    // create an element sequence from the trace type and data source factory
    yactfr::ElementSequence seq {*traceTypeMsUuidPair.first, factory};

    // print all the packet offsets and lengths (both in bits)
    const auto endIt = seq.end();
    auto it = seq.begin();
    yactfr::Index curPktOffset = 0;
    unsigned long curPktNumber = 0;

    while (it != endIt) {
        if (it->isPacketBeginningElement()) {
            // save packet beginning offset
            curPktOffset = it.offset();
        } else if (it->isPacketInfoElement()) {
            // this element contains the expected total length of the current packet
            auto& elem = it->asPacketInfoElement();

            assert(elem.expectedTotalLength());
            std::cout << "Packet #" << curPktNumber << ":    " <<
                         "Offset: " << std::setw(10) << curPktOffset << "    " <<
                         "Length: " << std::setw(10) << *elem.expectedTotalLength() <<
                         std::endl;
            ++curPktNumber;

            /*
             * Seek the next packet without iterating the intermediate
             * elements. The expected offset is in bytes, so we need to
             * divide what we have by 8.
             */
            it.seekPacket((curPktOffset + *elem.expectedTotalLength()) / 8);
            continue;
        }

        ++it;
    }
}

Contribute and report bugs

Please contribute with GitHub pull requests and report bugs as GitHub issues.

Community

See eepp.ca.

I’m eepp on Libera.Chat and OFTC.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
cmake		cmake
doc		doc
include		include
tests		tests
tools		tools
yactfr		yactfr
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
HOWITWORKS.adoc		HOWITWORKS.adoc
LICENSE		LICENSE
README.adoc		README.adoc

Navigation Menu

Repository navigation

About

Repository files navigation

yactfr

Notable features

Limitations

Performance

Build and install yactfr

Use yactfr in a CMake project

Run the tests

Usage examples

Contribute and report bugs

Community

About

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

yactfr

Notable features

Limitations

Performance

Build and install yactfr

Use yactfr in a CMake project

Run the tests

Usage examples

Contribute and report bugs

Community

About

Topics

Resources

Stars

Watchers

Forks

Used by

Contributors

Languages