Skip to content

Home

Jump to bottom
Jan Ciesko edited this page Jul 29, 2026 · 55 revisions

Kokkos Tools

Kokkos Tools provides a collection of lightweight profiling and debugging utilities that interface with instrumentation hooks in the Kokkos runtime. Relative to vendor-specific tools such as NVTX or ROCTx, these utilities emphasize kernel-centric analysis and preserve application labels attached to Kokkos constructs (for example, kernel launches and Views).

Profiling hooks are compiled into Kokkos executables by default. Compatible applications can therefore load tools at runtime without recompilation or changes to the application build.

Requirement: Dynamic loading of tools requires Kokkos_ENABLE_LIBDL=ON in the Kokkos configuration (the default in most builds).

General Use

A typical workflow is:

  1. Build one or more tool shared libraries.
  2. Identify those libraries to the application via KOKKOS_TOOLS_LIBS or --kokkos-tools-libs.
  3. Set any tool-specific environment variables, then run the application.

Example: sampler with memory events

Configure, build, and install with CMake (recommended):

cd ${YOUR_KTO_SRC_DIR}
mkdir -p mybuild && cd mybuild
cmake .. -DCMAKE_INSTALL_PREFIX=${YOUR_KTO_INSTALL}
make
make install

Load the sampler and memory-events libraries, configure sampling and fencing, and run:

export KOKKOS_TOOLS_LIBS="${YOUR_KTO_INSTALL}/lib/libkp_kokkos_sampler.so;${YOUR_KTO_INSTALL}/lib/libkp_memory_events.so"
export KOKKOS_TOOLS_SAMPLER_SKIP=1
export KOKKOS_TOOLS_GLOBALFENCES=1
./myKokkosApp.exe

In this configuration, KOKKOS_TOOLS_SAMPLER_SKIP=1 profiles every other Kokkos kernel invocation, and KOKKOS_TOOLS_GLOBALFENCES=1 inserts global fences so memory-event state is consistent at sampling points.

Many tools write output whose filenames incorporate the hostname and process ID.

Building against a specific Kokkos installation

To link or discover a particular Kokkos installation, pass -DKokkos_ROOT=${YOUR_KOKKOS_INSTALL} to CMake. This is especially important for connectors that depend on backend-specific APIs (for example, nvtx-connector).

Command-line selection of tools libraries

As an alternative to the environment variable, Kokkos accepts a tools-library path on the application command line:

./myApp.exe --kokkos-tools-libs="${YOUR_KTO_INSTALL}/lib/libkp_memory_events.so"

Makefile builds

Per-tool Makefiles remain supported. Shared libraries are produced in each tool’s source directory (often named kp_*.so rather than libkp_*.so):

cd ${YOUR_KTO_SRC_DIR}/profiling/memory-events && make
cd ${YOUR_KTO_SRC_DIR}/common/kokkos-sampler && make

Set KOKKOS_TOOLS_LIBS to those generated library paths before running the application.

Explicit Instrumentation

Applications and libraries may also call Kokkos profiling APIs directly. Region and section hooks use a push/pop model to delimit coarser scopes:

void foo() {
  Kokkos::Profiling::pushRegion("foo");
  bar();
  stool();
  Kokkos::Profiling::popRegion();
}

Tools

Utilities

  • KernelFilter — Restricts analysis tools to a selected subset of Kokkos kernels and regions.
  • KernelSampler — Restricts analysis tools to sampled Kokkos kernel invocations.

Memory Analysis

  • MemoryHighWater — Reports the high-water mark of application memory usage.
  • MemoryUsage — Produces a per–memory-space timeline of utilization and transfers.
  • MemoryEvents — Tracks allocations and deallocations, and provides MemoryUsage-style information.

Kernel Inspection

  • Space Time Stack — Reports nested, stack-based timing and memory usage across kernels and regions.
  • SimpleKernelTimer — Captures basic kernel timing information.
  • KernelLogger — Prints Kokkos kernel and region events at runtime.
  • Chrome Tracing — Emits Chrome trace events for timeline visualization.
  • Perfetto — Emits Perfetto trace data for system- and application-level analysis.

Third-Party Profiling Tool Hooks

Vendor-provided

  • nvtxConnector — Forwards Kokkos kernel names to NVTX for per-kernel analysis (formerly nvprof-connector).
  • nvtxFocusedConnector — As nvtxConnector, with profiling disabled for filtered-out kernels; intended for use with KernelFilter (formerly nvprof-focused-connector).
  • roctxConnector — Forwards Kokkos kernel names to ROCTx for per-kernel analysis.
  • VTuneConnector — Forwards Kokkos kernel names to Intel VTune.
  • VTuneFocusedConnector — As VTuneConnector, with profiling disabled for filtered-out kernels; intended for use with KernelFilter.

Open-source and external connectors

  • Timemory — Modular connector for timing, memory usage, hardware counters, and related metrics, with optional control of, or name forwarding to, VTune, CUDA profilers, TAU, NVTX, Caliper, and LIKWID. Timemory components may also be used as a basis for custom plug-ins with stdout, text, and JSON output.
  • PAPI — Connector for PAPI hardware-counter collection (KokkosTools_ENABLE_PAPI).
  • SystemTap — Connector for SystemTap-based probing (KokkosTools_ENABLE_SYSTEMTAP).
  • Variorum — Connector for Variorum power and energy interfaces (KokkosTools_ENABLE_VARIORUM).
  • Score-P — Connector for Score-P instrumentation and analysis (documented on the wiki).

Automated Tuning

  • Apex — Autotuning framework with Kokkos support; available via submodule (KokkosTools_ENABLE_APEX). Supports tuning of Kokkos execution parameters such as team size.
  • Apollo — Complements Apex with ML-guided tuning of performance parameters (documented on the wiki).

Automated Analysis

  • Caliper — Performance-analysis framework with Kokkos support (KokkosTools_ENABLE_CALIPER).

Performance Monitoring

  • LDMS connector — Integrates with the Lightweight Distributed Metric Service (LDMS) for HPC system monitoring (documented on the wiki).
  • Energy profiler — Utility for energy-oriented monitoring of Kokkos applications.

Contributing

Kokkos Tools combines connectors developed by the Kokkos team with contributions from the broader community. Contributions are encouraged in the following areas:

  1. Improvements to existing tools (connectors)
  2. New tools (connectors)
  3. Documentation
  4. Experience reports from production Kokkos applications (successes and limitations)

For items 1 and 2, open a GitHub issue and submit a pull request against develop.

For items 3 and 4, contact vlkale@sandia.gov and crtrott@sandia.gov, raise the topic on the Kokkos Team Slack when available, or open a documentation pull request (for example to README.md or Build.md). With contributor permission, representative case studies may be included in tutorials and presentations.

Additional guidance: General Development and Submitting a Pull Request.

Tutorials

Related Projects

Additional tooling support for Kokkos includes:

  1. HPCToolkit for Kokkos — John Mellor-Crummey (johnmc@rice.edu)
  2. TAU support for Kokkos — Sameer Shende (sameer@cs.uoregon.edu)
  3. Automated testing of Kokkos programs — Vivek Kale (vlkale@sandia.gov)

Additional Resources

  1. Examples: https://github.com/DavidPoliakoff/kokkos-tools-examples
  2. Project overview: https://kokkos.org
  3. Related discussion (nsys/ncu): https://github.com/NVIDIA/TensorRT-LLM/issues/183

SAND2017-3786

Home:

Introduction

Utilities

Memory Analysis

Kernel Inspection

3rd Party Profiling Tool Hooks

Automated Tuning

Automated Analysis

Performance Monitoring

Contributing

Clone this wiki locally