PhD Thesis | FSE 2026 TORAI | FSE 2026 EventADL | WWW 2025 RCAEval | ASE 2024 RCAEval v1 | FSE 2024 BARO
RCAEval is an open-source benchmark that offers nine datasets with 735 real failure cases, and an evaluation framework for root cause analysis (RCA) in microservice systems. It includes many reproducible baselines covering metric-based, trace-based, and multi-source RCA methods.
We recommend using machines equipped with at least 8 cores, 16GB RAM, and ~50GB available disk space with Ubuntu 22.04 or Ubuntu 20.04, and Python3.12 (Python 3.14 is also supported).
The default environment, which is used for most methods, can be easily installed as follows. Detailed installation instructions for all methods are in SETUP.md.
Open your terminal and run the following commands
sudo apt update -y
sudo apt install -y build-essential \
libxml2 libxml2-dev zlib1g-dev \
python3-tk graphvizClone RCAEval from GitHub
git clone https://github.com/phamquiluan/RCAEval.git && cd RCAEvalCreate virtual environment with Python 3.12 (refer SETUP.md to see how to install Python3.12 on Linux)
python3.12 -m venv env
. env/bin/activateInstall RCAEval using pip
pip install -e .[default]Or, install RCAEval from PyPI
# Install RCAEval from PyPI
pip install RCAEval[default]Test the installation
python -m pytest tests/test.py::test_basicExpected output after running the above command (it takes less than 1 minute)
$ pytest tests/test.py::test_basic
============================== test session starts ===============================
platform linux -- Python 3.12.12, pytest-7.3.1, pluggy-1.0.0
rootdir: /home/ubuntu/RCAEval
plugins: dvc-2.57.3, hydra-core-1.3.2
collected 1 item
tests/test.py . [100%]
=============================== 1 passed in 3.16s ================================The telemetry data must be presented as pandas.DataFrame. We require the data to have a column named time that stores the timestep. A sample of valid data could be downloaded using the download_data() or download_multi_source_data() method that we will demonstrate shortly below.
A basic example to use BARO, a metric-based RCA baseline, to perform RCA are presented as follows,
# You can put the code here to a file named test.py
from RCAEval.e2e import baro
from RCAEval.utility import download_data, read_data
# download a sample data to data.csv
download_data()
# read data from data.csv
data = read_data("data.csv")
anomaly_detected_timestamp = 1692569339
# perform root cause analysis
root_causes = baro(data, anomaly_detected_timestamp)["ranks"]
# print the top 5 root causes
print("Top 5 root causes:", root_causes[:5])Expected output after running the above code (it takes around 1 minute)
$ python test.py
Downloading data.csv..: 100%|████████████████████| 570k/570k [00:00<00:00, 19.8MiB/s]
Top 5 root causes: ['emailservice_mem', 'recommendationservice_mem', 'cartservice_mem', 'checkoutservice_latency', 'cartservice_latency']
A tutorial of using Multi-source BARO to diagnose failure using multi-source telemetry data (metrics, logs, and traces) is presented in docs/multi-source-rca-demo.ipynb.
A tutorial of using BARO to diagnose code-level faults is presented in docs/code-level-rca.ipynb.
RCAEval benchmark includes nine datasets organized into three benchmark suites (RE1, RE2, RE3), each covering three microservice systems (Online Boutique, Sock Shop, Train Ticket). Together, these datasets feature 735 failure cases with 11 fault types. Each failure case includes annotated root cause service and root cause indicator (e.g., specific metric or log indicating the root cause).
| Dataset | System | Cases | Fault Types | Metrics | Logs | Traces |
|---|---|---|---|---|---|---|
| RE1-OB | Online Boutique | 125 | cpu, mem, disk, delay, loss | 49-59 | N/A | N/A |
| RE1-SS | Sock Shop | 125 | cpu, mem, disk, delay, loss | 57-63 | N/A | N/A |
| RE1-TT | Train Ticket | 125 | cpu, mem, disk, delay, loss | 198-238 | N/A | N/A |
| RE2-OB | Online Boutique | 90 | cpu, mem, disk, delay, loss, socket | 69-77 | Yes | Yes |
| RE2-SS | Sock Shop | 90 | cpu, mem, disk, delay, loss, socket | 74-82 | Yes | N/A |
| RE2-TT | Train Ticket | 90 | cpu, mem, disk, delay, loss, socket | 340-376 | Yes | Yes |
| RE3-OB | Online Boutique | 30 | f1, f2, f3, f4, f5 | 68-101 | Yes | Yes |
| RE3-SS | Sock Shop | 30 | f1, f2, f3, f4 | 80-107 | Yes | N/A |
| RE3-TT | Train Ticket | 30 | f1, f2, f3, f4 | 294-322 | Yes | Yes |
| TORAI-OB | Online Boutique | 90 | cpu, mem, disk, delay, loss, socket | 69-77 | Yes | Yes |
| TORAI-SS | Sock Shop | 90 | cpu, mem, disk, delay, loss, socket | 74-82 | Yes | N/A |
| TORAI-TT | Train Ticket | 90 | cpu, mem, disk, delay, loss, socket | 340-376 | Yes | Yes |
RE1 Datasets (375 cases): Metric-only data supporting metric-based RCA methods. Five fault types (CPU, MEM, DISK, DELAY, LOSS) across five services per system, with five repetitions per fault-service pair.
RE2 Datasets (270 cases): Multi-source data (metrics, logs, traces) supporting multi-source RCA methods. Six fault types (adds SOCKET) across five services per system, with three repetitions per fault-service pair.
RE3 Datasets (90 cases): Multi-source data focusing on code-level faults (F1-F5). Supports diagnosing code-level faults through telemetry data, e.g., leveraging stack traces in logs or response codes in traces.
Each dataset directory follows the naming convention: {benchmark}_{service}_{fault}_{instance}
metrics.json: Time-series metrics datainject_time.txt: Fault injection timestamp (Unix timestamp)logs.csv: Log data (RE2 and RE3 only)traces.csv: Trace data (RE2 and RE3 only)
Our datasets and their description are publicly available with the following information:
- Hugging Face (recommended): https://huggingface.co/datasets/phamquiluan/RCAEval (Parquet, 3.4GB on disk, downloadable one suite or one case at a time)
- Figshare: https://figshare.com/articles/dataset/RCAEval_A_Benchmark_for_Root_Cause_Analysis_of_Microservice_Systems/31048672 (more structured format, with additional preprocessed series)
- Zenodo: https://zenodo.org/records/14590730 (DOI for citation: https://doi.org/10.5281/zenodo.14590730)
The Hugging Face copy stores the same 735 cases as Parquet instead of metrics.json /
logs.csv / traces.csv. It is 3.4GB to download against 5.2GB of zip archives on
Zenodo, and about eleven times smaller once extracted, since those archives expand to
roughly 38GB. Unlike a zip archive it can also be fetched a suite or a single case at a
time, so you never download telemetry you will not use:
from huggingface_hub import snapshot_download
# one suite (use "re2*" or "re3*" for the others, or omit to get all 735 cases)
snapshot_download(
repo_id="phamquiluan/RCAEval",
repo_type="dataset",
allow_patterns="re1*",
local_dir="data",
)cases.parquet in that repository indexes all 735 cases with their ground-truth root
cause service, fault type, injection time, and telemetry sizes, so you can select a
subset before downloading any telemetry:
import pandas as pd
idx = pd.read_parquet("hf://datasets/phamquiluan/RCAEval/cases.parquet")
print(idx[idx.dataset == "RE2-TT"][["case", "root_cause_service", "fault", "n_metrics"]])RCAEval.utility.read_metrics, read_logs and read_traces read either layout, so code
written against the original files works unchanged with the Parquet copy:
from RCAEval.utility import read_metrics
df = read_metrics("data/re1ob_adservice_cpu_1") # metrics.parquet or metrics.jsonWe also provide utility functions to download our datasets using Python. The downloaded datasets will be available at directory data.
from RCAEval.utility import (
download_re1_dataset,
download_re2_dataset,
download_re3_dataset,
)
download_re1_dataset()
download_re2_dataset()
download_re3_dataset()Expected output after running the above code (it takes half an hour to download and extract the datasets. )
$ python test.py
Downloading RE1.zip..: 100%|█████████████████████| 390M/390M [01:02<00:00, 6.22MiB/s]
Downloading RE2.zip..: 100%|███████████████████| 4.21G/4.21G [11:23<00:00, 6.17MiB/s]
Downloading RE3.zip..: 100%|█████████████████████| 534M/534M [01:29<00:00, 5.97MiB/s]
RCAEval stores all the RCA methods in the e2e module (implemented in RCAEval.e2e). There are many RCA baselines available: RUN, CausalRCA, CIRCA, RCD, MicroCause, EasyRCA, MSCRED, BARO, 𝜖-Diagnosis, TraceRCA, MicroRank, PDiagnose, Multi-source BARO, Multi-source RCD, Multi-source CIRCA, TORAI, EventADL.
We provide a script named main.py to assist in reproducing the results from our RCAEval paper. This script can be executed using Python with the following syntax:
python main.py [-h] [--dataset DATASET] [--method METHOD]
The available options and their descriptions are as follows:
options:
-h, --help Show this help message and exit
--dataset DATASET Choose a dataset. Valid options:
[re2-ob, re2-ss, re2-tt, etc.]
--method METHOD Choose a method (`causalrca`, `microcause`, `e_diagnosis`, `baro`, `rcd`, `circa`, etc.)
For example, in Table 6, BARO achieves Avg@5 of 0.72, 0.99, 1, 0.83, 0.64, and 0.8 for CPU, MEM, DISK, SOCKET, DELAY, LOSS, and AVERAGE on the Train Ticket dataset. To reproduce these results, you can run the following commands:
python main.py --method baro --dataset re2-ttThe expected output should be exactly as presented in the paper (it takes less than 1 minute to run the code)
$ python main.py --method baro --dataset re2-tt --length 20
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 90/90 [00:45<00:00, 1.98it/s]
--- Evaluation results ---
Avg@5-CPU: 0.72
Avg@5-MEM: 0.99
Avg@5-DISK: 1.0
Avg@5-SOCKET: 0.83
Avg@5-DELAY: 0.63
Avg@5-LOSS: 0.64
---
Avg speed: 0.51
We can replace the baro method with other methods (e.g., circa) and substitute re2-tt with other datasets to replicate the corresponding results shown in Table 6. This reproduction process is also integrated into our Continuous Integration (CI) setup. For more details, refer to the .circleci/config.yml file.
The multi-source methods (Multi-source BARO, Multi-source RCD, and Multi-source CIRCA) consume metrics, logs, and traces together. They run on the RE2/RE3 datasets, which include the required logts.csv, tracets_err.csv, and tracets_lat.csv files in each failure case:
# Multi-source BARO (Python 3.12, default environment)
python main.py --method mmbaro --dataset re2-ob
# Multi-source CIRCA (Python 3.12, default environment)
python main.py --method mmcirca --dataset re2-ob
# Multi-source RCD (requires the RCD environment: Python 3.8, `pip install -e .[rcd]`, see docs/SETUP.md)
python main.py --method mmrcd --dataset re2-obA function-level tutorial using Multi-source BARO on a single failure case is available in docs/multi-source-rca-demo.ipynb.
TORAI Datasets (270 cases): A processed version of the RE2 datasets, containing pre-aggregated time-series for metrics, logs, and traces (where available). These datasets are used by the TORAI method and are available separately on Figshare: https://doi.org/10.6084/m9.figshare.31925976.
TORAI requires a separate Python 3.8 environment and its own datasets. For full instructions, see docs/TORAI.md for installation and reproducibility instructions.
python main.py --method torai --dataset torai-ob --length 10Expected output (it takes less than 1 minute)
--- Evaluation results ---
Avg@5-CPU: 0.96
Avg@5-MEM: 0.93
Avg@5-DISK: 1.0
Avg@5-SOCKET: 0.93
Avg@5-DELAY: 0.8
Avg@5-LOSS: 0.84
EventADL localizes root-cause actors/resources over CloudTrail-style API-call event logs, rather than the metrics/traces/logs used by other RCAEval methods. It requires a separate Python 3.12 environment. For full instructions, see docs/EVENTADL.md for installation and usage instructions.
python main.py --method eventadl --dataset eventadl-falconAnomaly Detection and Root Cause Analysis for Microservice Systems
PhD Thesis, RMIT University, 2026 · Luan Pham
EventADL: Open-Box Anomaly Detection and Localization Framework for Events in Cloud-Based Service Systems
FSE 2026 · Luan Pham, Victor Nicolet, Joey Dodds, Hui Guan, Daniel Kroening
TORAI: Multi-source Root Cause Analysis for Blind Spots in Microservice Service Call Graph
FSE 2026 · Luan Pham, Huong Ha, Xiuzhen Zhang, Hongyu Zhang
Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition
NeurIPS 2026 (Evaluations and Datasets Track) · Hada Melino Muhammad, Luan Pham, Laure Barrière, Sachin Shetty, Leonardo Pulga, Flora D. Salim
RCAEval: A Benchmark for Root Cause Analysis of Microservice Systems with Telemetry Data
WWW 2025 (Companion) · Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, Xiuzhen Zhang
Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?
ASE 2024 · Luan Pham, Huong Ha, Hongyu Zhang
BARO: Robust Root Cause Analysis for Microservices via Multivariate Bayesian Online Change Point Detection
FSE 2024 · Luan Pham, Huong Ha, Hongyu Zhang
This repository includes code from various sources with different licenses. We have included their corresponding LICENSE into the LICENSES directory:
- BARO: Licensed under the MIT License. Original source: BARO GitHub Repository.
- CausalRCA: No License. Original source: CausalRCA GitHub Repository.
- CIRCA: Licensed under the BSD 3-Clause License. Original source: CIRCA GitHub Repository.
- E-Diagnosis: Licensed under the BSD 3-Clause License. Original source: PyRCA GitHub Repository.
- MicroCause: Licensed under the Apache License 2.0. Original source: MicroCause GitHub Repository.
- RCD: Licensed under the MIT License. Original source: RCD GitHub Repository.
- RUN: No License. Original source: RUN GitHub Repository.
For the code implemented by us and for our datasets, we distribute them under the MIT LICENSE.
We would like to express our sincere gratitude to the researchers and developers who created the baselines used in our study. Their work has been instrumental in making this project possible. We deeply appreciate the time, effort, and expertise that have gone into developing and maintaining these resources. This project would not have been feasible without their contributions.
- [Sep 2026] Added the retrieval–reranking decomposition of Where Root Cause Analysis Fails (NeurIPS'26).
main.py --report-decomposition KprintsRetrieval@KandRerank@1beneath eachAvg@5;K=alluses the whole returned ranking. - [Sep 2026] Fixed PageRank-based methods and CIRCA silently falling back to column order (scikit-network ≥0.32 and networkx ≥3.0 API removals, and a duplicate
timecolumn in 50 RE1-OB cases). The fallback now reports its error on stderr. - [Sep 2026] Added an opt-in chance-level baseline to the evaluator.
main.py --report-chanceprintsChance@5andLift@5beneath eachAvg@5, so an absolute score can be read against the floor a random ranker would reach on the same candidate set. Contributed by @mateenali66 in #79. - [Aug 2026] Published all nine datasets (735 cases) on Hugging Face as Parquet, 3.4GB on disk and downloadable one suite or one case at a time, with a
cases.parquetindex of every case and its ground-truth root cause. Addedread_metrics,read_logsandread_traces, which read either the original or the Parquet layout. - [Jul 2026] Made the multi-source RCA benchmark runnable via
main.py(Multi-source BARO/RCD/CIRCA) and fixed PyPI packaging so wheels include all subpackages. - [Jul 2026] Added EventADL, an event-log root cause localization method accepted at FSE'26.
- [Jun 2026] The version of RCAEval with TORAI used in our FSE'26 paper is available in the fse26 branch.
- [Apr 2026] Added TORAI, a multi-source RCA method accepted at FSE'26.
- [Mar 2025] The version of RCAEval used in our WWW'25 paper are available in the www25 branch.
- [Dec 2024] The prior version of RCAEval used in our ASE'24 paper are available in the ase24 branch.