+[](https://github.com/PathoGenOmics-Lab/BAMpiro/actions/workflows/ci.yml)
[](LICENSE)
[](https://www.nextflow.io/)
-[](CHANGELOG.md)
+[](CHANGELOG.md)
[](https://hub.docker.com/r/paururo/bampiro)
[](https://github.com/PathoGenOmics-Lab)
[](https://pathogenomics-lab.github.io/BAMpiro/)
@@ -44,7 +45,7 @@ tuberculosis* but is **organism-agnostic** - point it at any reference genome +
- Alignment-free MTBC lineage + WHO drug-resistance typing ([Pathotypr](docs/pathotypr.md))
- A self-contained, interactive [HTML QC report](docs/qc-report.md) with 21 panels
- Dual amino-acid numbering (used reference + H37Rv / Mycobrowser)
-- One pinned container with every tool and marker panel built in
+- One digest-pinned container with every tool and marker panel built in
## Features
@@ -56,19 +57,25 @@ tuberculosis* but is **organism-agnostic** - point it at any reference genome +
| 🩺 Lineage & drug resistance | Alignment-free MTBC lineage + WHO DR typing ([Pathotypr](docs/pathotypr.md)), reference-agnostic |
| 📊 Interactive QC report | Self-contained HTML dashboard, [21 panels](docs/qc-report.md) + per-sample `qc_flags.tsv` |
| 🔤 Dual amino-acid numbering | Protein changes in both the used reference and H37Rv/Mycobrowser numbering |
-| 📦 Containerized & reproducible | A single pinned image with every tool + bundled marker panels ([details](docs/installation.md)) |
+| 📦 Containerized & reproducible | A single image, pinned by digest, with every tool + bundled marker panels ([details](docs/installation.md)) |
| ⚙️ Fully configurable | Every step exposed as a Nextflow parameter ([reference](docs/configuration.md)) |
## Installation
Requires **Nextflow ≥ 24.04.2** and **Docker** or **Singularity**. The pipeline
-pulls a pinned `paururo/bampiro` image with every tool built in - nothing else to
-install.
+pulls the `paururo/bampiro` image, pinned by digest, with every tool built in - nothing
+else to install.
```bash
-nextflow run main.nf --tsv samples.tsv --outdir results_bampiro -profile standard
+nextflow run main.nf --tsv samples.tsv --outdir results_bampiro -profile local,docker
```
+`--tsv` is required, and `-profile` says where the work runs: `local` on this machine,
+`slurm` on any SLURM cluster, `garnatxa` on the I2SysBio one. With no `-profile` everything
+runs on the current host, which on a cluster login node means the login node - so choose
+one deliberately there. `nextflow run main.nf --help` lists every parameter with its
+default.
+
Full requirements and the bundled software versions: [Installation](docs/installation.md).
## Quick Start
@@ -79,7 +86,7 @@ amino-acid numbering) with:
```bash
nextflow run main.nf \
- --tsv samples.tsv --outdir results_bampiro -profile standard \
+ --tsv samples.tsv --outdir results_bampiro -profile local,docker \
--run_pathotypr true --annotate_canonical true
```
@@ -88,7 +95,7 @@ See the [Quick Start guide](docs/quickstart.md) for the full format and multi-ru
merging, and [Outputs](docs/outputs.md) for the result layout.
**New to BAMpiro?** Follow the hands-on
-[**Jupyter tutorial**](docs/tutorial/bampiro_tutorial.ipynb) — install → samplesheet →
+[**Jupyter tutorial**](docs/tutorial/bampiro_tutorial.ipynb) - install → samplesheet →
run → explore the outputs with `pandas` / `matplotlib` on a bundled 17-sample example
cohort (runnable without running the pipeline first).
@@ -96,13 +103,13 @@ cohort (runnable without running the pipeline first).
- 
+
▶ Open the live interactive demo report
- — a full 17-sample demo cohort, every panel populated, right in your browser
+ - a full 17-sample demo cohort, every panel populated, right in your browser
Every run writes a single self-contained `
_qc_report.html` (no internet,
@@ -118,13 +125,13 @@ Full panel list and interactive features: [Interactive QC Report](https://pathog
## Documentation
-📖 **[Read the full documentation online → pathogenomics-lab.github.io/BAMpiro](https://pathogenomics-lab.github.io/BAMpiro/)**
-— a searchable site with a pipeline diagram, a hover glossary, a step-by-step
+📖 **[Read the full documentation online → pathogenomics-lab.github.io/BAMpiro](https://pathogenomics-lab.github.io/BAMpiro/)**:
+a searchable site with a pipeline diagram, a hover glossary, a step-by-step
[**tutorial series**](https://pathogenomics-lab.github.io/BAMpiro/tutorials/), a hands-on
Jupyter notebook, and the [live QC-report demo](https://pathogenomics-lab.github.io/BAMpiro/examples/qc_report_demo.html).
> [!TIP]
-> **New here?** Follow the [**Tutorials**](https://pathogenomics-lab.github.io/BAMpiro/tutorials/) —
+> **New here?** Follow the [**Tutorials**](https://pathogenomics-lab.github.io/BAMpiro/tutorials/):
> a guided path from your first run to a finished phylogeny.
| Document | Description |
@@ -146,6 +153,17 @@ locally with `make docs-serve`
(needs [MkDocs Material](https://squidfunk.github.io/mkdocs-material/):
`pip install -r docs/requirements.txt`).
+## Tests
+
+```bash
+tests/run_tests.sh
+```
+
+Unit tests for the Python under `bin/`, tests for the report front-end's hand-written statistics
+(checked against SciPy), and a full `-stub-run` of the DAG over a 170 kB fixture cohort. No
+container, no reference genome, no network. [`tests/README.md`](tests/README.md) has the details;
+CI runs the same checks on every pull request.
+
## Why "BAMpiro"?
The name is a play on words combining bioinformatics and folklore:
@@ -172,7 +190,7 @@ If you use BAMpiro in your research, please cite:
title = {BAMpiro: bacterial short-read mapping, variant calling and lineage/drug-resistance typing},
author = {Ruiz-Rodriguez, Paula and Coscoll{\'a}, Mireia},
url = {https://github.com/PathoGenOmics-Lab/BAMpiro},
- version = {1.0.1},
+ version = {1.1.0},
license = {GPL-3.0}
}
```
diff --git a/RELEASING.md b/RELEASING.md
new file mode 100644
index 0000000..8bb2d92
--- /dev/null
+++ b/RELEASING.md
@@ -0,0 +1,70 @@
+# Releasing
+
+A release is a git tag. Everything else follows from it, so the order matters.
+
+## 1. Prepare
+
+- [ ] `tests/run_tests.sh` is green, and CI is green on `main`.
+- [ ] `CHANGELOG.md` has a section for the new version with today's date, and the
+ `## [Unreleased]` section is empty.
+- [ ] The version is bumped in **three** places, which must agree:
+ `nextflow.config` (`manifest.version`), `CITATION.cff` (`version`), and the
+ version badge in `README.md`.
+
+## 2. Publish the container first
+
+The pipeline pins its image **by digest**, so the image has to exist before the
+tag that references it.
+
+```bash
+gh workflow run docker-publish.yml -f tag=
+```
+
+It builds once and pushes to both GHCR and Docker Hub. Copy the `sha256:` digest
+the action prints into `nextflow.config`:
+
+```groovy
+container = "docker://paururo/bampiro@sha256:"
+```
+
+Commit that, then verify a run resolves it:
+
+```bash
+nextflow run main.nf -profile test -stub-run
+```
+
+!!! note
+ Whichever registry the cluster pulls from has to be **public**, or Singularity
+ needs `APPTAINER_DOCKER_USERNAME` / `APPTAINER_DOCKER_PASSWORD` in the
+ environment. Docker Hub is the public one today.
+
+## 3. Tag and release
+
+```bash
+git tag
+git push --tags
+gh release create --title "" --notes-file
+```
+
+The tag triggers `docker-publish.yml` again, which is harmless: it rebuilds the
+same content and republishes the same tags.
+
+## 4. Archive for citation
+
+Currently **blocked**: Zenodo can only archive a public repository, and this one
+is private. When it goes public:
+
+1. Sign in to [Zenodo](https://zenodo.org) with GitHub and enable the switch for
+ `PathoGenOmics-Lab/BAMpiro`.
+2. Cut a release. Zenodo archives it and mints a DOI, using the metadata in
+ `.zenodo.json`.
+3. Add the **concept DOI** (the one that always resolves to the newest version,
+ not the per-version DOI) to `CITATION.cff` as `doi:` and to the README badge
+ row.
+
+## 5. Docs
+
+`docs.yml` publishes GitHub Pages on push to `main`, gated on the repository
+variable `ENABLE_PAGES`. Pages on a private repository needs a paid plan, so
+while this repository is private the site is off and the README's docs badges do
+not resolve. Nothing to do at release time beyond checking the build passed.
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 0000000..ea16566
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,71 @@
+# Security Policy
+
+BAMpiro is a research pipeline, not a network service. It runs on a laptop or an HPC cluster, under
+your own account, on data you point it at. It listens on no port, has no users, no authentication and
+no persistent state beyond the files it writes to `--outdir` and the Nextflow work directory. That
+rules out most of what "vulnerability" usually means, so this policy is short and specific to what is
+actually plausible here.
+
+## Supported versions
+
+Only the latest release is supported. Fixes go into the next release; there are no backports to
+earlier tags.
+
+| Version | Supported |
+| :--- | :--- |
+| Latest release (see [CHANGELOG.md](CHANGELOG.md) and the [releases page](https://github.com/PathoGenOmics-Lab/BAMpiro/releases)) | Yes |
+| Anything older | No |
+
+## Reporting a vulnerability
+
+Please report privately rather than opening a public issue:
+
+- **GitHub private vulnerability reporting**: the *Security* tab of this repository, then
+ *Report a vulnerability*. This keeps the report visible only to the maintainers.
+- **Email**: paula.ruiz.rodriguez@csic.es
+
+Include what you found, how to reproduce it, and what an attacker would gain. If it involves an input
+file, a minimal example helps more than a real dataset (and please do not send us patient data).
+
+This is a small academic team with no on-call rotation, so we cannot promise a response time. We will
+acknowledge your report as soon as we see it and tell you what we intend to do about it. If you would
+like credit in the changelog entry for the fix, say so.
+
+## What is realistically in scope
+
+**The container's pinned dependencies.** Runs execute inside `paururo/bampiro`, which bundles
+BWA-MEM2, Samtools, BCFtools, FreeBayes, SnpEff, FastP, Kraken2, MUMmer, Python and a Java runtime,
+all pinned (versions are listed in [docs/installation.md](docs/installation.md)). A vulnerability in
+one of those reaches BAMpiro users through the image. Tell us if a bundled tool needs an urgent
+version bump.
+
+Two related notes. `nextflow.config` references the image by its mutable version **tag**, so the same
+tag can be rebuilt; re-pin to `docker://paururo/bampiro@sha256:` if you need byte-for-byte
+reproducibility, as documented in the installation page. And `modules/qc.nf` falls back to
+downloading `extract_kraken_reads.py` from `params.krakentools_url` over the network when the
+vendored copy under `bin/` is missing, which is a supply-chain surface worth knowing about; the
+vendored copy is there so the normal path never touches the network.
+
+**Untrusted input files.** The pipeline parses whatever you hand it: the TSV samplesheet, FASTQ
+reads, a reference FASTA, a GFF3, and optionally a Kraken2 database. Paths from the samplesheet reach
+shell commands inside process scripts. Running BAMpiro on files from a source you do not trust is
+therefore closer to running an untrusted script than to opening a data file. Treat a crafted
+samplesheet or reference that causes something other than a clean parse error as a bug worth
+reporting.
+
+**The QC report is your data.** Each run writes a single self-contained
+`_qc_report.html` that embeds sample identifiers, per-sample metrics and any metadata
+columns you supplied. It contains no remote references and phones nothing home, but it is not
+anonymised. Publishing or sharing one shares whatever was in your samplesheet.
+
+## What is out of scope
+
+- Denial of service through resource exhaustion. Kraken2 alone asks for around 80 GB of RAM by
+ design; an under-sized host being killed by the OOM reaper is a sizing problem, covered in
+ [Troubleshooting](docs/troubleshooting.md).
+- Anything requiring an attacker who already has your shell account, your cluster credentials or
+ write access to the work directory.
+- The security of Nextflow, Docker, Singularity or your scheduler. Report those upstream.
+
+Ordinary bugs, wrong results and crashes are not security issues. Open a
+[normal issue](https://github.com/PathoGenOmics-Lab/BAMpiro/issues/new/choose) for those.
diff --git a/assets/NO_FILE b/assets/NO_FILE
new file mode 100644
index 0000000..e69de29
diff --git a/assets/NO_FILE.README.md b/assets/NO_FILE.README.md
new file mode 100644
index 0000000..72bfc82
--- /dev/null
+++ b/assets/NO_FILE.README.md
@@ -0,0 +1,26 @@
+# `NO_FILE*` placeholders
+
+Nextflow has no notion of an optional process input: a `path` input must always
+receive something. When one of BAMpiro's optional features is off, `main.nf`
+stages one of these empty files instead, and the process recognises it by the
+`NO_FILE*` prefix and skips the corresponding argument.
+
+| File | Stands in for | Off when |
+| :--- | :--- | :--- |
+| `NO_FILE` | pathotypr lineage summary (`GENERATE_LEGACY_STATS`) | pathotypr did not type that sample |
+| `NO_FILE_DR` | drug-resistance calls (`QC_REPORT`) | `--run_pathotypr false` |
+| `NO_FILE_H37RV` | canonically annotated VCFs (`QC_REPORT`) | `--annotate_canonical false` |
+| `NO_FILE_LIFTOVER` | variant coordinate map (`QC_REPORT`) | `--variant_liftover false` |
+
+Two properties matter, and both are load-bearing:
+
+- **The names are distinct.** `QC_REPORT` takes three of these at once and all
+ three features are off by default. Nextflow refuses to stage two inputs of the
+ same task under the same filename, so sharing one `NO_FILE` fails the run with
+ `input file name collision`.
+- **The files exist and are empty.** A path that does not exist stages as a
+ dangling symlink, which happens to work with the default staging mode but fails
+ outright under `stageInMode 'copy'`. Empty keeps the `[ -s ... ]` guards false.
+
+They are referenced as `${projectDir}/assets/...`, so a file called `NO_FILE`
+sitting in someone's launch directory is never picked up by mistake.
diff --git a/assets/NO_FILE_DR b/assets/NO_FILE_DR
new file mode 100644
index 0000000..e69de29
diff --git a/assets/NO_FILE_H37RV b/assets/NO_FILE_H37RV
new file mode 100644
index 0000000..e69de29
diff --git a/assets/NO_FILE_LIFTOVER b/assets/NO_FILE_LIFTOVER
new file mode 100644
index 0000000..e69de29
diff --git a/bin/WGS_fasta_allpos.py b/bin/WGS_fasta_allpos.py
index 5859a55..62a418e 100644
--- a/bin/WGS_fasta_allpos.py
+++ b/bin/WGS_fasta_allpos.py
@@ -136,8 +136,11 @@ def _int(key: str) -> Optional[int]:
if ad and ad not in {".", ""}:
parts = ad.split(",")
try:
- ro = int(parts[0])
- ao = sum(int(x) for x in parts[1:]) if len(parts) > 1 else 0
+ # Both or neither: assigning ro before the ao sum can raise would leave a
+ # half-initialized (5, None) on a malformed AD such as "5,x".
+ ad_ro = int(parts[0])
+ ad_ao = sum(int(x) for x in parts[1:]) if len(parts) > 1 else 0
+ ro, ao = ad_ro, ad_ao
except ValueError:
pass
return ro, ao
@@ -187,11 +190,14 @@ def _load(self, path: str) -> None:
line = line.strip()
if not line or line.startswith("#"):
continue
- if line.lower().startswith("chrom"):
- continue
parts = line.split("\t")
if len(parts) < 3:
continue
+ # A header row is one whose coordinate columns are not numbers. Detecting it by
+ # the first field instead would drop a real interval on any contig named
+ # chrom*, e.g. "chromosome_1", which would then never be masked.
+ if not (parts[1].strip().isdigit() and parts[2].strip().isdigit()):
+ continue
chrom = parts[0].strip()
try:
start = int(parts[1]); end = int(parts[2])
@@ -319,7 +325,11 @@ def _load(self, fasta_path: str) -> None:
if line.startswith(">"):
if name is not None:
self.seqs[name] = "".join(parts).upper()
- name = line[1:].split()[0]
+ fields = line[1:].split()
+ if not fields:
+ raise RuntimeError(
+ f"{fasta_path}: FASTA header with no sequence name (a bare '>')")
+ name = fields[0]
parts = []
else:
parts.append(line.strip())
diff --git a/bin/backbone_allpos.awk b/bin/backbone_allpos.awk
new file mode 100644
index 0000000..bdb9843
--- /dev/null
+++ b/bin/backbone_allpos.awk
@@ -0,0 +1,54 @@
+# Turn a samtools mpileup stream into the all-positions backbone VCF body.
+#
+# samtools mpileup -aa ... | awk -v MINCOV=30 -v GAPDP=7 -v BAQ=1 -v OFS='\t' \
+# -f backbone_allpos.awk
+#
+# MINCOV params.allpos_min_cov - depth at or above this is a confident reference call
+# GAPDP params.consensus_min_dp - only read when BAQ=1
+# BAQ 1 when the input carries a second, BAQ-adjusted depth in column 10 (the two pileups
+# pasted together), 0 when it is a plain single mpileup
+#
+# The backbone supplies one record per reference position, which is what makes an all-sites VCF and
+# therefore a phylogenetic supermatrix possible. Both branches of CALL_BACKBONE used to carry their
+# own copy of count_bases; tests/unit/test_backbone_allpos.py covers this one.
+
+# Count reference and alternate observations in an mpileup base string.
+#
+# The format is not one character per read: a read start is '^' followed by a mapping-quality
+# character that must not be counted, and an indel is a sign, a length, and that many bases which
+# belong to the PREVIOUS position. Getting this wrong silently inflates or deflates the depth
+# evidence the consensus relies on.
+function count_bases(s, i, n, c, num) {
+ RO_G = 0; AO_G = 0; i = 1; n = length(s)
+ while (i <= n) {
+ c = substr(s, i, 1)
+ if (c == "^") { i += 2 } # ^ + mapping-quality char (read start)
+ else if (c == "$") { i++ } # read end
+ else if (c == "+" || c == "-") { # indel: skip sign + count + that many bases
+ i++; num = ""
+ while (i <= n && substr(s, i, 1) ~ /[0-9]/) { num = num substr(s, i, 1); i++ }
+ i += (num + 0)
+ }
+ else if (c == "." || c == ",") { RO_G++; i++ } # match to reference
+ else if (c ~ /[ACGTacgt]/) { AO_G++; i++ } # mismatch = alt observation
+ else { i++ } # * (del), <>, N, etc. -> ignore
+ }
+}
+
+{
+ chrom = $1; pos = $2; ref = $3; dp = ($4 + 0)
+ count_bases($5) # RO/AO from the no-BAQ pileup (real base evidence)
+
+ wt = (dp >= MINCOV ? 1 : 0)
+ nc = (dp >= MINCOV ? 0 : 1)
+ gt = (dp >= MINCOV ? "0" : "./.")
+
+ # A site is flagged baq_dropout when it is well covered but BAQ collapses it to at or below
+ # consensus_min_dp (the indel-adjacent homopolymer artefact), so the consensus masks it with X
+ # rather than leaving a gap. Without the BAQ column there is nothing to compare and no flag.
+ flt = "."
+ if (BAQ == 1) { dp_baq = ($10 + 0); if (dp > GAPDP && dp_baq <= GAPDP) flt = "baq_dropout" }
+
+ info = "ADP=" dp ";WT=" wt ";HET=0;HOM=0;NC=" nc
+ print chrom, pos, ".", ref, ".", ".", flt, info, "GT:DP:AD", gt ":" dp ":" RO_G "," AO_G
+}
diff --git a/bin/build_min_unique_len.py b/bin/build_min_unique_len.py
index aa01fda..2ba38a8 100755
--- a/bin/build_min_unique_len.py
+++ b/bin/build_min_unique_len.py
@@ -13,6 +13,8 @@
reference_start, so the read filter looks up min_unique_len[read.reference_start] directly.
"""
import argparse
+import sys
+
import numpy as np
@@ -41,7 +43,8 @@ def main():
for K, path in items: # ascending K -> first (smallest) unique wins
kv = np.uint16(min(K, a.sentinel - 1)) # never store the sentinel itself as a real length
for line in open(path):
- if not line or line[0] == "#":
+ # A blank line reads as "\n", never "", so testing `not line` alone never fires.
+ if not line.strip() or line[0] == "#":
continue
c, s, e, v = line.rstrip("\n").split("\t")
arr = tracks.get(c)
@@ -60,6 +63,22 @@ def main():
tail = (tracks[n] == SENT) & (~covered[n])
tracks[n][tail] = 0 # span >= 0 always -> such reads are kept
+ # A never-scored position becomes 0, i.e. "uniquely placeable at any read length", so
+ # the read filter keeps everything there. At a contig END that is deliberate: genmap
+ # cannot score a k-mer that runs off the sequence. A hole in the MIDDLE is a different
+ # thing entirely, and it would silently disable masking over that stretch, so say so
+ # rather than let the permissive default pass for a measurement.
+ if tail.any():
+ idx = np.flatnonzero(tail)
+ head_run = np.flatnonzero(covered[n])
+ first, last = (head_run[0], head_run[-1]) if head_run.size else (L, -1)
+ interior = idx[(idx > first) & (idx < last)]
+ if interior.size:
+ sys.stderr.write(
+ "[build_min_unique_len] WARNING: %s has %d unscored position(s) away from "
+ "the contig ends (first at %d); masking is disabled there\n"
+ % (n, interior.size, int(interior[0]) + 1))
+
np.savez(a.out_prefix + ".min_unique_len.npz", **tracks)
# Optional repeat BED: a base is unmappable iff NO read of length <= mask_window covering it
diff --git a/bin/build_snp_matrix.py b/bin/build_snp_matrix.py
index 427695c..905d235 100644
--- a/bin/build_snp_matrix.py
+++ b/bin/build_snp_matrix.py
@@ -60,6 +60,10 @@ def _af_dp(fmt, val):
alleles = [a for a in d.get('GT', './.').replace('|', '/').split('/') if a not in ('.', '')]
if alleles:
af = sum(1 for a in alleles if a != '0') / len(alleles)
+ # A single-element AD cannot give an allele fraction, but it still states a depth: without this
+ # the matrix emits an empty DP cell for a site whose depth is right there in the record.
+ if dp is None and ad:
+ dp = sum(ad)
return af, dp
diff --git a/bin/filter_reads_mappability.py b/bin/filter_reads_mappability.py
index 3ce5616..34f51cd 100755
--- a/bin/filter_reads_mappability.py
+++ b/bin/filter_reads_mappability.py
@@ -58,6 +58,8 @@ def anchored(A, idx, sp):
w = sys.stdout.write
for line in sys.stdin:
+ if not line.strip(): # a blank line has no fields to index
+ continue
if line[0] == "@":
if line.startswith("@SQ"):
for tok in line.split("\t"):
diff --git a/bin/format_snps_for_backbone.awk b/bin/format_snps_for_backbone.awk
new file mode 100644
index 0000000..a7a3a33
--- /dev/null
+++ b/bin/format_snps_for_backbone.awk
@@ -0,0 +1,51 @@
+# Rewrite a FreeBayes SNP VCF into the reduced form the backbone merge and the consensus step
+# expect: INFO becomes ADP/WT/HET/HOM/NC and the sample column becomes GT:DP.
+#
+# awk -v HETMIN=0.10 -f format_snps_for_backbone.awk valid_snps.vcf
+#
+# HETMIN is params.het_min_frac. Both FreeBayes processes pipe through this, so the genotype
+# decision below lives in exactly one place; tests/unit/test_format_snps_for_backbone.py covers it.
+
+BEGIN { OFS = "\t" }
+
+/^##/ { print; next }
+/^#CHROM/ { print; next }
+
+{
+ dp = 0; ro = 0; ao = 0
+
+ # INFO is tokenised on ';' and each key anchored (^DP=/^RO=/^AO=) so DP is never grabbed from a
+ # look-alike field such as MQMDP= or ADP=. AO is per-alt; after `bcftools norm -m -` the record is
+ # biallelic, so the first value is the one that matters.
+ ni = split($8, I, ";")
+ for (k = 1; k <= ni; k++) {
+ if (I[k] ~ /^DP=/) { v = I[k]; sub(/^DP=/, "", v); dp = v + 0 }
+ else if (I[k] ~ /^RO=/) { v = I[k]; sub(/^RO=/, "", v); ro = v + 0 }
+ else if (I[k] ~ /^AO=/) { v = I[k]; sub(/^AO=/, "", v); split(v, ax, ","); ao = ax[1] + 0 }
+ }
+
+ # Depth falls back through FORMAT/DP, then RO+AO, then 1 so the consensus never divides by zero.
+ # `+ 0` matters: FORMAT/DP may be ".", and without the coercion the comparisons below are string
+ # comparisons that "." fails, so every fallback is skipped and ADP=. reaches the consensus, which
+ # reads it as zero depth and turns a called SNP into a gap.
+ if (dp == 0) { n = split($9, fmt, ":"); m = split($10, dat, ":"); for (i = 1; i <= n; i++) if (fmt[i] == "DP") dp = dat[i] + 0 }
+ if (dp == 0) dp = ro + ao
+ if (dp == 0) dp = 1
+
+ split($10, b, ":"); gtype = b[1]
+
+ # Re-validate het: `bcftools norm -m -` splits a multiallelic 1/2 into biallelic records whose
+ # reference index 0 is an ARTIFACT (RO~0). A het whose reference allele is essentially
+ # unsupported is not a real het -> homozygous-alt (matches the af-based hom/het rule and stops
+ # the reference base leaking into the consensus IUPAC).
+ if ((gtype == "0/1" || gtype == "1/0") && (ro + ao) > 0 && ro / (ro + ao) < HETMIN) gtype = "1/1"
+
+ het = 0; hom = 0
+ if (gtype == "0/1" || gtype == "1/0") het = 1
+ else if (gtype == "1/1") hom = 1
+
+ $8 = "ADP=" dp ";WT=0;HET=" het ";HOM=" hom ";NC=0"
+ $9 = "GT:DP"
+ $10 = gtype ":" dp
+ print
+}
diff --git a/bin/pathotypr_liftover.py b/bin/pathotypr_liftover.py
index 39bbe68..aac8087 100644
--- a/bin/pathotypr_liftover.py
+++ b/bin/pathotypr_liftover.py
@@ -63,7 +63,9 @@ def _positions(path):
continue
c = line.replace(",", "\t").split("\t")
try:
- if len(c) >= 3 and c[1].lstrip("-").isdigit() and c[2].lstrip("-").isdigit():
+ # isdigit() only, not lstrip("-"): a BED coordinate is never negative, and accepting
+ # one silently expands to zero and negative positions that no reference has.
+ if len(c) >= 3 and c[1].isdigit() and c[2].isdigit():
a, b = int(c[1]), int(c[2]) # BED 0-based half-open -> 1-based positions a+1..b
for p in range(a + 1, b + 1):
pos.add(p)
diff --git a/bin/qc_report.py b/bin/qc_report.py
index fcea3e8..51ac807 100644
--- a/bin/qc_report.py
+++ b/bin/qc_report.py
@@ -13,1425 +13,32 @@
- Click any row, dot or point to highlight a sample everywhere; click a flag chip to filter.
With --gate it exits non-zero if any sample FAILs (at the default thresholds), so a downstream target can block.
Pure standard library (no numpy / matplotlib / CDN). Study-specific signatures live in a separate downstream gate.
+
+The implementation lives in the qcreport package next to this file: qcreport.parsers
+reads the pipeline's artefacts, qcreport.metrics holds the metric registry and the
+verdict engine, qcreport.panels builds the optional analysis panels and qcreport.render
+assembles the HTML. This file is the CLI: arguments in, payload assembled, two files out.
"""
from __future__ import annotations
import argparse
-import base64
-import gzip
-import html
-import json
import os
-import random
-import re
import sys
-import zlib
from datetime import datetime, timezone
-DEF = dict(depth_min=10.0, breadth_min=90.0, missing_max=10.0, dup_max=40.0,
- iupac_max=2.0, mapping_min=80.0, titv_min=0.0, snp_z=3.0,
- het_max_frac=3.0, mixed_min_frac=2.0) # het_max_frac + mixed_min_frac are PERCENTS
-
-# Ancient (aDNA) sample threshold view: looser cov/breadth/missing/mapping/dup/iupac (a 5x mummy must not FAIL the
-# modern gate), plus damage_min_ct = the terminal 5' C>T deamination floor for the DAMAGE_LOW authentication screen.
-ANC_DEF = dict(depth_min=3.0, breadth_min=30.0, missing_max=70.0, dup_max=60.0,
- iupac_max=5.0, mapping_min=20.0, damage_min_ct=0.05)
-FAIL_FLAGS = {"LOW_DEPTH", "LOW_BREADTH", "HIGH_MISSING", "NO_DATA"}
-NBINS = 200 # bins along the reference for the genome callability-landscape heatmap
-REPO_URL = "https://github.com/PathoGenOmics-Lab/BAMpiro" # surfaced in the report header + footer
-
-# metric definitions + typical acceptable ranges (surfaced as in-report help)
-DEFS = {
- "dose": ["Per-sample dose from the samplesheet (a numeric annotation, not a QC metric). Feeds the scatter/correlation axes and the dose x treatment test.", ""],
- "raw_reads": ["Read pairs/reads before trimming.", "context-dependent"],
- "trimmed_reads": ["Reads after fastp quality/adapter trimming.", ""],
- "read_length": ["Mean read length after trimming.", ""],
- "duplication_pct": ["% reads marked as duplicates (PCR/optical).", "typically < 30-40%"],
- "mapped_pct": ["% reads mapping to the reference.", "> 90% for a clean library"],
- "properly_paired_pct": ["% read pairs mapping in the expected orientation/insert.", "> 90%"],
- "mean_depth": ["Mean sequencing depth over the genome (sensitive to a few high-copy repeats; see Median depth).", ">= 10-20x for confident calls"],
- "median_depth": ["Median sequencing depth over the genome (robust to high-copy repeats that inflate the mean).", ">= 10-20x"],
- "breadth_pct": ["% of the genome covered at >= 1x. Note: 1x is not enough to CALL a base; see Callable % for usable genome.", ">= 90% at 1x to enter a phylogeny"],
- "breadth5x_pct": ["% of the genome covered at >= 5x.", "high for a confident consensus"],
- "breadth10x_pct": ["% of the genome covered at >= 10x.", ">= 90% for confident SNP calls"],
- "coverage_cv": ["Coverage evenness = coefficient of variation of depth (sd / mean). Lower is more even; spikes flag repeats/contamination.", "< ~0.5-1 for an even library"],
- "avg_base_quality": ["Mean per-base quality of mapped reads.", ">= 30"],
- "unmapped_pct": ["% of reads that did not map to the reference (an off-target / contamination proxy).", "low for a clean, on-target library"],
- "singleton_pct": ["% of reads mapped without their mate (approx: reads mapped minus reads mapped-and-paired).", "low"],
- "mapq0_pct": ["% of reads with MAPQ 0 (ambiguous multi-mapping, e.g. repeats / IS elements).", "low"],
- "error_rate_pct": ["samtools mismatch (error) rate, as a percent; elevated with damage or contamination.", "< ~1%"],
- "insert_size": ["Mean fragment/insert size (bp) of properly paired reads. For aDNA a short distribution is expected.", "library-dependent"],
- "insert_size_sd": ["Standard deviation of insert size; a very wide spread signals chimeras/contamination.", ""],
- "q20_pct": ["% of trimmed bases with quality >= Q20 (fastp).", "> 90%"],
- "q30_pct": ["% of trimmed bases with quality >= Q30 (fastp).", "> 80-90%"],
- "gc_pct": ["GC content of the trimmed reads (fastp); a large deviation from the reference GC flags contamination.", "match the reference GC"],
- "est_genome_cov": ["Estimated fold coverage = trimmed bases / genome length (fastp-based, pre-mapping).", "tracks mean_depth"],
- "total_variants": ["Total variants (SNPs + indels) called vs the reference.", "context-dependent"],
- "snps": ["SNPs called vs the reference. Dominated by true phylogenetic distance, NOT quality; the SNP outlier flag is a cohort distributional signal, not a defect.", "expected to be higher for divergent lineages"],
- "snp_density": ["SNP calls per callable megabase = snps / (callable_pct/100 * genome_len/1e6). Normalises SNP burden for how much of the genome was callable; compare within a lineage. Suppressed for a toy genome (< 100 kb).", "compare within a lineage"],
- "het_variants": ["Heterozygous variant calls. In a clonal/haploid organism these are expected ~0; a nonzero rate is consistent with mixture, contamination, or (for aDNA) post-mortem deamination.", "~0"],
- "homo_indels": ["Homozygous indels called.", ""],
- "ti_tv": ["Transition/transversion ratio (a spectrum sanity check, not a rate). Typical genome-wide MTBC ~1.7-2.0; M. bovis (A4) runs higher (~2.6) from its transition-biased spectrum, and aDNA deamination inflates it. LOW Ti/Tv (< ~1.5) is the QC concern (mapping/base-call noise is transversion-heavy); high is biology or damage, not a defect.", "~1.5-2.1 for human lineages"],
- "missing_pct": ["% of the consensus that is '-' or N (no confident base).", "< 10%"],
- "iupac_pct": ["% of the consensus that is an IUPAC ambiguity (mixed call).", "< 2%"],
- "callable_pct": ["% of the consensus that is a confident A/C/G/T (the honest, depth-aware usable-genome measure).", "> 90%"],
- "longest_n_run": ["Longest single run of consecutive N/gap in the consensus (one big deletion vs many scattered holes).", "context-dependent"],
- "n_gaps": ["Number of separate N/gap runs in the consensus.", "context-dependent"],
- "het_frac": ["Fraction of VARIANT genotypes that are heterozygous (het / (het+hom)). An illustrative within-sample mixture signal, not the textbook heterozygosity rate; the denominator is variant-count, so read it with care. Suppressed below a minimum variant count, and NOT applied to aDNA (deamination inflates it).", "~0 for a clonal isolate"],
- "n_lineages": ["Number of lineages (collapsed to main lineage) above the mixture cut-off AND above a minimum marker count; > 1 is consistent with a mixed infection or cross-contamination.", "1"],
- "damage_ct": ["Terminal 5' C>T deamination frequency (mapDamage2). Elevated in authentic aDNA; consistent with post-mortem damage but not proof of authenticity or endogeneity on its own.", "aDNA typically >= 0.05-0.40"],
- "damage_ga": ["Terminal 3' G>A deamination frequency (mapDamage2).", "aDNA typically >= 0.05-0.40"],
- "fraglen": ["Mean fragment (read) length from mapDamage2. Short (~30-70 bp) is expected for aDNA.", "aDNA ~30-70 bp"],
- "ann_high": ["snpEff HIGH-impact variants (stop_gained, frameshift, start/stop_lost, splice). A cohort-relative excess is a contamination / misassembly / wrong-reference-annotation signal, not a per-sample defect on its own.", "cohort-relative robust-z; high flags a look-here sample"],
- "missense_silent": ["Missense / synonymous variant-count ratio: a within-sample pN/pS PROXY (a spectrum sanity check, NOT dN/dS and NOT a selection test; no site-degeneracy correction, entangled with divergence and with how snpEff bins effects). Read within a lineage. For a real pN/pS use the eskaks panel.", "compare within a lineage; robust-z high tail only"],
- "lof_pct": ["Fraction of protein-coding-effect variants that are loss-of-function (stop_gained + frameshift + start/stop_lost + splice). Elevated LoF can be real pseudogenisation OR a frameshift storm from an indel-calling / assembly / annotation artifact.", "cohort-relative robust-z"],
- "coding_pct": ["Percent of annotated variants falling in a transcript (HIGH+MODERATE+LOW; MODIFIER is the intergenic / intron / UTR bucket). Tracks how gene-dense the reference annotation is; not a quality gate.", "reference-dependent, descriptive"],
- "annotated_pct": ["Percent of this sample's real variants that received an ANN annotation from snpEff. A low value flags a GFF-to-database mismatch: the annotation may be silently garbage.", "cohort-relative low + absolute floor"],
- "snpeff_warn": ["Variants carrying a snpEff ERROR/WARNING/INFO subfield (e.g. WARNING_TRANSCRIPT_INCOMPLETE, ERROR_CHROMOSOME_NOT_FOUND). A value near the total variant count means the reference database was built wrong for this contig; treat every annotation with suspicion.", "low; near-total = broken DB"],
-}
-# Contamination/mixture cut-offs (het_max_frac, mixed_min_frac) and the aDNA damage floor are ILLUSTRATIVE sensitivity
-# defaults, not validated gates; they are WARN-only and live-adjustable. Tune to your depth and organism.
-
-# (key, label, kind, direction). kind: int|float|pct ; direction: hi_good|hi_bad|neu
-METRICS = [
- ("raw_reads", "Raw reads", "int", "neu"),
- ("trimmed_reads", "Trimmed", "int", "neu"),
- ("read_length", "Read len", "float", "neu"),
- ("q20_pct", "Q20 %", "pct", "hi_good"),
- ("q30_pct", "Q30 %", "pct", "hi_good"),
- ("gc_pct", "GC %", "pct", "neu"),
- ("duplication_pct", "Dup %", "pct", "hi_bad"),
- ("mapped_pct", "Mapped %", "pct", "hi_good"),
- ("properly_paired_pct", "Proper %", "pct", "hi_good"),
- ("unmapped_pct", "Unmapped %", "pct", "hi_bad"),
- ("singleton_pct", "Singleton %", "pct", "hi_bad"),
- ("mapq0_pct", "MAPQ0 %", "pct", "hi_bad"),
- ("error_rate_pct", "Error %", "pct", "hi_bad"),
- ("insert_size", "Insert bp", "float", "neu"),
- ("insert_size_sd", "Insert sd", "float", "neu"),
- ("mean_depth", "Depth", "float", "hi_good"),
- ("median_depth", "Median depth", "float", "hi_good"),
- ("breadth_pct", "Breadth %", "pct", "hi_good"),
- ("breadth5x_pct", "Breadth 5x %", "pct", "hi_good"),
- ("breadth10x_pct", "Breadth 10x %", "pct", "hi_good"),
- ("coverage_cv", "Cov CV", "float", "hi_bad"),
- ("avg_base_quality", "Base Q", "float", "hi_good"),
- ("est_genome_cov", "Est cov", "float", "hi_good"),
- ("total_variants", "Variants", "int", "neu"),
- ("snps", "SNPs", "int", "neu"),
- ("het_variants", "Het", "int", "hi_bad"),
- ("homo_indels", "Indels", "int", "neu"),
- ("ti_tv", "Ti/Tv", "float", "neu"),
- ("missing_pct", "Missing %", "pct", "hi_bad"),
- ("iupac_pct", "IUPAC %", "pct", "hi_bad"),
- ("callable_pct", "Callable %", "pct", "hi_good"),
- ("longest_n_run", "Longest N-run", "int", "hi_bad"),
- ("n_gaps", "N-gaps", "int", "neu"),
- ("ann_high", "HIGH impact", "int", "hi_bad"),
- ("ann_moderate", "MODERATE", "int", "neu"),
- ("ann_low", "LOW impact", "int", "neu"),
- ("ann_modifier", "MODIFIER", "int", "neu"),
- ("missense_silent", "Missense/silent", "float", "neu"),
- ("lof_pct", "LoF %", "pct", "hi_bad"),
- ("coding_pct", "Coding %", "pct", "neu"),
- ("annotated_pct", "Annotated %", "pct", "hi_good"),
- ("snpeff_warn", "snpEff warn", "int", "hi_bad"),
-]
-DIST = ["mean_depth", "median_depth", "breadth_pct", "breadth10x_pct", "coverage_cv", "mapped_pct",
- "unmapped_pct", "properly_paired_pct", "duplication_pct", "mapq0_pct", "error_rate_pct",
- "missing_pct", "callable_pct", "iupac_pct", "snps", "ti_tv",
- "ann_high", "missense_silent", "annotated_pct"]
-
-# columns that are identifiers / metadata / already-surfaced -> NOT auto-detected as "extra" metrics
-KNOWN_KEYS = {k for k, *_ in METRICS} | {
- "sample_id", "lineage", "lineage_counts", "drug_resistance", "dr_counts", "date", "ancient",
- "consensus_id", "length", "genome_len", "mapped_reads", "homo_variants",
- "snp_profile", "het_profile", "indel_profile", # per-position variant tracks (huge strings, not metrics)
- "eff_missense", "eff_synonymous", "eff_stop_gained", "eff_stop_lost", "eff_start_lost",
- "eff_frameshift", "eff_inframe_indel", "eff_splice", "eff_intergenic", "eff_regulatory",
- "ann_present", "ann_annotated_frac", "ann_total", "ann_annotated", "coding_count", "lof_count", "ann_db_error",
-}
-_PCT_HINTS = ("_pct", "_rate", "percent", "fraction", "frac")
-_INT_HINTS = ("reads", "count", "_n", "num_", "variants", "snps", "indels", "bases")
-
-
-def parse_summary(path):
- rows = {}
- with open(path) as f:
- header = f.readline().rstrip("\n").split("\t")
- for line in f:
- if not line.strip():
- continue
- parts = line.rstrip("\n").split("\t")
- if len(parts) < len(header):
- parts += [""] * (len(header) - len(parts))
- d = dict(zip(header, parts))
- sid = (d.get("sample_id") or parts[0]).strip()
- if not sid:
- continue
- rows[sid] = d
- return rows
-
-
-def consensus_stats(path):
- op = gzip.open if str(path).endswith(".gz") else open
- name, chunks = None, []
- with op(path, "rt", encoding="utf-8", errors="replace") as fh:
- for line in fh:
- if line.startswith(">"):
- if name is None: # keep the first record's id, but keep reading ALL records so a
- name = line[1:].split()[0] # multi-contig consensus is measured whole-genome, not contig 1 only
- else:
- chunks.append(line.strip())
- seq = "".join(chunks).upper()
- L = len(seq) or 1
- gap, n = seq.count("-"), seq.count("N")
- iup = sum(seq.count(c) for c in "RYSWKMBDHV")
- acgt = sum(seq.count(c) for c in "ACGT")
- # gap-length view: one big deletion vs many scattered holes are very different for a phylogeny
- runs = re.findall(r"[-N]+", seq)
- longest_run = max((len(r) for r in runs), default=0)
- # binned missingness profile ALONG the reference (consensus is reference-aligned) -> the genome landscape heatmap.
- # A callability proxy for coverage: per bin, the fraction of positions that are N / gap (no confident base).
- prof = []
- for b in range(NBINS):
- s0, e0 = (b * L) // NBINS, ((b + 1) * L) // NBINS
- sub = seq[s0:e0]
- ln = len(sub) or 1
- prof.append(round((sub.count("-") + sub.count("N")) / ln, 3))
- return {"consensus_id": name, "length": len(seq), "missing_pct": 100.0 * (gap + n) / L,
- "iupac_pct": 100.0 * iup / L, "callable_pct": 100.0 * acgt / L,
- "longest_n_run": longest_run, "n_gaps": len(runs), "miss_profile": prof}
-
-
-def to_float(x):
- try:
- if x in (None, "", "NA", "."):
- return None
- return float(x)
- except (ValueError, TypeError):
- return None
-
-
-def clean_str(x):
- """A metadata string, or None for the NA/empty sentinels (so 'NA' lineage does not read as a real lineage)."""
- if x is None:
- return None
- s = str(x).strip()
- return s if s and s not in ("NA", ".", "-") else None
-
-
-_PROFILE_WARNED = set()
-
-
-def parse_profile(s):
- """A comma-joined per-bin variant-count profile string -> [int]*NBINS, or None if absent/unparseable.
- Coerced (pad with 0 / truncate) to NBINS: the genome-landscape JS assumes every profile has exactly
- NBINS bins so that a brushed span maps 1:1 onto the hotspot table. Upstream (stats_to_legacy.BIN_N)
- already emits NBINS, so a mismatch means a foreign / re-binned summary; warn once per observed length."""
- if not s or str(s) in ("NA", ".", ""):
- return None
- try:
- v = [int(float(x)) for x in str(s).split(",")]
- except (ValueError, TypeError):
- return None
- if not v:
- return None
- if len(v) != NBINS:
- if len(v) not in _PROFILE_WARNED:
- sys.stderr.write(
- "[qc_report] warning: variant profile has %d bins, expected %d; "
- "padding/truncating so the genome landscape stays aligned.\n" % (len(v), NBINS))
- _PROFILE_WARNED.add(len(v))
- v = (v + [0] * NBINS)[:NBINS]
- return v
-
-
-def parse_lineage_colors(path):
- """Read a 'lineage#hexdesc' palette (e.g. mycolorsTB) -> {lineage: '#hex'}. Empty if absent."""
- if not path or not os.path.exists(path):
- return {}
- out = {}
- try:
- with open(path, encoding="utf-8") as fh:
- for line in fh:
- if line.startswith("#") or not line.strip():
- continue
- c = line.rstrip("\n").split("\t")
- if len(c) >= 2 and c[1].strip().startswith("#") and len(c[1].strip()) in (4, 7):
- out[c[0].strip()] = c[1].strip()
- except OSError:
- return {}
- return out
-
-
-def parse_bed(path):
- """Parse a BED (0-based half-open) of regions to mask -> [(start, end)]. Empty if absent."""
- if not path or not os.path.exists(path):
- return []
- iv = []
- try:
- op = gzip.open if str(path).endswith(".gz") else open
- with op(path, "rt", encoding="utf-8", errors="replace") as fh:
- for line in fh:
- if not line.strip() or line.startswith(("#", "track", "browser")):
- continue
- c = line.split("\t")
- if len(c) >= 3:
- try:
- iv.append((int(c[1]), int(c[2])))
- except ValueError:
- pass
- except OSError:
- return []
- return iv
-
-
-def mask_profile(intervals, genome_len, nbins=NBINS):
- """Per-bin masked fraction (0-1) along the reference + the total masked %."""
- if not intervals or genome_len <= 0:
- return None, 0.0
- binbp = genome_len / nbins
- cov = [0.0] * nbins
- total = 0
- for s, e in intervals:
- s = max(0, s); e = min(genome_len, e)
- if e <= s:
- continue
- total += e - s
- b0 = int(s // binbp); b1 = min(nbins - 1, int((e - 1) // binbp))
- for b in range(b0, b1 + 1):
- bs = b * binbp; be = (b + 1) * binbp
- cov[b] += max(0.0, min(e, be) - max(s, bs))
- prof = [round(min(1.0, cov[b] / binbp), 3) if binbp > 0 else 0.0 for b in range(nbins)]
- return prof, round(100.0 * total / genome_len, 2)
-
-
-def parse_gene_burden(path):
- """Cohort gene-burden aux TSV (from collect_summary --gene-burden-out) -> list of gene dicts sorted by
- total_impactful desc. Optional file; empty on any problem. Bounded to the top 200 genes."""
- if not path or not os.path.exists(path):
- return []
- out = []
- try:
- with open(path, encoding="utf-8", errors="replace") as fh:
- header = fh.readline().rstrip("\n").split("\t")
- idx = {h: i for i, h in enumerate(header)}
-
- def sget(c, k):
- i = idx.get(k)
- return c[i] if (i is not None and i < len(c) and c[i] != "") else None
-
- def nget(c, k):
- v = sget(c, k)
- try:
- return int(float(v))
- except (TypeError, ValueError):
- return None
- for line in fh:
- if not line.strip():
- continue
- c = line.rstrip("\n").split("\t")
- gene = sget(c, "gene")
- if not gene:
- continue
- hi, mod = nget(c, "high") or 0, nget(c, "moderate") or 0
- tot = nget(c, "total_impactful")
- out.append({"gene": gene, "high": hi, "moderate": mod,
- "low": nget(c, "low"), "modifier": nget(c, "modifier"),
- "dominant_effect": sget(c, "dominant_effect") or sget(c, "dominant") or "NA",
- "n_samples": nget(c, "n_samples"),
- "total_impactful": tot if tot is not None else hi + mod,
- "start": nget(c, "start"), "end": nget(c, "end")})
- except Exception:
- return []
- out.sort(key=lambda r: (-(r.get("total_impactful") or 0), r["gene"]))
- return out[:200]
-
-
-def parse_pnps(path):
- """Optional cohort dN/dS (eskaks) per-gene TSV -> [{gene, pn, ps, pnps, n, effect}]. Empty on any problem
- (panel hidden). Real eskaks-aggregate columns: gene, gene_name, n_samples, n_pairs, mean_dN, mean_dS, dNdS,
- effect_dominant. inf/nan/empty are read as None, never blindly float()-ed."""
- if not path or not os.path.exists(path):
- return []
-
- def num(v):
- if v is None:
- return None
- v = str(v).strip().lower()
- if v in ("", "na", "nan", "inf", "-inf", "+inf"):
- return None
- return to_float(v)
- rows = []
- try:
- with open(path, encoding="utf-8", errors="replace") as fh:
- header = fh.readline().rstrip("\n").split("\t")
- idx = {h.strip().lower(): i for i, h in enumerate(header)}
-
- def col(c, p):
- i = idx.get(c)
- return p[i] if (i is not None and i < len(p)) else None
- for line in fh:
- if not line.strip():
- continue
- p = line.rstrip("\n").split("\t")
- g = col("gene_name", p) or col("gene", p)
- if not g:
- continue
- rows.append({"gene": g, "pn": num(col("mean_dn", p)), "ps": num(col("mean_ds", p)),
- "pnps": num(col("dnds", p)), "n": num(col("n_pairs", p)),
- "effect": col("effect_dominant", p)})
- except OSError:
- return []
- return rows[:300]
-
-
-_DR_DRUG_ORDER = ["RIF", "INH", "EMB", "PZA", "STR", "STM", "FQ", "LFX", "MFX", "OFX", "KAN", "AMK",
- "CAP", "ETH", "PTO", "LZD", "BDQ", "CFZ", "BDQ_CFZ", "DLM", "PAS", "CS"]
-
-
-def parse_dr(path):
- """Per-sample drug-resistance calls -> {'samples','drugs','calls':[{s,drug,gene,mutation,grade,gn,marker,
- af,dp}]} or None. Tab-separated with a header; recognised columns (case-insensitive): sample, drug, gene,
- mutation, grade, marker_name/marker, af, dp. `grade` may be a WHO string like '1) Assoc w R' -> gn=1."""
- if not path or not os.path.exists(path):
- return None
- calls, samples = [], []
- try:
- with open(path, encoding="utf-8", errors="replace") as fh:
- header = None
- for line in fh:
- if not line.strip() or line.startswith("#"):
- continue
- parts = line.rstrip("\n").split("\t")
- if header is None:
- header = [h.strip().lower() for h in parts]
- continue
- d = dict(zip(header, parts))
- s = (d.get("sample") or (parts[0] if parts else "")).strip()
- if not s:
- continue
- grade = (d.get("grade") or "").strip()
- mg = re.match(r"\s*(\d)", grade)
- dp = d.get("dp")
- try:
- dp = int(dp) if dp not in (None, "", "NA", ".") else None
- except ValueError:
- dp = None
- calls.append({"s": s, "drug": (d.get("drug") or "").strip(), "gene": (d.get("gene") or "").strip(),
- "mutation": (d.get("mutation") or "").strip(), "grade": grade,
- "gn": (int(mg.group(1)) if mg else None),
- "marker": (d.get("marker_name") or d.get("marker") or "").strip(),
- "af": to_float(d.get("af")), "dp": dp})
- if s not in samples:
- samples.append(s)
- except OSError:
- return None
- if not calls:
- return None
- drugs = sorted({c["drug"] for c in calls if c["drug"]},
- key=lambda x: (_DR_DRUG_ORDER.index(x) if x in _DR_DRUG_ORDER else 99, x))
- return {"samples": samples, "drugs": drugs, "calls": calls}
-
-
-def parse_kraken(paths):
- """Per-sample Kraken2 `.report` files (6 columns: clade%, clade reads, taxon reads, rank, taxid,
- name) -> {'samples':[{s, unclassified, classified, primary, secondary, top}]} or None. The sample id
- is the filename up to the first '__' (the pipeline writes '__.kraken.report'). The
- 'primary' taxon is the top species-level clade; contamination shows as a low primary% / large
- secondary taxon / high unclassified%."""
- out = []
- for p in paths or []:
- if not p or not os.path.exists(p):
- continue
- sid = os.path.basename(p).split("__")[0].split(".")[0]
- unclass, species = 0.0, []
- try:
- with open(p, encoding="utf-8", errors="replace") as fh:
- for line in fh:
- c = line.rstrip("\n").split("\t")
- if len(c) < 6:
- continue
- try:
- pct = float(c[0])
- except ValueError:
- continue
- rank, name = c[3].strip(), c[5].strip()
- if rank == "U":
- unclass = pct
- elif rank == "S": # species-level clades = the interpretable composition
- species.append((pct, name))
- except OSError:
- continue
- if not species and unclass == 0.0:
- continue
- species.sort(key=lambda x: -x[0])
- top = [{"name": n, "pct": round(p, 2)} for p, n in species[:6]]
- out.append({"s": sid, "unclassified": round(unclass, 2),
- "classified": round(100.0 - unclass, 2),
- "primary": (top[0] if top else None),
- "secondary": (top[1] if len(top) > 1 else None), "top": top})
- if not out:
- return None
- return {"samples": out}
-
-
-def parse_gff(path):
- """Best-effort GFF3 parse of gene/CDS features -> [{name, start, end}] (1-based). Empty on any problem."""
- if not path or not os.path.exists(path):
- return []
- genes = {}
- try:
- op = gzip.open if str(path).endswith(".gz") else open
- with op(path, "rt", encoding="utf-8", errors="replace") as fh:
- for line in fh:
- if line.startswith("#") or "\t" not in line:
- continue
- c = line.rstrip("\n").split("\t")
- if len(c) < 9 or c[2] not in ("gene", "CDS"):
- continue
- try:
- start, end = int(c[3]), int(c[4])
- except ValueError:
- continue
- name = None
- for key in ("Name=", "gene=", "locus_tag=", "ID="):
- i = c[8].find(key)
- if i >= 0:
- name = c[8][i + len(key):].split(";")[0].strip()
- break
- name = name or f"{start}-{end}"
- # prefer a 'gene' feature over a 'CDS' of the same name
- if name not in genes or (c[2] == "gene" and genes[name].get("type") != "gene"):
- genes[name] = {"name": name, "start": start, "end": end, "type": c[2]}
- except (OSError, ValueError):
- return []
- return sorted(({"name": g["name"], "start": g["start"], "end": g["end"]} for g in genes.values()),
- key=lambda g: g["start"])
-
-
-# Curated H37Rv locus tags (Mycobrowser) for the common MTB resistance genes, used as a fallback when
-# the run's reference GFF carries no H37Rv-style locus_tag for them (e.g. a non-H37Rv reference). These
-# are the WHO-catalogue first/second-line resistance genes.
-GENE_RV = {
- "rpoB": "Rv0667", "rpoC": "Rv0668", "katG": "Rv1908c", "inhA": "Rv1484", "fabG1": "Rv1483",
- "ahpC": "Rv2428", "gyrA": "Rv0006", "gyrB": "Rv0005", "embA": "Rv3794", "embB": "Rv3795",
- "embC": "Rv3793", "pncA": "Rv2043c", "rpsL": "Rv0682", "eis": "Rv2416c", "ethA": "Rv3854c",
- "ethR": "Rv3855", "gid": "Rv3919c", "gidB": "Rv3919c", "tlyA": "Rv1694", "Rv0678": "Rv0678",
- "atpE": "Rv1305", "pepQ": "Rv2535c", "folC": "Rv2447c", "thyA": "Rv2764c", "alr": "Rv3423c",
- "ddn": "Rv3547", "fgd1": "Rv0407", "fbiA": "Rv3261", "fbiB": "Rv3262", "fbiC": "Rv1173",
- "rrs": "MTB000019", "rrl": "MTB000020", "mmpL5": "Rv0676c", "mmpS5": "Rv0677c", "whiB7": "Rv3197A",
- "clpC1": "Rv3596c", "panD": "Rv3601c", "ddlA": "Rv2981c",
-}
-_RV_LOCUS_RE = re.compile(r'^(Rv\d|MTB\d)', re.I) # H37Rv (Mycobrowser) locus-tag schemes
-
-
-def parse_gene_locus(path):
- """{gene_display_name: locus_tag} from a GFF3 (e.g. rpoB -> Rv0667). Empty on any problem."""
- out = {}
- if not path or not os.path.exists(path):
- return out
- try:
- op = gzip.open if str(path).endswith(".gz") else open
- with op(path, "rt", encoding="utf-8", errors="replace") as fh:
- for line in fh:
- if line.startswith("#") or "\t" not in line:
- continue
- c = line.rstrip("\n").split("\t")
- if len(c) < 9 or c[2] not in ("gene", "CDS"):
- continue
- attrs = c[8]
-
- def attr(key):
- i = attrs.find(key)
- return attrs[i + len(key):].split(";")[0].strip() if i >= 0 else ""
- locus = attr("locus_tag=")
- name = attr("gene=") or attr("Name=")
- if name and locus and name not in out:
- out[name] = locus
- except (OSError, ValueError):
- return out
- return out
-
-
-def build_gene_map(gff_path):
- """{gene: Mycobrowser (H37Rv) locus tag}. Curated resistance-gene map, overlaid with any H37Rv-style
- locus tags from the run's GFF (so an H37Rv reference contributes every gene; a non-H37Rv reference's
- strain-specific tags are ignored in favour of the curated H37Rv equivalents)."""
- m = dict(GENE_RV)
- for gene, locus in parse_gene_locus(gff_path).items():
- if _RV_LOCUS_RE.match(locus):
- m[gene] = locus
- return m
-
-
-def robust(vals):
- v = sorted(x for x in vals if x is not None)
- if not v:
- return None, None
- n = len(v)
- med = v[n // 2] if n % 2 else (v[n // 2 - 1] + v[n // 2]) / 2
- devs = sorted(abs(x - med) for x in v)
- mad = devs[n // 2] if n % 2 else (devs[n // 2 - 1] + devs[n // 2]) / 2
- sigma = 1.4826 * mad
- if sigma <= 0:
- mean = sum(v) / n
- sigma = (sum((x - mean) ** 2 for x in v) / n) ** 0.5 or 1.0
- return med, sigma
-
-
-MIN_HET_VARIANTS = 50 # below this variant count the het fraction is too noisy to interpret -> None
-
-
-def het_frac(m):
- """Fraction of VARIANT genotypes that are heterozygous (het/(het+hom)). Illustrative mixture signal; the
- denominator is variant-count (not callable sites), so it is entangled with divergence -> read with care.
- Returns None below a minimum variant count (too noisy to interpret)."""
- het = to_float(m.get("het_variants"))
- hom = to_float(m.get("homo_variants"))
- if het is None or hom is None or (het + hom) < MIN_HET_VARIANTS:
- return None
- return het / (het + hom)
-
-
-def lineage_counts_parsed(lc):
- """Parse 'L4.1:3200;L4.3:120;L2:40' into {main_lineage: total_count}, COLLAPSING sub-lineages (L4.1 -> L4)
- so two sub-lineages of the same main lineage do not spuriously read as a mixed infection."""
- if not lc or str(lc) in ("NA", ".", ""):
- return {}
- out = {}
- for part in str(lc).replace(",", ";").split(";"):
- if ":" in part:
- k, v = part.rsplit(":", 1)
- n = to_float(v)
- if n is not None:
- main = k.strip().split(".")[0] # collapse sub-lineage to main lineage
- out[main] = out.get(main, 0.0) + n
- return out
-
-
-def lineage_fracs(lc):
- """{main_lineage: fraction} from the collapsed counts."""
- counts = lineage_counts_parsed(lc)
- tot = sum(counts.values())
- return {k: v / tot for k, v in counts.items()} if tot > 0 else {}
-
-
-def is_ancient(m):
- """True if the summary row marks this sample ancient (aDNA). Missing column -> False (modern)."""
- return str(m.get("ancient", "")).strip().lower() in ("yes", "y", "true", "1", "ancient")
-
-
-def infer_metric(key, values):
- """Infer (label, kind, dir) for an un-registered numeric summary column. Neutral dir = no auto-flag / no claim."""
- kl = key.lower()
- if any(h in kl for h in _PCT_HINTS):
- kind = "pct"
- elif all(v is None or float(v).is_integer() for v in values) and any(h in kl for h in _INT_HINTS):
- kind = "int"
- else:
- kind = "float"
- label = key.replace("_pct", " %").replace("_", " ").strip()
- label = label[:1].upper() + label[1:]
- return label, kind, "neu"
-
-
-def discover_extra_metrics(summ):
- """Any summary column not a KNOWN_KEY and numeric for >=1 sample -> an auto 'extra' metric (future-proofing)."""
- if not summ:
- return []
- seen = set()
- for mm in summ.values():
- for k in mm.keys():
- if k not in KNOWN_KEYS and k != "sample_id":
- seen.add(k)
- extras = []
- for k in sorted(seen):
- vals = [to_float(mm.get(k)) for mm in summ.values()]
- if any(v is not None for v in vals):
- label, kind, direction = infer_metric(k, vals)
- extras.append({"key": k, "label": label, "kind": kind, "dir": direction})
- return extras
-
-
-def _read_freq_file(path, max_pos=25):
- """Parse a mapDamage 5pCtoT_freq.txt / 3pGtoA_freq.txt ('posfreq'). Returns
- {'pos1': float|None, 'profile': [[pos,freq],...] up to max_pos} or None. Robust to header + blanks."""
- prof = []
- try:
- with open(path) as fh:
- for line in fh:
- line = line.strip()
- if not line or line.lower().startswith("pos"):
- continue
- parts = line.replace(",", "\t").split("\t")
- if len(parts) < 2:
- parts = line.split()
- if len(parts) < 2:
- continue
- p = to_float(parts[0]); v = to_float(parts[1])
- if p is None or v is None:
- continue
- prof.append((int(p), v))
- except (OSError, ValueError):
- return None
- if not prof:
- return None
- prof.sort(key=lambda x: x[0])
- pos1 = next((v for p, v in prof if p == 1), prof[0][1])
- return {"pos1": pos1, "profile": [[p, round(v, 5)] for p, v in prof[:max_pos]]}
-
-
-def _mean_fraglen(path):
- """Mean fragment length from lgdistribution.txt (StdLengthOccurences, '#'-comment lines)."""
- num = den = 0.0
- try:
- with open(path) as fh:
- for line in fh:
- if not line.strip() or line.startswith("#"):
- continue
- parts = line.split()
- if len(parts) < 3 or parts[1].lower() == "length":
- continue
- L = to_float(parts[1]); occ = to_float(parts[2])
- if L is None or occ is None:
- continue
- num += L * occ; den += occ
- except (OSError, ValueError):
- return None
- return (num / den) if den > 0 else None
-
-
-def mapdamage_stats(sample_dir):
- """Read a mapDamage2 output directory for one sample. Returns a dict or None if nothing parseable.
- keys: ct1 (5' C>T pos1), ga1 (3' G>A pos1), fraglen (mean), ct_profile [[pos,freq],...]."""
- if not sample_dir or not os.path.isdir(sample_dir):
- return None
- five = _read_freq_file(os.path.join(sample_dir, "5pCtoT_freq.txt"))
- three = _read_freq_file(os.path.join(sample_dir, "3pGtoA_freq.txt"))
- fl = _mean_fraglen(os.path.join(sample_dir, "lgdistribution.txt"))
- ct1 = five["pos1"] if five else None
- if five is None and three is None and fl is None and ct1 is None:
- return None
- return {"ct1": ct1, "ga1": (three["pos1"] if three else None), "fraglen": fl,
- "ct_profile": five["profile"] if five else None}
-
-
-def flag_sample(m, thr, snp_med, snp_sig, ancient=False, anc_thr=None, dmg=None):
- flags = []
- at = anc_thr if (ancient and anc_thr) else thr # ancient samples judged against the aDNA threshold view
- g = lambda k: to_float(m.get(k))
- # produced-nothing guard: a sample with no consensus AND no mapping metrics is a FAIL, not a quiet PASS-by-absence
- if all(g(k) is None for k in ("mean_depth", "breadth_pct", "missing_pct", "mapped_pct")):
- return "FAIL", ["NO_DATA"]
- if (v := g("mean_depth")) is not None and v < at["depth_min"]:
- flags.append("LOW_DEPTH")
- if (v := g("breadth_pct")) is not None and v < at["breadth_min"]:
- flags.append("LOW_BREADTH")
- if (v := g("missing_pct")) is not None and v > at["missing_max"]:
- flags.append("HIGH_MISSING")
- if (v := g("mapped_pct")) is not None and v < at["mapping_min"]:
- flags.append("MAPPING_LOW")
- if (v := g("duplication_pct")) is not None and v > at["dup_max"]:
- flags.append("HIGH_DUP")
- if (v := g("iupac_pct")) is not None and v > at["iupac_max"]:
- flags.append("HIGH_IUPAC")
- if thr["titv_min"] > 0 and (v := g("ti_tv")) is not None and v < thr["titv_min"]:
- flags.append("TITV_LOW")
- # SNP count is dominated by phylogenetic divergence, not quality -> a cohort distributional outlier (WARN only).
- # snp_med/sig are computed over MODERN samples; ancient SNP counts are low by design, so skip the z-flag for aDNA.
- if not ancient:
- snp = g("snps")
- if snp is not None and snp_med is not None and snp_sig:
- if snp < snp_med - thr["snp_z"] * snp_sig:
- flags.append("SNP_LOW")
- elif snp > snp_med + thr["snp_z"] * snp_sig:
- flags.append("SNP_HIGH")
- # het fraction: NOT applied to aDNA (post-mortem deamination inflates apparent het -> would false-flag mummies)
- if not ancient:
- hf = het_frac(m)
- if hf is not None and hf * 100.0 > thr["het_max_frac"]:
- flags.append("HET_HIGH")
- # MIXED: >1 MAIN lineage above the fraction cut-off AND above a minimum marker count (guards single-read noise)
- counts = lineage_counts_parsed(m.get("lineage_counts"))
- tot = sum(counts.values())
- if tot > 0 and sum(1 for c in counts.values()
- if (c / tot) * 100.0 >= thr["mixed_min_frac"] and c >= 3) > 1:
- flags.append("MIXED")
- # aDNA authentication: an ANCIENT sample whose terminal 5' C>T is below the floor -> possible modern contaminant
- if ancient and anc_thr is not None and dmg and dmg.get("ct1") is not None \
- and dmg["ct1"] < anc_thr["damage_min_ct"]:
- flags.append("DAMAGE_LOW")
- verdict = "FAIL" if any(f in FAIL_FLAGS for f in flags) else ("WARN" if flags else "PASS")
- return verdict, flags
-
-
-# ============================================================================= CSS / JS / SHELL
-# ---- report front-end assets (CSS / JS / HTML shell / logo) live next to this script in
-# report_assets/, one file per language so each stays editable with its own tooling; build_html()
-# splices them into the single self-contained output HTML. ----
-_ASSET_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "report_assets")
-
-
-def _asset(name):
- with open(os.path.join(_ASSET_DIR, name), encoding="utf-8") as fh:
- return fh.read()
-
-
-# CSS split into ordered modules under report_assets/css/ (base tokens -> layout -> components ->
-# panels); concatenated in cascade order, so the result is identical to a single stylesheet.
-_CSS_MODULES = ["css/01_base.css", "css/02_layout.css", "css/03_components.css", "css/04_panels.css"]
-CSS = "".join(_asset(m) for m in _CSS_MODULES)
-
-# The report front-end is one ES5 IIFE split into ordered modules under report_assets/js/ (one per
-# area) purely for editability; they are concatenated verbatim here, so the running code is identical
-# to a single file. Order matters: 01 opens the IIFE and sets up shared state; 14 wires events + closes it.
-_JS_MODULES = [
- "js/01_prelude.js", "js/02_qcspace.js", "js/03_state.js", "js/04_helpers.js",
- "js/05_insights_qc.js", "js/06_insights_genome.js", "js/07_render_core.js", "js/08_curation.js",
- "js/09_genome_genes.js", "js/10_dynamics.js", "js/11_epistasis.js", "js/12_snpmatrix.js",
- "js/13_drug_kraken.js", "js/14_boot.js",
-]
-JS = "".join(_asset(m) for m in _JS_MODULES)
-
-# BAMpiro header logo (bampiro2.png resized to ~90x100 and embedded so the report stays
-# self-contained - no external image request). Regenerate from .github/bampiro2.png if the logo changes.
-LOGO_DATA_URI = _asset("logo.b64")
-
-SHELL = _asset("shell.html") # named shell.html (not report_*.html) so .gitignore's report*.html rule can't swallow it
-
-
-def build_html(title, payload):
- # Embed the payload gzip-compressed + base64 so the whole cohort travels in a light, self-contained
- # HTML (JSON deflates ~5-10x); the browser inflates it natively at load. base64 has no ". Nothing is dropped for size - compression is what lets us keep it all.
- raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
- data_gz = base64.b64encode(gzip.compress(raw, 9)).decode("ascii")
- sys.stderr.write("[qc_report] payload %.2f MB JSON -> %.2f MB embedded (gzip+base64, %.0f%% smaller)\n"
- % (len(raw) / 1048576.0, len(data_gz) / 1048576.0,
- 100.0 * (1.0 - len(data_gz) / max(1, len(raw)))))
- # Inject content first, then substitute __TITLE__ LAST so a user --title that happens to
- # contain a placeholder token (e.g. "__JS__") can never pull in the CSS/JS/logo/JSON blob.
- return (SHELL.replace("__CSS__", CSS).replace("__LOGO__", LOGO_DATA_URI)
- .replace("__JS__", JS).replace("__JSON_GZ__", data_gz)
- .replace("__REPO__", html.escape(REPO_URL))
- .replace("__TITLE__", html.escape(title)))
-
-
-# ============================================================================
-# SNP DYNAMICS (optional): flexible metadata + per-sample VCFs -> per-connected-group
-# allele-frequency trajectories over time with flagged events. Self-hides if the metadata
-# has no time+group columns or no VCFs are given.
-# ============================================================================
-_DYN_SAMPLE_RE = re.compile(r'^(sample_?id|sampleid|sample|name|gid|strain|isolate)$', re.I)
-_DYN_TIME_RE = re.compile(r'(passage|pase|timepoint|time_?point|^time$|^day$|date|week|month|hour|generation|^tp$|visit|^t\d*$)', re.I)
-_DYN_GROUP_RE = re.compile(r'(group|series|patient|host|subject|cluster|experiment|^line$|replicate|chain|pair|lineage_?id|donor|case|animal|^samples$)', re.I)
-_DYN_NONSYN = re.compile(r'missense|stop_gained|stop_lost|start_lost|frameshift|inframe|splice|initiator', re.I)
-_DOSE_RE = re.compile(r'^(dose|dosis)$', re.I) # a numeric annotation -> a metric, not a categorical level
-_TX_RE = re.compile(r'(treatment|tratamiento|regimen|therapy|^arm$)', re.I) # the categorical column the dose x treatment test groups by
-
-
-def _dyn_open(path):
- return gzip.open(path, 'rt', encoding='utf-8', errors='replace') if str(path).endswith('.gz') \
- else open(path, 'r', encoding='utf-8', errors='replace')
-
-
-def _dyn_num(x):
- if x is None:
- return None
- m = re.search(r'-?\d+\.?\d*', str(x))
- return float(m.group()) if m else None
-
-
-_META_SKIP = {'r1', 'r2', 'reffasta', 'refgff', 'refid', 'taxid', 'runid'}
-
-
-def parse_sample_meta(path):
- """{'fields':[col...], 'rows':{sample:{col:value}}} for the user's annotation columns of the
- samplesheet (everything except the sample id and the pipeline's file/reference columns), in
- samplesheet order -> one column-header level per field. None if nothing usable."""
- if not path or not os.path.exists(path):
- return None
- header, rows = None, []
- try:
- with _dyn_open(path) as fh:
- for line in fh:
- if not line.strip() or line.startswith('#'):
- continue
- cells = line.rstrip('\n').split('\t')
- if header is None:
- header = [c.strip() for c in cells]
- continue
- rows.append(cells)
- except OSError:
- return None
- if not header:
- return None
- si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
- # the dose column (if any) is a numeric metric, not a categorical annotation -> keep it out of fields
- fields = [(i, h) for i, h in enumerate(header)
- if i != si and h.strip() and h.strip().lower() not in _META_SKIP
- and not _DOSE_RE.match(h.strip())]
- if not fields:
- return None
- out = {}
- for r in rows:
- if si >= len(r):
- continue
- s = r[si].strip()
- if not s or s in out:
- continue
- out[s] = {h: (r[i].strip() if i < len(r) else '') for i, h in fields}
- if not out:
- return None
- # tag the time / group columns (dynamics axes) and the treatment column (dose x treatment test)
- # so the report can route each correctly; they all stay as SNP-matrix header levels either way.
- time_field = next((h for _, h in fields if _DYN_TIME_RE.search(h.replace(' ', '_'))), None)
- group_field = next((h for _, h in fields if _DYN_GROUP_RE.search(h.replace(' ', '_'))), None)
- tx_field = next((h for _, h in fields if _TX_RE.search(h.replace(' ', '_'))), None)
- return {'fields': [h for _, h in fields], 'rows': out,
- 'time_field': time_field, 'group_field': group_field, 'tx_field': tx_field}
-
-
-def parse_dose(path):
- """{sample: float} for a numeric samplesheet column named 'dose'/'dosis' (a quantitative
- annotation exposed as a report metric). {} if the file or column is absent / non-numeric."""
- if not path or not os.path.exists(path):
- return {}
- header, rows = None, []
- try:
- with _dyn_open(path) as fh:
- for line in fh:
- if not line.strip() or line.startswith('#'):
- continue
- cells = line.rstrip('\n').split('\t')
- if header is None:
- header = [c.strip() for c in cells]
- continue
- rows.append(cells)
- except OSError:
- return {}
- if not header:
- return {}
- si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
- di = next((i for i, h in enumerate(header) if _DOSE_RE.match(h.strip())), None)
- if di is None:
- return {}
- out = {}
- for r in rows:
- if si >= len(r) or di >= len(r):
- continue
- s = r[si].strip()
- if not s or s in out:
- continue
- v = to_float(r[di].strip())
- if v is not None:
- out[s] = v
- return out
-
-
-def parse_metadata(path):
- """{sample: {'time','tnum','group'}} auto-detecting sample/time/group columns; {} if unusable."""
- if not path or not os.path.exists(path):
- return {}
- header, rows = None, []
- try:
- with _dyn_open(path) as fh:
- for line in fh:
- if not line.strip() or line.startswith('#'):
- continue
- cells = line.rstrip('\n').split('\t')
- if header is None:
- header = [c.strip() for c in cells]
- continue
- rows.append(cells)
- except OSError:
- return {}
- if not header:
- return {}
- def find(rx):
- for i, h in enumerate(header):
- if rx.search(h.replace(' ', '_')):
- return i
- return None
- si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
- ti, gi = find(_DYN_TIME_RE), find(_DYN_GROUP_RE)
- out = {}
- for r in rows:
- if si >= len(r):
- continue
- s = r[si].strip()
- if not s or s in out:
- continue
- out[s] = {'time': r[ti].strip() if (ti is not None and ti < len(r)) else None,
- 'tnum': _dyn_num(r[ti]) if (ti is not None and ti < len(r)) else None,
- 'group': r[gi].strip() if (gi is not None and gi < len(r)) else None}
- return out
-
-
-def _dyn_af(fmt, val):
- """First-alt allele fraction: AD/AO+RO (true AF) else GT dosage (hom 1.0 / het 0.5)."""
- d = dict(zip(fmt.split(':'), val.split(':')))
- if 'AD' in d:
- try:
- ad = [int(x) for x in d['AD'].split(',') if x not in ('.', '')]
- if len(ad) >= 2 and sum(ad) > 0:
- return sum(ad[1:]) / sum(ad)
- except ValueError:
- pass
- if 'AO' in d and 'RO' in d:
- try:
- ao = sum(int(x) for x in d['AO'].split(',') if x not in ('.', ''))
- ro = int(d['RO']) if d['RO'] not in ('.', '') else 0
- if ao + ro > 0:
- return ao / (ao + ro)
- except ValueError:
- pass
- alleles = [a for a in d.get('GT', './.').replace('|', '/').split('/') if a not in ('.', '')]
- if not alleles:
- return None
- return sum(1 for a in alleles if a != '0') / len(alleles)
-
-
-def _dyn_dp(fmt, val):
- """Total depth at the site: FORMAT DP, else sum(AD), else AO+RO."""
- d = dict(zip(fmt.split(':'), val.split(':')))
- if d.get('DP', '.') not in ('.', ''):
- try:
- return int(d['DP'])
- except ValueError:
- pass
- if 'AD' in d:
- try:
- ad = [int(x) for x in d['AD'].split(',') if x not in ('.', '')]
- if ad:
- return sum(ad)
- except ValueError:
- pass
- if 'AO' in d and 'RO' in d:
- try:
- return sum(int(x) for x in d['AO'].split(',') if x not in ('.', '')) + \
- (int(d['RO']) if d['RO'] not in ('.', '') else 0)
- except ValueError:
- pass
- return None
-
-
-def _dyn_ann(info):
- # snpEff ANN fields: 3=gene, 1=effect, 2=impact, 10=HGVS.p (protein change, e.g. p.Ser315Thr)
- for field in info.split(';'):
- if field.startswith('ANN='):
- a = field[4:].split(',')[0].split('|')
- return ((a[3] if len(a) > 3 else ''), (a[1] if len(a) > 1 else ''),
- (a[2] if len(a) > 2 else ''), (a[10] if len(a) > 10 else ''))
- return '', '', '', ''
-
-
-def _dyn_opos(info):
- """The ORIGINAL (used-reference) coordinate a liftover stamped onto a canonical-reference VCF record,
- so a used-ref variant can be paired with its reference-of-interest position WITHOUT assuming the two
- references share coordinates. Reads Picard LiftoverVcf's OriginalContig/OriginalStart, else a plain
- OPOS=. Returns 'contig:pos' (used-ref coordinates) or None."""
- if not info:
- return None
- oc = ost = None
- for field in info.split(';'):
- if field.startswith('OriginalContig='):
- oc = field[len('OriginalContig='):]
- elif field.startswith('OriginalStart='):
- ost = field[len('OriginalStart='):]
- elif field.startswith('OPOS='):
- return field[len('OPOS='):].strip() or None
- return (oc + ':' + ost) if (oc and ost) else None
-
-
-def parse_vcfs(paths):
- """{sample: {'chrom:pos': {ref,alt,gene,eff,imp,af}}} from annotated per-sample VCFs (SNPs only).
- Sample = the VCF #CHROM last column, so filenames are irrelevant."""
- out = {}
- for p in paths or []:
- if not p or not os.path.exists(p):
- continue
- try:
- with _dyn_open(p) as fh:
- sample = None
- for line in fh:
- if line.startswith('##'):
- continue
- if line.startswith('#CHROM'):
- cols = line.rstrip('\n').split('\t')
- sample = cols[9] if len(cols) > 9 else os.path.basename(p).split('.')[0]
- out.setdefault(sample, {})
- continue
- if sample is None:
- continue
- c = line.rstrip('\n').split('\t')
- if len(c) < 8:
- continue
- chrom, pos, ref, alt = c[0], c[1], c[3], c[4]
- if alt in ('.', '') or len(ref) != 1 or any(len(a) != 1 for a in alt.split(',')):
- continue
- af = _dyn_af(c[8], c[9]) if len(c) >= 10 else 1.0
- if af is None:
- continue
- dp = _dyn_dp(c[8], c[9]) if len(c) >= 10 else None
- gene, eff, imp, aa = _dyn_ann(c[7])
- out[sample][f'{chrom}:{pos}'] = {'ref': ref, 'alt': alt.split(',')[0], 'gene': gene,
- 'eff': eff, 'imp': imp, 'aa': aa, 'aa_h37rv': '',
- 'af': round(af, 4), 'dp': dp, 'opos': _dyn_opos(c[7])}
- except OSError:
- continue
- return out
-
-
-def build_dynamics(metadata, variants, sample_meta=None, emerge=0.25, fix=0.90, loss=0.10, min_points=2, min_move=0.15):
- """Per-connected-group AF trajectories over time, only for SNPs that move. None if nothing to show.
- sample_meta (parse_sample_meta output) attaches each group's metadata signature so the report can
- filter the series by any group-invariant field (e.g. patient / lineage)."""
- if not metadata:
- return None
- groups = {}
- for s, md in metadata.items():
- g = md.get('group')
- if g and s in variants:
- groups.setdefault(g, []).append(s)
- out_groups = []
- for g, samples in sorted(groups.items()):
- samples = sorted(samples, key=lambda s: (metadata[s]['tnum'] is None,
- metadata[s]['tnum'] if metadata[s]['tnum'] is not None else 0,
- str(metadata[s]['time'])))
- times = [metadata[s]['time'] for s in samples]
- if len(set(times)) < min_points:
- continue
- allpos = set()
- for s in samples:
- allpos.update(variants[s].keys())
- series = []
- for pos in sorted(allpos): # deterministic order (a set iterates in a PYTHONHASHSEED-dependent order)
- traj, dps, meta = [], [], None
- for s in samples:
- v = variants[s].get(pos)
- traj.append(v['af'] if v else 0.0)
- dps.append(v.get('dp') if v else None) # per-timepoint depth (None where not called)
- if v and meta is None:
- meta = v
- if max(traj) - min(traj) < min_move:
- continue
- flags = []
- if traj[0] <= loss and max(traj) >= emerge:
- flags.append('emergence')
- if traj[-1] >= fix and traj[0] < fix:
- flags.append('fixation')
- if traj[0] >= emerge and traj[-1] <= loss:
- flags.append('loss')
- if meta and _DYN_NONSYN.search(meta.get('eff', '')):
- flags.append('nonsyn')
- if meta and meta.get('imp', '') == 'HIGH':
- flags.append('high_impact')
- series.append({'pos': pos, 'gene': (meta or {}).get('gene', ''), 'eff': (meta or {}).get('eff', ''),
- 'imp': (meta or {}).get('imp', ''), 'alt': (meta or {}).get('alt', ''),
- 'aa': (meta or {}).get('aa', ''), 'aa_h37rv': (meta or {}).get('aa_h37rv', ''),
- 'pos_h37rv': (meta or {}).get('pos_h37rv', ''),
- 'traj': [round(x, 4) for x in traj], 'dp': dps, 'flags': flags})
- if not series:
- continue
- series.sort(key=lambda x: (-(max(x['traj']) - min(x['traj'])), -len(x['flags'])))
- gmeta = {}
- if sample_meta and sample_meta.get('fields'):
- rows = sample_meta.get('rows', {})
- for f in sample_meta['fields']:
- vals = sorted({v for v in ((rows.get(s) or {}).get(f) for s in samples) if v})
- if vals:
- gmeta[f] = vals
- out_groups.append({'group': g, 'samples': samples, 'times': times,
- 'tnums': [metadata[s]['tnum'] for s in samples], 'meta': gmeta,
- 'series': series, 'n_flagged': sum(1 for x in series if x['flags'])})
- if not out_groups:
- return None
- return {'groups': out_groups, 'thresholds': {'emerge': emerge, 'fix': fix, 'loss': loss}}
-
-
-def _pearson(x, y):
- """Pearson correlation of two equal-length trajectories, or None if undefined (n<3 or a flat one)."""
- n = len(x)
- if n < 3 or len(y) != n:
- return None
- mx, my = sum(x) / n, sum(y) / n
- sxx = sum((a - mx) ** 2 for a in x)
- syy = sum((b - my) ** 2 for b in y)
- if sxx <= 1e-9 or syy <= 1e-9:
- return None
- sxy = sum((a - mx) * (b - my) for a, b in zip(x, y))
- return sxy / ((sxx * syy) ** 0.5)
-
-
-def _perm_p(series_traj, obs_mean, B=2000, seed=0):
- """Permutation p-value for a pair's mean correlation: shuffle one trajectory's values within EACH
- series independently (breaking the time linkage) and recompute the mean |r|; p = fraction of the B
- nulls reaching the observed |r| (+1 smoothing). Deterministic (fixed per-pair seed) -> reproducible.
- With few timepoints the resolution is inherently coarse (a 4-point series has only 4!=24 orders), so
- a lone short series lands near p~0.04 while a pattern recurring across series drives p far lower."""
- obs = abs(obs_mean)
- rng = random.Random(seed)
- ge = 0
- for _ in range(B):
- tot, cnt = 0.0, 0
- for ta, tb in series_traj:
- perm = list(ta)
- rng.shuffle(perm)
- r = _pearson(perm, tb)
- if r is not None:
- tot += r
- cnt += 1
- if cnt and abs(tot / cnt) >= obs:
- ge += 1
- return (ge + 1) / (B + 1)
-
-
-def _amp(traj):
- """Amplitude (max-min) of a trajectory's non-null allele frequencies; 0 if flat/empty. Used to pick the
- most-variable variants when a series has too many to test all pairs of."""
- vals = [v for v in (traj or []) if v is not None]
- return (max(vals) - min(vals)) if len(vals) >= 2 else 0.0
-
-
-def build_epistasis(dynamics, min_r=0.8, min_points=3, top=300, perm=2000, max_vars=250, max_pairs_perm=800):
- """Candidate epistatic / linked variant pairs: two SNPs whose allele-frequency trajectories co-vary
- WITHIN a connected series - concordant (rise/fall together) or discordant (one rises as the other
- falls). Score = mean Pearson r of the two trajectories across every series where both move and the
- series has >= min_points timepoints. Significance = a permutation p-value (per pair), then a
- Benjamini-Hochberg FDR q-value across all reported pairs. Recurrence across independent series is
- tracked so a pattern seen in several series (low q) is separable from a lucky single short series.
- None if nothing qualifies. Built from the dynamics payload (moving variants per series).
-
- Scaling: the pairwise step is O(variants^2) per series and the permutation test is the dominant cost,
- so two caps keep it bounded for large cohorts (many variants and/or many samples -> long trajectories):
- each series contributes at most max_vars variants (the highest-amplitude / most-variable ones), and only
- the max_pairs_perm strongest-|r| candidate pairs get the (expensive) permutation p-value. Both caps are
- reported in the payload so the UI can say what was covered; the full trajectories stay in the dynamics/TSV."""
- if not dynamics or not dynamics.get('groups'):
- return None
- pairs, n_series = {}, 0
- vars_capped, n_vars_max = False, 0
- for g in dynamics['groups']:
- times = g.get('times') or []
- if len(set(times)) < min_points:
- continue
- n_series += 1
- series = g.get('series') or []
- n_vars_max = max(n_vars_max, len(series))
- if len(series) > max_vars: # too many variants to test every pair: keep the most-variable ones
- vars_capped = True
- series = sorted(series, key=lambda s: -_amp(s.get('traj')))[:max_vars]
- for i in range(len(series)):
- for j in range(i + 1, len(series)):
- a, b = series[i], series[j]
- r = _pearson(a['traj'], b['traj'])
- if r is None:
- continue
- lo, hi = (a, b) if str(a['pos']) <= str(b['pos']) else (b, a) # order-independent key
- key = (lo['pos'], hi['pos'])
- rec = pairs.get(key)
- if rec is None:
- rec = pairs[key] = {'A': lo, 'B': hi, 'rs': [], 'straj': [], 'bestn': -1,
- 'times': times, 'trajA': lo['traj'], 'trajB': hi['traj'], 'group': g['group']}
- rec['rs'].append(r)
- rec['straj'].append((lo['traj'], hi['traj'])) # every series' pair, for the permutation test
- if len(set(times)) > rec['bestn']: # keep the richest series for the mini-chart
- rec['bestn'] = len(set(times))
- rec['times'], rec['trajA'], rec['trajB'], rec['group'] = times, lo['traj'], hi['traj'], g['group']
- # Collect the pairs that clear the |r| threshold, then permutation-test only the strongest ones: the
- # permutation p-value is by far the dominant cost, so with many candidate pairs (common when trajectories
- # are short and spurious high correlations abound) we cap it to the top max_pairs_perm by |mean r|.
- cands = []
- for key, rec in pairs.items():
- rs = rec['rs']
- mean_r = sum(rs) / len(rs)
- if abs(mean_r) < min_r:
- continue
- cands.append((abs(mean_r), key, rec, mean_r))
- n_candidates = len(cands)
- pairs_capped = n_candidates > max_pairs_perm
- if pairs_capped:
- cands.sort(key=lambda c: -c[0])
- cands = cands[:max_pairs_perm]
- out = []
- for _absr, key, rec, mean_r in cands:
- rs = rec['rs']
- A, B = rec['A'], rec['B']
- n_pos = sum(1 for r in rs if r > 0)
- # seed from the STABLE pair key (positions), not the iteration index, so the permutation p-value is
- # reproducible regardless of dict/set iteration order. crc32 is deterministic (unlike hash()).
- seed = zlib.crc32(('%s|%s' % (key[0], key[1])).encode('utf-8'))
- p = _perm_p(rec['straj'], mean_r, B=perm, seed=seed)
- out.append({'geneA': A.get('gene', ''), 'posA': A['pos'], 'aaA': A.get('aa', ''), 'aaA_h37rv': A.get('aa_h37rv', ''), 'effA': A.get('eff', ''),
- 'geneB': B.get('gene', ''), 'posB': B['pos'], 'aaB': B.get('aa', ''), 'aaB_h37rv': B.get('aa_h37rv', ''), 'effB': B.get('eff', ''),
- 'r': round(mean_r, 3), 'n': len(rs), 'rmin': round(min(rs), 3), 'rmax': round(max(rs), 3),
- 'direction': 'concordant' if mean_r > 0 else 'discordant',
- 'p': p, 'consistent': (n_pos == len(rs) or n_pos == 0),
- 'times': rec['times'], 'trajA': rec['trajA'], 'trajB': rec['trajB'], 'group': rec['group']})
- if not out:
- return None
- # Benjamini-Hochberg FDR across every reported pair (multiple-testing correction)
- m = len(out)
- order = sorted(range(m), key=lambda i: out[i]['p'])
- qmin = 1.0
- for rank in range(m, 0, -1):
- i = order[rank - 1]
- qmin = min(qmin, out[i]['p'] * m / rank)
- out[i]['q'] = round(min(qmin, 1.0), 4)
- for pr in out:
- pr['tier'] = 'strong' if pr['q'] <= 0.05 else ('moderate' if pr['p'] <= 0.05 else 'weak')
- pr['p'] = round(pr['p'], 4)
- out.sort(key=lambda p: (p['q'], -abs(p['r'])))
- out = out[:top]
- # all-vs-all correlation matrix over the most-connected variants (INCLUDES sub-threshold cells, so it
- # is the full co-dynamics landscape, not only the reported pairs). Capped to bound the embedded size.
- allmeta, deg, meanr = {}, {}, {}
- for (a, b), rec in pairs.items():
- meanr[(a, b)] = sum(rec['rs']) / len(rec['rs'])
- deg[a] = deg.get(a, 0) + 1
- deg[b] = deg.get(b, 0) + 1
- for v in (rec['A'], rec['B']):
- allmeta.setdefault(v['pos'], {'pos': v['pos'], 'gene': v.get('gene', ''), 'aa': v.get('aa', '')})
- MAXNODES = 50
- node_pos = sorted(deg, key=lambda x: (-deg[x], x))[:MAXNODES] # keep the best-connected variants
- node_pos.sort(key=lambda x: (allmeta[x]['gene'] or 'zzz', x)) # display order: gene, then position
- nidx = {p: i for i, p in enumerate(node_pos)}
- cells = {}
- for (a, b), mr in meanr.items():
- if a in nidx and b in nidx:
- i, j = nidx[a], nidx[b]
- lo, hi = (i, j) if i <= j else (j, i)
- cells['%d,%d' % (lo, hi)] = round(mr, 3)
- matrix = {'nodes': [allmeta[p] for p in node_pos], 'cells': cells,
- 'truncated': len(deg) > MAXNODES, 'total_nodes': len(deg)}
- return {'pairs': out, 'min_r': min_r, 'min_points': min_points, 'n_series': n_series, 'perm': perm,
- 'matrix': matrix,
- 'vars_capped': vars_capped, 'max_vars': max_vars, 'n_vars_max': n_vars_max,
- 'pairs_capped': pairs_capped, 'n_candidates': n_candidates, 'max_pairs_perm': max_pairs_perm,
- 'n_concordant': sum(1 for p in out if p['direction'] == 'concordant'),
- 'n_discordant': sum(1 for p in out if p['direction'] == 'discordant'),
- 'n_strong': sum(1 for p in out if p['tier'] == 'strong')}
-
-
-def build_snp_matrix(variants, reference='', max_sites=50000):
- """Sparse SNP matrix for the report: samples + one row per SNP site with its ref/alt/annotation
- and only the cells (sample index -> [af, dp]) actually called. The payload is gzip-compressed in the
- HTML, so this cap is a high safety backstop (most-shared sites kept) rather than a size limit; the
- pipeline also writes the complete matrix TSV separately."""
- if not variants:
- return None
- samples = list(variants.keys())
- sidx = {s: i for i, s in enumerate(samples)}
- sites = {}
- for s, posmap in variants.items():
- for key, v in posmap.items():
- contig, _, pos = key.rpartition(':')
- try:
- ipos = int(pos)
- except ValueError:
- continue
- st = sites.setdefault(key, {'contig': contig, 'pos': ipos, 'ref': v.get('ref', ''),
- 'alt': set(), 'gene': '', 'eff': '', 'aa': '', 'aa_h37rv': '', 'pos_h37rv': '', 'cells': {}})
- if v.get('alt'):
- st['alt'].add(v['alt'])
- for k in ('gene', 'eff', 'aa', 'aa_h37rv', 'pos_h37rv'):
- if v.get(k) and not st[k]:
- st[k] = v[k]
- st['cells'][sidx[s]] = [v['af'], v.get('dp')]
- ordered = sorted(sites.values(), key=lambda x: (x['contig'], x['pos']))
- truncated = len(ordered) > max_sites
- if truncated:
- ordered = sorted(sites.values(), key=lambda x: -len(x['cells']))[:max_sites]
- ordered.sort(key=lambda x: (x['contig'], x['pos']))
- rows = [{'contig': x['contig'], 'pos': x['pos'], 'ref': x['ref'], 'alt': ','.join(sorted(x['alt'])),
- 'gene': x['gene'], 'eff': x['eff'], 'aa': x['aa'], 'aa_h37rv': x['aa_h37rv'],
- 'pos_h37rv': x['pos_h37rv'], 'n': len(x['cells']), 'cells': x['cells']}
- for x in ordered]
- return {'samples': samples, 'reference': reference, 'rows': rows,
- 'total_sites': len(sites), 'truncated': truncated}
-
-
-# Per-panel explanation shown by the (i) icon in each section heading: what the analysis is, how to read
-# it, and its caveats. Keyed by the section id in the HTML shell.
-SECTION_INFO = {
- "gstats": "Per-sample sequencing and mapping QC: reads, duplication, mapped %, mean/median depth, "
- "breadth, variant counts and Ti/Tv. Each numeric cell is shaded by where the sample sits in "
- "that column's range; the QC column flags PASS/WARN/FAIL against the live thresholds. Tick a "
- "box to basket a sample for exclusion; click a name for its full profile.",
- "linsum": "Median QC metrics grouped by the assigned MTBC lineage, with the number of samples, %PASS and "
- "how many are 'mixed' (more than one lineage above the mixture cut-off). Lineage comes from "
- "pathotypr; a mixed call can indicate co-infection or contamination.",
- "dist": "Per-metric value distributions across the cohort (a strip/bee plot, or a ranked bar). The shaded "
- "band is the acceptable range for the current thresholds; points outside it are outliers. Useful "
- "to see cohort spread and choose sensible cut-offs.",
- "corr": "A scatter of any two QC metrics, one point per sample, coloured by QC status or lineage. Drag a "
- "box to basket the enclosed samples. Reveals metric relationships (e.g. depth vs breadth) and "
- "samples that are joint outliers.",
- "corrmatrix": "Pairwise Pearson correlation between the QC metrics across the cohort, as a heatmap. Shows "
- "which metrics are redundant (strongly correlated) and which capture independent axes of quality.",
- "qcpca": "PCA of the standardised QC metrics: each point is a sample placed by its overall quality "
- "profile. Samples that cluster share a profile; a lone point is a multivariate outlier that no "
- "single metric flags.",
- "divcomp": "Each sample's genomic divergence (SNPs vs the reference) against its consensus completeness. "
- "Separates genuinely divergent samples from those that only look divergent because they are "
- "incomplete or contaminated.",
- "cons": "How much of the reference each sample's consensus recovers: callable %, missing % (N/gap), IUPAC "
- "(ambiguous) % and the longest gap. A phylogeny-oriented view of completeness, not just average depth.",
- "genome": "Per-position callability / variant density along the reference, binned into a heatmap per "
- "sample. Brush a region to list the genes under it. Reveals systematically low-callability "
- "regions (repeats, deletions) shared across samples.",
- "function": "The snpEff functional class of each sample's variants (HIGH/MODERATE/LOW/MODIFIER impact and "
- "effect types such as missense / synonymous). A per-sample mutational-impact profile.",
- "geneburden": "Genes carrying the most impactful (HIGH/MODERATE) variants across the cohort, with the "
- "dominant effect and how many samples are hit. A cohort-level view of which genes accumulate "
- "functional change.",
- "hotspots": "Genes with the highest SNP density across the cohort (variants per gene length). Flags the "
- "most variable loci - often repeats, antigens, or genes under diversifying selection.",
- "temporal": "The distribution of sampling dates, when the samplesheet provides them. Context for temporal "
- "analyses (e.g. SNP dynamics); not a QC metric itself.",
- "pnps": "Cohort-level pairwise dN/dS per gene (eskaks, Nei/Li). dN/dS > 1 suggests positive / diversifying "
- "selection at cohort scale. This is population genetics, NOT per-sample QC: noisy per gene, "
- "sensitive to alignment, and saturating at low divergence (NA when dS is near zero).",
- "adna": "For samples marked ancient: the characteristic post-mortem damage signal (elevated terminal 5' "
- "C to T). A sample below the damage floor may be a modern contaminant rather than authentic "
- "ancient DNA.",
- "dynamics": "For connected time-series (patient / passage line), each variant's allele-frequency "
- "trajectory over time, with per-timepoint read depth. Events flag emergence, fixation, loss "
- "and non-synonymous changes. Needs a time column and a group column in the samplesheet.",
- "epistasis": "Pairs of variants whose allele-frequency trajectories co-vary within a series: concordant "
- "(rise/fall together) or discordant (one rises as the other falls). Scored by the Pearson "
- "correlation, a permutation p-value and a Benjamini-Hochberg FDR q; a pattern recurring across "
- "independent series is what makes a pair 'strong'. Candidate linked / co-selected / competing "
- "SNPs, NOT proof of a functional interaction.",
- "snpmatrix": "Every SNP site (rows) by sample (columns); each cell is the allele frequency with its depth, "
- "plus a reference column and samplesheet metadata as column-header levels. Filter by gene / "
- "position or by metadata, and download the full matrix as a TSV.",
- "drug": "Resistance-associated mutations detected by pathotypr against the WHO catalogue (H37Rv numbering), "
- "as a sample x drug matrix (worst grade per drug) and a per-mutation table with the WHO confidence "
- "grade. Alignment-free (k-mer), so it works regardless of the mapping reference. A genomic screen, "
- "NOT a clinical drug-susceptibility result.",
- "flagged": "The samples the current thresholds flag as WARN / FAIL and the specific reason for each. This "
- "is the actionable QC summary; adjust the thresholds above to re-flag the whole report.",
- "dosetx": "The per-sample dose (a numeric samplesheet column) split by treatment group, drawn as a box (IQR "
- "+ median) with the individual samples, plus a Kruskal-Wallis rank test of whether dose differs "
- "across the treatment groups (a Mann-Whitney-equivalent when there are two groups). The test runs "
- "over the whole cohort, independent of the live filters; groups with fewer than two dosed samples "
- "are drawn but not tested. Click a point to highlight that sample across the report. Descriptive, "
- "not a claim about efficacy.",
- "vardose": "An association scan: for every variant site, the per-sample allele frequency (0 where the site is "
- "reference) is rank-correlated with dose across the dosed samples (Spearman rho + two-sided p), and "
- "a Benjamini-Hochberg FDR q is computed across all tested variants so the multiple testing is "
- "controlled. The table ranks the variants; click a row to plot its allele-frequency-vs-dose scatter, "
- "click a column header to re-sort, and click a point to highlight that sample. Sites with fewer than "
- "three carriers or no allele-frequency variation are skipped. A screen for dose-associated variants, "
- "NOT proof of causation.",
-}
+from qcreport.metrics import (ANC_DEF, DEF, DEFS, DIST, METRICS, build_gene_map, discover_extra_metrics,
+ flag_sample, het_frac, is_ancient, lineage_counts_parsed, lineage_fracs, robust)
+from qcreport.panels import build_dynamics, build_epistasis, build_snp_matrix
+from qcreport.parsers import (NBINS, clean_str, consensus_stats, mapdamage_stats, mask_profile, parse_bed,
+ parse_dose, parse_dr, parse_gene_burden, parse_gff, parse_kraken,
+ parse_lineage_colors, parse_metadata, parse_pnps, parse_profile,
+ parse_sample_meta, parse_summary, parse_vcfs, to_float)
+from qcreport.render import REPO_URL, SECTION_INFO, build_html
-def main():
+def build_parser():
+ """The CLI. Every threshold in DEF / ANC_DEF gets a --flag automatically, so adding a
+ threshold to the metrics module is all it takes to make it settable from the command line."""
ap = argparse.ArgumentParser(description="BAMpiro interactive QC report + flags.")
ap.add_argument("--summary", required=True)
ap.add_argument("--consensus", nargs="*", default=[])
@@ -1477,37 +84,13 @@ def main():
"reference-of-interest COORDINATE per variant, alignment-free. Fills pos_h37rv for the SNP "
"tables; an alternative to --vcfs-h37rv when the references do not share coordinates.")
ap.add_argument("--gate", action="store_true")
- args = ap.parse_args()
- thr = {k: getattr(args, k) for k in DEF}
- anc_thr = {k: getattr(args, "anc_" + k) for k in ANC_DEF}
-
- summ = parse_summary(args.summary)
- miss_by_sample = {} # per-sample binned missingness profile along the reference (genome landscape)
- for path in args.consensus:
- try:
- cs = consensus_stats(path)
- except Exception as e:
- sys.stderr.write(f"[qc_report] WARN consensus {path}: {e}\n")
- continue
- sid = cs.get("consensus_id")
- if sid not in summ:
- stem = os.path.basename(path).split(".")[0]
- sid = stem if stem in summ else sid
- keys = ("length", "missing_pct", "iupac_pct", "callable_pct", "longest_n_run", "n_gaps")
- if sid in summ:
- summ[sid].update({k: cs[k] for k in keys})
- elif sid:
- summ[sid] = {"sample_id": sid, **{k: cs[k] for k in keys}}
- if sid:
- miss_by_sample[sid] = cs.get("miss_profile")
+ return ap
- # SNP-count robust-z is computed over MODERN samples only (ancient counts are low by design and would skew it)
- snp_med, snp_sig = robust([to_float(m.get("snps")) for m in summ.values() if not is_ancient(m)])
- # representative genome length: consensus length, else the reference length column (works when make_consensus is off)
- genome_len = max([int(v) for v in (to_float(m.get("length")) or to_float(m.get("genome_len"))
- for m in summ.values()) if v], default=0)
- snp_density_ok = genome_len > 100000
+def collect_damage(mapdamage_dirs, summ):
+ """{sample: mapDamage stats} for the ancient samples, from the --mapdamage-dir path(s).
+ A dir may be the sample's own mapDamage output, a parent of per-sample subdirs, or the
+ published {OUTDIR}/{sample_dir}/mapDamage layout; all three are probed."""
# ---- aDNA damage authentication: map each mapDamage dir to a sample id ----
ancient_ids = {sid for sid, m in summ.items() if is_ancient(m)}
dmg_by_sample = {}
@@ -1534,7 +117,7 @@ def _index_dir(d):
if st:
dmg_by_sample[aid] = st
- for d in (args.mapdamage_dir or []):
+ for d in (mapdamage_dirs or []):
_index_dir(d)
if os.path.isdir(d):
for name in sorted(os.listdir(d)):
@@ -1546,6 +129,81 @@ def _index_dir(d):
if ancient_ids and not dmg_by_sample:
sys.stderr.write(f"[qc_report] {len(ancient_ids)} ancient sample(s) but no mapDamage output matched; "
f"DAMAGE authentication skipped.\n")
+ return dmg_by_sample
+
+
+def load_variants(args):
+ """{sample: {'chrom:pos': variant}} from --vcfs, with the canonical-reference amino acid
+ (--vcfs-h37rv) and/or the canonical coordinate (--pos-liftover) attached where available."""
+ _variants = parse_vcfs(args.vcfs) # parsed once, feeds both the dynamics panel and the SNP matrix
+ if args.vcfs_h37rv: # attach the reference-of-interest (H37Rv) amino-acid change AND coordinate per variant
+ _h37 = parse_vcfs(args.vcfs_h37rv)
+ for _s, _pm in _variants.items():
+ _hs = _h37.get(_s, {})
+ # Pair each used-ref variant with its canonical-reference record. Prefer the ORIGINAL used-ref
+ # coordinate a liftover stamped (OriginalContig/Start or OPOS) so the two references need NOT
+ # share coordinates; else fall back to the bare position (references that DO share H37Rv coords).
+ _by_opos, _by_pos = {}, {}
+ for _ck, _cv in _hs.items():
+ if _cv.get('opos'):
+ _by_opos[_cv['opos']] = (_ck, _cv)
+ _by_pos.setdefault(_ck.rpartition(':')[2], (_ck, _cv))
+ for _key, _v in _pm.items():
+ _hit = _by_opos.get(_key) or _by_pos.get(_key.rpartition(':')[2])
+ if _hit:
+ _ck, _cv = _hit
+ if _cv.get('aa'):
+ _v['aa_h37rv'] = _cv['aa']
+ _v['pos_h37rv'] = _ck # reference-of-interest coordinate (contig:pos), possibly != the mapping one
+ if args.pos_liftover: # alignment-free canonical coordinate per variant (pathotypr k-mer liftover map)
+ _lift = {}
+ try:
+ with open(args.pos_liftover, encoding="utf-8", errors="replace") as _fh:
+ for _line in _fh:
+ _c = _line.rstrip("\n").split("\t")
+ if len(_c) >= 2 and _c[0].strip().isdigit() and _c[1].strip().isdigit():
+ _lift[_c[0].strip()] = _c[1].strip()
+ except OSError as _e:
+ sys.stderr.write("[qc_report] WARN pos-liftover (%s): %s\n" % (args.pos_liftover, _e))
+ for _s, _pm in _variants.items():
+ for _key, _v in _pm.items():
+ _p = _key.rpartition(":")[2]
+ if not _v.get("pos_h37rv") and _p in _lift:
+ _v["pos_h37rv"] = "%s:%s" % (args.aa2_label, _lift[_p])
+ return _variants
+
+
+def build_payload(args, thr, anc_thr):
+ """Everything the report embeds: (payload, jsamples, counts). jsamples and counts are
+ returned alongside because the flags TSV and the --gate exit are built from them."""
+ summ = parse_summary(args.summary)
+ miss_by_sample = {} # per-sample binned missingness profile along the reference (genome landscape)
+ for path in args.consensus:
+ try:
+ cs = consensus_stats(path)
+ except Exception as e:
+ sys.stderr.write(f"[qc_report] WARN consensus {path}: {e}\n")
+ continue
+ sid = cs.get("consensus_id")
+ if sid not in summ:
+ stem = os.path.basename(path).split(".")[0]
+ sid = stem if stem in summ else sid
+ keys = ("length", "missing_pct", "iupac_pct", "callable_pct", "longest_n_run", "n_gaps")
+ if sid in summ:
+ summ[sid].update({k: cs[k] for k in keys})
+ elif sid:
+ summ[sid] = {"sample_id": sid, **{k: cs[k] for k in keys}}
+ if sid:
+ miss_by_sample[sid] = cs.get("miss_profile")
+
+ # SNP-count robust-z is computed over MODERN samples only (ancient counts are low by design and would skew it)
+ snp_med, snp_sig = robust([to_float(m.get("snps")) for m in summ.values() if not is_ancient(m)])
+ # representative genome length: consensus length, else the reference length column (works when make_consensus is off)
+ genome_len = max([int(v) for v in (to_float(m.get("length")) or to_float(m.get("genome_len"))
+ for m in summ.values()) if v], default=0)
+ snp_density_ok = genome_len > 100000
+
+ dmg_by_sample = collect_damage(args.mapdamage_dir, summ)
metric_keys = [k for k, _, _, _ in METRICS]
# fine-grained snpEff effect-class columns: carried into s.m for the Functional-annotation panel, not registered metrics
@@ -1592,41 +250,7 @@ def _index_dir(d):
k, v = kv.split("=", 1)
provenance[k.strip()] = v.strip()
- _variants = parse_vcfs(args.vcfs) # parsed once, feeds both the dynamics panel and the SNP matrix
- if args.vcfs_h37rv: # attach the reference-of-interest (H37Rv) amino-acid change AND coordinate per variant
- _h37 = parse_vcfs(args.vcfs_h37rv)
- for _s, _pm in _variants.items():
- _hs = _h37.get(_s, {})
- # Pair each used-ref variant with its canonical-reference record. Prefer the ORIGINAL used-ref
- # coordinate a liftover stamped (OriginalContig/Start or OPOS) so the two references need NOT
- # share coordinates; else fall back to the bare position (references that DO share H37Rv coords).
- _by_opos, _by_pos = {}, {}
- for _ck, _cv in _hs.items():
- if _cv.get('opos'):
- _by_opos[_cv['opos']] = (_ck, _cv)
- _by_pos.setdefault(_ck.rpartition(':')[2], (_ck, _cv))
- for _key, _v in _pm.items():
- _hit = _by_opos.get(_key) or _by_pos.get(_key.rpartition(':')[2])
- if _hit:
- _ck, _cv = _hit
- if _cv.get('aa'):
- _v['aa_h37rv'] = _cv['aa']
- _v['pos_h37rv'] = _ck # reference-of-interest coordinate (contig:pos), possibly != the mapping one
- if args.pos_liftover: # alignment-free canonical coordinate per variant (pathotypr k-mer liftover map)
- _lift = {}
- try:
- with open(args.pos_liftover, encoding="utf-8", errors="replace") as _fh:
- for _line in _fh:
- _c = _line.rstrip("\n").split("\t")
- if len(_c) >= 2 and _c[0].strip().isdigit() and _c[1].strip().isdigit():
- _lift[_c[0].strip()] = _c[1].strip()
- except OSError as _e:
- sys.stderr.write("[qc_report] WARN pos-liftover (%s): %s\n" % (args.pos_liftover, _e))
- for _s, _pm in _variants.items():
- for _key, _v in _pm.items():
- _p = _key.rpartition(":")[2]
- if not _v.get("pos_h37rv") and _p in _lift:
- _v["pos_h37rv"] = "%s:%s" % (args.aa2_label, _lift[_p])
+ _variants = load_variants(args)
_sample_meta = parse_sample_meta(args.metadata) # shared by the dynamics filter and the SNP matrix header
_dynamics = build_dynamics(parse_metadata(args.metadata), _variants, _sample_meta) # feeds dynamics + epistasis
# front-load 'dose' so it survives the correlation matrix's top-N view
@@ -1654,11 +278,12 @@ def _index_dir(d):
"metrics": [{"key": k, "label": l, "kind": kind, "dir": d} for k, l, kind, d in METRICS],
"extra": extra_metrics,
"samples": jsamples}
+ return payload, jsamples, counts
- os.makedirs(os.path.dirname(os.path.abspath(args.out_html)) or ".", exist_ok=True)
- with open(args.out_html, "w", encoding="utf-8") as fh:
- fh.write(build_html(args.title, payload))
- with open(args.out_flags, "w", encoding="utf-8") as fh:
+
+def write_flags(path, jsamples):
+ """The machine-readable verdict table: one row per sample, FAIL first, then WARN, then PASS."""
+ with open(path, "w", encoding="utf-8") as fh:
fh.write("sample\tverdict\tmean_depth\tbreadth_pct\tmapped_pct\tmissing_pct\tiupac_pct\tsnps\tti_tv\tflags\n")
for s in sorted(jsamples, key=lambda x: (x["v"] != "FAIL", x["v"] != "WARN", x["s"])):
m = s["m"]
@@ -1669,6 +294,18 @@ def g(k):
g("missing_pct"), g("iupac_pct"), g("snps"), g("ti_tv"),
",".join(s["f"]) or "."]) + "\n")
+
+def main():
+ args = build_parser().parse_args()
+ thr = {k: getattr(args, k) for k in DEF}
+ anc_thr = {k: getattr(args, "anc_" + k) for k in ANC_DEF}
+ payload, jsamples, counts = build_payload(args, thr, anc_thr)
+
+ os.makedirs(os.path.dirname(os.path.abspath(args.out_html)) or ".", exist_ok=True)
+ with open(args.out_html, "w", encoding="utf-8") as fh:
+ fh.write(build_html(args.title, payload))
+ write_flags(args.out_flags, jsamples)
+
sys.stderr.write(f"[qc_report] {len(jsamples)} samples -> {counts['PASS']} PASS, {counts['WARN']} WARN, "
f"{counts['FAIL']} FAIL. Wrote {args.out_html} + {args.out_flags}\n")
for s in sorted([x for x in jsamples if x["v"] == "FAIL"], key=lambda x: x["s"]):
diff --git a/bin/qcreport/__init__.py b/bin/qcreport/__init__.py
new file mode 100644
index 0000000..de2f6ea
--- /dev/null
+++ b/bin/qcreport/__init__.py
@@ -0,0 +1,13 @@
+"""Internals of the BAMpiro QC report.
+
+`bin/qc_report.py` is the CLI; everything it does lives here:
+
+ parsers.py read the pipeline's files (summary TSV, consensus FASTA, GFF, BED,
+ VCF, mapDamage, Kraken, samplesheet) into plain Python values
+ metrics.py the metric registry and the QC verdict engine (flag_sample)
+ panels.py the optional analysis panels built from parsed VCFs
+ render.py the front-end assets and the self-contained HTML assembly
+
+Nothing is re-exported here on purpose: each name is imported from the module that
+owns it, so the ownership stays visible at every call site.
+"""
diff --git a/bin/qcreport/metrics.py b/bin/qcreport/metrics.py
new file mode 100644
index 0000000..db7d718
--- /dev/null
+++ b/bin/qcreport/metrics.py
@@ -0,0 +1,304 @@
+"""The metric registry, the flag thresholds and the QC verdict engine.
+
+DEF / ANC_DEF are the default cut-offs (the modern view and the looser ancient-DNA
+view); METRICS is the ordered table of every metric the report knows about, DEFS
+its in-report documentation, and DIST the subset drawn as distributions. Around
+them sit the derived statistics (robust, het_frac, the lineage helpers) and
+flag_sample, which turns one summary row into a PASS / WARN / FAIL verdict plus
+the list of reasons.
+
+Everything here is pure in-memory logic over an already-parsed summary row.
+"""
+from __future__ import annotations
+
+import re
+
+from .parsers import parse_gene_locus, to_float
+
+DEF = dict(depth_min=10.0, breadth_min=90.0, missing_max=10.0, dup_max=40.0,
+ iupac_max=2.0, mapping_min=80.0, titv_min=0.0, snp_z=3.0,
+ het_max_frac=3.0, mixed_min_frac=2.0) # het_max_frac + mixed_min_frac are PERCENTS
+
+# Ancient (aDNA) sample threshold view: looser cov/breadth/missing/mapping/dup/iupac (a 5x mummy must not FAIL the
+# modern gate), plus damage_min_ct = the terminal 5' C>T deamination floor for the DAMAGE_LOW authentication screen.
+ANC_DEF = dict(depth_min=3.0, breadth_min=30.0, missing_max=70.0, dup_max=60.0,
+ iupac_max=5.0, mapping_min=20.0, damage_min_ct=0.05)
+FAIL_FLAGS = {"LOW_DEPTH", "LOW_BREADTH", "HIGH_MISSING", "NO_DATA"}
+
+# metric definitions + typical acceptable ranges (surfaced as in-report help)
+DEFS = {
+ "dose": ["Per-sample dose from the samplesheet (a numeric annotation, not a QC metric). Feeds the scatter/correlation axes and the dose x treatment test.", ""],
+ "raw_reads": ["Read pairs/reads before trimming.", "context-dependent"],
+ "trimmed_reads": ["Reads after fastp quality/adapter trimming.", ""],
+ "read_length": ["Mean read length after trimming.", ""],
+ "duplication_pct": ["% reads marked as duplicates (PCR/optical).", "typically < 30-40%"],
+ "mapped_pct": ["% reads mapping to the reference.", "> 90% for a clean library"],
+ "properly_paired_pct": ["% read pairs mapping in the expected orientation/insert.", "> 90%"],
+ "mean_depth": ["Mean sequencing depth over the genome (sensitive to a few high-copy repeats; see Median depth).", ">= 10-20x for confident calls"],
+ "median_depth": ["Median sequencing depth over the genome (robust to high-copy repeats that inflate the mean).", ">= 10-20x"],
+ "breadth_pct": ["% of the genome covered at >= 1x. Note: 1x is not enough to CALL a base; see Callable % for usable genome.", ">= 90% at 1x to enter a phylogeny"],
+ "breadth5x_pct": ["% of the genome covered at >= 5x.", "high for a confident consensus"],
+ "breadth10x_pct": ["% of the genome covered at >= 10x.", ">= 90% for confident SNP calls"],
+ "coverage_cv": ["Coverage evenness = coefficient of variation of depth (sd / mean). Lower is more even; spikes flag repeats/contamination.", "< ~0.5-1 for an even library"],
+ "avg_base_quality": ["Mean per-base quality of mapped reads.", ">= 30"],
+ "unmapped_pct": ["% of reads that did not map to the reference (an off-target / contamination proxy).", "low for a clean, on-target library"],
+ "singleton_pct": ["% of reads mapped without their mate (approx: reads mapped minus reads mapped-and-paired).", "low"],
+ "mapq0_pct": ["% of reads with MAPQ 0 (ambiguous multi-mapping, e.g. repeats / IS elements).", "low"],
+ "error_rate_pct": ["samtools mismatch (error) rate, as a percent; elevated with damage or contamination.", "< ~1%"],
+ "insert_size": ["Mean fragment/insert size (bp) of properly paired reads. For aDNA a short distribution is expected.", "library-dependent"],
+ "insert_size_sd": ["Standard deviation of insert size; a very wide spread signals chimeras/contamination.", ""],
+ "q20_pct": ["% of trimmed bases with quality >= Q20 (fastp).", "> 90%"],
+ "q30_pct": ["% of trimmed bases with quality >= Q30 (fastp).", "> 80-90%"],
+ "gc_pct": ["GC content of the trimmed reads (fastp); a large deviation from the reference GC flags contamination.", "match the reference GC"],
+ "est_genome_cov": ["Estimated fold coverage = trimmed bases / genome length (fastp-based, pre-mapping).", "tracks mean_depth"],
+ "total_variants": ["Total variants (SNPs + indels) called vs the reference.", "context-dependent"],
+ "snps": ["SNPs called vs the reference. Dominated by true phylogenetic distance, NOT quality; the SNP outlier flag is a cohort distributional signal, not a defect.", "expected to be higher for divergent lineages"],
+ "snp_density": ["SNP calls per callable megabase = snps / (callable_pct/100 * genome_len/1e6). Normalises SNP burden for how much of the genome was callable; compare within a lineage. Suppressed for a toy genome (< 100 kb).", "compare within a lineage"],
+ "het_variants": ["Heterozygous variant calls. In a clonal/haploid organism these are expected ~0; a nonzero rate is consistent with mixture, contamination, or (for aDNA) post-mortem deamination.", "~0"],
+ "homo_indels": ["Homozygous indels called.", ""],
+ "ti_tv": ["Transition/transversion ratio (a spectrum sanity check, not a rate). Typical genome-wide MTBC ~1.7-2.0; M. bovis (A4) runs higher (~2.6) from its transition-biased spectrum, and aDNA deamination inflates it. LOW Ti/Tv (< ~1.5) is the QC concern (mapping/base-call noise is transversion-heavy); high is biology or damage, not a defect.", "~1.5-2.1 for human lineages"],
+ "missing_pct": ["% of the consensus that is '-' or N (no confident base).", "< 10%"],
+ "iupac_pct": ["% of the consensus that is an IUPAC ambiguity (mixed call).", "< 2%"],
+ "callable_pct": ["% of the consensus that is a confident A/C/G/T (the honest, depth-aware usable-genome measure).", "> 90%"],
+ "longest_n_run": ["Longest single run of consecutive N/gap in the consensus (one big deletion vs many scattered holes).", "context-dependent"],
+ "n_gaps": ["Number of separate N/gap runs in the consensus.", "context-dependent"],
+ "het_frac": ["Fraction of VARIANT genotypes that are heterozygous (het / (het+hom)). An illustrative within-sample mixture signal, not the textbook heterozygosity rate; the denominator is variant-count, so read it with care. Suppressed below a minimum variant count, and NOT applied to aDNA (deamination inflates it).", "~0 for a clonal isolate"],
+ "n_lineages": ["Number of lineages (collapsed to main lineage) above the mixture cut-off AND above a minimum marker count; > 1 is consistent with a mixed infection or cross-contamination.", "1"],
+ "damage_ct": ["Terminal 5' C>T deamination frequency (mapDamage2). Elevated in authentic aDNA; consistent with post-mortem damage but not proof of authenticity or endogeneity on its own.", "aDNA typically >= 0.05-0.40"],
+ "damage_ga": ["Terminal 3' G>A deamination frequency (mapDamage2).", "aDNA typically >= 0.05-0.40"],
+ "fraglen": ["Mean fragment (read) length from mapDamage2. Short (~30-70 bp) is expected for aDNA.", "aDNA ~30-70 bp"],
+ "ann_high": ["snpEff HIGH-impact variants (stop_gained, frameshift, start/stop_lost, splice). A cohort-relative excess is a contamination / misassembly / wrong-reference-annotation signal, not a per-sample defect on its own.", "cohort-relative robust-z; high flags a look-here sample"],
+ "missense_silent": ["Missense / synonymous variant-count ratio: a within-sample pN/pS PROXY (a spectrum sanity check, NOT dN/dS and NOT a selection test; no site-degeneracy correction, entangled with divergence and with how snpEff bins effects). Read within a lineage. For a real pN/pS use the eskaks panel.", "compare within a lineage; robust-z high tail only"],
+ "lof_pct": ["Fraction of protein-coding-effect variants that are loss-of-function (stop_gained + frameshift + start/stop_lost + splice). Elevated LoF can be real pseudogenisation OR a frameshift storm from an indel-calling / assembly / annotation artifact.", "cohort-relative robust-z"],
+ "coding_pct": ["Percent of annotated variants falling in a transcript (HIGH+MODERATE+LOW; MODIFIER is the intergenic / intron / UTR bucket). Tracks how gene-dense the reference annotation is; not a quality gate.", "reference-dependent, descriptive"],
+ "annotated_pct": ["Percent of this sample's real variants that received an ANN annotation from snpEff. A low value flags a GFF-to-database mismatch: the annotation may be silently garbage.", "cohort-relative low + absolute floor"],
+ "snpeff_warn": ["Variants carrying a snpEff ERROR/WARNING/INFO subfield (e.g. WARNING_TRANSCRIPT_INCOMPLETE, ERROR_CHROMOSOME_NOT_FOUND). A value near the total variant count means the reference database was built wrong for this contig; treat every annotation with suspicion.", "low; near-total = broken DB"],
+}
+# Contamination/mixture cut-offs (het_max_frac, mixed_min_frac) and the aDNA damage floor are ILLUSTRATIVE sensitivity
+# defaults, not validated gates; they are WARN-only and live-adjustable. Tune to your depth and organism.
+# (key, label, kind, direction). kind: int|float|pct ; direction: hi_good|hi_bad|neu
+METRICS = [
+ ("raw_reads", "Raw reads", "int", "neu"),
+ ("trimmed_reads", "Trimmed", "int", "neu"),
+ ("read_length", "Read len", "float", "neu"),
+ ("q20_pct", "Q20 %", "pct", "hi_good"),
+ ("q30_pct", "Q30 %", "pct", "hi_good"),
+ ("gc_pct", "GC %", "pct", "neu"),
+ ("duplication_pct", "Dup %", "pct", "hi_bad"),
+ ("mapped_pct", "Mapped %", "pct", "hi_good"),
+ ("properly_paired_pct", "Proper %", "pct", "hi_good"),
+ ("unmapped_pct", "Unmapped %", "pct", "hi_bad"),
+ ("singleton_pct", "Singleton %", "pct", "hi_bad"),
+ ("mapq0_pct", "MAPQ0 %", "pct", "hi_bad"),
+ ("error_rate_pct", "Error %", "pct", "hi_bad"),
+ ("insert_size", "Insert bp", "float", "neu"),
+ ("insert_size_sd", "Insert sd", "float", "neu"),
+ ("mean_depth", "Depth", "float", "hi_good"),
+ ("median_depth", "Median depth", "float", "hi_good"),
+ ("breadth_pct", "Breadth %", "pct", "hi_good"),
+ ("breadth5x_pct", "Breadth 5x %", "pct", "hi_good"),
+ ("breadth10x_pct", "Breadth 10x %", "pct", "hi_good"),
+ ("coverage_cv", "Cov CV", "float", "hi_bad"),
+ ("avg_base_quality", "Base Q", "float", "hi_good"),
+ ("est_genome_cov", "Est cov", "float", "hi_good"),
+ ("total_variants", "Variants", "int", "neu"),
+ ("snps", "SNPs", "int", "neu"),
+ ("het_variants", "Het", "int", "hi_bad"),
+ ("homo_indels", "Indels", "int", "neu"),
+ ("ti_tv", "Ti/Tv", "float", "neu"),
+ ("missing_pct", "Missing %", "pct", "hi_bad"),
+ ("iupac_pct", "IUPAC %", "pct", "hi_bad"),
+ ("callable_pct", "Callable %", "pct", "hi_good"),
+ ("longest_n_run", "Longest N-run", "int", "hi_bad"),
+ ("n_gaps", "N-gaps", "int", "neu"),
+ ("ann_high", "HIGH impact", "int", "hi_bad"),
+ ("ann_moderate", "MODERATE", "int", "neu"),
+ ("ann_low", "LOW impact", "int", "neu"),
+ ("ann_modifier", "MODIFIER", "int", "neu"),
+ ("missense_silent", "Missense/silent", "float", "neu"),
+ ("lof_pct", "LoF %", "pct", "hi_bad"),
+ ("coding_pct", "Coding %", "pct", "neu"),
+ ("annotated_pct", "Annotated %", "pct", "hi_good"),
+ ("snpeff_warn", "snpEff warn", "int", "hi_bad"),
+]
+DIST = ["mean_depth", "median_depth", "breadth_pct", "breadth10x_pct", "coverage_cv", "mapped_pct",
+ "unmapped_pct", "properly_paired_pct", "duplication_pct", "mapq0_pct", "error_rate_pct",
+ "missing_pct", "callable_pct", "iupac_pct", "snps", "ti_tv",
+ "ann_high", "missense_silent", "annotated_pct"]
+
+# columns that are identifiers / metadata / already-surfaced -> NOT auto-detected as "extra" metrics
+KNOWN_KEYS = {k for k, *_ in METRICS} | {
+ "sample_id", "lineage", "lineage_counts", "drug_resistance", "dr_counts", "date", "ancient",
+ "consensus_id", "length", "genome_len", "mapped_reads", "homo_variants",
+ "snp_profile", "het_profile", "indel_profile", # per-position variant tracks (huge strings, not metrics)
+ "eff_missense", "eff_synonymous", "eff_stop_gained", "eff_stop_lost", "eff_start_lost",
+ "eff_frameshift", "eff_inframe_indel", "eff_splice", "eff_intergenic", "eff_regulatory",
+ "ann_present", "ann_annotated_frac", "ann_total", "ann_annotated", "coding_count", "lof_count", "ann_db_error",
+}
+_PCT_HINTS = ("_pct", "_rate", "percent", "fraction", "frac")
+_INT_HINTS = ("reads", "count", "_n", "num_", "variants", "snps", "indels", "bases")
+
+
+# Curated H37Rv locus tags (Mycobrowser) for the common MTB resistance genes, used as a fallback when
+# the run's reference GFF carries no H37Rv-style locus_tag for them (e.g. a non-H37Rv reference). These
+# are the WHO-catalogue first/second-line resistance genes.
+GENE_RV = {
+ "rpoB": "Rv0667", "rpoC": "Rv0668", "katG": "Rv1908c", "inhA": "Rv1484", "fabG1": "Rv1483",
+ "ahpC": "Rv2428", "gyrA": "Rv0006", "gyrB": "Rv0005", "embA": "Rv3794", "embB": "Rv3795",
+ "embC": "Rv3793", "pncA": "Rv2043c", "rpsL": "Rv0682", "eis": "Rv2416c", "ethA": "Rv3854c",
+ "ethR": "Rv3855", "gid": "Rv3919c", "gidB": "Rv3919c", "tlyA": "Rv1694", "Rv0678": "Rv0678",
+ "atpE": "Rv1305", "pepQ": "Rv2535c", "folC": "Rv2447c", "thyA": "Rv2764c", "alr": "Rv3423c",
+ "ddn": "Rv3547", "fgd1": "Rv0407", "fbiA": "Rv3261", "fbiB": "Rv3262", "fbiC": "Rv1173",
+ "rrs": "MTB000019", "rrl": "MTB000020", "mmpL5": "Rv0676c", "mmpS5": "Rv0677c", "whiB7": "Rv3197A",
+ "clpC1": "Rv3596c", "panD": "Rv3601c", "ddlA": "Rv2981c",
+}
+_RV_LOCUS_RE = re.compile(r'^(Rv\d|MTB\d)', re.I) # H37Rv (Mycobrowser) locus-tag schemes
+
+
+def build_gene_map(gff_path):
+ """{gene: Mycobrowser (H37Rv) locus tag}. Curated resistance-gene map, overlaid with any H37Rv-style
+ locus tags from the run's GFF (so an H37Rv reference contributes every gene; a non-H37Rv reference's
+ strain-specific tags are ignored in favour of the curated H37Rv equivalents)."""
+ m = dict(GENE_RV)
+ for gene, locus in parse_gene_locus(gff_path).items():
+ if _RV_LOCUS_RE.match(locus):
+ m[gene] = locus
+ return m
+
+
+def robust(vals):
+ v = sorted(x for x in vals if x is not None)
+ if not v:
+ return None, None
+ n = len(v)
+ med = v[n // 2] if n % 2 else (v[n // 2 - 1] + v[n // 2]) / 2
+ devs = sorted(abs(x - med) for x in v)
+ mad = devs[n // 2] if n % 2 else (devs[n // 2 - 1] + devs[n // 2]) / 2
+ sigma = 1.4826 * mad
+ if sigma <= 0:
+ mean = sum(v) / n
+ sigma = (sum((x - mean) ** 2 for x in v) / n) ** 0.5 or 1.0
+ return med, sigma
+
+
+MIN_HET_VARIANTS = 50 # below this variant count the het fraction is too noisy to interpret -> None
+
+
+def het_frac(m):
+ """Fraction of VARIANT genotypes that are heterozygous (het/(het+hom)). Illustrative mixture signal; the
+ denominator is variant-count (not callable sites), so it is entangled with divergence -> read with care.
+ Returns None below a minimum variant count (too noisy to interpret)."""
+ het = to_float(m.get("het_variants"))
+ hom = to_float(m.get("homo_variants"))
+ if het is None or hom is None or (het + hom) < MIN_HET_VARIANTS:
+ return None
+ return het / (het + hom)
+
+
+def lineage_counts_parsed(lc):
+ """Parse 'L4.1:3200;L4.3:120;L2:40' into {main_lineage: total_count}, COLLAPSING sub-lineages (L4.1 -> L4)
+ so two sub-lineages of the same main lineage do not spuriously read as a mixed infection."""
+ if not lc or str(lc) in ("NA", ".", ""):
+ return {}
+ out = {}
+ for part in str(lc).replace(",", ";").split(";"):
+ if ":" in part:
+ k, v = part.rsplit(":", 1)
+ n = to_float(v)
+ if n is not None:
+ main = k.strip().split(".")[0] # collapse sub-lineage to main lineage
+ out[main] = out.get(main, 0.0) + n
+ return out
+
+
+def lineage_fracs(lc):
+ """{main_lineage: fraction} from the collapsed counts."""
+ counts = lineage_counts_parsed(lc)
+ tot = sum(counts.values())
+ return {k: v / tot for k, v in counts.items()} if tot > 0 else {}
+
+
+def is_ancient(m):
+ """True if the summary row marks this sample ancient (aDNA). Missing column -> False (modern)."""
+ return str(m.get("ancient", "")).strip().lower() in ("yes", "y", "true", "1", "ancient")
+
+
+def infer_metric(key, values):
+ """Infer (label, kind, dir) for an un-registered numeric summary column. Neutral dir = no auto-flag / no claim."""
+ kl = key.lower()
+ if any(h in kl for h in _PCT_HINTS):
+ kind = "pct"
+ elif all(v is None or float(v).is_integer() for v in values) and any(h in kl for h in _INT_HINTS):
+ kind = "int"
+ else:
+ kind = "float"
+ label = key.replace("_pct", " %").replace("_", " ").strip()
+ label = label[:1].upper() + label[1:]
+ return label, kind, "neu"
+
+
+def discover_extra_metrics(summ):
+ """Any summary column not a KNOWN_KEY and numeric for >=1 sample -> an auto 'extra' metric (future-proofing)."""
+ if not summ:
+ return []
+ seen = set()
+ for mm in summ.values():
+ for k in mm.keys():
+ if k not in KNOWN_KEYS and k != "sample_id":
+ seen.add(k)
+ extras = []
+ for k in sorted(seen):
+ vals = [to_float(mm.get(k)) for mm in summ.values()]
+ if any(v is not None for v in vals):
+ label, kind, direction = infer_metric(k, vals)
+ extras.append({"key": k, "label": label, "kind": kind, "dir": direction})
+ return extras
+
+
+def flag_sample(m, thr, snp_med, snp_sig, ancient=False, anc_thr=None, dmg=None):
+ flags = []
+ at = anc_thr if (ancient and anc_thr) else thr # ancient samples judged against the aDNA threshold view
+ g = lambda k: to_float(m.get(k))
+ # produced-nothing guard: a sample with no consensus AND no mapping metrics is a FAIL, not a quiet PASS-by-absence
+ if all(g(k) is None for k in ("mean_depth", "breadth_pct", "missing_pct", "mapped_pct")):
+ return "FAIL", ["NO_DATA"]
+ if (v := g("mean_depth")) is not None and v < at["depth_min"]:
+ flags.append("LOW_DEPTH")
+ if (v := g("breadth_pct")) is not None and v < at["breadth_min"]:
+ flags.append("LOW_BREADTH")
+ if (v := g("missing_pct")) is not None and v > at["missing_max"]:
+ flags.append("HIGH_MISSING")
+ if (v := g("mapped_pct")) is not None and v < at["mapping_min"]:
+ flags.append("MAPPING_LOW")
+ if (v := g("duplication_pct")) is not None and v > at["dup_max"]:
+ flags.append("HIGH_DUP")
+ if (v := g("iupac_pct")) is not None and v > at["iupac_max"]:
+ flags.append("HIGH_IUPAC")
+ if thr["titv_min"] > 0 and (v := g("ti_tv")) is not None and v < thr["titv_min"]:
+ flags.append("TITV_LOW")
+ # SNP count is dominated by phylogenetic divergence, not quality -> a cohort distributional outlier (WARN only).
+ # snp_med/sig are computed over MODERN samples; ancient SNP counts are low by design, so skip the z-flag for aDNA.
+ if not ancient:
+ snp = g("snps")
+ if snp is not None and snp_med is not None and snp_sig:
+ if snp < snp_med - thr["snp_z"] * snp_sig:
+ flags.append("SNP_LOW")
+ elif snp > snp_med + thr["snp_z"] * snp_sig:
+ flags.append("SNP_HIGH")
+ # het fraction: NOT applied to aDNA (post-mortem deamination inflates apparent het -> would false-flag mummies)
+ if not ancient:
+ hf = het_frac(m)
+ if hf is not None and hf * 100.0 > thr["het_max_frac"]:
+ flags.append("HET_HIGH")
+ # MIXED: >1 MAIN lineage above the fraction cut-off AND above a minimum marker count (guards single-read noise)
+ counts = lineage_counts_parsed(m.get("lineage_counts"))
+ tot = sum(counts.values())
+ if tot > 0 and sum(1 for c in counts.values()
+ if (c / tot) * 100.0 >= thr["mixed_min_frac"] and c >= 3) > 1:
+ flags.append("MIXED")
+ # aDNA authentication: an ANCIENT sample whose terminal 5' C>T is below the floor -> possible modern contaminant
+ if ancient and anc_thr is not None and dmg and dmg.get("ct1") is not None \
+ and dmg["ct1"] < anc_thr["damage_min_ct"]:
+ flags.append("DAMAGE_LOW")
+ verdict = "FAIL" if any(f in FAIL_FLAGS for f in flags) else ("WARN" if flags else "PASS")
+ return verdict, flags
diff --git a/bin/qcreport/panels.py b/bin/qcreport/panels.py
new file mode 100644
index 0000000..4359d8b
--- /dev/null
+++ b/bin/qcreport/panels.py
@@ -0,0 +1,285 @@
+"""Optional analysis panels built from the parsed per-sample VCFs.
+
+build_dynamics turns metadata + variants into per-connected-group allele-frequency
+trajectories; build_epistasis scores pairs of those trajectories for co-variation;
+build_snp_matrix flattens the same variants into the sparse sample x site matrix.
+Each returns None when there is nothing to show, and the report hides the panel.
+
+_pearson / _perm_p / _amp are the statistics build_epistasis is built on.
+"""
+from __future__ import annotations
+
+import random
+import re
+import zlib
+
+_DYN_NONSYN = re.compile(r'missense|stop_gained|stop_lost|start_lost|frameshift|inframe|splice|initiator', re.I)
+
+
+def build_dynamics(metadata, variants, sample_meta=None, emerge=0.25, fix=0.90, loss=0.10, min_points=2, min_move=0.15):
+ """Per-connected-group AF trajectories over time, only for SNPs that move. None if nothing to show.
+ sample_meta (parse_sample_meta output) attaches each group's metadata signature so the report can
+ filter the series by any group-invariant field (e.g. patient / lineage)."""
+ if not metadata:
+ return None
+ groups = {}
+ for s, md in metadata.items():
+ g = md.get('group')
+ if g and s in variants:
+ groups.setdefault(g, []).append(s)
+ out_groups = []
+ for g, samples in sorted(groups.items()):
+ samples = sorted(samples, key=lambda s: (metadata[s]['tnum'] is None,
+ metadata[s]['tnum'] if metadata[s]['tnum'] is not None else 0,
+ str(metadata[s]['time'])))
+ times = [metadata[s]['time'] for s in samples]
+ if len(set(times)) < min_points:
+ continue
+ allpos = set()
+ for s in samples:
+ allpos.update(variants[s].keys())
+ series = []
+ for pos in sorted(allpos): # deterministic order (a set iterates in a PYTHONHASHSEED-dependent order)
+ traj, dps, meta = [], [], None
+ for s in samples:
+ v = variants[s].get(pos)
+ traj.append(v['af'] if v else 0.0)
+ dps.append(v.get('dp') if v else None) # per-timepoint depth (None where not called)
+ if v and meta is None:
+ meta = v
+ if max(traj) - min(traj) < min_move:
+ continue
+ flags = []
+ if traj[0] <= loss and max(traj) >= emerge:
+ flags.append('emergence')
+ if traj[-1] >= fix and traj[0] < fix:
+ flags.append('fixation')
+ if traj[0] >= emerge and traj[-1] <= loss:
+ flags.append('loss')
+ if meta and _DYN_NONSYN.search(meta.get('eff', '')):
+ flags.append('nonsyn')
+ if meta and meta.get('imp', '') == 'HIGH':
+ flags.append('high_impact')
+ series.append({'pos': pos, 'gene': (meta or {}).get('gene', ''), 'eff': (meta or {}).get('eff', ''),
+ 'imp': (meta or {}).get('imp', ''), 'alt': (meta or {}).get('alt', ''),
+ 'aa': (meta or {}).get('aa', ''), 'aa_h37rv': (meta or {}).get('aa_h37rv', ''),
+ 'pos_h37rv': (meta or {}).get('pos_h37rv', ''),
+ 'traj': [round(x, 4) for x in traj], 'dp': dps, 'flags': flags})
+ if not series:
+ continue
+ series.sort(key=lambda x: (-(max(x['traj']) - min(x['traj'])), -len(x['flags'])))
+ gmeta = {}
+ if sample_meta and sample_meta.get('fields'):
+ rows = sample_meta.get('rows', {})
+ for f in sample_meta['fields']:
+ vals = sorted({v for v in ((rows.get(s) or {}).get(f) for s in samples) if v})
+ if vals:
+ gmeta[f] = vals
+ out_groups.append({'group': g, 'samples': samples, 'times': times,
+ 'tnums': [metadata[s]['tnum'] for s in samples], 'meta': gmeta,
+ 'series': series, 'n_flagged': sum(1 for x in series if x['flags'])})
+ if not out_groups:
+ return None
+ return {'groups': out_groups, 'thresholds': {'emerge': emerge, 'fix': fix, 'loss': loss}}
+
+
+def _pearson(x, y):
+ """Pearson correlation of two equal-length trajectories, or None if undefined (n<3 or a flat one)."""
+ n = len(x)
+ if n < 3 or len(y) != n:
+ return None
+ mx, my = sum(x) / n, sum(y) / n
+ sxx = sum((a - mx) ** 2 for a in x)
+ syy = sum((b - my) ** 2 for b in y)
+ if sxx <= 1e-9 or syy <= 1e-9:
+ return None
+ sxy = sum((a - mx) * (b - my) for a, b in zip(x, y))
+ return sxy / ((sxx * syy) ** 0.5)
+
+
+def _perm_p(series_traj, obs_mean, B=2000, seed=0):
+ """Permutation p-value for a pair's mean correlation: shuffle one trajectory's values within EACH
+ series independently (breaking the time linkage) and recompute the mean |r|; p = fraction of the B
+ nulls reaching the observed |r| (+1 smoothing). Deterministic (fixed per-pair seed) -> reproducible.
+ With few timepoints the resolution is inherently coarse (a 4-point series has only 4!=24 orders), so
+ a lone short series lands near p~0.04 while a pattern recurring across series drives p far lower."""
+ obs = abs(obs_mean)
+ rng = random.Random(seed)
+ ge = 0
+ for _ in range(B):
+ tot, cnt = 0.0, 0
+ for ta, tb in series_traj:
+ perm = list(ta)
+ rng.shuffle(perm)
+ r = _pearson(perm, tb)
+ if r is not None:
+ tot += r
+ cnt += 1
+ if cnt and abs(tot / cnt) >= obs:
+ ge += 1
+ return (ge + 1) / (B + 1)
+
+
+def _amp(traj):
+ """Amplitude (max-min) of a trajectory's non-null allele frequencies; 0 if flat/empty. Used to pick the
+ most-variable variants when a series has too many to test all pairs of."""
+ vals = [v for v in (traj or []) if v is not None]
+ return (max(vals) - min(vals)) if len(vals) >= 2 else 0.0
+
+
+def build_epistasis(dynamics, min_r=0.8, min_points=3, top=300, perm=2000, max_vars=250, max_pairs_perm=800):
+ """Candidate epistatic / linked variant pairs: two SNPs whose allele-frequency trajectories co-vary
+ WITHIN a connected series - concordant (rise/fall together) or discordant (one rises as the other
+ falls). Score = mean Pearson r of the two trajectories across every series where both move and the
+ series has >= min_points timepoints. Significance = a permutation p-value (per pair), then a
+ Benjamini-Hochberg FDR q-value across all reported pairs. Recurrence across independent series is
+ tracked so a pattern seen in several series (low q) is separable from a lucky single short series.
+ None if nothing qualifies. Built from the dynamics payload (moving variants per series).
+
+ Scaling: the pairwise step is O(variants^2) per series and the permutation test is the dominant cost,
+ so two caps keep it bounded for large cohorts (many variants and/or many samples -> long trajectories):
+ each series contributes at most max_vars variants (the highest-amplitude / most-variable ones), and only
+ the max_pairs_perm strongest-|r| candidate pairs get the (expensive) permutation p-value. Both caps are
+ reported in the payload so the UI can say what was covered; the full trajectories stay in the dynamics/TSV."""
+ if not dynamics or not dynamics.get('groups'):
+ return None
+ pairs, n_series = {}, 0
+ vars_capped, n_vars_max = False, 0
+ for g in dynamics['groups']:
+ times = g.get('times') or []
+ if len(set(times)) < min_points:
+ continue
+ n_series += 1
+ series = g.get('series') or []
+ n_vars_max = max(n_vars_max, len(series))
+ if len(series) > max_vars: # too many variants to test every pair: keep the most-variable ones
+ vars_capped = True
+ series = sorted(series, key=lambda s: -_amp(s.get('traj')))[:max_vars]
+ for i in range(len(series)):
+ for j in range(i + 1, len(series)):
+ a, b = series[i], series[j]
+ r = _pearson(a['traj'], b['traj'])
+ if r is None:
+ continue
+ lo, hi = (a, b) if str(a['pos']) <= str(b['pos']) else (b, a) # order-independent key
+ key = (lo['pos'], hi['pos'])
+ rec = pairs.get(key)
+ if rec is None:
+ rec = pairs[key] = {'A': lo, 'B': hi, 'rs': [], 'straj': [], 'bestn': -1,
+ 'times': times, 'trajA': lo['traj'], 'trajB': hi['traj'], 'group': g['group']}
+ rec['rs'].append(r)
+ rec['straj'].append((lo['traj'], hi['traj'])) # every series' pair, for the permutation test
+ if len(set(times)) > rec['bestn']: # keep the richest series for the mini-chart
+ rec['bestn'] = len(set(times))
+ rec['times'], rec['trajA'], rec['trajB'], rec['group'] = times, lo['traj'], hi['traj'], g['group']
+ # Collect the pairs that clear the |r| threshold, then permutation-test only the strongest ones: the
+ # permutation p-value is by far the dominant cost, so with many candidate pairs (common when trajectories
+ # are short and spurious high correlations abound) we cap it to the top max_pairs_perm by |mean r|.
+ cands = []
+ for key, rec in pairs.items():
+ rs = rec['rs']
+ mean_r = sum(rs) / len(rs)
+ if abs(mean_r) < min_r:
+ continue
+ cands.append((abs(mean_r), key, rec, mean_r))
+ n_candidates = len(cands)
+ pairs_capped = n_candidates > max_pairs_perm
+ if pairs_capped:
+ cands.sort(key=lambda c: -c[0])
+ cands = cands[:max_pairs_perm]
+ out = []
+ for _absr, key, rec, mean_r in cands:
+ rs = rec['rs']
+ A, B = rec['A'], rec['B']
+ n_pos = sum(1 for r in rs if r > 0)
+ # seed from the STABLE pair key (positions), not the iteration index, so the permutation p-value is
+ # reproducible regardless of dict/set iteration order. crc32 is deterministic (unlike hash()).
+ seed = zlib.crc32(('%s|%s' % (key[0], key[1])).encode('utf-8'))
+ p = _perm_p(rec['straj'], mean_r, B=perm, seed=seed)
+ out.append({'geneA': A.get('gene', ''), 'posA': A['pos'], 'aaA': A.get('aa', ''), 'aaA_h37rv': A.get('aa_h37rv', ''), 'effA': A.get('eff', ''),
+ 'geneB': B.get('gene', ''), 'posB': B['pos'], 'aaB': B.get('aa', ''), 'aaB_h37rv': B.get('aa_h37rv', ''), 'effB': B.get('eff', ''),
+ 'r': round(mean_r, 3), 'n': len(rs), 'rmin': round(min(rs), 3), 'rmax': round(max(rs), 3),
+ 'direction': 'concordant' if mean_r > 0 else 'discordant',
+ 'p': p, 'consistent': (n_pos == len(rs) or n_pos == 0),
+ 'times': rec['times'], 'trajA': rec['trajA'], 'trajB': rec['trajB'], 'group': rec['group']})
+ if not out:
+ return None
+ # Benjamini-Hochberg FDR across every reported pair (multiple-testing correction)
+ m = len(out)
+ order = sorted(range(m), key=lambda i: out[i]['p'])
+ qmin = 1.0
+ for rank in range(m, 0, -1):
+ i = order[rank - 1]
+ qmin = min(qmin, out[i]['p'] * m / rank)
+ out[i]['q'] = round(min(qmin, 1.0), 4)
+ for pr in out:
+ pr['tier'] = 'strong' if pr['q'] <= 0.05 else ('moderate' if pr['p'] <= 0.05 else 'weak')
+ pr['p'] = round(pr['p'], 4)
+ out.sort(key=lambda p: (p['q'], -abs(p['r'])))
+ out = out[:top]
+ # all-vs-all correlation matrix over the most-connected variants (INCLUDES sub-threshold cells, so it
+ # is the full co-dynamics landscape, not only the reported pairs). Capped to bound the embedded size.
+ allmeta, deg, meanr = {}, {}, {}
+ for (a, b), rec in pairs.items():
+ meanr[(a, b)] = sum(rec['rs']) / len(rec['rs'])
+ deg[a] = deg.get(a, 0) + 1
+ deg[b] = deg.get(b, 0) + 1
+ for v in (rec['A'], rec['B']):
+ allmeta.setdefault(v['pos'], {'pos': v['pos'], 'gene': v.get('gene', ''), 'aa': v.get('aa', '')})
+ MAXNODES = 50
+ node_pos = sorted(deg, key=lambda x: (-deg[x], x))[:MAXNODES] # keep the best-connected variants
+ node_pos.sort(key=lambda x: (allmeta[x]['gene'] or 'zzz', x)) # display order: gene, then position
+ nidx = {p: i for i, p in enumerate(node_pos)}
+ cells = {}
+ for (a, b), mr in meanr.items():
+ if a in nidx and b in nidx:
+ i, j = nidx[a], nidx[b]
+ lo, hi = (i, j) if i <= j else (j, i)
+ cells['%d,%d' % (lo, hi)] = round(mr, 3)
+ matrix = {'nodes': [allmeta[p] for p in node_pos], 'cells': cells,
+ 'truncated': len(deg) > MAXNODES, 'total_nodes': len(deg)}
+ return {'pairs': out, 'min_r': min_r, 'min_points': min_points, 'n_series': n_series, 'perm': perm,
+ 'matrix': matrix,
+ 'vars_capped': vars_capped, 'max_vars': max_vars, 'n_vars_max': n_vars_max,
+ 'pairs_capped': pairs_capped, 'n_candidates': n_candidates, 'max_pairs_perm': max_pairs_perm,
+ 'n_concordant': sum(1 for p in out if p['direction'] == 'concordant'),
+ 'n_discordant': sum(1 for p in out if p['direction'] == 'discordant'),
+ 'n_strong': sum(1 for p in out if p['tier'] == 'strong')}
+
+
+def build_snp_matrix(variants, reference='', max_sites=50000):
+ """Sparse SNP matrix for the report: samples + one row per SNP site with its ref/alt/annotation
+ and only the cells (sample index -> [af, dp]) actually called. The payload is gzip-compressed in the
+ HTML, so this cap is a high safety backstop (most-shared sites kept) rather than a size limit; the
+ pipeline also writes the complete matrix TSV separately."""
+ if not variants:
+ return None
+ samples = list(variants.keys())
+ sidx = {s: i for i, s in enumerate(samples)}
+ sites = {}
+ for s, posmap in variants.items():
+ for key, v in posmap.items():
+ contig, _, pos = key.rpartition(':')
+ try:
+ ipos = int(pos)
+ except ValueError:
+ continue
+ st = sites.setdefault(key, {'contig': contig, 'pos': ipos, 'ref': v.get('ref', ''),
+ 'alt': set(), 'gene': '', 'eff': '', 'aa': '', 'aa_h37rv': '', 'pos_h37rv': '', 'cells': {}})
+ if v.get('alt'):
+ st['alt'].add(v['alt'])
+ for k in ('gene', 'eff', 'aa', 'aa_h37rv', 'pos_h37rv'):
+ if v.get(k) and not st[k]:
+ st[k] = v[k]
+ st['cells'][sidx[s]] = [v['af'], v.get('dp')]
+ ordered = sorted(sites.values(), key=lambda x: (x['contig'], x['pos']))
+ truncated = len(ordered) > max_sites
+ if truncated:
+ ordered = sorted(sites.values(), key=lambda x: -len(x['cells']))[:max_sites]
+ ordered.sort(key=lambda x: (x['contig'], x['pos']))
+ rows = [{'contig': x['contig'], 'pos': x['pos'], 'ref': x['ref'], 'alt': ','.join(sorted(x['alt'])),
+ 'gene': x['gene'], 'eff': x['eff'], 'aa': x['aa'], 'aa_h37rv': x['aa_h37rv'],
+ 'pos_h37rv': x['pos_h37rv'], 'n': len(x['cells']), 'cells': x['cells']}
+ for x in ordered]
+ return {'samples': samples, 'reference': reference, 'rows': rows,
+ 'total_sites': len(sites), 'truncated': truncated}
diff --git a/bin/qcreport/parsers.py b/bin/qcreport/parsers.py
new file mode 100644
index 0000000..2c1eca2
--- /dev/null
+++ b/bin/qcreport/parsers.py
@@ -0,0 +1,744 @@
+"""File parsers for the QC report: pipeline artefacts in, plain Python values out.
+
+Every parser is defensive by design. The optional inputs (BED, GFF, gene burden,
+pN/pS, drug resistance, Kraken, mapDamage, samplesheet, VCF) return an empty
+value instead of raising, so a missing or malformed optional file hides its panel
+rather than failing the run. The two REQUIRED inputs, parse_summary and
+consensus_stats, deliberately raise: the pipeline always hands them a file it has
+just produced, so a problem there is a bug and must be loud.
+
+The low-level `_dyn_*` helpers live here rather than in panels.py because they are
+parsers too (of a VCF FORMAT/INFO field), and the `parse_*` functions above them
+are their only callers.
+"""
+from __future__ import annotations
+
+import gzip
+import os
+import re
+import sys
+
+NBINS = 200 # bins along the reference for the genome callability-landscape heatmap
+
+
+def parse_summary(path):
+ rows = {}
+ with open(path) as f:
+ header = f.readline().rstrip("\n").split("\t")
+ for line in f:
+ if not line.strip():
+ continue
+ parts = line.rstrip("\n").split("\t")
+ if len(parts) < len(header):
+ parts += [""] * (len(header) - len(parts))
+ d = dict(zip(header, parts))
+ sid = (d.get("sample_id") or parts[0]).strip()
+ if not sid:
+ continue
+ rows[sid] = d
+ return rows
+
+
+def consensus_stats(path):
+ op = gzip.open if str(path).endswith(".gz") else open
+ name, chunks = None, []
+ with op(path, "rt", encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ if line.startswith(">"):
+ if name is None: # keep the first record's id, but keep reading ALL records so a
+ name = line[1:].split()[0] # multi-contig consensus is measured whole-genome, not contig 1 only
+ else:
+ chunks.append(line.strip())
+ seq = "".join(chunks).upper()
+ L = len(seq) or 1
+ gap, n = seq.count("-"), seq.count("N")
+ iup = sum(seq.count(c) for c in "RYSWKMBDHV")
+ acgt = sum(seq.count(c) for c in "ACGT")
+ # gap-length view: one big deletion vs many scattered holes are very different for a phylogeny
+ runs = re.findall(r"[-N]+", seq)
+ longest_run = max((len(r) for r in runs), default=0)
+ # binned missingness profile ALONG the reference (consensus is reference-aligned) -> the genome landscape heatmap.
+ # A callability proxy for coverage: per bin, the fraction of positions that are N / gap (no confident base).
+ prof = []
+ for b in range(NBINS):
+ s0, e0 = (b * L) // NBINS, ((b + 1) * L) // NBINS
+ sub = seq[s0:e0]
+ ln = len(sub) or 1
+ prof.append(round((sub.count("-") + sub.count("N")) / ln, 3))
+ return {"consensus_id": name, "length": len(seq), "missing_pct": 100.0 * (gap + n) / L,
+ "iupac_pct": 100.0 * iup / L, "callable_pct": 100.0 * acgt / L,
+ "longest_n_run": longest_run, "n_gaps": len(runs), "miss_profile": prof}
+
+
+def to_float(x):
+ try:
+ if x in (None, "", "NA", "."):
+ return None
+ return float(x)
+ except (ValueError, TypeError):
+ return None
+
+
+def clean_str(x):
+ """A metadata string, or None for the NA/empty sentinels (so 'NA' lineage does not read as a real lineage)."""
+ if x is None:
+ return None
+ s = str(x).strip()
+ return s if s and s not in ("NA", ".", "-") else None
+
+
+_PROFILE_WARNED = set()
+
+
+def parse_profile(s):
+ """A comma-joined per-bin variant-count profile string -> [int]*NBINS, or None if absent/unparseable.
+ Coerced (pad with 0 / truncate) to NBINS: the genome-landscape JS assumes every profile has exactly
+ NBINS bins so that a brushed span maps 1:1 onto the hotspot table. Upstream (stats_to_legacy.BIN_N)
+ already emits NBINS, so a mismatch means a foreign / re-binned summary; warn once per observed length."""
+ if not s or str(s) in ("NA", ".", ""):
+ return None
+ try:
+ v = [int(float(x)) for x in str(s).split(",")]
+ except (ValueError, TypeError):
+ return None
+ if not v:
+ return None
+ if len(v) != NBINS:
+ if len(v) not in _PROFILE_WARNED:
+ sys.stderr.write(
+ "[qc_report] warning: variant profile has %d bins, expected %d; "
+ "padding/truncating so the genome landscape stays aligned.\n" % (len(v), NBINS))
+ _PROFILE_WARNED.add(len(v))
+ v = (v + [0] * NBINS)[:NBINS]
+ return v
+
+
+def parse_lineage_colors(path):
+ """Read a 'lineage#hexdesc' palette (e.g. mycolorsTB) -> {lineage: '#hex'}. Empty if absent."""
+ if not path or not os.path.exists(path):
+ return {}
+ out = {}
+ try:
+ with open(path, encoding="utf-8") as fh:
+ for line in fh:
+ if line.startswith("#") or not line.strip():
+ continue
+ c = line.rstrip("\n").split("\t")
+ if len(c) >= 2 and c[1].strip().startswith("#") and len(c[1].strip()) in (4, 7):
+ out[c[0].strip()] = c[1].strip()
+ except OSError:
+ return {}
+ return out
+
+
+def parse_bed(path):
+ """Parse a BED (0-based half-open) of regions to mask -> [(start, end)]. Empty if absent."""
+ if not path or not os.path.exists(path):
+ return []
+ iv = []
+ try:
+ op = gzip.open if str(path).endswith(".gz") else open
+ with op(path, "rt", encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ if not line.strip() or line.startswith(("#", "track", "browser")):
+ continue
+ c = line.split("\t")
+ if len(c) >= 3:
+ try:
+ iv.append((int(c[1]), int(c[2])))
+ except ValueError:
+ pass
+ except OSError:
+ return []
+ return iv
+
+
+def mask_profile(intervals, genome_len, nbins=NBINS):
+ """Per-bin masked fraction (0-1) along the reference + the total masked %."""
+ if not intervals or genome_len <= 0:
+ return None, 0.0
+ binbp = genome_len / nbins
+ cov = [0.0] * nbins
+ kept = []
+ for s, e in intervals:
+ s = max(0, s); e = min(genome_len, e)
+ if e <= s:
+ continue
+ kept.append((s, e))
+ b0 = int(s // binbp); b1 = min(nbins - 1, int((e - 1) // binbp))
+ for b in range(b0, b1 + 1):
+ bs = b * binbp; be = (b + 1) * binbp
+ cov[b] += max(0.0, min(e, be) - max(s, bs))
+ # the total is the UNION of the intervals: an un-merged BED with overlapping records must not
+ # count the same base twice (that reported >100% masked), so sweep them in coordinate order
+ total = 0
+ reach = 0 # rightmost end already counted
+ for s, e in sorted(kept):
+ if e > reach:
+ total += e - max(s, reach)
+ reach = e
+ prof = [round(min(1.0, cov[b] / binbp), 3) if binbp > 0 else 0.0 for b in range(nbins)]
+ return prof, round(100.0 * total / genome_len, 2)
+
+
+def parse_gene_burden(path):
+ """Cohort gene-burden aux TSV (from collect_summary --gene-burden-out) -> list of gene dicts sorted by
+ total_impactful desc. Optional file; empty on any problem. Bounded to the top 200 genes."""
+ if not path or not os.path.exists(path):
+ return []
+ out = []
+ try:
+ with open(path, encoding="utf-8", errors="replace") as fh:
+ header = fh.readline().rstrip("\n").split("\t")
+ idx = {h: i for i, h in enumerate(header)}
+
+ def sget(c, k):
+ i = idx.get(k)
+ return c[i] if (i is not None and i < len(c) and c[i] != "") else None
+
+ def nget(c, k):
+ v = sget(c, k)
+ try:
+ return int(float(v))
+ except (TypeError, ValueError):
+ return None
+ for line in fh:
+ if not line.strip():
+ continue
+ c = line.rstrip("\n").split("\t")
+ gene = sget(c, "gene")
+ if not gene:
+ continue
+ hi, mod = nget(c, "high") or 0, nget(c, "moderate") or 0
+ tot = nget(c, "total_impactful")
+ out.append({"gene": gene, "high": hi, "moderate": mod,
+ "low": nget(c, "low"), "modifier": nget(c, "modifier"),
+ "dominant_effect": sget(c, "dominant_effect") or sget(c, "dominant") or "NA",
+ "n_samples": nget(c, "n_samples"),
+ "total_impactful": tot if tot is not None else hi + mod,
+ "start": nget(c, "start"), "end": nget(c, "end")})
+ except Exception:
+ return []
+ out.sort(key=lambda r: (-(r.get("total_impactful") or 0), r["gene"]))
+ return out[:200]
+
+
+def parse_pnps(path):
+ """Optional cohort dN/dS (eskaks) per-gene TSV -> [{gene, pn, ps, pnps, n, effect}]. Empty on any problem
+ (panel hidden). Real eskaks-aggregate columns: gene, gene_name, n_samples, n_pairs, mean_dN, mean_dS, dNdS,
+ effect_dominant. inf/nan/empty are read as None, never blindly float()-ed."""
+ if not path or not os.path.exists(path):
+ return []
+
+ def num(v):
+ if v is None:
+ return None
+ v = str(v).strip().lower()
+ if v in ("", "na", "nan", "inf", "-inf", "+inf"):
+ return None
+ return to_float(v)
+ rows = []
+ try:
+ with open(path, encoding="utf-8", errors="replace") as fh:
+ header = fh.readline().rstrip("\n").split("\t")
+ idx = {h.strip().lower(): i for i, h in enumerate(header)}
+
+ def col(c, p):
+ i = idx.get(c)
+ return p[i] if (i is not None and i < len(p)) else None
+ for line in fh:
+ if not line.strip():
+ continue
+ p = line.rstrip("\n").split("\t")
+ g = col("gene_name", p) or col("gene", p)
+ if not g:
+ continue
+ rows.append({"gene": g, "pn": num(col("mean_dn", p)), "ps": num(col("mean_ds", p)),
+ "pnps": num(col("dnds", p)), "n": num(col("n_pairs", p)),
+ "effect": col("effect_dominant", p)})
+ except OSError:
+ return []
+ return rows[:300]
+
+
+_DR_DRUG_ORDER = ["RIF", "INH", "EMB", "PZA", "STR", "STM", "FQ", "LFX", "MFX", "OFX", "KAN", "AMK",
+ "CAP", "ETH", "PTO", "LZD", "BDQ", "CFZ", "BDQ_CFZ", "DLM", "PAS", "CS"]
+
+
+def parse_dr(path):
+ """Per-sample drug-resistance calls -> {'samples','drugs','calls':[{s,drug,gene,mutation,grade,gn,marker,
+ af,dp}]} or None. Tab-separated with a header; recognised columns (case-insensitive): sample, drug, gene,
+ mutation, grade, marker_name/marker, af, dp. `grade` may be a WHO string like '1) Assoc w R' -> gn=1."""
+ if not path or not os.path.exists(path):
+ return None
+ calls, samples = [], []
+ try:
+ with open(path, encoding="utf-8", errors="replace") as fh:
+ header = None
+ for line in fh:
+ if not line.strip() or line.startswith("#"):
+ continue
+ parts = line.rstrip("\n").split("\t")
+ if header is None:
+ header = [h.strip().lower() for h in parts]
+ continue
+ d = dict(zip(header, parts))
+ s = (d.get("sample") or (parts[0] if parts else "")).strip()
+ if not s:
+ continue
+ grade = (d.get("grade") or "").strip()
+ mg = re.match(r"\s*(\d)", grade)
+ dp = d.get("dp")
+ try: # via float(): a depth written as '40.0' is still an integer depth, not a parse failure
+ dp = int(float(dp)) if dp not in (None, "", "NA", ".") else None
+ except ValueError:
+ dp = None
+ calls.append({"s": s, "drug": (d.get("drug") or "").strip(), "gene": (d.get("gene") or "").strip(),
+ "mutation": (d.get("mutation") or "").strip(), "grade": grade,
+ "gn": (int(mg.group(1)) if mg else None),
+ "marker": (d.get("marker_name") or d.get("marker") or "").strip(),
+ "af": to_float(d.get("af")), "dp": dp})
+ if s not in samples:
+ samples.append(s)
+ except OSError:
+ return None
+ if not calls:
+ return None
+ drugs = sorted({c["drug"] for c in calls if c["drug"]},
+ key=lambda x: (_DR_DRUG_ORDER.index(x) if x in _DR_DRUG_ORDER else 99, x))
+ return {"samples": samples, "drugs": drugs, "calls": calls}
+
+
+def parse_kraken(paths):
+ """Per-sample Kraken2 `.report` files (6 columns: clade%, clade reads, taxon reads, rank, taxid,
+ name) -> {'samples':[{s, unclassified, classified, primary, secondary, top}]} or None. The sample id
+ is the filename up to the first '__' (the pipeline writes '__.kraken.report'). The
+ 'primary' taxon is the top species-level clade; contamination shows as a low primary% / large
+ secondary taxon / high unclassified%."""
+ out = []
+ for p in paths or []:
+ if not p or not os.path.exists(p):
+ continue
+ sid = os.path.basename(p).split("__")[0].split(".")[0]
+ unclass, species = 0.0, []
+ try:
+ with open(p, encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ c = line.rstrip("\n").split("\t")
+ if len(c) < 6:
+ continue
+ try:
+ pct = float(c[0])
+ except ValueError:
+ continue
+ rank, name = c[3].strip(), c[5].strip()
+ if rank == "U":
+ unclass = pct
+ elif rank == "S": # species-level clades = the interpretable composition
+ species.append((pct, name))
+ except OSError:
+ continue
+ if not species and unclass == 0.0:
+ continue
+ species.sort(key=lambda x: -x[0])
+ top = [{"name": n, "pct": round(p, 2)} for p, n in species[:6]]
+ out.append({"s": sid, "unclassified": round(unclass, 2),
+ "classified": round(100.0 - unclass, 2),
+ "primary": (top[0] if top else None),
+ "secondary": (top[1] if len(top) > 1 else None), "top": top})
+ if not out:
+ return None
+ return {"samples": out}
+
+
+def _gff_attr(attrs, key):
+ """Value of GFF3 attribute `key` from the column-9 string, or None when absent. The key is matched
+ WHOLE against each ';'-separated field (a substring search would read 'locus_tag' out of
+ 'old_locus_tag' and 'gene' out of 'pseudogene', both legal GFF3 and routine in RefSeq/Prokka).
+ Only the first '=' splits, so a value that itself contains '=' survives."""
+ for field in attrs.split(";"):
+ k, sep, v = field.partition("=")
+ if sep and k.strip() == key:
+ return v.strip()
+ return None
+
+
+def parse_gff(path):
+ """Best-effort GFF3 parse of gene/CDS features -> [{name, start, end}] (1-based). Empty on any problem."""
+ if not path or not os.path.exists(path):
+ return []
+ genes = {}
+ try:
+ op = gzip.open if str(path).endswith(".gz") else open
+ with op(path, "rt", encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ if line.startswith("#") or "\t" not in line:
+ continue
+ c = line.rstrip("\n").split("\t")
+ if len(c) < 9 or c[2] not in ("gene", "CDS"):
+ continue
+ try:
+ start, end = int(c[3]), int(c[4])
+ except ValueError:
+ continue
+ name = None
+ for key in ("Name", "gene", "locus_tag", "ID"):
+ name = _gff_attr(c[8], key)
+ if name is not None:
+ break
+ name = name or f"{start}-{end}"
+ # prefer a 'gene' feature over a 'CDS' of the same name
+ if name not in genes or (c[2] == "gene" and genes[name].get("type") != "gene"):
+ genes[name] = {"name": name, "start": start, "end": end, "type": c[2]}
+ except (OSError, ValueError):
+ return []
+ return sorted(({"name": g["name"], "start": g["start"], "end": g["end"]} for g in genes.values()),
+ key=lambda g: g["start"])
+
+
+def parse_gene_locus(path):
+ """{gene_display_name: locus_tag} from a GFF3 (e.g. rpoB -> Rv0667). Empty on any problem."""
+ out = {}
+ if not path or not os.path.exists(path):
+ return out
+ try:
+ op = gzip.open if str(path).endswith(".gz") else open
+ with op(path, "rt", encoding="utf-8", errors="replace") as fh:
+ for line in fh:
+ if line.startswith("#") or "\t" not in line:
+ continue
+ c = line.rstrip("\n").split("\t")
+ if len(c) < 9 or c[2] not in ("gene", "CDS"):
+ continue
+ attrs = c[8]
+ locus = _gff_attr(attrs, "locus_tag")
+ name = _gff_attr(attrs, "gene") or _gff_attr(attrs, "Name")
+ if name and locus and name not in out:
+ out[name] = locus
+ except (OSError, ValueError):
+ return out
+ return out
+
+
+def _read_freq_file(path, max_pos=25):
+ """Parse a mapDamage 5pCtoT_freq.txt / 3pGtoA_freq.txt ('posfreq'). Returns
+ {'pos1': float|None, 'profile': [[pos,freq],...] up to max_pos} or None. Robust to header + blanks."""
+ prof = []
+ try:
+ with open(path) as fh:
+ for line in fh:
+ line = line.strip()
+ if not line or line.lower().startswith("pos"):
+ continue
+ parts = line.replace(",", "\t").split("\t")
+ if len(parts) < 2:
+ parts = line.split()
+ if len(parts) < 2:
+ continue
+ p = to_float(parts[0]); v = to_float(parts[1])
+ if p is None or v is None:
+ continue
+ prof.append((int(p), v))
+ except (OSError, ValueError):
+ return None
+ if not prof:
+ return None
+ prof.sort(key=lambda x: x[0])
+ pos1 = next((v for p, v in prof if p == 1), prof[0][1])
+ return {"pos1": pos1, "profile": [[p, round(v, 5)] for p, v in prof[:max_pos]]}
+
+
+def _mean_fraglen(path):
+ """Mean fragment length from lgdistribution.txt (StdLengthOccurences, '#'-comment lines)."""
+ num = den = 0.0
+ try:
+ with open(path) as fh:
+ for line in fh:
+ if not line.strip() or line.startswith("#"):
+ continue
+ parts = line.split()
+ if len(parts) < 3 or parts[1].lower() == "length":
+ continue
+ L = to_float(parts[1]); occ = to_float(parts[2])
+ if L is None or occ is None:
+ continue
+ num += L * occ; den += occ
+ except (OSError, ValueError):
+ return None
+ return (num / den) if den > 0 else None
+
+
+def mapdamage_stats(sample_dir):
+ """Read a mapDamage2 output directory for one sample. Returns a dict or None if nothing parseable.
+ keys: ct1 (5' C>T pos1), ga1 (3' G>A pos1), fraglen (mean), ct_profile [[pos,freq],...]."""
+ if not sample_dir or not os.path.isdir(sample_dir):
+ return None
+ five = _read_freq_file(os.path.join(sample_dir, "5pCtoT_freq.txt"))
+ three = _read_freq_file(os.path.join(sample_dir, "3pGtoA_freq.txt"))
+ fl = _mean_fraglen(os.path.join(sample_dir, "lgdistribution.txt"))
+ ct1 = five["pos1"] if five else None
+ if five is None and three is None and fl is None and ct1 is None:
+ return None
+ return {"ct1": ct1, "ga1": (three["pos1"] if three else None), "fraglen": fl,
+ "ct_profile": five["profile"] if five else None}
+
+
+# ---------------------------------------------------------------- samplesheet + VCF (dynamics inputs)
+_DYN_SAMPLE_RE = re.compile(r'^(sample_?id|sampleid|sample|name|gid|strain|isolate)$', re.I)
+_DYN_TIME_RE = re.compile(r'(passage|pase|timepoint|time_?point|^time$|^day$|date|week|month|hour|generation|^tp$|visit|^t\d*$)', re.I)
+_DYN_GROUP_RE = re.compile(r'(group|series|patient|host|subject|cluster|experiment|^line$|replicate|chain|pair|lineage_?id|donor|case|animal|^samples$)', re.I)
+_DOSE_RE = re.compile(r'^(dose|dosis)$', re.I) # a numeric annotation -> a metric, not a categorical level
+_TX_RE = re.compile(r'(treatment|tratamiento|regimen|therapy|^arm$)', re.I) # the categorical column the dose x treatment test groups by
+
+
+def _dyn_open(path):
+ return gzip.open(path, 'rt', encoding='utf-8', errors='replace') if str(path).endswith('.gz') \
+ else open(path, 'r', encoding='utf-8', errors='replace')
+
+
+def _dyn_num(x):
+ if x is None:
+ return None
+ m = re.search(r'-?\d+\.?\d*', str(x))
+ return float(m.group()) if m else None
+
+
+_META_SKIP = {'r1', 'r2', 'reffasta', 'refgff', 'refid', 'taxid', 'runid'}
+
+
+def parse_sample_meta(path):
+ """{'fields':[col...], 'rows':{sample:{col:value}}} for the user's annotation columns of the
+ samplesheet (everything except the sample id and the pipeline's file/reference columns), in
+ samplesheet order -> one column-header level per field. None if nothing usable."""
+ if not path or not os.path.exists(path):
+ return None
+ header, rows = None, []
+ try:
+ with _dyn_open(path) as fh:
+ for line in fh:
+ if not line.strip() or line.startswith('#'):
+ continue
+ cells = line.rstrip('\n').split('\t')
+ if header is None:
+ header = [c.strip() for c in cells]
+ continue
+ rows.append(cells)
+ except OSError:
+ return None
+ if not header:
+ return None
+ si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
+ # the dose column (if any) is a numeric metric, not a categorical annotation -> keep it out of fields
+ fields = [(i, h) for i, h in enumerate(header)
+ if i != si and h.strip() and h.strip().lower() not in _META_SKIP
+ and not _DOSE_RE.match(h.strip())]
+ if not fields:
+ return None
+ out = {}
+ for r in rows:
+ if si >= len(r):
+ continue
+ s = r[si].strip()
+ if not s or s in out:
+ continue
+ out[s] = {h: (r[i].strip() if i < len(r) else '') for i, h in fields}
+ if not out:
+ return None
+ # tag the time / group columns (dynamics axes) and the treatment column (dose x treatment test)
+ # so the report can route each correctly; they all stay as SNP-matrix header levels either way.
+ time_field = next((h for _, h in fields if _DYN_TIME_RE.search(h.replace(' ', '_'))), None)
+ group_field = next((h for _, h in fields if _DYN_GROUP_RE.search(h.replace(' ', '_'))), None)
+ tx_field = next((h for _, h in fields if _TX_RE.search(h.replace(' ', '_'))), None)
+ return {'fields': [h for _, h in fields], 'rows': out,
+ 'time_field': time_field, 'group_field': group_field, 'tx_field': tx_field}
+
+
+def parse_dose(path):
+ """{sample: float} for a numeric samplesheet column named 'dose'/'dosis' (a quantitative
+ annotation exposed as a report metric). {} if the file or column is absent / non-numeric."""
+ if not path or not os.path.exists(path):
+ return {}
+ header, rows = None, []
+ try:
+ with _dyn_open(path) as fh:
+ for line in fh:
+ if not line.strip() or line.startswith('#'):
+ continue
+ cells = line.rstrip('\n').split('\t')
+ if header is None:
+ header = [c.strip() for c in cells]
+ continue
+ rows.append(cells)
+ except OSError:
+ return {}
+ if not header:
+ return {}
+ si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
+ di = next((i for i, h in enumerate(header) if _DOSE_RE.match(h.strip())), None)
+ if di is None:
+ return {}
+ out = {}
+ for r in rows:
+ if si >= len(r) or di >= len(r):
+ continue
+ s = r[si].strip()
+ if not s or s in out:
+ continue
+ v = to_float(r[di].strip())
+ if v is not None:
+ out[s] = v
+ return out
+
+
+def parse_metadata(path):
+ """{sample: {'time','tnum','group'}} auto-detecting sample/time/group columns; {} if unusable."""
+ if not path or not os.path.exists(path):
+ return {}
+ header, rows = None, []
+ try:
+ with _dyn_open(path) as fh:
+ for line in fh:
+ if not line.strip() or line.startswith('#'):
+ continue
+ cells = line.rstrip('\n').split('\t')
+ if header is None:
+ header = [c.strip() for c in cells]
+ continue
+ rows.append(cells)
+ except OSError:
+ return {}
+ if not header:
+ return {}
+ def find(rx):
+ for i, h in enumerate(header):
+ if rx.search(h.replace(' ', '_')):
+ return i
+ return None
+ si = next((i for i, h in enumerate(header) if _DYN_SAMPLE_RE.match(h.strip())), 0)
+ ti, gi = find(_DYN_TIME_RE), find(_DYN_GROUP_RE)
+ out = {}
+ for r in rows:
+ if si >= len(r):
+ continue
+ s = r[si].strip()
+ if not s or s in out:
+ continue
+ out[s] = {'time': r[ti].strip() if (ti is not None and ti < len(r)) else None,
+ 'tnum': _dyn_num(r[ti]) if (ti is not None and ti < len(r)) else None,
+ 'group': r[gi].strip() if (gi is not None and gi < len(r)) else None}
+ return out
+
+
+def _dyn_af(fmt, val):
+ """First-alt allele fraction: AD/AO+RO (true AF) else GT dosage (hom 1.0 / het 0.5)."""
+ d = dict(zip(fmt.split(':'), val.split(':')))
+ if 'AD' in d:
+ try:
+ ad = [int(x) for x in d['AD'].split(',') if x not in ('.', '')]
+ if len(ad) >= 2 and sum(ad) > 0:
+ return sum(ad[1:]) / sum(ad)
+ except ValueError:
+ pass
+ if 'AO' in d and 'RO' in d:
+ try:
+ ao = sum(int(x) for x in d['AO'].split(',') if x not in ('.', ''))
+ ro = int(d['RO']) if d['RO'] not in ('.', '') else 0
+ if ao + ro > 0:
+ return ao / (ao + ro)
+ except ValueError:
+ pass
+ alleles = [a for a in d.get('GT', './.').replace('|', '/').split('/') if a not in ('.', '')]
+ if not alleles:
+ return None
+ return sum(1 for a in alleles if a != '0') / len(alleles)
+
+
+def _dyn_dp(fmt, val):
+ """Total depth at the site: FORMAT DP, else sum(AD), else AO+RO."""
+ d = dict(zip(fmt.split(':'), val.split(':')))
+ if d.get('DP', '.') not in ('.', ''):
+ try:
+ return int(d['DP'])
+ except ValueError:
+ pass
+ if 'AD' in d:
+ try:
+ ad = [int(x) for x in d['AD'].split(',') if x not in ('.', '')]
+ if ad:
+ return sum(ad)
+ except ValueError:
+ pass
+ if 'AO' in d and 'RO' in d:
+ try:
+ return sum(int(x) for x in d['AO'].split(',') if x not in ('.', '')) + \
+ (int(d['RO']) if d['RO'] not in ('.', '') else 0)
+ except ValueError:
+ pass
+ return None
+
+
+def _dyn_ann(info):
+ # snpEff ANN fields: 3=gene, 1=effect, 2=impact, 10=HGVS.p (protein change, e.g. p.Ser315Thr)
+ for field in info.split(';'):
+ if field.startswith('ANN='):
+ a = field[4:].split(',')[0].split('|')
+ return ((a[3] if len(a) > 3 else ''), (a[1] if len(a) > 1 else ''),
+ (a[2] if len(a) > 2 else ''), (a[10] if len(a) > 10 else ''))
+ return '', '', '', ''
+
+
+def _dyn_opos(info):
+ """The ORIGINAL (used-reference) coordinate a liftover stamped onto a canonical-reference VCF record,
+ so a used-ref variant can be paired with its reference-of-interest position WITHOUT assuming the two
+ references share coordinates. Reads Picard LiftoverVcf's OriginalContig/OriginalStart, else a plain
+ OPOS=. Returns 'contig:pos' (used-ref coordinates) or None."""
+ if not info:
+ return None
+ oc = ost = None
+ for field in info.split(';'):
+ if field.startswith('OriginalContig='):
+ oc = field[len('OriginalContig='):]
+ elif field.startswith('OriginalStart='):
+ ost = field[len('OriginalStart='):]
+ elif field.startswith('OPOS='):
+ return field[len('OPOS='):].strip() or None
+ return (oc + ':' + ost) if (oc and ost) else None
+
+
+def parse_vcfs(paths):
+ """{sample: {'chrom:pos': {ref,alt,gene,eff,imp,af}}} from annotated per-sample VCFs (SNPs only).
+ Sample = the VCF #CHROM last column, so filenames are irrelevant."""
+ out = {}
+ for p in paths or []:
+ if not p or not os.path.exists(p):
+ continue
+ try:
+ with _dyn_open(p) as fh:
+ sample = None
+ for line in fh:
+ if line.startswith('##'):
+ continue
+ if line.startswith('#CHROM'):
+ cols = line.rstrip('\n').split('\t')
+ sample = cols[9] if len(cols) > 9 else os.path.basename(p).split('.')[0]
+ out.setdefault(sample, {})
+ continue
+ if sample is None:
+ continue
+ c = line.rstrip('\n').split('\t')
+ if len(c) < 8:
+ continue
+ chrom, pos, ref, alt = c[0], c[1], c[3], c[4]
+ if alt in ('.', '') or len(ref) != 1 or any(len(a) != 1 for a in alt.split(',')):
+ continue
+ af = _dyn_af(c[8], c[9]) if len(c) >= 10 else 1.0
+ if af is None:
+ continue
+ dp = _dyn_dp(c[8], c[9]) if len(c) >= 10 else None
+ gene, eff, imp, aa = _dyn_ann(c[7])
+ out[sample][f'{chrom}:{pos}'] = {'ref': ref, 'alt': alt.split(',')[0], 'gene': gene,
+ 'eff': eff, 'imp': imp, 'aa': aa, 'aa_h37rv': '',
+ 'af': round(af, 4), 'dp': dp, 'opos': _dyn_opos(c[7])}
+ except OSError:
+ continue
+ return out
diff --git a/bin/qcreport/render.py b/bin/qcreport/render.py
new file mode 100644
index 0000000..da0a225
--- /dev/null
+++ b/bin/qcreport/render.py
@@ -0,0 +1,152 @@
+"""The report front-end: the CSS/JS/HTML assets and the self-contained HTML assembly.
+
+The assets are read from bin/report_assets/ at IMPORT time, so importing this
+module touches the disk. build_html() splices them into shell.html together with
+the gzip+base64 payload, producing the single file the pipeline publishes.
+
+SECTION_INFO lives here rather than with the metrics because it is report copy:
+the text behind the (i) icon of each section of the HTML shell.
+"""
+from __future__ import annotations
+
+import base64
+import gzip
+import html
+import json
+import os
+import sys
+
+REPO_URL = "https://github.com/PathoGenOmics-Lab/BAMpiro" # surfaced in the report header + footer
+
+
+# ============================================================================= CSS / JS / SHELL
+# ---- report front-end assets (CSS / JS / HTML shell / logo) live next to bin/qc_report.py in
+# report_assets/, one file per language so each stays editable with its own tooling; build_html()
+# splices them into the single self-contained output HTML. ----
+# NOTE: report_assets/ sits beside bin/qc_report.py, i.e. one level ABOVE this package, so the
+# path below walks bin/qcreport/render.py -> bin/qcreport -> bin -> bin/report_assets. The extra
+# dirname() is what keeps it resolving after the move out of bin/qc_report.py; drop it and every
+# asset read fails at import time.
+_ASSET_DIR = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "report_assets")
+
+
+def _asset(name):
+ with open(os.path.join(_ASSET_DIR, name), encoding="utf-8") as fh:
+ return fh.read()
+
+
+# CSS split into ordered modules under report_assets/css/ (base tokens -> layout -> components ->
+# panels); concatenated in cascade order, so the result is identical to a single stylesheet.
+_CSS_MODULES = ["css/01_base.css", "css/02_layout.css", "css/03_components.css", "css/04_panels.css"]
+CSS = "".join(_asset(m) for m in _CSS_MODULES)
+
+# The report front-end is one ES5 IIFE split into ordered modules under report_assets/js/ (one per
+# area) purely for editability; they are concatenated verbatim here, so the running code is identical
+# to a single file. Order matters: 01 opens the IIFE and sets up shared state; 14 wires events + closes it.
+_JS_MODULES = [
+ "js/01_prelude.js", "js/02_qcspace.js", "js/03_state.js", "js/04_helpers.js",
+ "js/05_insights_qc.js", "js/06_insights_genome.js", "js/07_render_core.js", "js/08_curation.js",
+ "js/09_genome_genes.js", "js/10_dynamics.js", "js/11_epistasis.js", "js/12_snpmatrix.js",
+ "js/13_drug_kraken.js", "js/14_boot.js",
+]
+JS = "".join(_asset(m) for m in _JS_MODULES)
+
+# BAMpiro header logo (bampiro2.png resized to ~90x100 and embedded so the report stays
+# self-contained - no external image request). Regenerate from .github/bampiro2.png if the logo changes.
+LOGO_DATA_URI = _asset("logo.b64")
+
+SHELL = _asset("shell.html") # named shell.html (not report_*.html) so .gitignore's report*.html rule can't swallow it
+
+
+def build_html(title, payload):
+ # Embed the payload gzip-compressed + base64 so the whole cohort travels in a light, self-contained
+ # HTML (JSON deflates ~5-10x); the browser inflates it natively at load. base64 has no ". Nothing is dropped for size - compression is what lets us keep it all.
+ raw = json.dumps(payload, separators=(",", ":")).encode("utf-8")
+ data_gz = base64.b64encode(gzip.compress(raw, 9)).decode("ascii")
+ sys.stderr.write("[qc_report] payload %.2f MB JSON -> %.2f MB embedded (gzip+base64, %.0f%% smaller)\n"
+ % (len(raw) / 1048576.0, len(data_gz) / 1048576.0,
+ 100.0 * (1.0 - len(data_gz) / max(1, len(raw)))))
+ # Inject content first, then substitute __TITLE__ LAST so a user --title that happens to
+ # contain a placeholder token (e.g. "__JS__") can never pull in the CSS/JS/logo/JSON blob.
+ return (SHELL.replace("__CSS__", CSS).replace("__LOGO__", LOGO_DATA_URI)
+ .replace("__JS__", JS).replace("__JSON_GZ__", data_gz)
+ .replace("__REPO__", html.escape(REPO_URL))
+ .replace("__TITLE__", html.escape(title)))
+
+
+# Per-panel explanation shown by the (i) icon in each section heading: what the analysis is, how to read
+# it, and its caveats. Keyed by the section id in the HTML shell.
+SECTION_INFO = {
+ "gstats": "Per-sample sequencing and mapping QC: reads, duplication, mapped %, mean/median depth, "
+ "breadth, variant counts and Ti/Tv. Each numeric cell is shaded by where the sample sits in "
+ "that column's range; the QC column flags PASS/WARN/FAIL against the live thresholds. Tick a "
+ "box to basket a sample for exclusion; click a name for its full profile.",
+ "linsum": "Median QC metrics grouped by the assigned MTBC lineage, with the number of samples, %PASS and "
+ "how many are 'mixed' (more than one lineage above the mixture cut-off). Lineage comes from "
+ "pathotypr; a mixed call can indicate co-infection or contamination.",
+ "dist": "Per-metric value distributions across the cohort (a strip/bee plot, or a ranked bar). The shaded "
+ "band is the acceptable range for the current thresholds; points outside it are outliers. Useful "
+ "to see cohort spread and choose sensible cut-offs.",
+ "corr": "A scatter of any two QC metrics, one point per sample, coloured by QC status or lineage. Drag a "
+ "box to basket the enclosed samples. Reveals metric relationships (e.g. depth vs breadth) and "
+ "samples that are joint outliers.",
+ "corrmatrix": "Pairwise Pearson correlation between the QC metrics across the cohort, as a heatmap. Shows "
+ "which metrics are redundant (strongly correlated) and which capture independent axes of quality.",
+ "qcpca": "PCA of the standardised QC metrics: each point is a sample placed by its overall quality "
+ "profile. Samples that cluster share a profile; a lone point is a multivariate outlier that no "
+ "single metric flags.",
+ "divcomp": "Each sample's genomic divergence (SNPs vs the reference) against its consensus completeness. "
+ "Separates genuinely divergent samples from those that only look divergent because they are "
+ "incomplete or contaminated.",
+ "cons": "How much of the reference each sample's consensus recovers: callable %, missing % (N/gap), IUPAC "
+ "(ambiguous) % and the longest gap. A phylogeny-oriented view of completeness, not just average depth.",
+ "genome": "Per-position callability / variant density along the reference, binned into a heatmap per "
+ "sample. Brush a region to list the genes under it. Reveals systematically low-callability "
+ "regions (repeats, deletions) shared across samples.",
+ "function": "The snpEff functional class of each sample's variants (HIGH/MODERATE/LOW/MODIFIER impact and "
+ "effect types such as missense / synonymous). A per-sample mutational-impact profile.",
+ "geneburden": "Genes carrying the most impactful (HIGH/MODERATE) variants across the cohort, with the "
+ "dominant effect and how many samples are hit. A cohort-level view of which genes accumulate "
+ "functional change.",
+ "hotspots": "Genes with the highest SNP density across the cohort (variants per gene length). Flags the "
+ "most variable loci - often repeats, antigens, or genes under diversifying selection.",
+ "temporal": "The distribution of sampling dates, when the samplesheet provides them. Context for temporal "
+ "analyses (e.g. SNP dynamics); not a QC metric itself.",
+ "pnps": "Cohort-level pairwise dN/dS per gene (eskaks, Nei/Li). dN/dS > 1 suggests positive / diversifying "
+ "selection at cohort scale. This is population genetics, NOT per-sample QC: noisy per gene, "
+ "sensitive to alignment, and saturating at low divergence (NA when dS is near zero).",
+ "adna": "For samples marked ancient: the characteristic post-mortem damage signal (elevated terminal 5' "
+ "C to T). A sample below the damage floor may be a modern contaminant rather than authentic "
+ "ancient DNA.",
+ "dynamics": "For connected time-series (patient / passage line), each variant's allele-frequency "
+ "trajectory over time, with per-timepoint read depth. Events flag emergence, fixation, loss "
+ "and non-synonymous changes. Needs a time column and a group column in the samplesheet.",
+ "epistasis": "Pairs of variants whose allele-frequency trajectories co-vary within a series: concordant "
+ "(rise/fall together) or discordant (one rises as the other falls). Scored by the Pearson "
+ "correlation, a permutation p-value and a Benjamini-Hochberg FDR q; a pattern recurring across "
+ "independent series is what makes a pair 'strong'. Candidate linked / co-selected / competing "
+ "SNPs, NOT proof of a functional interaction.",
+ "snpmatrix": "Every SNP site (rows) by sample (columns); each cell is the allele frequency with its depth, "
+ "plus a reference column and samplesheet metadata as column-header levels. Filter by gene / "
+ "position or by metadata, and download the full matrix as a TSV.",
+ "drug": "Resistance-associated mutations detected by pathotypr against the WHO catalogue (H37Rv numbering), "
+ "as a sample x drug matrix (worst grade per drug) and a per-mutation table with the WHO confidence "
+ "grade. Alignment-free (k-mer), so it works regardless of the mapping reference. A genomic screen, "
+ "NOT a clinical drug-susceptibility result.",
+ "flagged": "The samples the current thresholds flag as WARN / FAIL and the specific reason for each. This "
+ "is the actionable QC summary; adjust the thresholds above to re-flag the whole report.",
+ "dosetx": "The per-sample dose (a numeric samplesheet column) split by treatment group, drawn as a box (IQR "
+ "+ median) with the individual samples, plus a Kruskal-Wallis rank test of whether dose differs "
+ "across the treatment groups (a Mann-Whitney-equivalent when there are two groups). The test runs "
+ "over the whole cohort, independent of the live filters; groups with fewer than two dosed samples "
+ "are drawn but not tested. Click a point to highlight that sample across the report. Descriptive, "
+ "not a claim about efficacy.",
+ "vardose": "An association scan: for every variant site, the per-sample allele frequency (0 where the site is "
+ "reference) is rank-correlated with dose across the dosed samples (Spearman rho + two-sided p), and "
+ "a Benjamini-Hochberg FDR q is computed across all tested variants so the multiple testing is "
+ "controlled. The table ranks the variants; click a row to plot its allele-frequency-vs-dose scatter, "
+ "click a column header to re-sort, and click a point to highlight that sample. Sites with fewer than "
+ "three carriers or no allele-frequency variation are skipped. A screen for dose-associated variants, "
+ "NOT proof of causation.",
+}
diff --git a/bin/safe_tabix b/bin/safe_tabix
new file mode 100755
index 0000000..1e62bdb
--- /dev/null
+++ b/bin/safe_tabix
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+#
+# Index a bgzipped VCF, tolerating the one case where tabix legitimately fails.
+#
+# safe_tabix sample.vcf.gz
+#
+# tabix exits non-zero on a VCF with no data rows, which is a normal outcome here: a sample can
+# genuinely have no variants left after filtering. That case gets an empty .tbi so the pipeline
+# carries on. Anything else (a truncated or malformed file) must fail loudly rather than get a
+# fake index that hides the corruption from every downstream step.
+#
+# The record count comes from `bcftools view -H`, which reads the whole file: a truncated VCF makes
+# bcftools itself error out, so the distinction is real rather than a guess about the header.
+
+set -euo pipefail
+
+gz="${1:?usage: safe_tabix }"
+idx="${gz}.tbi"
+
+if tabix -f -p vcf "$gz"; then
+ exit 0
+fi
+status=$?
+
+nrec=$(bcftools view -H "$gz" | wc -l)
+if [ "$nrec" -ne 0 ]; then
+ echo "ERROR: tabix failed on $gz with $nrec records (malformed/truncated) -> refusing a fake index" >&2
+ exit "$status"
+fi
+
+: > "$idx"
diff --git a/bin/stats_to_legacy.py b/bin/stats_to_legacy.py
index 3216276..007ae5a 100644
--- a/bin/stats_to_legacy.py
+++ b/bin/stats_to_legacy.py
@@ -134,7 +134,12 @@ def get_genome_size(fai_file):
for line in f:
parts = line.split('\t')
if len(parts) >= 2:
- total += int(parts[1])
+ # per-LINE guard: a bad row is skipped, never truncating the total (a partial genome
+ # length silently propagates into mean depth, breadth, evenness and every density bin)
+ try:
+ total += int(parts[1])
+ except ValueError as e:
+ sys.stderr.write(f"[stats_to_legacy] WARN get_genome_size({fai_file}): skipping row: {e}\n")
except Exception as e:
sys.stderr.write(f"[stats_to_legacy] WARN get_genome_size({fai_file}): {e}\n")
return total # 0 signals unknown; every consumer guards g_size > 0 and emits NA (no organism-specific fallback)
@@ -151,8 +156,13 @@ def get_contig_offsets(fai_file):
for line in f:
parts = line.split('\t')
if len(parts) >= 2:
+ try: # per-LINE guard: a bad row must not drop every contig after it
+ length = int(parts[1])
+ except ValueError as e:
+ sys.stderr.write(f"[stats_to_legacy] WARN get_contig_offsets({fai_file}): skipping row: {e}\n")
+ continue
offsets[parts[0]] = running
- running += int(parts[1])
+ running += length
except Exception as e:
sys.stderr.write(f"[stats_to_legacy] WARN get_contig_offsets({fai_file}): {e}\n")
return offsets
@@ -184,30 +194,33 @@ def get_bam_stats(stats_file):
key = parts[1].rstrip(':')
val = parts[2].strip()
- if key == 'raw total sequences':
- result['total'] = int(val)
- elif key == 'reads mapped':
- result['mapped'] = int(val)
- elif key == 'reads mapped and paired':
- result['mapped_paired'] = int(val)
- elif key == 'reads properly paired':
- result['properly_paired'] = int(val)
- elif key == 'reads duplicated':
- result['duplicates'] = int(val)
- elif key == 'reads unmapped':
- result['reads_unmapped'] = int(val)
- elif key == 'reads MQ0':
- result['reads_mq0'] = int(val)
- elif key == 'average quality':
- result['avg_quality'] = float(val)
- elif key == 'insert size average':
- result['insert_size_avg'] = float(val)
- elif key == 'insert size standard deviation':
- result['insert_size_sd'] = float(val)
- elif key == 'error rate':
- result['error_rate'] = float(val)
- elif key == 'bases mapped (cigar)':
- result['bases_mapped'] = int(val)
+ try: # per-LINE guard: an unparsable value skips that metric, not the rest of the block
+ if key == 'raw total sequences':
+ result['total'] = int(val)
+ elif key == 'reads mapped':
+ result['mapped'] = int(val)
+ elif key == 'reads mapped and paired':
+ result['mapped_paired'] = int(val)
+ elif key == 'reads properly paired':
+ result['properly_paired'] = int(val)
+ elif key == 'reads duplicated':
+ result['duplicates'] = int(val)
+ elif key == 'reads unmapped':
+ result['reads_unmapped'] = int(val)
+ elif key == 'reads MQ0':
+ result['reads_mq0'] = int(val)
+ elif key == 'average quality':
+ result['avg_quality'] = float(val)
+ elif key == 'insert size average':
+ result['insert_size_avg'] = float(val)
+ elif key == 'insert size standard deviation':
+ result['insert_size_sd'] = float(val)
+ elif key == 'error rate':
+ result['error_rate'] = float(val)
+ elif key == 'bases mapped (cigar)':
+ result['bases_mapped'] = int(val)
+ except ValueError as e:
+ sys.stderr.write(f"[stats_to_legacy] WARN get_bam_stats({stats_file}): skipping '{key}': {e}\n")
except Exception as e:
sys.stderr.write(f"[stats_to_legacy] WARN get_bam_stats({stats_file}): {e}\n")
return result
@@ -323,82 +336,90 @@ def analyze_vcf(vcf_file, g_size=0, nbins=BIN_N, contig_offsets=None):
try:
with opener(vcf_file, 'rt') as f:
for line in f:
- if line.startswith('#'): continue
- parts = line.strip().split('\t')
- if len(parts) < 10: continue
- ref = parts[3]
- alt = parts[4]
- if alt in ['.', '*']: continue
- fmt = parts[8].split(':')
- if 'GT' not in fmt: continue
- gt_idx = fmt.index('GT')
- sample = parts[9].split(':')
- if len(sample) <= gt_idx: continue
- gt = sample[gt_idx]
- if gt in ['./.', '0/0', '0|0']: continue
-
- # snpEff functional annotation of this real variant (no-op when the VCF has no ANN=)
- info_str = parts[7] if len(parts) > 7 else ''
- stats['ann_total'] += 1
- ann = _parse_ann_field(info_str)
- if ann is not None:
- cls, impact, gene, has_error, db_error = ann
- stats['ann_present'] = True
- stats['ann_records'] += 1
- stats['ann_impact'][impact] += 1
- stats['ann_class'][cls] += 1
- if has_error:
- stats['ann_warn'] += 1
- if db_error:
- stats['ann_db_error'] = True
- if gene:
- g = stats['ann_gene'].get(gene)
- if g is None:
- g = [0, 0, {}]
- stats['ann_gene'][gene] = g
- if impact == 'HIGH':
- g[0] += 1
- elif impact == 'MODERATE':
- g[1] += 1
- g[2][cls] = g[2].get(cls, 0) + 1
-
- is_snp = (len(ref) == 1 and len(alt) == 1)
- # homozygous = every present allele is identical: covers diploid '1/1' AND haploid '1'
- # (freebayes_ploidy=1). The old 'len==2 and equal' test miscounted every haploid call as het.
- alleles = [a for a in gt.replace('|', '/').split('/') if a != '.']
- is_homo = bool(alleles) and len(set(alleles)) == 1
-
- # positional bin for the genome-landscape variant tracks. POS restarts per contig, so add the
- # contig's cumulative offset to get a global coordinate before binning over the whole genome.
- b = None
- if g_size > 0:
- try:
- gpos = contig_offsets.get(parts[0], 0) + int(parts[1])
- b = min(nbins - 1, (gpos * nbins) // g_size)
- except ValueError:
- b = None
-
- if is_snp:
- stats['snps'] += 1
- r = ref.upper(); a = alt.upper()
- if r in 'ACGT' and a in 'ACGT' and r != a:
- if frozenset((r, a)) in _TRANSITIONS:
- stats['ti'] += 1
- else:
- stats['tv'] += 1
- if b is not None and is_homo:
- stats['snp_prof'][b] += 1
- else:
- stats['indels'] += 1
- if is_homo: stats['homo_indels'] += 1
- if b is not None:
- stats['indel_prof'][b] += 1
- if is_homo:
- stats['homo_total'] += 1
- else:
- stats['het_total'] += 1
- if b is not None:
- stats['het_prof'][b] += 1
+ # per-RECORD guard: one malformed record is skipped, it never discards the rest of the VCF
+ try:
+ if line.startswith('#'): continue
+ parts = line.strip().split('\t')
+ if len(parts) < 10: continue
+ ref = parts[3]
+ alt = parts[4]
+ if alt in ['.', '*']: continue
+ fmt = parts[8].split(':')
+ if 'GT' not in fmt: continue
+ gt_idx = fmt.index('GT')
+ sample = parts[9].split(':')
+ if len(sample) <= gt_idx: continue
+ gt = sample[gt_idx]
+ # Ploidy-agnostic: decide from the present (non-'.') alleles instead of a list of diploid
+ # spellings, so the haploid forms freebayes_ploidy=1 emits are handled too. A no-call
+ # ('./.', '.') has no alleles and a reference call ('0/0', '0|0', '0') only allele 0;
+ # neither is a variant. Keeps the haploid '1' = homozygous behaviour below.
+ alleles = [a for a in gt.replace('|', '/').split('/') if a != '.']
+ if not alleles or set(alleles) == {'0'}: continue
+
+ # snpEff functional annotation of this real variant (no-op when the VCF has no ANN=)
+ info_str = parts[7] if len(parts) > 7 else ''
+ stats['ann_total'] += 1
+ ann = _parse_ann_field(info_str)
+ if ann is not None:
+ cls, impact, gene, has_error, db_error = ann
+ stats['ann_present'] = True
+ stats['ann_records'] += 1
+ stats['ann_impact'][impact] += 1
+ stats['ann_class'][cls] += 1
+ if has_error:
+ stats['ann_warn'] += 1
+ if db_error:
+ stats['ann_db_error'] = True
+ if gene:
+ g = stats['ann_gene'].get(gene)
+ if g is None:
+ g = [0, 0, {}]
+ stats['ann_gene'][gene] = g
+ if impact == 'HIGH':
+ g[0] += 1
+ elif impact == 'MODERATE':
+ g[1] += 1
+ g[2][cls] = g[2].get(cls, 0) + 1
+
+ is_snp = (len(ref) == 1 and len(alt) == 1)
+ # homozygous = every present allele is identical: covers diploid '1/1' AND haploid '1'
+ # (freebayes_ploidy=1). The old 'len==2 and equal' test miscounted every haploid call as het.
+ is_homo = len(set(alleles)) == 1
+
+ # positional bin for the genome-landscape variant tracks. POS restarts per contig, so add the
+ # contig's cumulative offset to get a global coordinate before binning over the whole genome.
+ b = None
+ if g_size > 0:
+ try:
+ gpos = contig_offsets.get(parts[0], 0) + int(parts[1])
+ b = min(nbins - 1, (gpos * nbins) // g_size)
+ except ValueError:
+ b = None
+
+ if is_snp:
+ stats['snps'] += 1
+ r = ref.upper(); a = alt.upper()
+ if r in 'ACGT' and a in 'ACGT' and r != a:
+ if frozenset((r, a)) in _TRANSITIONS:
+ stats['ti'] += 1
+ else:
+ stats['tv'] += 1
+ if b is not None and is_homo:
+ stats['snp_prof'][b] += 1
+ else:
+ stats['indels'] += 1
+ if is_homo: stats['homo_indels'] += 1
+ if b is not None:
+ stats['indel_prof'][b] += 1
+ if is_homo:
+ stats['homo_total'] += 1
+ else:
+ stats['het_total'] += 1
+ if b is not None:
+ stats['het_prof'][b] += 1
+ except Exception as e:
+ sys.stderr.write(f"[stats_to_legacy] WARN analyze_vcf({vcf_file}): skipping record: {e}\n")
except Exception as e:
sys.stderr.write(f"[stats_to_legacy] WARN analyze_vcf({vcf_file}): {e} (Ti/Tv & SNP counts may be partial)\n")
return stats
@@ -509,8 +530,9 @@ def wmean(key, wkey):
tot_after_bases = sum(l['after_bases'] for l in lanes)
r_len = wmean('r1_len', 'after_reads') or 0.0
- data['TRD_OUT'] = str(int(trd)) if trd > 0 else 'NA'
- data['TRM_OUT'] = str(int(trm)) if trm > 0 else 'NA'
+ # inside `if lanes`, so a fastp JSON WAS read: 0 reads is a MEASURED (failed) sample, not a missing one
+ data['TRD_OUT'] = str(int(trd))
+ data['TRM_OUT'] = str(int(trm))
data['RDL_OUT'] = f"{r_len:.1f}"
data['DUP_OUT'] = f"{(wmean('dup', 'before_reads') or 0.0)*100:.2f}"
q20 = wmean('q20', 'after_bases')
@@ -536,8 +558,8 @@ def wmean(key, wkey):
if bam['total'] > 0:
data['PP_OUT'] = f"{(bam['properly_paired'] / bam['total']) * 100:.2f}"
- # Average base quality
- if bam['avg_quality'] > 0:
+ # Average base quality; measured as soon as the SN block reports reads, so a genuine 0 is emitted as 0
+ if bam['total'] > 0:
data['STD_OUT'] = f"{bam['avg_quality']:.1f}"
# Coverage: depth from CIGAR-based mapped bases (more accurate than reads * read_length)
@@ -549,16 +571,20 @@ def wmean(key, wkey):
# Breadth of coverage (% genome covered at >= 1x)
breadth = calc_breadth_of_coverage(args.bam_stats, g_size, min_dp=1)
- if breadth > 0:
+ if g_size > 0: # 0% breadth is a real (failed) sample; NA is for an unknown genome length, as below
data['COV_BREADTH'] = f"{breadth * 100:.2f}"
# Extended coverage metrics from the same COV histogram (no extra tool)
ch = cov_histogram_stats(args.bam_stats, g_size)
if g_size > 0:
- if ch['median_depth'] > 0:
- data['COV_MEDIAN'] = f"{ch['median_depth']:.2f}"
+ # a median depth of 0 is the RIGHT answer when over half the genome is uncovered, so it is emitted
+ # like the two breadths below: guarding it on > 0 made a failed sample look like an unmeasured one
+ data['COV_MEDIAN'] = f"{ch['median_depth']:.2f}"
data['COV_BREADTH5'] = f"{ch['breadth5']:.2f}"
data['COV_BREADTH10'] = f"{ch['breadth10']:.2f}"
+ # evenness (CoV) is UNDEFINED at zero mean depth, and cov_histogram_stats returns 0.0 for both that
+ # and a perfectly flat genome; this guard is the only thing separating them, so it stays (a 0.000
+ # here would read as ideal evenness for a sample with no coverage at all).
if ch['evenness'] > 0:
data['COV_EVENNESS'] = f"{ch['evenness']:.3f}"
@@ -568,11 +594,13 @@ def wmean(key, wkey):
data['MQ0_PCT'] = f"{(bam['reads_mq0'] / bam['total']) * 100:.2f}"
singletons = max(0, bam['mapped'] - bam['mapped_paired']) # approx (no flagstat passed)
data['SINGLE_PCT'] = f"{(singletons / bam['total']) * 100:.2f}"
+ # samtools reports a 0 mean insert size when there is no paired insert at all (single-end library or no
+ # proper pairs) -> not applicable, keep NA. A 0 sd under a REAL mean is measured (a degenerate library of
+ # identical inserts), so the sd follows the mean instead of guarding itself away.
if bam['insert_size_avg'] > 0:
data['ISIZE_MEAN'] = f"{bam['insert_size_avg']:.1f}"
- if bam['insert_size_sd'] > 0:
data['ISIZE_SD'] = f"{bam['insert_size_sd']:.1f}"
- if bam['error_rate'] > 0:
+ if bam['bases_mapped'] > 0: # the error rate's own denominator: 0 mismatches over mapped bases is a real 0
data['ERR_RATE'] = f"{bam['error_rate']*100:.3f}" # emit as a percent
# 3. Variants
diff --git a/bin/vcf_filter_rules.py b/bin/vcf_filter_rules.py
new file mode 100755
index 0000000..ff017fa
--- /dev/null
+++ b/bin/vcf_filter_rules.py
@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""Print the bcftools filter expression that defines a homozygous or heterozygous SNP call.
+
+ vcf_filter_rules.py hom --min-dp 30 --min-alt-fwd 2 --min-alt-rev 2 \
+ --hom-threshold 0.90 --het-min-frac 0.10
+
+This is the scientific definition of a call, not plumbing: which depth and strand support a site
+needs, and where the allele-fraction boundary between heterozygous and homozygous sits. It lived
+inline in two FreeBayes processes, written out twice, which is how the two drift apart. Keeping it
+here means one definition and tests/unit/test_vcf_filter_rules.py covering it.
+
+The expressions are consumed as `bcftools view -i ""`.
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+# Depth may be recorded per-sample or per-site depending on the caller, so either satisfies it.
+# SAF/SAR are the forward and reverse alt-supporting reads: requiring both rejects a variant seen
+# on one strand only, the classic short-read artefact.
+BASE = ("(FMT/DP >= {min_dp} || INFO/DP >= {min_dp})"
+ " && INFO/SAF[0] >= {min_alt_fwd} && INFO/SAR[0] >= {min_alt_rev}")
+
+# Guard the division: a site with neither ref nor alt observations has no defined allele fraction.
+AF_GUARD = "(INFO/AO[0] + INFO/RO) > 0"
+AF_CALC = "((INFO/AO[0] * 1.0) / (INFO/AO[0] + INFO/RO))"
+
+
+def base_filter(min_dp, min_alt_fwd, min_alt_rev) -> str:
+ return BASE.format(min_dp=min_dp, min_alt_fwd=min_alt_fwd, min_alt_rev=min_alt_rev)
+
+
+def hom_rule(min_dp, min_alt_fwd, min_alt_rev, hom_threshold) -> str:
+ """Alt allele at or above the homozygous fraction."""
+ return (f"{base_filter(min_dp, min_alt_fwd, min_alt_rev)} && {AF_GUARD}"
+ f" && {AF_CALC} >= {hom_threshold}")
+
+
+def het_rule(min_dp, min_alt_fwd, min_alt_rev, hom_threshold, het_min_frac) -> str:
+ """Alt allele between the heterozygous floor and the homozygous threshold, upper bound exclusive."""
+ return (f"{base_filter(min_dp, min_alt_fwd, min_alt_rev)} && {AF_GUARD}"
+ f" && {AF_CALC} >= {het_min_frac} && {AF_CALC} < {hom_threshold}")
+
+
+def parse_args(argv=None):
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("rule", choices=["hom", "het", "either", "base"])
+ p.add_argument("--min-dp", required=True)
+ p.add_argument("--min-alt-fwd", required=True)
+ p.add_argument("--min-alt-rev", required=True)
+ p.add_argument("--hom-threshold", required=True)
+ p.add_argument("--het-min-frac", required=True)
+ return p.parse_args(argv)
+
+
+def main(argv=None) -> int:
+ a = parse_args(argv)
+ hom = hom_rule(a.min_dp, a.min_alt_fwd, a.min_alt_rev, a.hom_threshold)
+ het = het_rule(a.min_dp, a.min_alt_fwd, a.min_alt_rev, a.hom_threshold, a.het_min_frac)
+ out = {
+ "hom": hom,
+ "het": het,
+ "either": f"{hom} || {het}",
+ "base": base_filter(a.min_dp, a.min_alt_fwd, a.min_alt_rev),
+ }[a.rule]
+ sys.stdout.write(out + "\n")
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/conf/README.md b/conf/README.md
index c3c5442..e6887ea 100644
--- a/conf/README.md
+++ b/conf/README.md
@@ -1,21 +1,40 @@
-# `conf/` — ready-made example configs
+# `conf/` - ready-made configs
+
+## Site configs (applied by a profile)
+
+- **[`garnatxa.config`](garnatxa.config)** - the I2SysBio / University of Valencia cluster, applied by
+ `-profile garnatxa`: SLURM QoS per step, `module load singularity`, the Singularity bind paths, the
+ shared Kraken2 database, a reusable mappability cache and a work directory on scratch. Nothing to pass
+ with `-c`, and it works when the pipeline is pulled straight from GitHub.
+
+ ```bash
+ nextflow run PathoGenOmics-Lab/BAMpiro --tsv samples.tsv --outdir results -profile garnatxa
+ ```
+
+ On any other cluster, start from the generic `-profile slurm` and copy this file as the template for
+ your own site config. `test.config` / `test_full.config` back `-profile test` / `-profile test_full`.
+
+## Organism configs (applied with `-c`)
Layer either on top of BAMpiro's defaults with `-c`:
-- **[`tuberculosis.config`](tuberculosis.config)** — the recommended *M. tuberculosis* run (the **default**
- organism): turns on the full MTBC feature set — lineage + drug-resistance typing, dual H37Rv amino-acid
- numbering, the H37Rv coordinate liftover, and blind-spot masking.
+- **[`tuberculosis.config`](tuberculosis.config)** - the recommended *M. tuberculosis* run (the **default**
+ organism): turns on the full MTBC feature set, that is lineage + drug-resistance typing, dual H37Rv
+ amino-acid numbering, the H37Rv coordinate liftover, and blind-spot masking.
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard -c conf/tuberculosis.config
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker -c conf/tuberculosis.config
```
-- **[`organism.config`](organism.config)** — a commented, copy-and-edit template for **any other organism**:
+- **[`organism.config`](organism.config)** - a commented, copy-and-edit template for **any other organism**:
ploidy 1, the MTBC-only features off, and placeholders for a Kraken DB / canonical numbering / gate
thresholds. Pair it with the built-in `generic` profile.
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard,generic -c my_organism.config
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker,generic -c my_organism.config
```
+Swap `local,docker` for the executor profile you actually want (`slurm`, `garnatxa`, …); profiles compose
+with a comma.
+
See [docs/configuration.md → Example configs](../docs/configuration.md#example-configs) for the details.
diff --git a/conf/garnatxa.config b/conf/garnatxa.config
new file mode 100644
index 0000000..2db996e
--- /dev/null
+++ b/conf/garnatxa.config
@@ -0,0 +1,60 @@
+/*
+ * ============================================================================
+ * BAMpiro - site configuration for GARNATXA (SLURM, I2SysBio / University of Valencia)
+ * ============================================================================
+ * Applied by `-profile garnatxa`, so a run on that cluster needs nothing else:
+ *
+ * nextflow run PathoGenOmics-Lab/BAMpiro -r 1.0.2 \
+ * --tsv samples.tsv --outdir results -profile garnatxa
+ *
+ * Combine it with the option profiles as usual, e.g. `-profile garnatxa,generic`
+ * for a non-tuberculosis organism.
+ *
+ * Everything here is specific to one cluster: its paths, its QoS names and its
+ * module system. Use it as the template for your own site, and note that only the
+ * DRIVER needs a long-running allocation; the values below apply to the tasks.
+ * ============================================================================
+ */
+
+process {
+ executor = 'slurm'
+ clusterOptions = '--qos=short'
+ beforeScript = 'module load singularity'
+
+ // Steps that outgrow the short QoS. Adjust the QoS names to whatever
+ // `sacctmgr show qos format=name,maxwall` reports for your account.
+ withName: 'MERGE_AND_MARKDUP|CALL_FREEBAYES|KRAKEN_FILTER_.*' {
+ clusterOptions = '--qos=medium'
+ }
+
+ // Local scratch on the compute nodes is much faster than the shared filesystem;
+ // Nextflow stages the outputs back to the work directory on completion.
+ scratch = true
+}
+
+executor {
+ queueSize = 200 // concurrent jobs from one run
+ submitRateLimit = '50/1min'
+ pollInterval = '30s'
+}
+
+singularity {
+ // A task only sees the host paths bound into it. autoMounts covers the work and launch
+ // directories; everything else (reads, references, the Kraken database, and the pipeline
+ // itself when Nextflow pulls it into $HOME/.nextflow/assets) has to be listed here.
+ // Comma-separated, no spaces, and bind PARENT directories.
+ runOptions = "--bind /scr/ruizro,/storage/shared_datasets,${System.getenv('HOME') ?: '/home'}"
+}
+
+params {
+ // Shared Kraken2 database on this cluster (~80 GB resident, the run's memory ceiling).
+ kraken2_db = '/storage/shared_datasets/kraken2'
+
+ // The per-reference mappability track is computed once and reused. An absolute, shared path
+ // means every future run on this cluster skips that step.
+ mappability_dir = '/scr/ruizro/test_pipes/bampiro/mappability_cache'
+}
+
+// Keep the work directory on scratch: never on $HOME, which is quota-limited and slow, and which
+// -resume needs to stay intact.
+workDir = '/scr/ruizro/test_pipes/bampiro/work'
diff --git a/conf/test.config b/conf/test.config
new file mode 100644
index 0000000..05f271c
--- /dev/null
+++ b/conf/test.config
@@ -0,0 +1,66 @@
+/*
+ * ============================================================================
+ * BAMpiro - minimal test profile
+ * ============================================================================
+ * Runs the bundled fixture cohort (tests/data/: a 5 kb reference, three samples,
+ * one of them single-end and one split over two runs). Intended for `-stub-run`,
+ * which executes each process's `stub:` block instead of the real tool, so the
+ * whole DAG is validated in seconds with no container and no reference data:
+ *
+ * nextflow run main.nf -profile test -stub-run
+ *
+ * Launch it from the repository root: the fixture samplesheet points at the reads
+ * with repository-relative paths. `tests/run_tests.sh` does that for you.
+ *
+ * This profile covers the DEFAULT feature set. For a run that also walks the
+ * optional branches (Kraken, pathotypr, canonical annotation, liftover, virgin
+ * consensus) use `-profile test_full`.
+ * ============================================================================
+ */
+
+params {
+ tsv = "${projectDir}/tests/data/samplesheet.tsv"
+ outdir = "results_test"
+
+ // No Kraken database in the fixtures: main.nf warns and disables the branch.
+ kraken2_db = null
+
+ // The fixture genome is 5 kb at ~30x, far below the production defaults, so the
+ // thresholds are relaxed to keep the QC gate meaningful rather than always red.
+ filter_min_dp = 1
+ allpos_min_cov = 1
+ consensus_min_dp = 1
+ report_depth_min = 1
+ report_gate = false
+
+ // Cap the per-process requests declared in the modules (up to 80 GB for Kraken) so the run
+ // fits a 2-core CI runner instead of queueing forever against a 7 GB host.
+ max_cpus = 2
+ max_memory = '4.GB'
+ max_time = '30.m'
+
+ // The k sweep is a per-reference cost that dominates a tiny test run.
+ genmap_max_k = 100
+ genmap_step = 25
+
+ // BUILD_MAPPABILITY uses storeDir, so a shared cache would make the second run skip it and
+ // the test would stop covering that process. Keep the cache inside the throwaway directory
+ // that tests/run_tests.sh clears, so every run is cold and hermetic.
+ mappability_dir = "${projectDir}/.test_cache/mappability"
+}
+
+process {
+ executor = 'local'
+ beforeScript = ''
+ clusterOptions = ''
+ scratch = false
+}
+
+// A stub run executes plain shell in the task directory; no image needed.
+singularity.enabled = false
+docker.enabled = false
+
+// Keep the fixture run out of the results tree the pipeline reports on.
+timeline.enabled = false
+report.enabled = false
+trace.enabled = false
diff --git a/conf/test_full.config b/conf/test_full.config
new file mode 100644
index 0000000..eaa2f7d
--- /dev/null
+++ b/conf/test_full.config
@@ -0,0 +1,42 @@
+/*
+ * ============================================================================
+ * BAMpiro - test profile with every optional branch ON
+ * ============================================================================
+ * Layered on top of conf/test.config. It switches on the features that are OFF by
+ * default so a single stub run instantiates EVERY process in the pipeline:
+ * Kraken filtering, pathotypr lineage + DR typing, canonical (H37Rv) annotation,
+ * the k-mer coordinate liftover, and the virgin (unmasked) consensus path.
+ *
+ * nextflow run main.nf -profile test_full -stub-run
+ *
+ * The paths below point at fixtures, not at real panels. They exist only so
+ * Nextflow can stage them; a stub run never reads their contents.
+ * ============================================================================
+ */
+
+params {
+ // Kraken: main.nf enables the branch when the DB path exists AND the samplesheet
+ // carries a taxId. The fixture "database" is an empty directory.
+ kraken2_db = "${projectDir}/tests/data/kraken2_db"
+
+ // Pathotypr lineage + drug-resistance typing.
+ run_pathotypr = true
+ pathotypr_ref = "${projectDir}/tests/data/pathotypr/reference.fasta"
+ pathotypr_markers = "${projectDir}/tests/data/pathotypr/lineage_markers.tsv"
+ pathotypr_dr_markers = "${projectDir}/tests/data/pathotypr/dr_markers.tsv"
+
+ // Dual amino-acid numbering and the k-mer coordinate liftover.
+ annotate_canonical = true
+ variant_liftover = true
+ canonical_ref = "${projectDir}/tests/data/pathotypr/reference.fasta"
+
+ // Virgin consensus path: CALL_BACKBONE_RAW, MERGE_VCFS_RAW, CONSENSUS_FASTA_RAW,
+ // and (with virgin_full_freebayes) CALL_FREEBAYES_RAW.
+ dynamic_read_filter = true
+ keep_virgin_consensus = true
+ virgin_full_freebayes = true
+
+ // Blind-spot masking, exercised inside PREPARE_REFERENCE.
+ mask_blindspots = true
+ blindspot_liftover = true
+}
diff --git a/docs/configuration.md b/docs/configuration.md
index 63d1632..f6ed2ba 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -3,26 +3,74 @@
Customize execution by passing parameters on the command line (e.g.
`--threads 16`) or by editing `nextflow.config`.
+`nextflow run main.nf --help` prints every parameter with its default and exits, so the
+table below is never the only source of truth. A parameter BAMpiro does not know is
+**rejected** before any work starts:
+
+```console
+$ nextflow run main.nf --tsv samples.tsv --freebayes_ploydi 1
+Unrecognised parameter(s): --freebayes_ploydi
+ --freebayes_ploydi: did you mean --freebayes_ploidy?
+Run with --help to list every parameter.
+```
+
+Previously a misspelled flag was accepted, ignored, and the run finished with the default.
+
+## Execution profiles
+
+`-profile` decides **where** and **how** tasks execute. Nothing site-specific is baked into
+the defaults, so a fresh clone (or `nextflow run PathoGenOmics-Lab/BAMpiro`) runs correctly
+anywhere. Pick one executor profile and add as many option profiles as you like,
+comma-separated and with no spaces.
+
+| Profile | Sets | Notes |
+| :--- | :--- | :--- |
+| `standard` | `executor = local` | The default, applied when you pass no `-profile`. Runs on the machine you launched from. |
+| `local` | `executor = local` | An explicit alias for `standard`, for a command that says what it does. |
+| `slurm` | `executor = slurm` | Any SLURM cluster. No queue, account or module system assumed; add yours with `-c`. |
+| `garnatxa` | `conf/garnatxa.config` | The I2SysBio / University of Valencia cluster: QoS per step, `module load singularity`, bind paths, the shared Kraken2 DB, a mappability cache and a work directory on scratch. |
+| `docker` | Docker on, Singularity off | Combine it with an executor profile: `-profile local,docker`. |
+| `generic` | ploidy 1, MTBC features off | A non-tuberculosis organism (see [below](#example-configs)). |
+| `test` · `test_full` | `conf/test.config` (+ `test_full`) | The bundled fixture cohort, sized for `nextflow run main.nf -profile test -stub-run`. |
+
+```bash
+-profile local,docker # a laptop, with Docker
+-profile slurm # any SLURM cluster
+-profile garnatxa # the I2SysBio cluster, with its paths already set
+-profile slurm,generic # a SLURM cluster, non-tuberculosis organism
+```
+
+!!! warning "On a cluster, choose the executor profile deliberately"
+
+ Because the default executor is `local`, launching from a cluster login node with no
+ `-profile` runs the **whole pipeline on the login node** instead of submitting it. The
+ startup banner prints the profile, executor, container and Kraken state it resolved:
+ check the `Executor` line before walking away.
+
+Adapting the pipeline to another cluster is covered in
+[Configuring a run](tutorials/configuring-a-run.md#on-an-hpc-cluster-make-your-files-visible-to-the-container),
+with `conf/garnatxa.config` as the worked template.
+
## Feature toggles (what to turn on / off)
Each of these is a boolean you flip on the command line (e.g. `--run_pathotypr true`).
This is the quick on/off overview; every flag also appears with its default in the full
table below.
-**Opt-in — OFF by default, enable with `-- true`:**
+**Opt-in - OFF by default, enable with `-- true`:**
| Flag | Default | Turns on |
| :--- | :---: | :--- |
| `--run_pathotypr` | :octicons-x-circle-fill-16:{ .red } off | Alignment-free lineage + WHO drug-resistance typing ([pathotypr](pathotypr.md)) |
| `--annotate_canonical` | :octicons-x-circle-fill-16:{ .red } off | Dual **amino-acid** numbering (re-annotate vs H37Rv) + Mycobrowser gene links |
-| `--variant_liftover` | :octicons-x-circle-fill-16:{ .red } off | Dual **coordinate** — the H37Rv position per variant (alignment-free k-mer liftover) |
+| `--variant_liftover` | :octicons-x-circle-fill-16:{ .red } off | Dual **coordinate** - the H37Rv position per variant (alignment-free k-mer liftover) |
| `--mask_blindspots` | :octicons-x-circle-fill-16:{ .red } off | Mask the H37Rv Illumina blind-spots ([Zenodo 3701840](https://zenodo.org/records/3701840)) |
| `--output_cram` | :octicons-x-circle-fill-16:{ .red } off | Publish alignments as CRAM instead of BAM |
| `--report_gate` | :octicons-x-circle-fill-16:{ .red } off | Fail the run if any sample is flagged **FAIL** |
| `--publish_prefilter_bam` | :octicons-x-circle-fill-16:{ .red } off | Also publish the pre-filter `final.bam` |
| `--publish_virgin_allpos_vcf` | :octicons-x-circle-fill-16:{ .red } off | Also publish the virgin (`.raw`) per-position VCF |
-**On by default — DISABLE with `-- false`:**
+**On by default - DISABLE with `-- false`:**
| Flag | Default | Turns off |
| :--- | :---: | :--- |
@@ -39,33 +87,35 @@ table below.
## Example configs
-Two documented, ready-to-use configs live in
-[`conf/`](https://github.com/PathoGenOmics-Lab/BAMpiro/tree/indel-mask/conf) — apply either with `-c`:
+Two documented, ready-to-use **organism** configs live in
+[`conf/`](https://github.com/PathoGenOmics-Lab/BAMpiro/tree/indel-mask/conf) - apply either with `-c`.
+(The same directory also holds `garnatxa.config`, the **site** config behind
+`-profile garnatxa`, and the `test*` configs behind `-profile test`.)
=== "M. tuberculosis (default)"
BAMpiro's built-in defaults are already TB-tuned, so a plain run is a valid TB run.
[`conf/tuberculosis.config`](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/indel-mask/conf/tuberculosis.config)
- additionally turns on the **full recommended MTBC feature set** — lineage +
+ additionally turns on the **full recommended MTBC feature set** - lineage +
drug-resistance typing, dual H37Rv amino-acid numbering, the H37Rv coordinate
liftover, and blind-spot masking (all bundled in the container):
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard -c conf/tuberculosis.config
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker -c conf/tuberculosis.config
```
- *(Running with no `-c` is still a valid TB run — it just leaves those optional features off.)*
+ *(Running with no `-c` is still a valid TB run - it just leaves those optional features off.)*
=== "Another organism"
BAMpiro is organism-agnostic, but its defaults are TB-tuned (diploid calling for
- mixed infections + the MTBC-only features). Add the built-in **`generic`** profile —
- it sets **ploidy 1** and forces the MTBC-only features off — and copy the commented
+ mixed infections + the MTBC-only features). Add the built-in **`generic`** profile,
+ which sets **ploidy 1** and forces the MTBC-only features off, and copy the commented
template [`conf/organism.config`](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/indel-mask/conf/organism.config)
for the rest (Kraken DB, a canonical numbering, gate thresholds):
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard,generic -c my_organism.config
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker,generic -c my_organism.config
```
What `generic` changes versus the TB defaults:
@@ -79,7 +129,7 @@ Two documented, ready-to-use configs live in
your genome, set `--kraken2_db` and the `taxId` column for your species, and the
report's TB-only panels (lineage, drug resistance) simply **self-hide** when there is
no such data. To keep a canonical amino-acid numbering for your organism, set
- `--annotate_canonical true` with `--canonical_snpeff_db` / `--canonical_label` — but
+ `--annotate_canonical true` with `--canonical_snpeff_db` / `--canonical_label` - but
note **only the H37Rv SnpEff DB is bundled** in the image, so for another organism
either add its genome to the image and rebuild, or make sure a bare
`canonical_snpeff_db` id can be downloaded by SnpEff at run time (needs internet).
@@ -88,16 +138,19 @@ Two documented, ready-to-use configs live in
| Category | Parameter | Default | Description |
| :--- | :--- | :--- | :--- |
-| **Input/Output** | `--tsv` | `samples_legio.tsv` | Path to the input sample sheet (TSV). |
+| **Input/Output** | `--tsv` | *(required, no default)* | Path to the input sample sheet (TSV). Omitting it stops the run at once with a message naming the expected columns. |
| | `--outdir` | `results_bampiro` | Directory where results will be saved. |
| | `--threads` | `8` | CPUs for the **BWA-MEM2 mapping** steps only; every other process has its own fixed `cpus` (see [Resource requirements](installation.md#resource-requirements)). |
-| | `--container` | `docker://paururo/bampiro:1.0.1` | Container image. The default is the mutable `1.0.1` Docker Hub **tag** (not a digest); re-pin to `docker://paururo/bampiro@sha256:` for byte-exact reproducibility, or override. |
+| | `--container` | `docker://paururo/bampiro@sha256:c3bc851…` | Container image, **pinned by digest** (the 1.0.1 image) so two runs months apart cannot pick up a rebuilt tag. Override it to use your own copy; the resolved value is recorded in the report's provenance footer. See [the container](installation.md#the-container). |
| | `--nested_output` | `true` | Nest per-sample folders (e.g. `MP00091` → `MP/00/09/1`). |
| | `--publish_mode` | `copy` | `copy` duplicates outputs into `outdir`; `link` hardlinks them to the work dir. |
| | `--output_cram` | `false` | Publish the alignment as CRAM (~40-50% smaller) instead of BAM. |
| | `--publish_prefilter_bam` | `false` | Also publish the pre-filter `final.bam`. With the length-aware filter on (`--dynamic_read_filter`, the default), `filtered.bam` is the analysis BAM and `final.bam` is a ~redundant second full BAM, so it is dropped by default. |
| | `--publish_allpos_vcf` | `true` | Publish the masked per-position (all-positions) VCF the consensus is built from. Set `false` to drop it and save disk. |
| | `--publish_virgin_allpos_vcf` | `false` | Also publish the virgin (`.raw`) per-position VCF (needs `--publish_allpos_vcf`); redundant with the virgin consensus, so off by default. |
+| **Resource ceilings** | `--max_cpus` | *(none)* | Cap **every** process request to what the machine can actually give, e.g. `--max_cpus 4 --max_memory 8.GB`. Without a cap the requests stand as written, which is what you want on a cluster but not on a laptop (Kraken2 alone asks for 80 GB). |
+| | `--max_memory` | *(none)* | As above, for memory. Nextflow units: `8.GB`, `16.GB`. |
+| | `--max_time` | *(none)* | As above, for wall time. Nextflow units: `30.m`, `2.h`. |
| **Lineage & DR (Pathotypr)** | `--run_pathotypr` | `false` | Enable alignment-free MTBC lineage + WHO drug-resistance typing. |
| | `--pathotypr_bin` | `pathotypr` | Executable name (on `PATH` inside the container). |
| | `--pathotypr_ref` | `/opt/pathotypr/reference.fasta` | MTBC-ancestor FASTA the markers are defined on (bundled). |
@@ -108,7 +161,7 @@ Two documented, ready-to-use configs live in
| | `--canonical_snpeff_db` | `Mycobacterium_tuberculosis_h37rv` | Canonical snpEff genome for the second annotation. |
| | `--canonical_label` | `H37Rv` | How that numbering is labelled in the report. |
| **Canonical coords & masking** | `--variant_liftover` | `false` | Fill the H37Rv **coordinate** per variant via the alignment-free k-mer liftover (report SNP tables); works for any reference. |
-| | `--canonical_ref` | `/opt/pathotypr/reference.fasta` | FASTA the canonical (H37Rv-colinear) coordinates are defined on — the liftover / blind-spot **target**. |
+| | `--canonical_ref` | `/opt/pathotypr/reference.fasta` | FASTA the canonical (H37Rv-colinear) coordinates are defined on - the liftover / blind-spot **target**. |
| | `--mask_blindspots` | `false` | Add the H37Rv Illumina blind-spots ([Zenodo 3701840](https://zenodo.org/records/3701840)) to the exclusion mask. |
| | `--blindspot_bed` | `assets/H37Rv_blindspots.bed` | The blind-spots BED (H37Rv / NC_000962.3 coordinates). |
| | `--blindspot_liftover` | `true` | Lift the blind-spots onto the run reference (k-mer liftover) so the mask is correct on **any** reference. |
@@ -121,7 +174,7 @@ Two documented, ready-to-use configs live in
| | `--report_dup_max` | `40` | Gate: max duplication %. |
| | `--report_iupac_max` | `2` | Gate: max IUPAC (ambiguous) %. |
| | `--report_mapping_min` | `80` | Gate: min mapped %. |
-| **QC & Filter** | `--kraken2_db` | *(path)* | Path to the Kraken2 database directory. |
+| **QC & Filter** | `--kraken2_db` | *(none)* | Path to a Kraken2 database directory. **No default**: without one the contamination screen is skipped and the report's taxonomy panel hides itself. `-profile garnatxa` sets the shared database on that cluster. |
| | `--kraken_memory_mapping` | `true` | Read the Kraken2 DB via mmap so parallel tasks share it in RAM (large peak-memory saving; slightly slower per task on a cold page cache). |
| | `--fastp_min_length` | `35` | Discard reads shorter than this length. |
| **Mapping/Backbone**| `--allpos_min_cov` | `30` | Minimum coverage to call a site "WT" (otherwise "NC"). |
diff --git a/docs/installation.md b/docs/installation.md
index 7eff3a2..96c2f3c 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -6,28 +6,68 @@
- **Singularity** or **Docker**
- **Java** (version 17 or later, as required by Nextflow >= 24.04.2)
-The pipeline automatically pulls the container `docker://paururo/bampiro:1.0.1`,
-which contains every tool it needs (BWA-MEM2, Samtools, FreeBayes, SnpEff,
-Pathotypr, Python, …). `nextflow.config` references the image by its mutable `1.0.1`
-tag; for byte-for-byte reproducibility re-pin it to the digest the image was built
-from (`docker://paururo/bampiro@sha256:`). Override either with `--container`.
+The pipeline is parsed by both the classic and the strict Nextflow parser, so it runs
+unchanged on 24.04.2 and on 25.10 or newer without setting `NXF_SYNTAX_PARSER=v1`.
+
+A run needs two things: a samplesheet (`--tsv`, required) and a profile that says where to
+execute (`-profile`, defaulting to the current host).
```bash
-nextflow run main.nf --tsv samples.tsv --outdir results_bampiro -profile standard
+nextflow run main.nf --tsv samples.tsv --outdir results_bampiro -profile local,docker
```
See [Quick Start](quickstart.md) for a full example and the
-[samplesheet format](quickstart.md#samplesheet).
+[samplesheet format](quickstart.md#samplesheet). `nextflow run main.nf --help` lists every
+parameter with its default and exits.
+
+## The container
+
+The pipeline pulls `paururo/bampiro` automatically; it contains every tool it needs
+(BWA-MEM2, Samtools, FreeBayes, SnpEff, Pathotypr, Python, …). `nextflow.config` pins the
+image **by digest**, not by a tag:
+
+```groovy
+container = "docker://paururo/bampiro@sha256:c3bc85157c866154490162603ce8cdcffe86e41d3f065064104d6eb875a1cd19"
+```
+
+A tag can be repointed at a rebuilt image, so two runs a month apart could silently use
+different tool versions; a digest cannot. That digest is the 1.0.1 image. Override it with
+`--container`; either way the value actually used is recorded in the report's provenance
+footer.
+
+The same image is published to Docker Hub (`paururo/bampiro`, public) and to GHCR
+(`ghcr.io/pathogenomics-lab/bampiro`, currently private) with the same digest.
+
+## Choosing where it runs
+
+`-profile` decides **where** each task is executed. Nothing site-specific is baked into
+the defaults: with no `-profile` Nextflow applies `standard`, which runs every task on the
+current host, so a fresh clone works anywhere without being edited.
+
+| Profile | Executor | Use it for |
+| :--- | :--- | :--- |
+| `standard` *(applied when you pass no `-profile`)* | `local` | Whatever machine you launched from |
+| `local` | `local` | The same thing, spelled out in the command |
+| `slurm` | `slurm` | Any SLURM cluster. No queue, account or module system assumed |
+| `garnatxa` | `slurm` | The I2SysBio / University of Valencia cluster, configured end to end |
+| `docker` | *(leaves it alone)* | Docker instead of Singularity (combine it: `local,docker`) |
+| `generic` | *(leaves it alone)* | A non-tuberculosis organism (ploidy 1, the MTBC-only features off) |
+| `test` · `test_full` | `local` | The bundled fixture cohort, for `-stub-run` |
+
+Pick one executor profile and add as many option profiles as you like, comma-separated
+and with no spaces: `-profile local,docker`, `-profile slurm,generic`.
-## Running without SLURM
+!!! warning "On a cluster, choose the executor profile deliberately"
-The **default** (`-profile standard`) targets the authors' SLURM cluster — it uses the `slurm` executor,
-`module load singularity`, and site-specific Singularity `--bind` paths. On any other machine those fail
-before the pipeline runs, so use the `local` (and, for Docker, `docker`) profiles, which drop all of that:
+ The default runs on the machine you launched from. On a cluster login node that means
+ the entire pipeline runs **on the login node**: nothing is ever submitted to the
+ scheduler, and your admins will notice. Pass `-profile slurm` (or `-profile garnatxa`)
+ for real work. The startup banner prints the profile, executor, container and Kraken
+ state it resolved, so read the `Executor` line before walking away.
=== "Local (Singularity/Apptainer)"
- On a single machine, no SLURM and no environment-modules:
+ On a single machine, no scheduler and no environment modules:
```bash
nextflow run main.nf --tsv samples.tsv --outdir results -profile local
@@ -39,14 +79,42 @@ before the pipeline runs, so use the `local` (and, for Docker, `docker`) profile
nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker
```
-`local` runs every task on the current host with the executor set to `local` and the cluster-only bind
-paths cleared. Point `--kraken2_db` at your own Kraken2 database — if it lives outside the launch directory,
-Singularity may need it bound explicitly, e.g. `-profile local` plus
-`--kraken2_db /data/kraken2` and `export NXF_SINGULARITY_RUN_OPTIONS="--bind /data/kraken2"`.
+=== "SLURM cluster"
+
+ ```bash
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile slurm
+ ```
+
+ `slurm` submits tasks with `sbatch` and assumes nothing else: no queue, no account, no
+ environment modules. Add your site's QoS / partition and your Singularity bind paths in
+ a small `-c` config, as shown in
+ [Configuring a run](tutorials/configuring-a-run.md#on-an-hpc-cluster-make-your-files-visible-to-the-container).
+
+=== "Garnatxa (I2SysBio)"
+
+ ```bash
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile garnatxa
+ ```
+
+ Everything that cluster needs is already in `conf/garnatxa.config`: the SLURM QoS per
+ step, `module load singularity`, the Singularity bind paths, the shared Kraken2
+ database, a reusable mappability cache and a work directory on scratch. It also works
+ when the pipeline is pulled straight from GitHub
+ (`nextflow run PathoGenOmics-Lab/BAMpiro -profile garnatxa`), so there is nothing to
+ write or copy first.
+
+## Kraken2 is opt-in
+
+`--kraken2_db` has no default: without one the contamination screen is skipped and the
+report's taxonomy panel hides itself. To run it, point `--kraken2_db` at your own database
+(and give the samplesheet a `taxId` column). If the database lives outside the launch
+directory, Singularity needs it bound explicitly, e.g. `--kraken2_db /data/kraken2` plus
+`export NXF_SINGULARITY_RUN_OPTIONS="--bind /data/kraken2"`. `-profile garnatxa` already
+sets both the database and its bind.
## Container contents (software versions)
-The Docker container (`paururo/bampiro:1.0.1`) bundles the following tools:
+The container (`paururo/bampiro`, the 1.0.1 image pinned above) bundles the following tools:
| Tool | Version | Purpose |
| :--- | :--- | :--- |
@@ -80,20 +148,24 @@ run offline and reproducibly.
Nextflow schedules each process with its own CPU / RAM request (retrying with more RAM
on an out-of-memory kill). The peak driver is **Kraken2**, so size your machine / queue
-for it:
+for it whenever you set `--kraken2_db`:
| Step | CPUs | Memory | Notes |
| :--- | :--- | :--- | :--- |
-| **Kraken2** (contamination) | — | **~80 GB** | loads the whole DB; the run's memory ceiling. `--kraken_memory_mapping` (default on) mmaps it so parallel tasks share RAM |
+| **Kraken2** (contamination) | - | **~80 GB** | loads the whole DB; the run's memory ceiling. `--kraken_memory_mapping` (default on) mmaps it so parallel tasks share RAM |
| Mappability track (`genmap`) | 8 | 16 GB | once per reference (cached across runs) |
| Reference prep / SnpEff DB build | 4 | 8 GB | once per reference |
| Mapping (`bwa-mem2`) | `--threads` (8) | scales with genome | the only step `--threads` controls |
| Pathotypr typing | 4 | 8 GB | only with `--run_pathotypr` |
-| Variant calling / consensus / report | 1–4 | 2–4 GB | — |
+| Variant calling / consensus / report | 1–4 | 2–4 GB | - |
!!! warning "Kraken2 sets the memory ceiling (~80 GB)"
Kraken2 loads the entire database into RAM, so it is the single biggest memory
- request in the run — an under-sized host kills it with an **OOM (exit 137)**. On a
- laptop, use a smaller Kraken2 DB (or skip a DB you don't have) so the step fits.
- Disk is roughly a few GB per sample (BAM/CRAM + VCFs + consensus).
+ request in the run - an under-sized host kills it with an **OOM (exit 137)**. On a
+ laptop, use a smaller Kraken2 DB, or leave `--kraken2_db` unset and the whole screen
+ is skipped. Disk is roughly a few GB per sample (BAM/CRAM + VCFs + consensus).
+
+ To make the whole pipeline fit a small machine, cap every request at once with
+ `--max_cpus` / `--max_memory` / `--max_time` (e.g. `--max_cpus 4 --max_memory 8.GB`).
+ Uncapped is the right choice on a cluster, where the per-process numbers are the point.
diff --git a/docs/introduction.md b/docs/introduction.md
index 7a35f7e..72e739b 100644
--- a/docs/introduction.md
+++ b/docs/introduction.md
@@ -32,7 +32,8 @@ sequences, and a consolidated interactive quality-control report.
---
- `FastP` cleaning and `Kraken2` taxonomic contamination checks on every sample.
+ `FastP` cleaning on every sample, plus `Kraken2` taxonomic contamination checks
+ whenever you point `--kraken2_db` at a database.
- :material-target:{ .lg .middle } **Variant calling & backbone**
@@ -46,7 +47,7 @@ sequences, and a consolidated interactive quality-control report.
---
Alignment-free (k-mer) MTBC lineage + WHO drug-resistance genotyping with
- [Pathotypr](pathotypr.md) — reference-agnostic and bundled in the container.
+ [Pathotypr](pathotypr.md) - reference-agnostic and bundled in the container.
- :material-chart-box:{ .lg .middle } **Interactive QC report**
@@ -100,7 +101,8 @@ flowchart TD
```
1. **Reference** - indexing + repeat masking + SnpEff DB building.
-2. **QC** - read validation → Kraken2 (taxonomy) → FastP (trimming).
+2. **QC** - read validation → Kraken2 (taxonomy, only with `--kraken2_db`) → FastP
+ (trimming).
3. **Pathotypr** *(optional)* - alignment-free MTBC lineage (nested sub-lineage)
**and** WHO drug-resistance typing from reads.
4. **Mapping** - `bwa-mem2` alignment → merge runs → mark duplicates (`samtools`) →
diff --git a/docs/outputs.md b/docs/outputs.md
index 6e88e6a..26da852 100644
--- a/docs/outputs.md
+++ b/docs/outputs.md
@@ -54,7 +54,7 @@ results_bampiro/
├── MP00091.LENS.filter_mqc.tsv # -> Length-aware read-filter drop counts (input/kept/dropped; MultiQC table)
├── MP00091.LENS.mask_sites.tsv # -> Specific positions masked due to low confidence
├── MP00091...fastp.html/.json # -> Trimming quality reports
- ├── MP00091...kraken.report # -> Taxonomic classification report
+ ├── MP00091...kraken.report # -> Taxonomic classification report (only with --kraken2_db)
├── MP00091.LENS.snpeff.csv # -> Variant effect statistics
├── MP00091__.dr_mutations.tsv # -> Pathotypr per-sample DR mutations (only if --run_pathotypr)
└── Locus_to_exclude_LENS.txt # -> List of repetitive regions excluded from calling
@@ -72,7 +72,7 @@ live-adjustable inside the HTML report.
| `LOW_DEPTH` | mean depth below | `--report_depth_min` | :octicons-x-circle-fill-16:{ .red } **FAIL** |
| `LOW_BREADTH` | genome breadth below | `--report_breadth_min` | :octicons-x-circle-fill-16:{ .red } **FAIL** |
| `HIGH_MISSING` | missing % above | `--report_missing_max` | :octicons-x-circle-fill-16:{ .red } **FAIL** |
-| `NO_DATA` | no QC metrics for the sample | — | :octicons-x-circle-fill-16:{ .red } **FAIL** |
+| `NO_DATA` | no QC metrics for the sample | - | :octicons-x-circle-fill-16:{ .red } **FAIL** |
| `MAPPING_LOW` | mapped % below | `--report_mapping_min` | :octicons-alert-fill-16:{ .amber } WARN |
| `HIGH_DUP` | duplication % above | `--report_dup_max` | :octicons-alert-fill-16:{ .amber } WARN |
| `HIGH_IUPAC` | ambiguous/IUPAC % above | `--report_iupac_max` | :octicons-alert-fill-16:{ .amber } WARN |
@@ -85,34 +85,63 @@ live-adjustable inside the HTML report.
```text
BAMpiro/
-├── bin/ # Python helpers
+├── main.nf # Entry point: reads the samplesheet, then calls each stage in order
+├── subworkflows/ # One per pipeline stage, with an explicit take/emit
+│ ├── prepare_references.nf # 1. Reference indexing + SnpEff DB
+│ ├── read_qc.nf # 2-3. Read validation, Kraken2 screen, FastP
+│ ├── lineage_typing.nf # 4. Pathotypr lineage + drug resistance
+│ ├── map_reads.nf # 5. Mapping, merge/markdup, length-aware read filter
+│ ├── call_variants.nf # 6. FreeBayes, backbone, merge
+│ ├── make_consensus.nf # 7. Masked and virgin consensus
+│ ├── annotate.nf # 8. SnpEff (main / legacy)
+│ ├── legacy_stats.nf # 9. Per-sample metrics
+│ ├── multiqc_report.nf # 10. MultiQC
+│ └── cohort_report.nf # 11. Cohort summary, SNP matrix, QC report
+├── modules/ # The processes themselves (Nextflow DSL2)
+│ ├── qc.nf # FastP, Kraken, MultiQC, software versions
+│ ├── mapping.nf # BWA-MEM2, MarkDup, length-aware read filter
+│ ├── variants.nf # FreeBayes, Backbone, Merge
+│ ├── annotation.nf # SnpEff (main / legacy / canonical), legacy stats
+│ ├── consensus.nf # Consensus Fasta
+│ ├── report.nf # Cohort summary, SNP matrix, DR collection, QC report
+│ ├── pathotypr.nf # Lineage + drug-resistance typing
+│ ├── reference.nf # Reference Prep
+│ └── utils.nf # Publish-path routing, parameter validation, --help
+├── bin/ # Everything a process actually runs, kept out of the process scripts
+│ ├── qc_report.py # CLI for the interactive self-contained HTML QC report
+│ ├── qcreport/ # The package behind it
+│ │ ├── parsers.py # Read every input file the report consumes
+│ │ ├── metrics.py # Thresholds and the PASS/WARN/FAIL verdict
+│ │ ├── panels.py # SNP dynamics, epistasis, SNP matrix
+│ │ └── render.py # CSS/JS assets and the HTML assembly
│ ├── stats_to_legacy.py # Per-sample metrics -> legacy log (parses the pathotypr lineage call)
│ ├── collect_summary.py # Aggregate per-sample logs -> cohort summary + gene-burden TSVs
│ ├── collect_dr.py # Aggregate pathotypr DR calls -> run drug-resistance TSV
│ ├── pathotypr_liftover.py # Alignment-free k-mer coordinate liftover (mapping ref <-> H37Rv)
│ ├── build_snp_matrix.py # Build the master SNP matrix (site × sample)
-│ ├── qc_report.py # Build the interactive self-contained HTML QC report
│ ├── WGS_fasta_allpos.py # Consensus FASTA from the all-positions VCF
│ ├── build_min_unique_len.py # Per-reference mappability track (genmap)
│ ├── filter_reads_mappability.py # Length-aware read filter
-│ └── extract_kraken_reads.py # Pull reads of a given taxon from Kraken2 output (decontamination)
+│ ├── vcf_filter_rules.py # The bcftools expressions defining a hom / het call
+│ ├── format_snps_for_backbone.awk # Genotype re-validation before the backbone merge
+│ ├── backbone_allpos.awk # mpileup -> one VCF record per reference position
+│ ├── safe_tabix # Index a VCF, tolerating only a genuinely empty one
+│ └── extract_kraken_reads.py # Vendored from KrakenTools (decontamination)
├── assets/
│ ├── mycolorsTB_nature.tsv # Canonical MTBC lineage colour palette (report)
│ ├── H37Rv_blindspots.bed # H37Rv Illumina blind-spots (Zenodo 3701840; --mask_blindspots)
-│ └── H37Rv_blindspots.README.md # Provenance / derivation of the blind-spots BED
-├── modules/ # Nextflow DSL2 Modules
-│ ├── qc.nf # FastP, Kraken, MultiQC, software versions
-│ ├── mapping.nf # BWA-MEM2, MarkDup, length-aware read filter
-│ ├── variants.nf # FreeBayes, Backbone, Merge
-│ ├── annotation.nf # SnpEff (main / legacy / canonical), legacy stats
-│ ├── consensus.nf # Consensus Fasta
-│ ├── report.nf # Cohort summary, SNP matrix, DR collection, QC report
-│ ├── pathotypr.nf # Lineage + drug-resistance typing
-│ ├── reference.nf # Reference Prep
-│ └── utils.nf # Publish-path routing / clean publish dir
+│ ├── H37Rv_blindspots.README.md # Provenance / derivation of the blind-spots BED
+│ └── NO_FILE* # Placeholders for the optional QC report inputs
+├── tests/ # See tests/README.md: unit, front-end and stub-run legs
+├── conf/ # Ready-made configs: sites (garnatxa.config -> -profile garnatxa),
+│ # organisms (tuberculosis.config, organism.config), tests (test*.config)
├── .github/dockerfile/ # Container recipe (bundles pathotypr + Zenodo panels + H37Rv snpEff DB)
-├── conf/ # Ready-made example configs (tuberculosis.config, organism.config)
├── docs/ # This documentation (MkDocs Material; built via mkdocs.yml)
-├── nextflow.config # Global configuration & params
-└── main.nf # Main workflow entry point
+└── nextflow.config # Global configuration & params
```
+
+!!! note "Why the science lives in `bin/`"
+
+ A process script cannot be tested: `-stub-run` replaces it, and nothing can import it. Anything
+ with a decision in it - the hom/het thresholds, the genotype re-validation, the pileup parser -
+ is a file under `bin/` that the process calls, so the test suite can exercise it directly.
diff --git a/docs/qc-report.md b/docs/qc-report.md
index 965f4c5..d0f58ac 100644
--- a/docs/qc-report.md
+++ b/docs/qc-report.md
@@ -8,12 +8,12 @@ by `bin/qc_report.py` and controlled by `--make_qc_report` (default `true`).
!!! tip "See it live"
- Explore a full example report built from a 17-sample demo cohort — every panel
+ Explore a full example report built from a 17-sample demo cohort - every panel
populated, fully interactive (live thresholds, the exclusion basket, dark mode):
[:octicons-play-16: Open the interactive demo report](examples/qc_report_demo.html){ .md-button .md-button--primary target="_blank" rel="noopener" }
- *Synthetic demo data. It opens as a standalone page — the very same self-contained
+ *Synthetic demo data. It opens as a standalone page - the very same self-contained
HTML file each real run produces.*
## Live & interactive
@@ -52,7 +52,7 @@ Grouped as in the sidebar:
| Group | Panels |
| :--- | :--- |
-| **Overview** | **Executive summary** (cohort health & headline findings at a glance — the first panel) · General statistics (value-coloured, sortable, filterable, TSV export) · **Flagged samples** (worst-first, each failing margin shown inline) · Per-lineage summary (canonical *mycolorsTB* palette) · Distributions (beeswarm / bar / histogram) · **Taxonomic composition** (Kraken2: primary taxon / contaminants / unclassified, worst-first) |
+| **Overview** | **Executive summary** (cohort health & headline findings at a glance - the first panel) · General statistics (value-coloured, sortable, filterable, TSV export) · **Flagged samples** (worst-first, each failing margin shown inline) · Per-lineage summary (canonical *mycolorsTB* palette) · Distributions (beeswarm / bar / histogram) · **Taxonomic composition** (Kraken2: primary taxon / contaminants / unclassified, worst-first) |
| **Correlation & structure** | Metric-pair scatter (box-select to basket, with a Spearman *r* + *p* read-out) · Metric correlation heatmap · QC-space PCA (+ most-unusual-samples table) · Divergence vs completeness · **Dose × treatment** (per-group dose distribution + Kruskal–Wallis test) |
| **Genome & genes** | Consensus completeness · Genome landscape (per-position callability / variant heatmap, gene search, mask-region toggle) · Functional annotation (snpEff classes) · Functional gene burden · Variable genes (SNP-density hotspots) |
| **Evolution** | Temporal sampling overview · Selection pN/pS (dN/dS, eskaks) · aDNA damage authentication (mapDamage) |
@@ -60,7 +60,7 @@ Grouped as in the sidebar:
## Optional inputs
-Nothing below is required — each optional input just lights up the matching panel or
+Nothing below is required - each optional input just lights up the matching panel or
feature, and any panel with no data hides itself and its nav link. The pipeline wires
these from the run's own outputs; you only get what you have.
@@ -69,8 +69,8 @@ these from the run's own outputs; you only get what you have.
| `--metadata ` | SNP-dynamics grouping + the SNP-matrix column-header levels (see **Optional metadata** below) |
| `--vcfs ` | Per-SNP allele frequencies → SNP dynamics, epistasis, SNP matrix |
| `--gff ` | Variable-genes hotspots, genome-landscape gene search, and the **gene → Mycobrowser (H37Rv) locus-tag** links |
-| `--dr-report ` | **Drug resistance** panel (WHO-grade sample × drug matrix) — the [pathotypr](pathotypr.md) DR calls |
-| `--kraken ` | Taxonomic composition / contamination panel (Kraken2) |
+| `--dr-report ` | **Drug resistance** panel (WHO-grade sample × drug matrix) - the [pathotypr](pathotypr.md) DR calls |
+| `--kraken ` | Taxonomic composition / contamination panel (Kraken2). The pipeline only has these reports to pass when you set `--kraken2_db`, which has no default, so the panel hides itself otherwise |
| `--gene-burden ` | Functional gene-burden panel |
| `--pnps ` | Selection pN/pS panel ([eskaks](https://github.com/PathoGenOmics-Lab/eskaks)) |
| `--mapdamage-dir ` | aDNA damage-authentication panel (mapDamage) |
@@ -85,7 +85,7 @@ these from the run's own outputs; you only get what you have.
The report reads richer context from the run **samplesheet** (or any TSV passed as
`--metadata`). Every column is optional and matched **by name** (case-insensitive), so
-there is nothing to configure — add a column and the matching panel reacts.
+there is nothing to configure - add a column and the matching panel reacts.
| Column (matched by name) | Examples | What it drives |
| :--- | :--- | :--- |
@@ -95,7 +95,7 @@ there is nothing to configure — add a column and the matching panel reacts.
| **any other column** | `treatment`, `site`, `region`, `ward`, `batch`… | A categorical annotation - becomes a **cohort filter** dropdown in the toolbar (restrict the whole QC view to one value) and a SNP-matrix header level. |
| **dose** | `dose`, `dosis` | A **numeric** column - becomes a first-class metric (selectable on the scatter axes + the correlation matrix, with a Spearman *r* + *p* read-out) and drives the **Dose × treatment** test. |
-**Every** annotation column — including the time and group ones — also becomes a
+**Every** annotation column - including the time and group ones - also becomes a
**column-header level** in the **SNP matrix** (filter the matrix by it, hover a header
for its value); the time and group columns *additionally* drive the dynamics, and each
plain categorical column *additionally* becomes a **cohort filter** in the toolbar. The
@@ -122,9 +122,9 @@ singleton `TB-2020-C` (no series) simply has no dynamics trajectory.
### Dose × treatment and quantitative metadata
A numeric **`dose`** column is treated as a quantitative variable rather than a category.
-It becomes a first-class **metric** — pick it on either **Correlations** scatter axis (or
+It becomes a first-class **metric** - pick it on either **Correlations** scatter axis (or
in the **Metric correlation** matrix) to get a Spearman *r* with a two-sided *p*-value
-against any QC or genomic metric — and it powers a dedicated **Dose × treatment** panel:
+against any QC or genomic metric - and it powers a dedicated **Dose × treatment** panel:
a per-treatment dose distribution (box + points) with a **Kruskal–Wallis** rank test of
whether dose differs across the treatment groups (a Mann–Whitney-equivalent when there
are two groups). The test runs over the whole cohort; groups with fewer than two dosed
@@ -137,7 +137,7 @@ When per-sample **VCFs** are also present, a **Variant × dose** panel (under *V
over time*) runs an association scan: for every variant site it correlates the per-sample
**allele frequency** (0 where the site is reference) with dose across the dosed samples
(**Spearman** ρ + two-sided *p*), applies a **Benjamini–Hochberg FDR** across all tested
-variants, and ranks them — so you can see which mutations track the dose while the
+variants, and ranks them - so you can see which mutations track the dose while the
multiple-testing correction keeps incidental hits in check. Click any row to plot that
variant's allele-frequency-vs-dose scatter, click a column header to re-sort (by ρ, *p*,
*q* or carrier count), search by gene / position / amino acid, and click a point to
diff --git a/docs/quickstart.md b/docs/quickstart.md
index 3752f40..5e15c30 100644
--- a/docs/quickstart.md
+++ b/docs/quickstart.md
@@ -2,33 +2,40 @@
By default BAMpiro assumes *M. tuberculosis* settings (ploidy = 2 to detect mixed
infections). Lineage/DR typing is **off** by default - enable it with
-`--run_pathotypr true`. **Working with a different organism?** Add `-profile
-standard,generic` (ploidy 1 + the MTBC-only features off) — see the
-[example configs](configuration.md#example-configs) (a ready-made TB config and a
-non-TB template).
+`--run_pathotypr true`. **Working with a different organism?** Add `generic` to your
+profile list (ploidy 1 + the MTBC-only features off), e.g. `-profile local,docker,generic`.
+The [example configs](configuration.md#example-configs) have a ready-made TB config and a
+non-TB template.
```bash
nextflow run main.nf \
--tsv samples.tsv \ # (1)!
--outdir results_bampiro \ # (2)!
- -profile standard # (3)!
+ -profile local,docker # (3)!
```
-1. **Required.** Your Tab-Separated samplesheet — one row per `(sample, run, reference)`. There is no usable default.
+1. **Required.** Your Tab-Separated samplesheet, one row per `(sample, run, reference)`. There is no default.
2. Where results are written (the directory is created if it does not exist).
-3. Execution profile. `standard` targets a **SLURM** cluster; use `local` (or `local,docker`) off-cluster.
+3. Execution profile. `local` runs on this machine; use `slurm` on a cluster, or `garnatxa` on the I2SysBio one.
-`--tsv` is **required** — there is no usable default samplesheet, so omitting it fails
-with `Samplesheet not found: samples_legio.tsv` (a leftover placeholder). `-profile
-standard` targets a **SLURM cluster**; on a laptop / VM / non-SLURM host use
-`-profile local` (or `-profile local,docker` for Docker) — see
-[Running without SLURM](installation.md#running-without-slurm).
+`--tsv` is **required**: there is no default samplesheet, so omitting it stops the run
+immediately with a message naming the columns it expects (`sampleId`, `runId`, `r1`, `r2`,
+`refId`, `refFasta`, `refGff`, `taxId`). `-profile` decides where the work lands: with no
+`-profile` everything runs on the **current host**, which on a cluster login node means the
+login node. Pass `-profile slurm` (or `-profile garnatxa`) to submit to the scheduler - see
+[Choosing where it runs](installation.md#choosing-where-it-runs).
+
+Two things the command line does for you:
+
+- `nextflow run main.nf --help` prints every parameter with its default, then exits.
+- A misspelled `--parameter` is **rejected** with a suggestion instead of being silently
+ ignored, so a typo can no longer cost you a run that quietly used the default.
Enable alignment-free lineage / drug-resistance typing and dual amino-acid numbering:
```bash
nextflow run main.nf \
- --tsv samples.tsv --outdir results_bampiro -profile standard \
+ --tsv samples.tsv --outdir results_bampiro -profile local,docker \
--run_pathotypr true --annotate_canonical true
```
@@ -40,7 +47,7 @@ the full result layout and [Configuration](configuration.md) for every parameter
The [Jupyter tutorial](tutorial/bampiro_tutorial.ipynb) runs the same journey
end-to-end and lets you explore an example cohort's outputs with `pandas` /
- `matplotlib` — no pipeline run needed.
+ `matplotlib` - no pipeline run needed.
## Samplesheet
@@ -69,7 +76,7 @@ Reads, `refFasta` and `refGff` may be **plain or gzipped** (`.gz` is auto-detect
### Reference requirements
-- **`refGff` is required** (a mandatory column), even if you don't care about annotation —
+- **`refGff` is required** (a mandatory column), even if you don't care about annotation:
it builds the per-sample SnpEff database. If you truly have no annotation, give a minimal
GFF3 stub: a `##gff-version 3` line plus one `region`/`gene` feature whose seqid matches
the FASTA header.
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 7ae1baf..e2f6ad0 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -1,22 +1,24 @@
# Troubleshooting
When a run fails, Nextflow names the failed process, its exit code, and the task's work
-directory — `cd` there and read `.command.err` / `.command.log` / `.command.sh`. The
+directory - `cd` there and read `.command.err` / `.command.log` / `.command.sh`. The
most common first-run problems:
| Symptom (error) | Cause | Fix |
| :--- | :--- | :--- |
-| `Unknown configuration profile: ''` | that profile isn't defined | the profiles are `standard` (SLURM), `local`, `docker`, `generic` — combine as needed (`-profile local,docker`) |
-| `Cannot run program "sbatch"` / jobs never start | default targets SLURM | you're not on a SLURM cluster — add `-profile local` (see [Running without SLURM](installation.md#running-without-slurm)) |
-| `module: command not found` in `.command.run` | base config runs `module load singularity` | use `-profile local` (or `docker`), which drops the module-load |
-| `FATAL … mount source /scr/… no such file or directory` | the default binds site-specific Singularity paths | use `-profile local` (clears those binds); bind your own paths with `NXF_SINGULARITY_RUN_OPTIONS="--bind /your/path"` |
-| `manifest for paururo/bampiro:1.0.1 not found` / auth denied | the image isn't pullable on your setup | pull it from the registry the maintainers published (Docker Hub or GHCR), or point `--container` at your own copy |
-| Kraken2 task killed (OOM, exit 137) | Kraken2 loads the whole DB — it asks for **~80 GB** | run on a host / partition with enough RAM, or use a smaller DB; `--kraken_memory_mapping true` (default) mmaps the DB so parallel tasks share it |
-| `Samplesheet not found: samples_legio.tsv` | `--tsv` was omitted | `--tsv` is required (no usable default) — pass your samplesheet |
-| `Missing required column(s): refGff …` / SnpEff DB build fails | `refGff` is a **required** samplesheet column, and its seqid must match the FASTA | give every sample a GFF3 whose first column equals the FASTA header id — see [reference requirements](quickstart.md#reference-requirements) |
+| `Unknown configuration profile: ''` | that profile isn't defined | the profiles are `standard` (the default, this host), `local`, `slurm`, `garnatxa`, `docker`, `generic`, `test` - combine as needed (`-profile local,docker`) |
+| The run executes on the login node, nothing reaches the queue | no executor profile was chosen; the default runs on the current host | add `-profile slurm` (or `-profile garnatxa`) - see [Choosing where it runs](installation.md#choosing-where-it-runs). The startup banner's `Executor` line tells you which one you got |
+| `Unrecognised parameter(s): --…` | a misspelled or invented `--flag` | fix the spelling (the error suggests the nearest match); `nextflow run main.nf --help` lists every parameter |
+| `module: command not found` in `.command.run` | a site profile ran `module load singularity`, but this machine has no module system | that comes from `-profile garnatxa`; on another cluster use `-profile slurm` plus your own `-c` config |
+| `FATAL … mount source /scr/… no such file or directory` | `-profile garnatxa` binds paths that exist only on that cluster | use `-profile slurm` and bind your own paths (`-c` config, or `NXF_SINGULARITY_RUN_OPTIONS="--bind /your/path"`) |
+| `manifest for paururo/bampiro@sha256:… not found` / auth denied | the pinned image isn't pullable on your setup | pull it from the registry the maintainers published (Docker Hub or GHCR), or point `--container` at your own copy |
+| Kraken2 task killed (OOM, exit 137) | Kraken2 loads the whole DB - it asks for **~80 GB** | run on a host / partition with enough RAM, use a smaller DB, or drop `--kraken2_db` and skip the screen; `--kraken_memory_mapping true` (default) mmaps the DB so parallel tasks share it |
+| `--tsv is required: a Tab-Separated samplesheet …` | `--tsv` was omitted | `--tsv` has no default - pass your samplesheet (the message lists the columns) |
+| `Missing required column(s): refGff …` / SnpEff DB build fails | `refGff` is a **required** samplesheet column, and its seqid must match the FASTA | give every sample a GFF3 whose first column equals the FASTA header id - see [reference requirements](quickstart.md#reference-requirements) |
| Report has no **lineage** / **drug-resistance** panel | those need `--run_pathotypr true` | expected without pathotypr; panels self-hide when there is no lineage/DR data |
+| Report has no **taxonomic composition** panel | `--kraken2_db` has no default, so the screen was skipped | point `--kraken2_db` at a database (and give the samplesheet a `taxId` column) |
| Report has no **SNP dynamics** / **epistasis** panel | need per-sample VCFs + `--metadata` with a time+group column | add the metadata columns (see [Optional metadata](qc-report.md#optional-metadata)) |
-| A sample failed with `SLURM … TIMEOUT` | the task exceeded the queue time limit | raise the time / use a longer `--qos` / partition for that step |
+| A sample failed with `SLURM … TIMEOUT` | the task exceeded the queue time limit | give that step a longer QoS / partition through `process.clusterOptions` in your own `-c` config; `conf/garnatxa.config` shows the pattern (a `withName:` block for the slow steps) |
!!! question "Still stuck?"
diff --git a/docs/tutorial/bampiro_tutorial.ipynb b/docs/tutorial/bampiro_tutorial.ipynb
index a27ffe5..89856c6 100644
--- a/docs/tutorial/bampiro_tutorial.ipynb
+++ b/docs/tutorial/bampiro_tutorial.ipynb
@@ -45,8 +45,8 @@
"cd BAMpiro\n",
"```\n",
"\n",
- "The container is referenced by tag (`docker://paururo/bampiro:1.0.1`); override with `--container`,\n",
- "or re-pin it to a `@sha256:` digest for byte-exact reproducibility. See\n",
+ "The container is pinned **by digest** (`docker://paururo/bampiro@sha256:c3bc851…`), not by a tag, so a\n",
+ "run repeated months later uses the same binaries. Override it with `--container`. See\n",
"[Installation](../installation.md)."
]
},
@@ -229,39 +229,49 @@
"\n",
"```bash\n",
"nextflow run main.nf \\\n",
- " --tsv samples.tsv --outdir results_bampiro -profile standard \\\n",
+ " --tsv samples.tsv --outdir results_bampiro -profile slurm \\\n",
" --run_pathotypr true --annotate_canonical true --variant_liftover true\n",
"```\n",
"\n",
+ "`-profile` says where the work runs: `local` (or `local,docker`) on one machine, `slurm` on any SLURM\n",
+ "cluster, `garnatxa` on the I2SysBio one. With no `-profile` everything runs on the current host, which\n",
+ "on a login node means the login node, so choose one deliberately there.\n",
+ "\n",
"### How to set parameters\n",
"\n",
"Two equivalent ways, no code changes needed:\n",
"\n",
- "- **On the command line** — `-- ` (e.g. `--freebayes_ploidy 1 --report_depth_min 20`).\n",
+ "- **On the command line**: `-- ` (e.g. `--freebayes_ploidy 1 --report_depth_min 20`).\n",
" Overrides everything; best for one-off runs.\n",
- "- **In `nextflow.config`** — edit the `params { … }` block for your lab's persistent defaults.\n",
+ "- **In `nextflow.config`**: edit the `params { … }` block for your lab's persistent defaults.\n",
+ "\n",
+ "`nextflow run main.nf --help` lists every parameter with its default, and a `--flag` BAMpiro does not\n",
+ "know stops the run with a suggestion instead of being silently ignored.\n",
"\n",
"### What to turn on / off\n",
"\n",
- "Most features are booleans. **Opt-in — OFF by default, enable with `-- true`:**\n",
+ "Most features are booleans. **Opt-in, OFF by default, enable with `-- true`:**\n",
"\n",
"| Flag | Turns on |\n",
"| :--- | :--- |\n",
"| `--run_pathotypr` | alignment-free lineage + WHO drug-resistance typing |\n",
"| `--annotate_canonical` | dual **amino-acid** numbering (H37Rv) + Mycobrowser gene links |\n",
- "| `--variant_liftover` | dual **coordinate** — the H37Rv position per variant (k-mer liftover) |\n",
+ "| `--variant_liftover` | dual **coordinate**: the H37Rv position per variant (k-mer liftover) |\n",
"| `--mask_blindspots` | mask the H37Rv Illumina blind-spots (Zenodo 3701840) |\n",
"| `--output_cram` | publish CRAM (~40-50% smaller) instead of BAM |\n",
"| `--report_gate` | fail the run if any sample is flagged FAIL |\n",
"\n",
- "**On by default — DISABLE with `-- false`:** `--make_qc_report`, `--make_snp_matrix`,\n",
+ "**On by default, DISABLE with `-- false`:** `--make_qc_report`, `--make_snp_matrix`,\n",
"`--make_consensus`, `--exclude_repeats`, `--dynamic_read_filter`, `--annotate_main_vcf`,\n",
"`--annotate_legacy_vcfs`, `--nested_output`, `--blindspot_liftover`.\n",
"\n",
+ "`--kraken2_db` has no default: without one the contamination screen is skipped and the report's\n",
+ "taxonomy panel hides itself.\n",
+ "\n",
"### QC gate thresholds\n",
"\n",
"The PASS / WARN / FAIL verdicts (and, with `--report_gate`, whether the run fails) come from a handful\n",
- "of cut-offs — you can set them at launch, and they are also **live-adjustable inside the HTML report**\n",
+ "of cut-offs. You can set them at launch, and they are also **live-adjustable inside the HTML report**\n",
"(drag a slider and the whole cohort re-flags instantly, no re-run):\n",
"\n",
"| Parameter | Default | Gate |\n",
@@ -946,7 +956,6 @@
}
],
"source": [
- "import numpy as np\n",
"snpm = pd.read_csv('example_outputs/samples_demo_snp_matrix.tsv', sep='\\t')\n",
"af = snpm[[c for c in snpm.columns if c.endswith('|AF')]]\n",
"af.columns = [c.replace('|AF', '') for c in af.columns]\n",
diff --git a/docs/tutorials/configuring-a-run.md b/docs/tutorials/configuring-a-run.md
index a5b149a..79921a9 100644
--- a/docs/tutorials/configuring-a-run.md
+++ b/docs/tutorials/configuring-a-run.md
@@ -2,11 +2,11 @@
By now you've completed a run and built a samplesheet. This page shows you how to point BAMpiro at *your* machine and *your* organism: picking an execution profile, setting parameters, and switching on the optional MTBC features. Budget about 15 minutes to read and adapt to your setup.
-For the exhaustive parameter list (every flag, its default and description), keep [Configuration](../configuration.md) open alongside this page — here we cover only the choices that matter most for a first tuning.
+For the exhaustive parameter list (every flag, its default and description), keep [Configuration](../configuration.md) open alongside this page - here we cover only the choices that matter most for a first tuning.
## 1. Choose an execution profile
-A `-profile` tells Nextflow *where* and *how* to run each task. The default (`standard`) targets the authors' SLURM cluster, so on any other machine you'll want a different one.
+A `-profile` tells Nextflow *where* and *how* to run each task. Nothing site-specific is baked into the defaults, so the choice is entirely yours: with no `-profile`, Nextflow applies `standard`, which runs everything on the machine you launched from.
=== "Laptop / workstation (Docker)"
@@ -20,41 +20,94 @@ A `-profile` tells Nextflow *where* and *how* to run each task. The default (`st
nextflow run main.nf --tsv samples.tsv --outdir results -profile local
```
-=== "SLURM cluster (default)"
+=== "SLURM cluster"
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile slurm
```
-`local` runs every task on the current host and clears the cluster-only `module load` and Singularity bind paths; add `docker` if you'd rather use Docker than Singularity. Profiles compose with a comma (e.g. `local,docker`).
+ Submits every task with `sbatch`. It assumes nothing about your site: no queue, no account, no environment modules. Add those in a small `-c` config, [as shown below](#on-an-hpc-cluster-make-your-files-visible-to-the-container).
+
+=== "Garnatxa (I2SysBio)"
+
+ ```bash
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile garnatxa
+ ```
+
+ That cluster's QoS, module load, bind paths, Kraken2 database and scratch work directory are already set. Nothing else to write.
+
+`local` runs every task on the current host; add `docker` if you'd rather use Docker than Singularity. Profiles compose with a comma and no spaces (e.g. `local,docker`, `slurm,generic`).
+
+!!! warning "The default runs where you launched it"
+
+ Because `standard` means "this host", launching from a cluster **login node** with no `-profile` runs the entire pipeline on the login node: nothing is ever submitted to the scheduler, and it will be noticed. Pick `slurm` or `garnatxa` deliberately for real work.
+
+ You do not have to guess which one you got. The startup banner prints the resolved profile, executor, container and Kraken state:
+
+ ```text
+ Profile(s) : slurm
+ Executor : slurm
+ Container : docker://paururo/bampiro@sha256:c3bc851…
+ Kraken2 : disabled
+ ```
+
+ `Kraken2` reads `disabled` until you point `--kraken2_db` at a database, which has no default. While it is disabled the contamination screen is skipped and the report's taxonomy panel hides itself.
!!! warning "Kraken2 sets the memory ceiling (~80 GB)"
- Kraken2 loads its whole database into RAM and is the single biggest request in the run — an under-sized host kills it with an OOM (exit 137). On a laptop, point `--kraken2_db` at a smaller database. See [Resource requirements](../installation.md#resource-requirements).
+ Kraken2 loads its whole database into RAM and is the single biggest request in the run - an under-sized host kills it with an OOM (exit 137). On a laptop, point `--kraken2_db` at a smaller database, or leave it unset and the screen is skipped entirely. To make every step fit a small machine in one go, cap the requests with `--max_cpus 4 --max_memory 8.GB`. See [Resource requirements](../installation.md#resource-requirements).
### On an HPC cluster: make your files visible to the container
-On a cluster BAMpiro runs each task through **Singularity / Apptainer**, and a container only sees the host paths that are **bound** into it. Nextflow auto-mounts the work and launch directories, but your **reads, references and Kraken2 database usually live elsewhere on a shared filesystem** — so the container can't read them and tasks fail with *"No such file or directory"* even though the paths are correct on the login node.
+On a cluster BAMpiro runs each task through **Singularity / Apptainer**, and a container only sees the host paths that are **bound** into it. Nextflow auto-mounts the work and launch directories, but your **reads, references and Kraken2 database usually live elsewhere on a shared filesystem** - so the container can't read them and tasks fail with *"No such file or directory"* even though the paths are correct on the login node.
-The default `standard` profile carries the *authors'* site-specific bind paths (e.g. `/scr/…`, `/storage/…/kraken2`) and a `--qos=short` SLURM option — none of which exist on your cluster. Replace them with **your** paths in a small config and pass it with `-c`:
+Binds are therefore per-site, like the queue names and the way Singularity is provided. BAMpiro keeps all of that in **site profiles** instead of in the defaults, so there are two cases.
-```groovy title="cluster.config"
-singularity {
- // every host directory your samplesheet paths + --kraken2_db sit under,
- // comma-separated, no spaces (parent dirs are fine — /data covers /data/*):
- runOptions = '--bind /scratch/me/reads,/data/refs,/shared/kraken2'
-}
+=== "Garnatxa (I2SysBio)"
-process {
- executor = 'slurm' // your scheduler
- clusterOptions = '--qos=normal' // your QoS / partition / account
- beforeScript = 'module load singularity' // however your site provides Singularity
-}
-```
+ Nothing to write. `-profile garnatxa` applies [`conf/garnatxa.config`](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/indel-mask/conf/garnatxa.config), which already carries that cluster's bind paths, its `--qos` per step, `module load singularity`, the shared Kraken2 database, a reusable mappability cache and a work directory on scratch:
-```bash
-nextflow run main.nf --tsv samples.tsv --outdir results -profile standard -c cluster.config
-```
+ ```bash
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile garnatxa
+ ```
+
+ It also works when the pipeline is pulled straight from GitHub, with no clone to configure:
+
+ ```bash
+ nextflow run PathoGenOmics-Lab/BAMpiro --tsv samples.tsv --outdir results -profile garnatxa
+ ```
+
+=== "Any other cluster"
+
+ Start from `-profile slurm`, which submits with `sbatch` and assumes nothing else, and add your site's specifics in a small config passed with `-c`. Copy `conf/garnatxa.config` as the template and replace the paths and queue names with yours:
+
+ ```groovy title="my_site.config"
+ process {
+ clusterOptions = '--qos=normal' // your QoS / partition / account
+ beforeScript = 'module load singularity' // however your site provides Singularity
+
+ // the steps that outgrow a short queue
+ withName: 'MERGE_AND_MARKDUP|CALL_FREEBAYES|KRAKEN_FILTER_.*' {
+ clusterOptions = '--qos=long'
+ }
+ }
+
+ singularity {
+ // every host directory your samplesheet paths + --kraken2_db sit under,
+ // comma-separated, no spaces (parent dirs are fine: /data covers /data/*).
+ // Add $HOME too if you let Nextflow pull the pipeline into ~/.nextflow/assets.
+ runOptions = "--bind /scratch/me/reads,/data/refs,/shared/kraken2,${System.getenv('HOME') ?: '/home'}"
+ }
+
+ // keep work/ off a quota-limited home; -resume needs it to survive
+ workDir = '/scratch/me/bampiro/work'
+ ```
+
+ ```bash
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile slurm -c my_site.config
+ ```
+
+ If your site is one other people use, that file is worth contributing back as a profile of its own.
!!! tip "Binding without a config file"
@@ -64,7 +117,7 @@ nextflow run main.nf --tsv samples.tsv --outdir results -profile standard -c clu
export NXF_SINGULARITY_RUN_OPTIONS="--bind /scratch/me/reads,/data/refs,/shared/kraken2"
```
- Bind **parent** directories so every file underneath is visible. See [Installation](../installation.md) for the site-specific defaults you are overriding.
+ Bind **parent** directories so every file underneath is visible.
## 2. Set parameters
@@ -75,7 +128,22 @@ nextflow run main.nf --tsv samples.tsv --outdir results \
-profile local,docker --threads 16 --freebayes_ploidy 1
```
-For anything you want to reuse, collect your overrides in a config file and pass it with `-c` — command-line `--flags` still win over the file:
+To see what is available, ask the pipeline rather than the docs:
+
+```bash
+nextflow run main.nf --help # every parameter, with its default, then exits
+```
+
+Typos are caught before any work starts. A `--flag` BAMpiro does not know stops the run with the nearest match instead of being silently ignored:
+
+```console
+$ nextflow run main.nf --tsv samples.tsv --freebayes_ploydi 1
+Unrecognised parameter(s): --freebayes_ploydi
+ --freebayes_ploydi: did you mean --freebayes_ploidy?
+Run with --help to list every parameter.
+```
+
+For anything you want to reuse, collect your overrides in a config file and pass it with `-c` - command-line `--flags` still win over the file:
```groovy title="my_run.config"
params {
@@ -106,7 +174,7 @@ The `generic` profile sets `freebayes_ploidy 1` (a clonal bacterium is haploid)
!!! tip "Not working with TB?"
The dedicated tutorial [**Running a non-TB organism**](other-organisms.md) walks through
- the whole thing end to end — your reference, the `generic` profile, keeping the typing off,
+ the whole thing end to end - your reference, the `generic` profile, keeping the typing off,
tuning thresholds, and a worked *E. coli* example.
## 4. Turn on the MTBC features
@@ -119,7 +187,7 @@ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker \
```
- `--run_pathotypr true` enables alignment-free MTBC lineage + WHO drug-resistance typing ([Lineage & Drug-Resistance Typing](../pathotypr.md)).
-- `--annotate_canonical true` adds dual **amino-acid** numbering — re-annotating variants against H37Rv, with Mycobrowser gene links.
+- `--annotate_canonical true` adds dual **amino-acid** numbering - re-annotating variants against H37Rv, with Mycobrowser gene links.
!!! tip "Or use the ready-made TB config"
@@ -127,7 +195,7 @@ nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker \
## 5. A word on the QC gate
-Two things are easy to confuse. The interactive QC report lets you drag the cut-offs and re-flags samples **live** in your browser — that never changes any file. The pipeline's own pass/fail **gate** is separate: it uses the `--report_*` thresholds from your config, and only stops the run when you opt in with `--report_gate true`.
+Two things are easy to confuse. The interactive QC report lets you drag the cut-offs and re-flags samples **live** in your browser - that never changes any file. The pipeline's own pass/fail **gate** is separate: it uses the `--report_*` thresholds from your config, and only stops the run when you opt in with `--report_gate true`.
```bash
# Fail the whole run if any sample is flagged FAIL, using a stricter depth cut-off
@@ -140,16 +208,16 @@ The gate thresholds (`--report_depth_min`, `--report_breadth_min`, `--report_mis
Runs are long, and clusters kill jobs. Nextflow caches every finished task in the `work/` directory, so you rarely have to start over.
-- **Resume** — if a run fails or you cancel it, re-run the **exact same command** with `-resume` added. Nextflow reuses the completed tasks and only recomputes what's actually needed:
+- **Resume** - if a run fails or you cancel it, re-run the **exact same command** with `-resume` added. Nextflow reuses the completed tasks and only recomputes what's actually needed:
```bash
nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker -resume
```
-- **Execution reports** — add these Nextflow flags to profile the run (handy for right-sizing CPU / memory / time requests on a cluster):
+- **Execution reports** - every run already writes a resource-usage report, a per-task trace and a timeline under `/pipeline_info/`; they are what you need to right-size CPU / memory / time requests on a cluster. To send them somewhere else, name them yourself:
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard \
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile slurm \
-with-report report.html -with-trace trace.txt -with-timeline timeline.html
```
@@ -157,7 +225,7 @@ Runs are long, and clusters kill jobs. Nextflow caches every finished task in th
!!! note "Keep `work/`… until you're done"
- `-resume` only works while the `work/` directory survives — clean it (`nextflow clean` or `rm -rf work`) **after** a run finishes and you've collected your `--outdir` results, not before.
+ `-resume` only works while the `work/` directory survives - clean it (`nextflow clean` or `rm -rf work`) **after** a run finishes and you've collected your `--outdir` results, not before.
## What's next
diff --git a/docs/tutorials/first-run.md b/docs/tutorials/first-run.md
index 7a49e2a..aa219fc 100644
--- a/docs/tutorials/first-run.md
+++ b/docs/tutorials/first-run.md
@@ -6,14 +6,14 @@ By the end you'll have launched BAMpiro on a tiny samplesheet and opened the sel
## Before you start
-You only need two things on your machine — the container carries every bioinformatics tool (BWA-MEM2, Samtools, FreeBayes, SnpEff, Pathotypr, …), so there is nothing else to install.
+You only need two things on your machine - the container carries every bioinformatics tool (BWA-MEM2, Samtools, FreeBayes, SnpEff, Pathotypr, …), so there is nothing else to install.
- **Nextflow** `>= 24.04.2` (which needs **Java 17+**)
- **Docker** *or* **Singularity / Apptainer**
!!! tip "Installing Nextflow in a conda environment"
- The cleanest way to get Nextflow — **and the Java it needs** — without touching your system is a
+ The cleanest way to get Nextflow - **and the Java it needs** - without touching your system is a
dedicated **conda / mamba** environment:
```bash
@@ -28,10 +28,12 @@ You only need two things on your machine — the container carries every bioinfo
!!! note "The container does the heavy lifting"
- On the first run, BAMpiro automatically pulls its pinned image
- (`docker://paururo/bampiro:1.0.1`) with every tool built in. That download is a
- few GB and happens once — later runs reuse the cached image. Full requirements
- and the bundled software versions are in [Installation](../installation.md).
+ On the first run, BAMpiro automatically pulls its image with every tool built in.
+ The image is pinned **by digest** (`docker://paururo/bampiro@sha256:c3bc851…`), not by
+ a tag, so the run you do today and the one you repeat next year use the same binaries.
+ That download is a few GB and happens once - later runs reuse the cached image. Full
+ requirements and the bundled software versions are in
+ [Installation](../installation.md#the-container).
## 1. Get BAMpiro
@@ -44,24 +46,24 @@ cd BAMpiro
## 2. Prepare a tiny samplesheet
-The only required input is a Tab-Separated samplesheet passed with `--tsv` — there is no default. For your first run, keep it to a single sample. Each row points at your reads and a reference genome + GFF:
+The only required input is a Tab-Separated samplesheet passed with `--tsv` - there is no default. For your first run, keep it to a single sample. Each row points at your reads and a reference genome + GFF:
```tsv
sampleId runId r1 r2 refId refFasta refGff taxId
T00001 RUN1 /data/T1_R1.fq.gz /data/T1_R2.fq.gz H37Rv /refs/tb.fa /refs/tb.gff 1773
```
-Reads, `refFasta` and `refGff` may be plain or gzipped. Don't worry about the finer points yet — the full column reference and multi-run merging are covered in [The samplesheet](samplesheet.md).
+Reads, `refFasta` and `refGff` may be plain or gzipped. Don't worry about the finer points yet - the full column reference and multi-run merging are covered in [The samplesheet](samplesheet.md).
!!! tip "Just want to look around first?"
A bundled **17-sample MTBC demo cohort** and a runnable Jupyter notebook let you
- explore a finished run's outputs with `pandas` / `matplotlib` — no pipeline run
+ explore a finished run's outputs with `pandas` / `matplotlib` - no pipeline run
needed. See the [Quick Start](../quickstart.md) for that shortcut.
## 3. Launch the run
-Run the pipeline, pointing `--tsv` at your samplesheet and `--outdir` at where results should land. Pick the tab for your container engine — both use `-profile local`, which runs everything on the current host:
+Run the pipeline, pointing `--tsv` at your samplesheet and `--outdir` at where results should land. Pick the tab for your container engine - both use `-profile local`, which runs everything on the current host:
=== "Docker"
@@ -81,15 +83,20 @@ Run the pipeline, pointing `--tsv` at your samplesheet and `--outdir` at where r
-profile local
```
-!!! warning "`-profile standard` targets a SLURM cluster"
+!!! warning "On a cluster, say where you want the work to run"
- The default `standard` profile (as shown in the [Quick Start](../quickstart.md))
- uses the `slurm` executor and site-specific paths. On a laptop or any non-SLURM
- host, use `local` (or `local,docker`) as above, or the run fails before it
- starts. See [Running without SLURM](../installation.md#running-without-slurm).
+ `local` is also what you get with no `-profile` at all, because the default profile
+ (`standard`) runs on the machine you launched from. That is what makes a fresh clone
+ work anywhere, but it cuts both ways: launch it from a cluster **login node** and the
+ whole pipeline runs on the login node instead of being submitted to the scheduler.
- Running on **your own cluster**? You'll also need to bind your files into the
- Singularity container so tasks can read them — see
+ On a cluster, choose the executor deliberately: `-profile slurm` for a generic SLURM
+ site, or `-profile garnatxa` on the I2SysBio cluster. The startup banner prints the
+ `Executor` it resolved, so you can confirm it in the first seconds of a run. See
+ [Choosing where it runs](../installation.md#choosing-where-it-runs).
+
+ On **your own cluster** you will also need to bind your files into the Singularity
+ container so tasks can read them - see
[On an HPC cluster](configuring-a-run.md#on-an-hpc-cluster-make-your-files-visible-to-the-container).
Nextflow prints a live table of processes as they run. Behind that table it QCs and trims your reads, screens for contamination, aligns to the reference, masks repeats and low-mappability regions, calls and annotates variants, builds a consensus, and folds everything into one report.
@@ -104,7 +111,7 @@ When the run finishes, everything is under your `--outdir`. The one file to open
results_bampiro/_qc_report.html
```
-Open it in any browser — double-click it, or:
+Open it in any browser - double-click it, or:
```bash
open results_bampiro/*_qc_report.html # macOS (Linux: xdg-open)
@@ -114,4 +121,4 @@ It needs no internet and no server: every panel, threshold and theme toggle is b
## What's next
-You've completed an end-to-end run. Next, learn how to describe **any** cohort — multiple samples, multiple runs, single-end reads and several references — in [**The samplesheet**](samplesheet.md).
+You've completed an end-to-end run. Next, learn how to describe **any** cohort - multiple samples, multiple runs, single-end reads and several references - in [**The samplesheet**](samplesheet.md).
diff --git a/docs/tutorials/other-organisms.md b/docs/tutorials/other-organisms.md
index 2726119..c4c80be 100644
--- a/docs/tutorials/other-organisms.md
+++ b/docs/tutorials/other-organisms.md
@@ -1,8 +1,8 @@
# Running a non-TB organism
-BAMpiro maps, calls variants and builds its QC report for **any** bacterium — only its *defaults* are TB-tuned. This page shows you the handful of switches to flip so a non-tuberculosis organism runs correctly, and which TB-specific features to leave off. Budget about 10 minutes.
+BAMpiro maps, calls variants and builds its QC report for **any** bacterium - only its *defaults* are TB-tuned. This page shows you the handful of switches to flip so a non-tuberculosis organism runs correctly, and which TB-specific features to leave off. Budget about 10 minutes.
-If you have not run the pipeline before, do [Your first run](first-run.md) first — the mechanics are identical; here we only change the reference, one profile and a couple of flags.
+If you have not run the pipeline before, do [Your first run](first-run.md) first - the mechanics are identical; here we only change the reference, one profile and a couple of flags.
## The key idea
@@ -10,7 +10,7 @@ BAMpiro's built-in defaults assume *M. tuberculosis*: **diploid** calling (ploid
## 1. Point the samplesheet at your reference
-Everything organism-specific lives in the samplesheet columns — no code changes. In each row set:
+Everything organism-specific lives in the samplesheet columns - no code changes. In each row set:
- **`refFasta` / `refGff`** to *your* organism's genome and annotation (GFF3),
- **`taxId`** to your species' NCBI TaxID, which drives the Kraken2 contamination screen (e.g. `562` for *E. coli*).
@@ -19,7 +19,7 @@ See [The samplesheet](samplesheet.md) for the full column reference.
!!! warning "The GFF seqid must match the FASTA header"
- `refGff`'s first column (seqid) must be identical to the `refFasta` header id (up to the first whitespace), or the per-sample SnpEff database build fails or annotates nothing. If you have no annotation, pass a minimal GFF3 stub — see [Reference requirements](../quickstart.md#reference-requirements).
+ `refGff`'s first column (seqid) must be identical to the `refFasta` header id (up to the first whitespace), or the per-sample SnpEff database build fails or annotates nothing. If you have no annotation, pass a minimal GFF3 stub - see [Reference requirements](../quickstart.md#reference-requirements).
## 2. Add the `generic` profile
@@ -28,7 +28,7 @@ See [The samplesheet](samplesheet.md) for the full column reference.
=== "SLURM cluster"
```bash
- nextflow run main.nf --tsv samples.tsv --outdir results -profile standard,generic
+ nextflow run main.nf --tsv samples.tsv --outdir results -profile slurm,generic
```
=== "Laptop / workstation (Docker)"
@@ -41,7 +41,7 @@ What `generic` changes versus the TB defaults:
| Setting | TB default | `generic` |
| :--- | :--- | :--- |
-| `freebayes_ploidy` | `2` (diploid — detects mixed infections) | **`1`** (a clonal bacterium is haploid) |
+| `freebayes_ploidy` | `2` (diploid - detects mixed infections) | **`1`** (a clonal bacterium is haploid) |
| `run_pathotypr` · `variant_liftover` · `mask_blindspots` · `annotate_canonical` | optional / off | **forced off** (all MTBC-specific) |
Everything else is already generic. For the full table see [Configuration](../configuration.md).
@@ -50,7 +50,7 @@ Everything else is already generic. For the full table see [Configuration](../co
The `generic` profile already handles this, but it is worth understanding:
-- **Lineage / DR typing** (`--run_pathotypr`) is MTBC-specific and **off by default** — leave it off. See [Lineage & Drug-Resistance Typing](../pathotypr.md).
+- **Lineage / DR typing** (`--run_pathotypr`) is MTBC-specific and **off by default** - leave it off. See [Lineage & Drug-Resistance Typing](../pathotypr.md).
- **`--annotate_canonical`** stays **off**: H37Rv amino-acid numbering is meaningless for a non-MTBC organism. If you *do* want a canonical numbering for your species, set `--annotate_canonical true` with `--canonical_snpeff_db` (and `--canonical_label`) pointing at another SnpEff genome.
!!! note "Only the H37Rv SnpEff DB is bundled"
@@ -59,13 +59,13 @@ The `generic` profile already handles this, but it is worth understanding:
## 4. Tune the QC thresholds
-The pass/fail gate defaults (depth, breadth, missing, and so on) were picked for a ~4.4 Mb TB genome at typical MTBC depths. A different genome size or expected coverage may warrant different cut-offs — adjust the `--report_*` parameters. See [Configuring a run](configuring-a-run.md) for how the gate works and which knobs to turn.
+The pass/fail gate defaults (depth, breadth, missing, and so on) were picked for a ~4.4 Mb TB genome at typical MTBC depths. A different genome size or expected coverage may warrant different cut-offs - adjust the `--report_*` parameters. See [Configuring a run](configuring-a-run.md) for how the gate works and which knobs to turn.
## 5. The QC report is organism-agnostic
-The interactive report still builds in full. The MTBC-only panels (per-lineage summary, drug resistance, dual amino-acid numbering) simply **self-hide** when there is no such data — everything else (depth/breadth, SNP matrix, genome landscape, gene burden, SNP dynamics…) behaves exactly as for TB. See the [Interactive QC Report](../qc-report.md).
+The interactive report still builds in full. The MTBC-only panels (per-lineage summary, drug resistance, dual amino-acid numbering) simply **self-hide** when there is no such data - everything else (depth/breadth, SNP matrix, genome landscape, gene burden, SNP dynamics…) behaves exactly as for TB. See the [Interactive QC Report](../qc-report.md).
-## Worked example — *E. coli* K-12
+## Worked example - *E. coli* K-12
A haploid *E. coli* run on a laptop with Docker, screened against TaxID `562`:
diff --git a/main.nf b/main.nf
index 0591ad6..9d8e3e3 100644
--- a/main.nf
+++ b/main.nf
@@ -7,49 +7,50 @@ nextflow.enable.dsl=2
===============================================================================
*/
+// --- SUB-WORKFLOW IMPORTS ---
+// One sub-workflow per numbered stage; each owns the processes it runs and the channel plumbing
+// that feeds them, so a stage can be read and changed on its own. The processes themselves still
+// live in modules/.
+include { PREPARE_REFERENCES } from './subworkflows/prepare_references' // 1. Reference preparation
+include { READ_QC } from './subworkflows/read_qc' // 2-3. Validation, Kraken, FastP
+include { LINEAGE_TYPING } from './subworkflows/lineage_typing' // 4. Pathotypr
+include { MAP_READS } from './subworkflows/map_reads' // 5. Mapping, merge, read filter
+include { CALL_VARIANTS } from './subworkflows/call_variants' // 6. Variant calling
+include { MAKE_CONSENSUS } from './subworkflows/make_consensus' // 7. Consensus
+include { ANNOTATE } from './subworkflows/annotate' // 8. Annotation
+include { LEGACY_STATS } from './subworkflows/legacy_stats' // 9. Legacy stats
+include { MULTIQC_REPORT } from './subworkflows/multiqc_report' // 10. MultiQC
+include { COHORT_REPORT } from './subworkflows/cohort_report' // 11. Cohort outputs
+
// --- MODULE IMPORTS ---
-include { PREPARE_REFERENCE; SNPEFF_BUILD_DB; BUILD_MAPPABILITY } from './modules/reference'
-include { VALIDATE_RAW_READS_PE; VALIDATE_RAW_READS_SE; KRAKEN_FILTER_PE; KRAKEN_FILTER_SE; FASTP_PE; FASTP_SE; MULTIQC; DUMP_VERSIONS } from './modules/qc'
-include { RUN_PATHOTYPR_PE; RUN_PATHOTYPR_SE } from './modules/pathotypr'
-include { MAPPING_PE; MAPPING_SE; MERGE_AND_MARKDUP; FILTER_READS } from './modules/mapping'
-include { CALL_FREEBAYES; CALL_BACKBONE; MERGE_VCFS; CALL_FREEBAYES_RAW } from './modules/variants'
-include { CONSENSUS_FASTA } from './modules/consensus'
-// Aliases for the parallel virgin (unmasked) consensus path (a DSL2 process runs once per name)
-include { CALL_BACKBONE as CALL_BACKBONE_RAW; MERGE_VCFS as MERGE_VCFS_RAW } from './modules/variants'
-include { CONSENSUS_FASTA as CONSENSUS_FASTA_RAW } from './modules/consensus'
-include { ANNOTATE_LEGACY_VCF; ANNOTATE_MAIN_VCF; ANNOTATE_CANONICAL; GENERATE_LEGACY_STATS } from './modules/annotation'
-include { COLLECT_SUMMARY; COLLECT_DR; LIFT_VARIANTS; QC_REPORT; SNP_MATRIX } from './modules/report'
-include { cleanStr; nullish; sanitizeId } from './modules/utils'
-
-/* ----------------------------- Configuration Logic ----------------------------- */
-
-def final_outdir = file(params.outdir).toAbsolutePath().toString()
-
-// Logic to extract the base name of the TSV (e.g., "samples.tsv" -> "samples")
-// This is used to name the final MultiQC report dynamically
-def tsv_name = file(params.tsv).simpleName
-def multiqc_report_filename = "${tsv_name}_multiqc_report"
+include { cleanStr; nullish; sanitizeId; validateParams; paramsHelp } from './modules/utils'
/* ----------------------------- Helpers ----------------------------- */
// cleanStr / nullish / sanitizeId are shared with the modules; imported from modules/utils.nf above.
+//
+// Everything below is a FUNCTION, and every statement lives inside `workflow`. That is what the
+// strict parser (the default from Nextflow 25.10) requires: a script may declare includes,
+// processes, workflows and functions, but may not run statements at the top level.
-def inferRunId = { r1 ->
+def inferRunId(String r1) {
def name = new File(r1).getName()
name = name.replaceAll(/(\.fastq|\.fq)(\.gz|\.bz2)?$/,'')
name = name.replaceAll(/(\.gz|\.bz2)$/,'')
- sanitizeId(name)
+ return sanitizeId(name)
}
-// 1. Output Directory Strategy
-def SAMPLE_PUBLISH_DIR = { sid -> "${final_outdir}/${sid}" }
-
/* ----------------------------- TSV Parsing ----------------------------- */
-def tsvFile = new File(params.tsv as String)
-if (!tsvFile.exists()) throw new RuntimeException("Samplesheet not found: ${params.tsv}")
+// Read and validate the samplesheet, returning the parsed cohort as
+// [refMap, refGffMap, expectedMap, peList, seList, hasTax]
+// Throws with every problem listed at once, so the user fixes them in a single pass.
+def parseSamplesheet(String tsvPath) {
+
+def tsvFile = new File(tsvPath)
+if (!tsvFile.exists()) throw new RuntimeException("Samplesheet not found: ${tsvPath}")
def lines = tsvFile.readLines()
-if (lines.size() < 2) throw new RuntimeException("Samplesheet has a header but no data rows: ${params.tsv}")
+if (lines.size() < 2) throw new RuntimeException("Samplesheet has a header but no data rows: ${tsvPath}")
def header = lines[0].replace('\r','').split('\t')*.trim()
def col = [:]
@@ -63,7 +64,7 @@ def errors = []
if (!col.containsKey(req)) errors << "header is missing the required column '${req}'"
}
if (errors) {
- log.error "Samplesheet ${params.tsv} is unusable:\n" + errors.collect{ " - ${it}" }.join('\n')
+ log.error "Samplesheet ${tsvPath} is unusable:\n" + errors.collect{ " - ${it}" }.join('\n')
throw new RuntimeException("Samplesheet header invalid (${errors.size()} problem(s)); see the list above.")
}
@@ -84,8 +85,8 @@ lines.drop(1).eachWithIndex { raw, idx ->
// split(-1) only keeps trailing empties when the tabs are physically present; pad short rows so a
// row that omits trailing columns becomes empty fields (reported below) instead of crashing on an
// out-of-bounds Java-array read.
- def p = line.split('\t', -1).toList()
- while (p.size() < header.size()) p << ''
+ def cells = line.split('\t', -1).toList()
+ def p = cells.size() < header.size() ? cells + ([''] * (header.size() - cells.size())) : cells
def sampleId = sanitizeId(p[col.sampleId])
def r1Str = cleanStr(p[col.r1])
@@ -117,8 +118,11 @@ lines.drop(1).eachWithIndex { raw, idx ->
// Guarantee a unique run token per (sampleId, refId): lane-split inputs with no runId column
// infer the same basename for every lane -> identical BAM names collide in the merge group.
+ // Kept as two statements: the strict (v2) parser used by Nextflow >= 25.10 rejects an
+ // assignment used as an expression, so `def n = (map[k] = map[k] + 1)` fails to compile.
def runKey = "${sampleId}||${refId}||${runId}"
- def runN = (seenRuns[runKey] = seenRuns[runKey] + 1)
+ seenRuns[runKey] = seenRuns[runKey] + 1
+ def runN = seenRuns[runKey]
if (runN > 1) runId = "${runId}_${runN}"
// Input files: record every missing path for this row, then skip the row.
@@ -148,363 +152,126 @@ lines.drop(1).eachWithIndex { raw, idx ->
}
if (errors) {
- log.error "Samplesheet ${params.tsv} has ${errors.size()} problem(s):\n" + errors.collect{ " - ${it}" }.join('\n')
+ log.error "Samplesheet ${tsvPath} has ${errors.size()} problem(s):\n" + errors.collect{ " - ${it}" }.join('\n')
throw new RuntimeException("Samplesheet validation failed with ${errors.size()} problem(s); fix the rows listed above and rerun.")
}
if (peList.isEmpty() && seList.isEmpty()) {
- throw new RuntimeException("Samplesheet ${params.tsv} produced no usable samples (every data row was blank or commented out).")
+ throw new RuntimeException("Samplesheet ${tsvPath} produced no usable samples (every data row was blank or commented out).")
}
-// Check if Kraken DB exists
-def KRAKEN_ENABLED = false
-if (params.kraken2_db && hasTax) {
- def db = new File(params.kraken2_db as String)
- if (db.exists()) KRAKEN_ENABLED = true
- else log.warn "Kraken DB not found at: ${params.kraken2_db} -> Kraken disabled"
+return [ refMap: refMap, refGffMap: refGffMap, expectedMap: expectedMap,
+ peList: peList, seList: seList, hasTax: hasTax ]
}
-log.info """
+/* ----------------------------- Main Workflow ----------------------------- */
+
+workflow {
+
+ /* ----------------------------- Configuration ----------------------------- */
+
+ if (params.help) {
+ log.info paramsHelp("${projectDir}/nextflow.config", workflow.manifest.version)
+ return
+ }
+
+ // A misspelled --flag is otherwise accepted, ignored, and the run finishes with the default.
+ validateParams(params, "${projectDir}/nextflow.config")
+
+ // Checked first: everything below dereferences params.tsv, so without this the user gets
+ // "Argument of `file()` function cannot be null" instead of being told what to pass.
+ if (!params.tsv) {
+ throw new RuntimeException(
+ "--tsv is required: a Tab-Separated samplesheet with one row per (sample, run, reference).\n" +
+ " Columns: sampleId, runId, r1, r2, refId, refFasta, refGff, taxId\n" +
+ " Example: nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker\n" +
+ " See docs/tutorials/samplesheet.md")
+ }
+
+ def final_outdir = file(params.outdir).toAbsolutePath().toString()
+ // Base name of the TSV ("samples.tsv" -> "samples"), used to name the cohort-level outputs.
+ def tsv_name = file(params.tsv).simpleName
+ def multiqc_report_filename = "${tsv_name}_multiqc_report"
+
+ def sheet = parseSamplesheet(params.tsv as String)
+ def refMap = sheet.refMap
+ def refGffMap = sheet.refGffMap
+ def expectedMap = sheet.expectedMap
+
+ // Check if Kraken DB exists
+ def KRAKEN_ENABLED = false
+ if (params.kraken2_db && sheet.hasTax) {
+ def db = new File(params.kraken2_db as String)
+ if (db.exists()) KRAKEN_ENABLED = true
+ else log.warn "Kraken DB not found at: ${params.kraken2_db} -> Kraken disabled"
+ }
+
+ // Executor and profile are shown because WHERE the run lands is the easiest thing to get wrong:
+ // with no -profile everything runs on the current host, which on a cluster is the login node.
+ log.info """
================================================================
- BAMpiro Pipeline 🧛♂️🧬
+ BAMpiro Pipeline 🧛♂️🧬 v${workflow.manifest.version}
================================================================
TSV : ${params.tsv}
Output Absolute : ${final_outdir}
Report Name : ${multiqc_report_filename}.html
+Profile(s) : ${workflow.profile}
+Executor : ${workflow.session.config.navigate('process.executor') ?: 'local'}
+Container : ${params.container}
+Kraken2 : ${KRAKEN_ENABLED ? params.kraken2_db : 'disabled'}
Annotate Legacy : ${params.annotate_legacy_vcfs}
Run Pathotypr : ${params.run_pathotypr}
================================================================
"""
-/* ----------------------------- Channels ----------------------------- */
+ /* ----------------------------- Channels ----------------------------- */
-def uniqueRefsList = refMap.collect { k,v -> [k, file(v)] }
-def ref_in_ch = Channel.fromList(uniqueRefsList).map { refId, fasta -> tuple(refId, fasta) }
+ def ref_in_ch = Channel.fromList(refMap.collect { k, v -> [k, file(v)] })
+ .map { refId, fasta -> tuple(refId, fasta) }
-def snpeff_db_in = Channel.fromList(refMap.keySet().collect { rid ->
- [rid, file(refMap[rid]), file(refGffMap[rid])]
-}).map { rid, fa, gff -> tuple(rid, fa, gff) }
+ def snpeff_db_in = Channel.fromList(refMap.keySet().collect { rid ->
+ [rid, file(refMap[rid]), file(refGffMap[rid])]
+ }).map { rid, fa, gff -> tuple(rid, fa, gff) }
-def pe_reads_ch = Channel.fromList(peList).map { sId, runId, r1, r2, refId, taxId -> tuple(sId, runId, r1, r2, refId, taxId) }
-def se_reads_ch = Channel.fromList(seList).map { sId, runId, r1, refId, taxId -> tuple(sId, runId, r1, refId, taxId) }
+ def pe_reads_ch = Channel.fromList(sheet.peList).map { sId, runId, r1, r2, refId, taxId -> tuple(sId, runId, r1, r2, refId, taxId) }
+ def se_reads_ch = Channel.fromList(sheet.seList).map { sId, runId, r1, refId, taxId -> tuple(sId, runId, r1, refId, taxId) }
-/* ----------------------------- Main Workflow ----------------------------- */
-
-workflow {
+ /* ----------------------------- Stages ----------------------------- */
+ // Each stage lives in subworkflows/ and keeps its own conditionals, so a branch that is off
+ // still emits the Channel.empty() placeholder the next stage mixes in.
// 1. Reference Preparation
- def ref_bundle = PREPARE_REFERENCE(ref_in_ch)
- def snpeff_db = SNPEFF_BUILD_DB(snpeff_db_in)
-
- // 2. Read Validation
- def checked_pe = VALIDATE_RAW_READS_PE(pe_reads_ch)
- def checked_se = VALIDATE_RAW_READS_SE(se_reads_ch)
- def pe_final = checked_pe.reads
- def se_final = checked_se.reads
-
- // Channel to collect Kraken reports if enabled
- def ch_kraken_reports = Channel.empty()
-
- if (KRAKEN_ENABLED) {
- def pe_with_tax = checked_pe.reads.filter { it[5] != null }
- def pe_no_tax = checked_pe.reads.filter { it[5] == null }
- def se_with_tax = checked_se.reads.filter { it[4] != null }
- def se_no_tax = checked_se.reads.filter { it[4] == null }
-
- def krk_pe = KRAKEN_FILTER_PE(pe_with_tax, params.kraken2_db)
- def krk_se = KRAKEN_FILTER_SE(se_with_tax, params.kraken2_db)
-
- pe_final = krk_pe.reads.mix(pe_no_tax)
- se_final = krk_se.reads.mix(se_no_tax)
-
- // Accumulate reports for MultiQC
- ch_kraken_reports = krk_pe.report.mix(krk_se.report)
- }
+ def refs = PREPARE_REFERENCES(ref_in_ch, snpeff_db_in)
- // 3. FastP QC
- def fastp_pe = FASTP_PE(pe_final)
- def fastp_se = FASTP_SE(se_final)
-
- // 4. Pathotypr (Optional lineage + drug-resistance typing; k-mer, reference-agnostic)
- def patho_results = Channel.empty() // lineage summary per sample
- def patho_dr_results = Channel.empty() // drug-resistance summary per sample
-
- if (params.run_pathotypr) {
- // Reshape channels for Pathotypr
- def patho_pe_ch = fastp_pe.pe_reads.map { sId, runId, r1, r2, refId, taxId -> tuple(refId, sId, runId, r1, r2) }
- def patho_se_ch = fastp_se.se_reads.map { sId, runId, rSE, refId, taxId -> tuple(refId, sId, runId, rSE) }
-
- def run_pe = RUN_PATHOTYPR_PE(
- patho_pe_ch,
- file(params.pathotypr_ref),
- file(params.pathotypr_markers),
- file(params.pathotypr_dr_markers),
- params.pathotypr_bin
- )
-
- def run_se = RUN_PATHOTYPR_SE(
- patho_se_ch,
- file(params.pathotypr_ref),
- file(params.pathotypr_markers),
- file(params.pathotypr_dr_markers),
- params.pathotypr_bin
- )
-
- // Combine summaries for later statistics / the QC report
- patho_results = run_pe.summary.mix(run_se.summary)
- patho_dr_results = run_pe.dr_mutations.mix(run_se.dr_mutations)
- }
-
- // 5. Mapping (BWA)
- def map_pe = fastp_pe.pe_reads.map { sId, runId, r1, r2, refId, taxId -> tuple(refId, "PE", sId, runId, r1, r2, taxId) }
- def map_se1 = fastp_pe.se_reads.map { sId, runId, rSE, refId, taxId -> tuple(refId, "SE", sId, runId, rSE, null, taxId) }
- def map_se2 = fastp_se.se_reads.map { sId, runId, rSE, refId, taxId -> tuple(refId, "SE", sId, runId, rSE, null, taxId) }
- def map_all = map_pe.mix(map_se1).mix(map_se2)
-
- // Join reads with references (matching RefID)
- def joined = map_all.combine(ref_bundle.bundle).filter { it[0] == it[7] }.map { refId, mode, sId, runId, r1, r2, taxId, refId2, ref_fa, indices, exclude_txt -> tuple(refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt) }
- def branched = joined.branch { pe: it[1] == "PE"; se: it[1] == "SE" }
-
- def map_pe_in = branched.pe.map { refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt -> tuple(refId, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt) }
- def map_se_in = branched.se.map { refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt -> tuple(refId, sId, runId, r1, taxId, ref_fa, indices, exclude_txt) }
-
- def mapped_pe = MAPPING_PE(map_pe_in)
- def mapped_se = MAPPING_SE(map_se_in)
- def all_bams = mapped_pe.bam.mix(mapped_se.bam)
+ // 2. Read Validation + 3. FastP QC (with the optional Kraken filter in between)
+ def reads = READ_QC(pe_reads_ch, se_reads_ch, KRAKEN_ENABLED)
- // Group BAMs by SampleID + RefID for merging
- def bams_grouped = all_bams.map { sId, rId, bam, ref_fa, exclude_txt ->
- def key = "${sId}||${rId}"
- def nExp = expectedMap[key]
- tuple(groupKey(key, nExp as int), sId, rId, bam, ref_fa, exclude_txt)
- }.groupTuple(by: 0).map { gk, sIds, rIds, bams, ref_fas, excls -> tuple(sIds[0], rIds[0], bams, ref_fas[0], excls[0]) }
+ // 4. Pathotypr (Optional lineage + drug-resistance typing; k-mer, reference-agnostic)
+ def typing = LINEAGE_TYPING(reads.pe_reads, reads.se_reads)
- def final_bams = MERGE_AND_MARKDUP(bams_grouped)
+ // 5. Mapping (BWA), per-sample merge, and the optional length-aware read filter
+ def bams = MAP_READS(reads.pe_reads, reads.pe_orphans, reads.se_reads, refs.bundle, expectedMap)
// 6. Variant Calling
- // Optional length-aware read filter: drop reads too short to map uniquely at their locus
- // (short-read false positives in near-repeats), then call/consensus on the filtered BAM.
- // Feature off -> call on the dedup BAM unchanged.
- def vbase
- def filter_stats = Channel.empty()
- if (params.dynamic_read_filter) {
- def mapp = BUILD_MAPPABILITY( ref_bundle.bundle.map { rId, fa, idx, excl -> tuple(rId, fa) } )
- def track_bed = mapp.track.join(mapp.repeat_bed, by: 0) // (rId, npz, repeat_bed)
- def filt_in = final_bams.final_bam
- .map { sId, rId, bam, bai, fa, excl -> tuple(rId, sId, bam, bai, fa, excl) }
- .combine(track_bed, by: 0) // fan the per-reference track+bed to each sample
- .map { rId, sId, bam, bai, fa, excl, npz, bed -> tuple(sId, rId, bam, bai, fa, excl, npz, bed) }
- def filt = FILTER_READS(filt_in)
- vbase = filt.filtered_bam // exclude_txt now = nucmer + genmap repeats
- filter_stats = filt.stats
- } else {
- vbase = final_bams.final_bam
- }
-
- // Run FreeBayes and Backbone parallel
- def fb_out = CALL_FREEBAYES(vbase)
- def bb_in = vbase.map { sId, rId, bam, bai, ref_fa, exclude_txt -> tuple(sId, rId, bam, bai, ref_fa) }
- def bb_out = CALL_BACKBONE(bb_in)
-
- // Merge specific variants with backbone for All-Positions VCF (outLabel "" = masked path)
- def merge_in = fb_out.snps.join(bb_out.backbone, by: [0,1]).join(bb_out.header, by: [0,1])
- .map { sId, rId, sv, st, bv, bt, hdr -> tuple(sId, rId, sv, st, bv, bt, hdr, "") }
- def vcf_ch = MERGE_VCFS(merge_in)
+ def variants = CALL_VARIANTS(bams.variant_base)
// 7. Consensus Generation (Optional)
- def masked_consensus = Channel.empty() // captured for the cohort QC report (section 11)
- if (params.make_consensus) {
- // Prepare inputs: VCF + Reference + Mask sites
- // Note: Script is called from bin/ directly in the module
- // refmeta from vbase so the masked consensus gets the combined (nucmer + genmap) exclude
- def refmeta = vbase.map { sId, rId, bam, bai, ref_fa, exclude_txt -> tuple(sId, rId, ref_fa, exclude_txt) }
- def allpos_mask = vcf_ch.allpos.join(fb_out.mask_sites, by: [0,1])
- def cons_in = allpos_mask.join(refmeta, by: [0,1]).map { sId, rId, vcf_gz, tbi, mask, ref_fa, exclude_txt -> tuple(sId, rId, vcf_gz, tbi, mask, ref_fa, exclude_txt, "") }
-
- masked_consensus = CONSENSUS_FASTA(cons_in).fasta
-
- // 7b. Virgin (unmasked) consensus from the ORIGINAL dedup BAM, in parallel with the masked one.
- if (params.keep_virgin_consensus && params.dynamic_read_filter) {
- def raw_bb_in = final_bams.final_bam.map { sId, rId, bam, bai, ref_fa, excl -> tuple(sId, rId, bam, bai, ref_fa) }
- def bb_raw = CALL_BACKBONE_RAW(raw_bb_in)
- // Virgin SNPs: re-call FreeBayes on the raw bam (shows pre-filter variants) or reuse masked SNPs.
- def raw_snps = params.virgin_full_freebayes ? CALL_FREEBAYES_RAW(final_bams.final_bam).snps : fb_out.snps
- def merge_raw = raw_snps.join(bb_raw.backbone, by: [0,1]).join(bb_raw.header, by: [0,1])
- .map { sId, rId, sv, st, bv, bt, hdr -> tuple(sId, rId, sv, st, bv, bt, hdr, ".raw") }
- def vcf_raw = MERGE_VCFS_RAW(merge_raw)
- def refmeta_raw = final_bams.final_bam.map { sId, rId, bam, bai, ref_fa, excl -> tuple(sId, rId, ref_fa, excl) }
- def cons_raw = vcf_raw.allpos.join(fb_out.mask_sites, by: [0,1]).join(refmeta_raw, by: [0,1])
- .map { sId, rId, vcf_gz, tbi, mask, ref_fa, excl -> tuple(sId, rId, vcf_gz, tbi, mask, ref_fa, excl, ".raw") }
- CONSENSUS_FASTA_RAW(cons_raw)
- }
- }
+ def consensus = MAKE_CONSENSUS(bams.variant_base, bams.dedup_bam,
+ variants.allpos, variants.snps, variants.mask_sites)
// 8. Annotation
- // Annotate Legacy split VCFs (Homo/Het/Indel/Raw)
- def freebayes_ann = Channel.empty() // annotated freebayes.raw (AD + snpEff ANN) -> SNP dynamics
- if (params.annotate_legacy_vcfs) {
- def leg_inputs = Channel.empty()
- .mix(fb_out.homo_snp.map { sId, rId, vcf -> tuple(rId, sId, "var.homo.SNPs", vcf) })
- .mix(fb_out.het_snp.map { sId, rId, vcf -> tuple(rId, sId, "var.het.SNPs", vcf) })
- .mix(fb_out.homo_indel.map { sId, rId, vcf -> tuple(rId, sId, "var.homo.indel", vcf) })
- .mix(fb_out.raw_fb.map { sId, rId, vcf, tbi -> tuple(rId, sId, "freebayes.raw", vcf) })
-
- def ann_leg_in = leg_inputs.combine(snpeff_db.db, by: 0)
- .map { rId, sId, lbl, vcf, cfg, dat -> tuple(sId, rId, lbl, vcf, cfg, dat) }
-
- def ann_leg = ANNOTATE_LEGACY_VCF(ann_leg_in)
- // Keep the annotated freebayes VCF: it carries per-alt AD (true AF) AND gene/effect (ANN),
- // so the dynamics panel gets continuous allele frequencies without losing gene annotation.
- freebayes_ann = ann_leg.out
- .filter { sId, rId, vcf, tbi -> vcf.name.contains('freebayes.raw') }
- .map { sId, rId, vcf, tbi -> vcf }
- }
-
- // Annotate Main VCF
- def vcf_for_stats = Channel.empty()
- def ch_snpeff_stats = Channel.empty()
-
- if (params.annotate_main_vcf) {
- def main_pre = vcf_ch.main_vcf.map { sId, rId, vcf, tbi -> tuple(rId, sId, vcf, tbi) }
- def ann_main_in = main_pre.combine(snpeff_db.db, by: 0)
- .map { rId, sId, vcf, tbi, cfg, dat -> tuple(sId, rId, vcf, tbi, cfg, dat) }
-
- def ann_out = ANNOTATE_MAIN_VCF(ann_main_in)
-
- vcf_for_stats = ann_out.vcf_ann.map { sId, rId, vcf, tbi -> tuple(sId, rId, vcf) }
- ch_snpeff_stats = ann_out.stats // Collect SnpEff stats CSV for MultiQC
- } else {
- vcf_for_stats = vcf_ch.allpos.map { sId, rId, vcf, tbi -> tuple(sId, rId, vcf) }
- }
+ def annotated = ANNOTATE(variants.homo_snp, variants.het_snp, variants.homo_indel,
+ variants.raw_fb, variants.main_vcf, variants.allpos, refs.snpeff)
// 9. Legacy Stats Generation
- // Prepare Reference Index channel
- def ref_fai_ch = ref_bundle.bundle.map { rId, fa, indices, excl ->
- def fai_file = indices.find { it.name.endsWith('.fai') }
- tuple(rId, fai_file)
- }
-
- // Prepare FastP JSONs (grouped by sample). A multi-lane sample has one JSON per lane; pass them ALL
- // so GENERATE_LEGACY_STATS aggregates read counts (SUM) and quality rates (read/base-weighted) instead
- // of silently keeping a single lane. Grouped on the real sampleId val (do NOT re-derive it from the
- // filename: a sampleId containing '__' would be truncated by split('__')). Sorted for a stable -resume.
- def json_ch = fastp_pe.json.mix(fastp_se.json) // [sId, json]
- .groupTuple() // -> [sId, [json_lane1, json_lane2, ...]]
- .map { sId, jsons -> tuple(sId, jsons.toSorted { it.name }) }
-
- // BAM stats already carry (sampleId, refId) as vals -- do NOT re-parse the filename with
- // tokenize('.'), which truncates any dotted refId (e.g. NC_000962.3) and breaks the join.
- def bam_stats_ch = final_bams.stats // [sId, rId, stats]
-
- // Join logic for Stats
- def vcf_bam_joined = vcf_for_stats.join(bam_stats_ch, by: [0,1]) // [sId, rId, vcf, stats]
- // combine (not join) on sampleId: one json per sample must fan out to EVERY (sample,ref)
- // row, else a sample mapped to >1 reference silently loses all but one reference's stats.
- def vcf_bam_fastp = vcf_bam_joined.combine(json_ch, by: 0) // [sId, rId, vcf, stats, json]
-
- // Join with Reference Index (Using COMBINE to reuse reference)
- def ready_no_patho = vcf_bam_fastp
- .map { sId, rId, vcf, stats, json -> tuple(rId, sId, vcf, stats, json) }
- .combine(ref_fai_ch)
- .filter { it[0] == it[5] } // Filter where sample RefID == FAI RefID
- .map { rId, sId, vcf, stats, json, rId2, fai -> tuple(sId, rId, json, stats, vcf, fai) }
-
- // Join with Pathotypr (Using remainder to handle missing files safely)
- def final_stats_input = ready_no_patho
- .join(patho_results, by: 0, remainder: true)
- .map { list ->
- // list size < 7 means missing left-side data (e.g. mapping failed)
- if (list.size() < 7) return null
-
- def sId = list[0]
- def rId = list[1]
- def json = list[2]
- def stats = list[3]
- def vcf = list[4]
- def fai = list[5]
- def patho = list[6] // Can be null
-
- // Fallback placeholder file
- def real_patho = patho ? patho : file("NO_FILE")
- tuple(sId, rId, json, stats, vcf, fai, real_patho)
- }
- .filter { it != null }
-
-
- def legacy_stats = GENERATE_LEGACY_STATS(final_stats_input)
+ def legacy = LEGACY_STATS(refs.bundle, reads.pe_json, reads.se_json,
+ bams.stats, annotated.stats_vcf, typing.summary)
// 10. MultiQC Report
- // Collect all relevant metrics from previous processes.
- // FastP JSONs and Kraken reports are reference-independent (QC of the raw reads), so a sample
- // mapped to >1 reference emits identically-named copies from parallel tasks. Deduplicate by
- // filename before collecting, otherwise MultiQC hits a fatal input-name collision.
- def fastp_json_mqc = fastp_pe.json.mix(fastp_se.json).map { sId, json -> json }.unique { it.name }
- def kraken_mqc = ch_kraken_reports.unique { it.name }
- // Provenance: dump the container's tool versions once and surface them in the report.
- def versions_mqc = DUMP_VERSIONS().mqc
- def qc_collection = Channel.empty()
- .mix(fastp_json_mqc) // FastP
- .mix(final_bams.stats.map { s, r, st -> st }) // Samtools (drop the (sId,rId) key -> bare path)
- .mix(kraken_mqc) // Kraken
- .mix(ch_snpeff_stats) // SnpEff
- .mix(filter_stats) // Length-aware read filter (drop rate)
- .mix(versions_mqc) // Software versions
- .collect()
-
- MULTIQC(qc_collection, multiqc_report_filename)
+ MULTIQC_REPORT(reads.pe_json, reads.se_json, bams.stats, reads.kraken_reports,
+ annotated.snpeff_stats, bams.filter_mqc, multiqc_report_filename)
// 11. Cohort-level outputs (QC report + master SNP matrix).
- // Per-sample annotated VCFs, shared by both: prefer the annotated freebayes VCF (per-alt AD ->
- // continuous AF + gene/effect); fall back to the annotated main VCF (GT-based AF) if legacy
- // annotation is off. reference name(s) the samples were mapped against.
- def report_vcfs = (params.annotate_legacy_vcfs ? freebayes_ann
- : vcf_for_stats.map { sId, rId, vcf -> vcf })
- .collect().ifEmpty([])
- def ref_name = refMap.keySet().join(',')
-
- // 11a. Consolidated QC report: aggregate every per-sample legacy log into one summary TSV, then
- // render the self-contained interactive HTML + PASS/WARN/FAIL flags.
- if (params.make_qc_report) {
- def all_logs = legacy_stats.legacy_log.collect()
- def summ = COLLECT_SUMMARY(all_logs, tsv_name)
- def cons_files = masked_consensus.map { sId, rId, fa -> fa }.collect().ifEmpty([])
- // Reference-level extras (cohort report -> take the reference bundle; single-ref is the norm):
- // GFF enables the per-gene SNP-density panel, the nucmer/repeat BED the masked-regions panel.
- def report_ref = refGffMap.keySet().toList().first() // deterministic: the first reference
- def report_gff = file(refGffMap[report_ref])
- def report_mask = ref_bundle.bundle.filter { rId, fa, idx, excl -> rId == report_ref }
- .map { rId, fa, idx, excl -> excl }.first()
- // Provenance footer: pinned container digest + reference(s).
- def provenance = (["container=${params.container}"] + refMap.keySet().collect { "reference=${it}" })
- .collect { "\"${it}\"" }.join(' ')
- // SNP dynamics: the samplesheet is the metadata source (auto-detects time/group columns; the
- // panel self-hides if absent).
- def report_meta = file(params.tsv)
- // Drug-resistance calls (pathotypr DR run -> one run TSV); NO_FILE when pathotypr is off.
- def dr_report = params.run_pathotypr
- ? COLLECT_DR(patho_dr_results.map { sId, f -> f }.collect().ifEmpty([]), tsv_name).dr
- : file("NO_FILE")
- // Canonical-reference-annotated VCFs for the dual amino-acid numbering (off unless annotate_canonical).
- // vcf_for_stats is the per-sample (sId,rId,vcf) channel; its positions match report_vcfs, so the
- // report's sample+position merge finds each variant's canonical amino-acid change.
- def report_vcfs_h37rv = params.annotate_canonical
- ? ANNOTATE_CANONICAL(vcf_for_stats, params.canonical_snpeff_db).out
- .map { sId, vcf -> vcf }.collect().ifEmpty([])
- : file("NO_FILE")
- // Kraken2 per-sample reports (deduped) -> Taxonomic composition panel; empty when Kraken is off.
- def report_kraken = ch_kraken_reports.unique { it.name }.collect().ifEmpty([])
- // Alignment-free canonical COORDINATE per variant via pathotypr (alternative to the --vcfs-h37rv path):
- // lift the run's variant positions (mapping-reference coords) onto H37Rv and hand the map to the report.
- def report_ref_fa = ref_bundle.bundle.filter { rId, fa, idx, excl -> rId == report_ref }
- .map { rId, fa, idx, excl -> fa }.first()
- def pos_liftover = params.variant_liftover
- ? LIFT_VARIANTS(report_vcfs, report_ref_fa, report_ref, tsv_name).map
- : file("NO_FILE")
- QC_REPORT(summ.summary, summ.gene_burden, cons_files, report_gff, report_mask,
- report_meta, report_vcfs, report_vcfs_h37rv, pos_liftover, dr_report, report_kraken, provenance, tsv_name)
- }
-
- // 11b. Master SNP matrix: rows = SNP sites, columns = reference/annotation + per-sample AF & depth.
- if (params.make_snp_matrix) {
- SNP_MATRIX(report_vcfs, ref_name, tsv_name)
- }
+ COHORT_REPORT(legacy.legacy_log, consensus.fasta, refs.bundle, reads.kraken_reports,
+ typing.dr_mutations, annotated.stats_vcf, annotated.freebayes_vcf,
+ refMap, refGffMap, tsv_name)
}
diff --git a/modules/annotation.nf b/modules/annotation.nf
index e3fa301..43d2576 100644
--- a/modules/annotation.nf
+++ b/modules/annotation.nf
@@ -15,7 +15,7 @@ process ANNOTATE_CANONICAL {
// canonical annotation remains (qc_report.py reads the first ANN). Positions must line up with the
// canonical reference (true when the mapping reference shares its coordinates).
tag "AnnCanonical: ${sampleId}"
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 2
memory { 6.GB * task.attempt }
@@ -37,12 +37,17 @@ process ANNOTATE_CANONICAL {
| bgzip -c > ${sampleId}.${refId}.canonical.ann.vcf.gz
tabix -f -p vcf ${sampleId}.${refId}.canonical.ann.vcf.gz 2>/dev/null || : > ${sampleId}.${refId}.canonical.ann.vcf.gz.tbi
"""
+
+ stub:
+ """
+ touch ${sampleId}.${refId}.canonical.ann.vcf.gz
+ """
}
process ANNOTATE_LEGACY_VCF {
tag "AnnLegacy: ${sampleId}"
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 2
memory { 6.GB * task.attempt }
@@ -58,14 +63,6 @@ process ANNOTATE_LEGACY_VCF {
set -euo pipefail
# Function to safely index VCFs, handling potential empty files
- safe_tabix () {
- local gz="\$1"; local idx="\${gz}.tbi"
- set +e; tabix -f -p vcf "\$gz"; st=\$?; set -e
- if [ \$st -ne 0 ]; then
- # Check if file has variants or is just header
- if zgrep -vq '^#' "\$gz"; then exit \$st; else : > "\$idx"; fi
- fi
- }
# SnpEff requires the data directory structure to be present locally
if [ ! -e data ]; then ln -s !{data_dir} data; fi
@@ -80,12 +77,20 @@ process ANNOTATE_LEGACY_VCF {
safe_tabix !{sampleId}.!{refId}.!{label}.ann.vcf.gz
"""
+
+ // label must stay in the name: main.nf picks the freebayes.raw VCF by filename
+ stub:
+ """
+ touch ${sampleId}.${refId}.${label}.ann.vcf.gz
+ touch ${sampleId}.${refId}.${label}.ann.vcf.gz.tbi
+ touch ${sampleId}.${refId}.${label}.snpeff.stderr.log
+ """
}
process ANNOTATE_MAIN_VCF {
tag "AnnMain: ${sampleId}"
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 2
memory { 8.GB * task.attempt }
@@ -101,11 +106,6 @@ process ANNOTATE_MAIN_VCF {
"""
set -euo pipefail
- safe_tabix () {
- local gz="\$1"; local idx="\${gz}.tbi"
- set +e; tabix -f -p vcf "\$gz"; st=\$?; set -e
- if [ \$st -ne 0 ]; then if zgrep -vq '^#' "\$gz"; then exit \$st; else : > "\$idx"; fi; fi
- }
# Link SnpEff database directory
if [ ! -e data ]; then ln -s !{data_dir} data; fi
@@ -117,12 +117,20 @@ process ANNOTATE_MAIN_VCF {
safe_tabix !{sampleId}.!{refId}.ann.vcf.gz
"""
+
+ stub:
+ """
+ touch ${sampleId}.${refId}.ann.vcf.gz
+ touch ${sampleId}.${refId}.ann.vcf.gz.tbi
+ touch ${sampleId}.${refId}.snpeff.csv
+ touch ${sampleId}.${refId}.snpeff.stderr.log
+ """
}
process GENERATE_LEGACY_STATS {
tag "Stats: ${sampleId}"
// Use getSampleDir. getSavePath automatically places .log files into the 'stats/' subfolder.
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 1
input:
@@ -150,4 +158,9 @@ process GENERATE_LEGACY_STATS {
${patho_arg}
mv "${sampleId}.log" "${sampleId}.${refId}.log"
"""
+
+ stub:
+ """
+ touch ${sampleId}.${refId}.log
+ """
}
diff --git a/modules/consensus.nf b/modules/consensus.nf
index 47b59f0..9041dc2 100644
--- a/modules/consensus.nf
+++ b/modules/consensus.nf
@@ -12,7 +12,7 @@ process CONSENSUS_FASTA {
tag "Consensus: ${sampleId}"
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 1
memory { 4.GB * task.attempt }
@@ -74,4 +74,10 @@ process CONSENSUS_FASTA {
sed -i "s/^>.*/>${sampleId}${outLabel}/" ${sampleId}.${refId}${outLabel}.consensus.fasta
fi
"""
+
+ stub:
+ """
+ touch ${sampleId}.${refId}${outLabel}.consensus.fasta
+ touch ${sampleId}.${refId}${outLabel}.consensus.log
+ """
}
diff --git a/modules/mapping.nf b/modules/mapping.nf
index 23c1486..ba290dc 100644
--- a/modules/mapping.nf
+++ b/modules/mapping.nf
@@ -36,6 +36,12 @@ process MAPPING_PE {
!{ref_fa} !{r1} !{r2} | \
samtools sort -@ !{task.cpus} -l 1 -o !{sampleId}__!{runId}.pe.sorted.bam -
'''
+
+ // Stubs create only the NEW outputs; ref_fa and exclude_txt are inputs re-emitted as-is.
+ stub:
+ """
+ touch ${sampleId}__${runId}.pe.sorted.bam
+ """
}
process MAPPING_SE {
@@ -63,6 +69,11 @@ process MAPPING_SE {
!{ref_fa} !{reads_se} | \
samtools sort -@ !{task.cpus} -l 1 -o !{sampleId}__!{runId}.se.sorted.bam -
'''
+
+ stub:
+ """
+ touch ${sampleId}__${runId}.se.sorted.bam
+ """
}
process MERGE_AND_MARKDUP {
@@ -70,7 +81,7 @@ process MERGE_AND_MARKDUP {
// Use getSampleDir for nested output support
// This handles putting .stats in the stats/ folder and keeping the final .bam in the sample root
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
// --- OOM (Out of Memory) Protection Strategy ---
// If the process fails with exit code 137 (OOM), it retries with more memory
@@ -135,6 +146,15 @@ process MERGE_AND_MARKDUP {
# 4. Cleanup (only the merge intermediate exists, and only in the multi-BAM case)
rm -f merged.bam
'''
+
+ stub:
+ """
+ touch ${sampleId}.${refId}.final.bam
+ touch ${sampleId}.${refId}.final.bam.bai
+ touch ${sampleId}.${refId}.dedup.stats
+ touch ${sampleId}.${refId}.final.cram
+ touch ${sampleId}.${refId}.final.cram.crai
+ """
}
process FILTER_READS {
@@ -143,7 +163,7 @@ process FILTER_READS {
// if EITHER mate anchors uniquely (concordant-pair rescue). One streaming pass, no re-sort.
tag "Filter: ${sampleId}"
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 4
memory { 4.GB * task.attempt }
@@ -212,4 +232,14 @@ process FILTER_READS {
echo "[FILTER_READS] WARNING: !{sampleId}.!{refId} dropped ${pct}% of reads (> !{params.filter_max_drop_pct}%) -- check contamination / read length / reference" >&2
fi
'''
+
+ stub:
+ """
+ touch ${sampleId}.${refId}.filtered.bam
+ touch ${sampleId}.${refId}.filtered.bam.bai
+ touch ${sampleId}.${refId}.exclude.txt
+ touch ${sampleId}.${refId}.filter_mqc.tsv
+ touch ${sampleId}.${refId}.filtered.cram
+ touch ${sampleId}.${refId}.filtered.cram.crai
+ """
}
diff --git a/modules/pathotypr.nf b/modules/pathotypr.nf
index 1090139..ae4c033 100644
--- a/modules/pathotypr.nf
+++ b/modules/pathotypr.nf
@@ -19,7 +19,7 @@ include { getSavePath; getSampleDir } from './utils'
process RUN_PATHOTYPR_PE {
tag "PathotyprPE: ${sampleId}"
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 4
memory '8 GB'
@@ -51,11 +51,18 @@ process RUN_PATHOTYPR_PE {
# Fix the DR detail file name to carry our sampleId+runId (pathotypr names the sample after the FASTQ)
cp "\$(ls ${sampleId}__${runId}.pathotypr.dr_*_mutations.tsv | head -1)" ${sampleId}__${runId}.dr_mutations.tsv
"""
+
+ // The lineage summary also satisfies the pathotypr.* glob (emit: results), so two files cover all three outputs.
+ stub:
+ """
+ touch ${sampleId}__${runId}.pathotypr.lineage_summary.tsv
+ touch ${sampleId}__${runId}.dr_mutations.tsv
+ """
}
process RUN_PATHOTYPR_SE {
tag "PathotyprSE: ${sampleId}"
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
cpus 4
memory '8 GB'
@@ -83,4 +90,10 @@ process RUN_PATHOTYPR_SE {
--output-prefix ${sampleId}__${runId}.pathotypr.dr --min-alt-percent ${params.pathotypr_min_alt} --threads ${task.cpus}
cp "\$(ls ${sampleId}__${runId}.pathotypr.dr_*_mutations.tsv | head -1)" ${sampleId}__${runId}.dr_mutations.tsv
"""
+
+ stub:
+ """
+ touch ${sampleId}__${runId}.pathotypr.lineage_summary.tsv
+ touch ${sampleId}__${runId}.dr_mutations.tsv
+ """
}
diff --git a/modules/qc.nf b/modules/qc.nf
index e87f895..36c8e4a 100644
--- a/modules/qc.nf
+++ b/modules/qc.nf
@@ -26,6 +26,12 @@ process VALIDATE_RAW_READS_PE {
if [[ "!{r1}" == *.gz ]]; then gzip -t "!{r1}" >/dev/null; else [[ -s "!{r1}" ]]; fi
if [[ "!{r2}" == *.gz ]]; then gzip -t "!{r2}" >/dev/null; else [[ -s "!{r2}" ]]; fi
'''
+
+ stub:
+ """
+ # the outputs re-emit the staged input reads, so there is nothing to create
+ true
+ """
}
process VALIDATE_RAW_READS_SE {
@@ -43,6 +49,11 @@ process VALIDATE_RAW_READS_SE {
set -euo pipefail
if [[ "!{r1}" == *.gz ]]; then gzip -t "!{r1}" >/dev/null; else [[ -s "!{r1}" ]]; fi
'''
+
+ stub:
+ """
+ true
+ """
}
process KRAKEN_FILTER_PE {
@@ -51,7 +62,7 @@ process KRAKEN_FILTER_PE {
memory '80 GB'
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
input:
tuple val(sampleId), val(runId), path(r1), path(r2), val(refId), val(taxId)
@@ -93,6 +104,13 @@ process KRAKEN_FILTER_PE {
gzip -f ${prefix}_R1.kraken.fq
gzip -f ${prefix}_R2.kraken.fq
'''
+
+ stub:
+ """
+ touch ${sampleId}__${runId}_R1.kraken.fq.gz
+ touch ${sampleId}__${runId}_R2.kraken.fq.gz
+ touch ${sampleId}__${runId}.kraken.report
+ """
}
process KRAKEN_FILTER_SE {
@@ -101,7 +119,7 @@ process KRAKEN_FILTER_SE {
memory '80 GB'
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
input:
tuple val(sampleId), val(runId), path(r1), val(refId), val(taxId)
@@ -136,6 +154,12 @@ process KRAKEN_FILTER_SE {
gzip -f ${prefix}.kraken.fq
'''
+
+ stub:
+ """
+ touch ${sampleId}__${runId}.kraken.fq.gz
+ touch ${sampleId}__${runId}.kraken.report
+ """
}
process FASTP_PE {
@@ -144,7 +168,7 @@ process FASTP_PE {
memory { 4.GB * task.attempt }
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
input:
tuple val(sampleId), val(runId), path(r1), path(r2), val(refId), val(taxId)
@@ -179,6 +203,15 @@ process FASTP_PE {
cat ${prefix}_merged.fq.gz ${prefix}_u1.fq.gz ${prefix}_u2.fq.gz > ${prefix}_se_combined.fq.gz
rm -f ${prefix}_merged.fq.gz ${prefix}_u1.fq.gz ${prefix}_u2.fq.gz
'''
+
+ stub:
+ """
+ touch ${sampleId}__${runId}_R1.clean.fq.gz
+ touch ${sampleId}__${runId}_R2.clean.fq.gz
+ touch ${sampleId}__${runId}_se_combined.fq.gz
+ touch ${sampleId}__${runId}_fastp.json
+ touch ${sampleId}__${runId}_fastp.html
+ """
}
process FASTP_SE {
@@ -187,7 +220,7 @@ process FASTP_SE {
memory { 4.GB * task.attempt }
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
input:
tuple val(sampleId), val(runId), path(r1), val(refId), val(taxId)
@@ -208,6 +241,13 @@ process FASTP_SE {
--length_required !{params.fastp_min_length} \
--json ${prefix}_fastp.json --html ${prefix}_fastp.html
'''
+
+ stub:
+ """
+ touch ${sampleId}__${runId}_SE.clean.fq.gz
+ touch ${sampleId}__${runId}_fastp.json
+ touch ${sampleId}__${runId}_fastp.html
+ """
}
process MULTIQC {
@@ -243,6 +283,12 @@ process MULTIQC {
# -n : Set dynamic output name
multiqc . -c multiqc_config.yaml -n ${report_name}
"""
+
+ stub:
+ """
+ touch ${report_name}.html
+ mkdir -p ${report_name}_data
+ """
}
process DUMP_VERSIONS {
@@ -302,4 +348,10 @@ process DUMP_VERSIONS {
echo ' '
} > software_versions_mqc.yml
'''
+
+ stub:
+ """
+ touch software_versions.txt
+ touch software_versions_mqc.yml
+ """
}
diff --git a/modules/reference.nf b/modules/reference.nf
index e11fe79..265f349 100644
--- a/modules/reference.nf
+++ b/modules/reference.nf
@@ -11,7 +11,7 @@ process PREPARE_REFERENCE {
// We keep custom logic here because we NEED to publish the index files (.fai, .bwt, etc.),
// which the global 'getSavePath' function would filter out.
- publishDir "${params.outdir}/references/${refId}", mode: params.publish_mode, saveAs: { filename ->
+ publishDir path: { "${params.outdir}/references/${refId}" }, mode: params.publish_mode, saveAs: { filename ->
// Hide the raw copy of reference.fa to save space (the original input already exists),
// BUT keep it when publishing CRAM so the outputs are self-decodable.
if (filename == "reference.fa" && !params.output_cram) return null
@@ -111,6 +111,14 @@ process PREPARE_REFERENCE {
fi
fi
'''
+
+ stub:
+ """
+ touch reference.fa
+ # main.nf picks the .fai out of this index list, so it has to be part of the glob
+ touch reference.fa.amb reference.fa.ann reference.fa.bwt.2bit.64 reference.fa.pac reference.fa.0123 reference.fa.fai
+ touch Locus_to_exclude_${refId}.txt
+ """
}
@@ -169,12 +177,18 @@ process BUILD_MAPPABILITY {
--mask-window !{params.genmap_max_k} \
--out-prefix !{refId}
'''
+
+ stub:
+ """
+ touch ${refId}.min_unique_len.npz
+ touch Locus_to_exclude_mappability_${refId}.txt
+ """
}
process SNPEFF_BUILD_DB {
tag "SnpEff DB: ${refId}"
- publishDir "${params.outdir}/references/${refId}/snpeff", mode: params.publish_mode
+ publishDir path: { "${params.outdir}/references/${refId}/snpeff" }, mode: params.publish_mode
cpus 1
memory '8 GB'
@@ -214,4 +228,10 @@ EOF
# Build Database
snpEff build -c snpEff.config -gff3 -noCheckCds -noCheckProtein -v !{refId}
'''
+
+ stub:
+ """
+ touch snpEff.config
+ mkdir -p data/${refId}
+ """
}
diff --git a/modules/report.nf b/modules/report.nf
index 2031df2..47b4fc9 100644
--- a/modules/report.nf
+++ b/modules/report.nf
@@ -32,6 +32,12 @@ process COLLECT_SUMMARY {
# guarantee it exists (header only) so the channel/QC_REPORT always have a file.
[ -f ${basename}_gene_burden.tsv ] || printf 'gene\\thigh\\tmoderate\\tlow\\tmodifier\\tdominant_effect\\tn_samples\\ttotal_impactful\\tstart\\tend\\n' > ${basename}_gene_burden.tsv
"""
+
+ stub:
+ """
+ touch ${basename}_summary.tsv
+ touch ${basename}_gene_burden.tsv
+ """
}
process COLLECT_DR {
@@ -52,6 +58,11 @@ process COLLECT_DR {
set -euo pipefail
python3 ${projectDir}/bin/collect_dr.py -o ${basename}_dr.tsv ${dr_mutations}
"""
+
+ stub:
+ """
+ touch ${basename}_dr.tsv
+ """
}
process LIFT_VARIANTS {
@@ -86,6 +97,11 @@ process LIFT_VARIANTS {
printf 'src_pos\\ttgt_pos\\n' > ${basename}_pos_liftover.tsv
fi
"""
+
+ stub:
+ """
+ printf 'src_pos\\ttgt_pos\\n' > ${basename}_pos_liftover.tsv
+ """
}
process QC_REPORT {
@@ -124,9 +140,10 @@ process QC_REPORT {
MASK_ARG=""; [ -s "${mask_bed}" ] && MASK_ARG="--mask-bed ${mask_bed}"
MD_ARG=""; [ -s "${metadata}" ] && MD_ARG="--metadata ${metadata}"
VCF_ARG=""; [ -n "${vcfs}" ] && VCF_ARG="--vcfs ${vcfs}"
- VH_ARG=""; case "${vcfs_h37rv}" in ""|NO_FILE) ;; *) VH_ARG="--vcfs-h37rv ${vcfs_h37rv}";; esac
- PL_ARG=""; case "${pos_liftover}" in ""|NO_FILE) ;; *) [ -s "${pos_liftover}" ] && PL_ARG="--pos-liftover ${pos_liftover}";; esac
- DR_ARG=""; [ -s "${dr_report}" ] && [ "${dr_report}" != "NO_FILE" ] && DR_ARG="--dr-report ${dr_report}"
+ # NO_FILE* are the per-input placeholders main.nf stages when a feature is off (assets/NO_FILE_*).
+ VH_ARG=""; case "${vcfs_h37rv}" in ""|NO_FILE*) ;; *) VH_ARG="--vcfs-h37rv ${vcfs_h37rv}";; esac
+ PL_ARG=""; case "${pos_liftover}" in ""|NO_FILE*) ;; *) [ -s "${pos_liftover}" ] && PL_ARG="--pos-liftover ${pos_liftover}";; esac
+ DR_ARG=""; case "${dr_report}" in ""|NO_FILE*) ;; *) [ -s "${dr_report}" ] && DR_ARG="--dr-report ${dr_report}";; esac
KRK_ARG=""; [ -n "${kraken_reports}" ] && KRK_ARG="--kraken ${kraken_reports}"
python3 ${projectDir}/bin/qc_report.py \\
--summary ${summary} \\
@@ -148,6 +165,12 @@ process QC_REPORT {
--mapping-min ${params.report_mapping_min} \\
${gate_arg}
"""
+
+ stub:
+ """
+ touch ${basename}_qc_report.html
+ touch ${basename}_qc_flags.tsv
+ """
}
process SNP_MATRIX {
@@ -172,4 +195,9 @@ process SNP_MATRIX {
--reference "${reference}" \\
-o ${basename}_snp_matrix.tsv
"""
+
+ stub:
+ """
+ touch ${basename}_snp_matrix.tsv
+ """
}
diff --git a/modules/utils.nf b/modules/utils.nf
index b309b02..4f56ce5 100644
--- a/modules/utils.nf
+++ b/modules/utils.nf
@@ -164,3 +164,90 @@ def getSavePath(filename, params) {
// E. DEFAULT PUBLISH
return name
}
+
+/**
+ * Parse the declared parameters straight out of nextflow.config.
+ *
+ * Reading the config rather than keeping a second list of names means the two can never drift.
+ * Returns a list of [name, default, description, section] in declaration order.
+ */
+def declaredParams(String configPath) {
+ def f = new File(configPath)
+ if (!f.exists()) return []
+
+ def out = []
+ def section = 'General'
+ def depth = 0
+ def inside = false
+
+ f.readLines().each { raw ->
+ def line = raw.trim()
+ if (!inside) {
+ if (line ==~ /^params\s*\{.*/) { inside = true; depth = 1 }
+ return
+ }
+ // Track nesting by line so the matching close ends the block and a later one is ignored.
+ // A balanced "${...}" inside a value nets to zero, which is what we want.
+ def net = depth + line.count('{') - line.count('}')
+ if (net <= 0) { inside = false; return }
+ depth = net
+
+ def head = (line =~ /^\/\/\s*---\s*(.+?)\s*-*\s*$/)
+ if (head) { section = head[0][1]; return }
+ if (line.startsWith('//') || !line) return
+ // A quoted value is matched whole first, so a "//" inside it (a URL, a docker:// ref) is
+ // not mistaken for the start of a trailing comment.
+ def m = (line =~ /^([A-Za-z_][A-Za-z0-9_]*)\s*=\s*("[^"]*"|'[^']*'|.+?)\s*(?:\/\/\s*(.*))?$/)
+ if (m) out << [name: m[0][1], value: m[0][2], description: (m[0][3] ?: '').trim(), section: section]
+ }
+ return out
+}
+
+/**
+ * Reject an unknown --parameter instead of accepting it and doing nothing.
+ *
+ * Nextflow puts every --flag into params whether or not the pipeline declares it, so a typo like
+ * `--treads 16` is silently ignored and the run completes with the default. Anything Nextflow
+ * itself injects is allowed through.
+ */
+def validateParams(Map params, String configPath) {
+ def declared = declaredParams(configPath)*.name as Set
+ if (!declared) return // config unreadable: do not block the run over a validation helper
+
+ // Nextflow-injected and convention keys that are never declared in the params block.
+ def allowed = ['help', 'validate_params', 'monochrome_logs', 'igenomes_base'] as Set
+
+ def unknown = params.keySet().findAll { !declared.contains(it) && !allowed.contains(it) }.sort()
+ if (!unknown) return
+
+ def msg = new StringBuilder("Unrecognised parameter(s): ${unknown.collect{ "--${it}" }.join(', ')}\n")
+ unknown.each { u ->
+ // Cheap edit-distance suggestion: most real cases are one transposed or dropped letter.
+ def near = declared.findAll { d ->
+ d.size() > 3 && (d.toLowerCase().contains(u.toLowerCase()) || u.toLowerCase().contains(d.toLowerCase())
+ || (d.size() - u.size()).abs() <= 2 && (d.toSet().intersect(u.toSet()).size() >= u.size() - 1))
+ }.sort().take(3)
+ if (near) msg << " --${u}: did you mean ${near.collect{ "--${it}" }.join(' or ')}?\n"
+ }
+ msg << "Run with --help to list every parameter."
+ throw new RuntimeException(msg.toString())
+}
+
+/**
+ * Print every declared parameter with its default, grouped by the section headings in the config.
+ */
+def paramsHelp(String configPath, String version) {
+ def declared = declaredParams(configPath)
+ def out = new StringBuilder("\nBAMpiro ${version}\n\n")
+ out << " nextflow run main.nf --tsv samples.tsv --outdir results -profile local,docker\n\n"
+ out << "Profiles: standard (default, this host), local, slurm, garnatxa, docker, generic, test\n"
+ declared.groupBy{ it.section }.each { section, entries ->
+ out << "\n${section}\n"
+ entries.each { p ->
+ def val = p.value.replaceAll(/\s+/, ' ')
+ if (val.size() > 46) val = val.take(43) + '...'
+ out << String.format(" --%-26s %s\n", p.name, val)
+ }
+ }
+ return out.toString()
+}
diff --git a/modules/variants.nf b/modules/variants.nf
index e0a73cb..2b3db3f 100644
--- a/modules/variants.nf
+++ b/modules/variants.nf
@@ -13,7 +13,7 @@ process CALL_FREEBAYES {
// Use getSampleDir for nested output support.
// getSavePath handles the filtering of intermediate files if needed.
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
// FreeBayes is single-threaded unless region-parallel is enabled; only bgzip uses extra cores.
cpus { params.freebayes_parallel ? (params.freebayes_parallel_jobs as int) : 2 }
@@ -39,21 +39,6 @@ process CALL_FREEBAYES {
if [ -n "\${SLURM_TMPDIR:-}" ]; then export TMPDIR="\$SLURM_TMPDIR"; else export TMPDIR="./tmp_fb"; fi
mkdir -p "\$TMPDIR"
- # Helper function for safe tabix indexing
- safe_tabix () {
- local gz="\$1"; local idx="\${gz}.tbi"
- set +e; tabix -f -p vcf "\$gz"; st=\$?; set -e
- if [ \$st -ne 0 ]; then
- # Only a genuinely header-only VCF gets an empty index. bcftools reads the whole file,
- # so a truncated/malformed VCF errors under set -e (fails loud) instead of a fake index.
- local nrec; nrec=\$(bcftools view -H "\$gz" | wc -l)
- if [ "\$nrec" -ne 0 ]; then
- echo "ERROR: tabix failed on \$gz with \$nrec records (malformed/truncated) -> refusing a fake index" >&2
- exit \$st
- fi
- : > "\$idx"
- fi
- }
# 1. Handle Excluded Regions (Repeats)
# Keep only numeric-coordinate rows -> strips EVERY header (the combined nucmer+genmap exclude has two)
@@ -101,15 +86,12 @@ process CALL_FREEBAYES {
# Ensure file exists even if empty
[ -s !{sampleId}.!{refId}.mask_sites.tsv ] || : > !{sampleId}.!{refId}.mask_sites.tsv
- # 5. Define Filter Rules (Bash variables for bcftools expressions)
- base_filt="(FMT/DP >= !{params.filter_min_dp} || INFO/DP >= !{params.filter_min_dp}) && INFO/SAF[0] >= !{params.min_alt_fwd} && INFO/SAR[0] >= !{params.min_alt_rev}"
- af_guard="(INFO/AO[0] + INFO/RO) > 0"
- af_calc="((INFO/AO[0] * 1.0) / (INFO/AO[0] + INFO/RO))"
-
- # Homozygous Rule: High Allele Fraction
- hom_rule="\$base_filt && \$af_guard && \$af_calc >= !{params.hom_threshold}"
- # Heterozygous Rule: Mid Allele Fraction
- het_rule="\$base_filt && \$af_guard && \$af_calc >= !{params.het_min_frac} && \$af_calc < !{params.hom_threshold}"
+ # Hom / het are defined in ONE place (bin/vcf_filter_rules.py) so the two FreeBayes paths
+ # cannot drift apart. Run it with --help to see the expression it builds.
+ hom_rule=\$(vcf_filter_rules.py hom --min-dp !{params.filter_min_dp} --min-alt-fwd !{params.min_alt_fwd} --min-alt-rev !{params.min_alt_rev} \
+ --hom-threshold !{params.hom_threshold} --het-min-frac !{params.het_min_frac})
+ het_rule=\$(vcf_filter_rules.py het --min-dp !{params.filter_min_dp} --min-alt-fwd !{params.min_alt_fwd} --min-alt-rev !{params.min_alt_rev} \
+ --hom-threshold !{params.hom_threshold} --het-min-frac !{params.het_min_frac})
# 6. Apply Filters
bcftools view \$EXCL_ARG --types snps -i "\$hom_rule || \$het_rule" normalized.vcf > valid_snps.vcf || true
@@ -124,37 +106,24 @@ process CALL_FREEBAYES {
# Adds ADP, WT, HET, HOM, NC tags to INFO/FORMAT for the consensus step. INFO is tokenised on ';'
# (anchored ^DP=/^RO=/^AO=) so DP is never grabbed from a look-alike field, and the genotype is
# re-validated with RO/AO.
- awk -v HETMIN=!{params.het_min_frac} 'BEGIN{OFS="\\t"}
- /^##/ { print; next }
- /^#CHROM/ { print; next }
- {
- dp=0; ro=0; ao=0
- ni=split(\$8,I,";")
- for(k=1;k<=ni;k++){
- if(I[k] ~ /^DP=/){ v=I[k]; sub(/^DP=/,"",v); dp=v+0 }
- else if(I[k] ~ /^RO=/){ v=I[k]; sub(/^RO=/,"",v); ro=v+0 }
- else if(I[k] ~ /^AO=/){ v=I[k]; sub(/^AO=/,"",v); split(v,ax,","); ao=ax[1]+0 }
- }
- if (dp==0) { n=split(\$9,fmt,":"); m=split(\$10,dat,":"); for(i=1;i<=n;i++) if(fmt[i]=="DP") dp=dat[i] }
- if (dp==0) dp=ro+ao
- if (dp==0) dp=1
- split(\$10,b,":"); gtype=b[1]
- # Re-validate het: 'bcftools norm -m -' splits a multiallelic 1/2 into biallelic records whose
- # reference index 0 is an ARTIFACT (RO~0). A het whose reference allele is essentially
- # unsupported is not a real het -> homozygous-alt (matches the af-based hom/het rule and stops
- # the reference base leaking into the consensus IUPAC).
- if ((gtype=="0/1" || gtype=="1/0") && (ro+ao)>0 && ro/(ro+ao) < HETMIN) gtype="1/1"
- het=0; hom=0
- if (gtype=="0/1" || gtype=="1/0") het=1
- else if (gtype=="1/1") hom=1
- \$8="ADP="dp";WT=0;HET="het";HOM="hom";NC=0"
- \$9="GT:DP"
- \$10=gtype":"dp
- print
- }' valid_snps.vcf | bgzip -@ !{task.cpus} -c > valid_snps_formatted.vcf.gz
+ awk -v HETMIN=!{params.het_min_frac} \
+ -f !{projectDir}/bin/format_snps_for_backbone.awk valid_snps.vcf | bgzip -@ !{task.cpus} -c > valid_snps_formatted.vcf.gz
safe_tabix valid_snps_formatted.vcf.gz
"""
+
+ // Stubs create only the NEW outputs; note valid_snps_formatted.* is a fixed, non-interpolated name.
+ stub:
+ """
+ touch valid_snps_formatted.vcf.gz
+ touch valid_snps_formatted.vcf.gz.tbi
+ touch ${sampleId}.${refId}.mask_sites.tsv
+ touch ${sampleId}.${refId}.freebayes.raw.vcf.gz
+ touch ${sampleId}.${refId}.freebayes.raw.vcf.gz.tbi
+ touch ${sampleId}.${refId}.var.homo.SNPs.vcf
+ touch ${sampleId}.${refId}.var.het.SNPs.vcf
+ touch ${sampleId}.${refId}.var.homo.indel.vcf
+ """
}
process CALL_BACKBONE {
@@ -177,18 +146,6 @@ process CALL_BACKBONE {
if [ -n "\${SLURM_TMPDIR:-}" ]; then export TMPDIR="\$SLURM_TMPDIR"; else export TMPDIR="./tmp_bb"; fi
mkdir -p "\$TMPDIR"
- safe_tabix () {
- local gz="\$1"; local idx="\${gz}.tbi"
- set +e; tabix -f -p vcf "\$gz"; st=\$?; set -e
- if [ \$st -ne 0 ]; then
- local nrec; nrec=\$(bcftools view -H "\$gz" | wc -l)
- if [ "\$nrec" -ne 0 ]; then
- echo "ERROR: tabix failed on \$gz with \$nrec records (malformed/truncated) -> refusing a fake index" >&2
- exit \$st
- fi
- : > "\$idx"
- fi
- }
# Ensure index exists
samtools faidx !{ref_fa} >/dev/null 2>&1 || true
@@ -221,73 +178,31 @@ process CALL_BACKBONE {
n1=\$(wc -l < mpileup_nobaq.txt); n2=\$(wc -l < mpileup_baq.txt)
if [ "\$n1" -ne "\$n2" ]; then echo "ERROR: mpileup line counts differ (\$n1 vs \$n2) -- truncated stream" >&2; exit 1; fi
paste mpileup_nobaq.txt mpileup_baq.txt \\
- | awk -v MINCOV=!{params.allpos_min_cov} -v GAPDP=!{params.consensus_min_dp} -v OFS="\\t" '
- function count_bases(s, i,n,c,num){
- RO_G=0; AO_G=0; i=1; n=length(s)
- while(i<=n){
- c=substr(s,i,1)
- if(c=="^"){ i+=2 } # ^ + mapping-quality char (read start)
- else if(c=="\$"){ i++ } # read end
- else if(c=="+"||c=="-"){ # indel: skip sign + count + that many bases
- i++; num=""
- while(i<=n && substr(s,i,1) ~ /[0-9]/){ num=num substr(s,i,1); i++ }
- i+=(num+0)
- }
- else if(c=="."||c==","){ RO_G++; i++ } # match to reference
- else if(c ~ /[ACGTacgt]/){ AO_G++; i++ } # mismatch = alt observation
- else{ i++ } # * (del), <>, N, etc. -> ignore
- }
- }
- {
- chrom=\$1; pos=\$2; ref=\$3; dp=(\$4+0); dp_baq=(\$10+0);
- count_bases(\$5) # RO/AO from the no-BAQ pileup (real base evidence)
- wt=(dp>=MINCOV?1:0);
- nc=(dp>=MINCOV?0:1);
- gt=(dp>=MINCOV?"0":"./.");
- flt=(dp>GAPDP && dp_baq<=GAPDP)?"baq_dropout":".";
- info="ADP="dp";WT="wt";HET=0;HOM=0;NC="nc;
- print chrom, pos, ".", ref, ".", ".", flt, info, "GT:DP:AD", gt":"dp":"RO_G","AO_G
- }' | cat header_template.txt - | bgzip -@ !{task.cpus} -c > backbone.vcf.gz
+ | awk -v MINCOV=!{params.allpos_min_cov} -v GAPDP=!{params.consensus_min_dp} -v BAQ=1 \\
+ -v OFS="\\t" -f !{projectDir}/bin/backbone_allpos.awk | cat header_template.txt - | bgzip -@ !{task.cpus} -c > backbone.vcf.gz
else
samtools mpileup -aa -f !{ref_fa} -Q !{params.allpos_min_bq} -d !{params.allpos_max_depth} !{bam} \\
- | awk -v MINCOV=!{params.allpos_min_cov} -v OFS="\\t" '
- function count_bases(s, i,n,c,num){
- RO_G=0; AO_G=0; i=1; n=length(s)
- while(i<=n){
- c=substr(s,i,1)
- if(c=="^"){ i+=2 }
- else if(c=="\$"){ i++ }
- else if(c=="+"||c=="-"){
- i++; num=""
- while(i<=n && substr(s,i,1) ~ /[0-9]/){ num=num substr(s,i,1); i++ }
- i+=(num+0)
- }
- else if(c=="."||c==","){ RO_G++; i++ }
- else if(c ~ /[ACGTacgt]/){ AO_G++; i++ }
- else{ i++ }
- }
- }
- {
- chrom=\$1; pos=\$2; ref=\$3; dp=(\$4+0);
- count_bases(\$5)
- wt=(dp>=MINCOV?1:0);
- nc=(dp>=MINCOV?0:1);
- gt=(dp>=MINCOV?"0":"./.");
- info="ADP="dp";WT="wt";HET=0;HOM=0;NC="nc;
- print chrom, pos, ".", ref, ".", ".", ".", info, "GT:DP:AD", gt":"dp":"RO_G","AO_G
- }' | cat header_template.txt - | bgzip -@ !{task.cpus} -c > backbone.vcf.gz
+ | awk -v MINCOV=!{params.allpos_min_cov} -v BAQ=0 \\
+ -v OFS="\\t" -f !{projectDir}/bin/backbone_allpos.awk | cat header_template.txt - | bgzip -@ !{task.cpus} -c > backbone.vcf.gz
fi
safe_tabix backbone.vcf.gz
rm -f mpileup_nobaq.txt mpileup_baq.txt
"""
+
+ stub:
+ """
+ touch backbone.vcf.gz
+ touch backbone.vcf.gz.tbi
+ touch header_template.txt
+ """
}
process MERGE_VCFS {
tag "Merge: ${sampleId}"
// Use getSampleDir for nested output support
- publishDir "${params.outdir}/${getSampleDir(sampleId, params)}", mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
+ publishDir path: { "${params.outdir}/${getSampleDir(sampleId, params)}" }, mode: params.publish_mode, saveAs: { filename -> getSavePath(filename, params) }
// bcftools (concat/sort/view) on a bacterial VCF is light and mostly single-threaded.
cpus 2
@@ -307,21 +222,6 @@ process MERGE_VCFS {
if [ -n "\${SLURM_TMPDIR:-}" ]; then export TMPDIR="\$SLURM_TMPDIR"; else export TMPDIR="./tmp_merge"; fi
mkdir -p "\$TMPDIR"
- safe_tabix () {
- local gz="\$1"; local idx="\${gz}.tbi"
- set +e; tabix -f -p vcf "\$gz"; st=\$?; set -e
- if [ \$st -ne 0 ]; then
- # Only a genuinely header-only VCF gets an empty index. bcftools reads the whole
- # file, so a truncated/malformed VCF errors under set -e (fails loud) instead of
- # getting a fake index that would let a short VCF flow into the consensus.
- local nrec; nrec=\$(bcftools view -H "\$gz" | wc -l)
- if [ "\$nrec" -ne 0 ]; then
- echo "ERROR: tabix failed on \$gz with \$nrec records (malformed/truncated) -> refusing a fake index" >&2
- exit \$st
- fi
- : > "\$idx"
- fi
- }
# 1. Apply proper header to SNP VCF
bcftools reheader -h !{hdr_template} !{snps_vcf} > clean_snps.vcf.gz
@@ -362,6 +262,14 @@ process MERGE_VCFS {
bgzip -t !{sampleId}.!{refId}!{outLabel}.all.pos.vcf.gz
tabix -f -p vcf !{sampleId}.!{refId}!{outLabel}.all.pos.vcf.gz
"""
+
+ stub:
+ """
+ touch ${sampleId}.${refId}${outLabel}.all.pos.vcf.gz
+ touch ${sampleId}.${refId}${outLabel}.all.pos.vcf.gz.tbi
+ touch ${sampleId}.${refId}${outLabel}.vcf.gz
+ touch ${sampleId}.${refId}${outLabel}.vcf.gz.tbi
+ """
}
process CALL_FREEBAYES_RAW {
@@ -411,41 +319,26 @@ process CALL_FREEBAYES_RAW {
bcftools norm -m - -a -f !{ref_fa} raw_freebayes.vcf | bcftools view -e 'GT="0/0" || GT="0"' -Ov -o normalized.vcf
- base_filt="(FMT/DP >= !{params.filter_min_dp} || INFO/DP >= !{params.filter_min_dp}) && INFO/SAF[0] >= !{params.min_alt_fwd} && INFO/SAR[0] >= !{params.min_alt_rev}"
- af_guard="(INFO/AO[0] + INFO/RO) > 0"
- af_calc="((INFO/AO[0] * 1.0) / (INFO/AO[0] + INFO/RO))"
- hom_rule="\$base_filt && \$af_guard && \$af_calc >= !{params.hom_threshold}"
- het_rule="\$base_filt && \$af_guard && \$af_calc >= !{params.het_min_frac} && \$af_calc < !{params.hom_threshold}"
+ # Hom / het are defined in ONE place (bin/vcf_filter_rules.py) so the two FreeBayes paths
+ # cannot drift apart. Run it with --help to see the expression it builds.
+ hom_rule=\$(vcf_filter_rules.py hom --min-dp !{params.filter_min_dp} --min-alt-fwd !{params.min_alt_fwd} --min-alt-rev !{params.min_alt_rev} \
+ --hom-threshold !{params.hom_threshold} --het-min-frac !{params.het_min_frac})
+ het_rule=\$(vcf_filter_rules.py het --min-dp !{params.filter_min_dp} --min-alt-fwd !{params.min_alt_fwd} --min-alt-rev !{params.min_alt_rev} \
+ --hom-threshold !{params.hom_threshold} --het-min-frac !{params.het_min_frac})
bcftools view \$EXCL_ARG --types snps -i "\$hom_rule || \$het_rule" normalized.vcf > valid_snps.vcf || true
[ -s valid_snps.vcf ] || bcftools view -h normalized.vcf > valid_snps.vcf
- awk -v HETMIN=!{params.het_min_frac} 'BEGIN{OFS="\\t"}
- /^##/ { print; next }
- /^#CHROM/ { print; next }
- {
- dp=0; ro=0; ao=0
- ni=split(\$8,I,";")
- for(k=1;k<=ni;k++){
- if(I[k] ~ /^DP=/){ v=I[k]; sub(/^DP=/,"",v); dp=v+0 }
- else if(I[k] ~ /^RO=/){ v=I[k]; sub(/^RO=/,"",v); ro=v+0 }
- else if(I[k] ~ /^AO=/){ v=I[k]; sub(/^AO=/,"",v); split(v,ax,","); ao=ax[1]+0 }
- }
- if (dp==0) { n=split(\$9,fmt,":"); m=split(\$10,dat,":"); for(i=1;i<=n;i++) if(fmt[i]=="DP") dp=dat[i] }
- if (dp==0) dp=ro+ao
- if (dp==0) dp=1
- split(\$10,b,":"); gtype=b[1]
- if ((gtype=="0/1" || gtype=="1/0") && (ro+ao)>0 && ro/(ro+ao) < HETMIN) gtype="1/1"
- het=0; hom=0
- if (gtype=="0/1" || gtype=="1/0") het=1
- else if (gtype=="1/1") hom=1
- \$8="ADP="dp";WT=0;HET="het";HOM="hom";NC=0"
- \$9="GT:DP"
- \$10=gtype":"dp
- print
- }' valid_snps.vcf | bgzip -@ !{task.cpus} -c > valid_snps_formatted.vcf.gz
+ awk -v HETMIN=!{params.het_min_frac} \
+ -f !{projectDir}/bin/format_snps_for_backbone.awk valid_snps.vcf | bgzip -@ !{task.cpus} -c > valid_snps_formatted.vcf.gz
set +e; tabix -f -p vcf valid_snps_formatted.vcf.gz; st=\$?; set -e
if [ \$st -ne 0 ]; then : > valid_snps_formatted.vcf.gz.tbi; fi
"""
+
+ stub:
+ """
+ touch valid_snps_formatted.vcf.gz
+ touch valid_snps_formatted.vcf.gz.tbi
+ """
}
diff --git a/nextflow.config b/nextflow.config
index 70fbb16..06af717 100644
--- a/nextflow.config
+++ b/nextflow.config
@@ -10,22 +10,35 @@
// 1. Pipeline Parameters
params {
// --- Inputs & System ---
- tsv = "samples_legio.tsv"
+ help = false // --help lists every parameter with its default, then exits
+ // Required. There is no sensible default samplesheet, so the run stops with a clear message.
+ tsv = null
outdir = "results_bampiro"
threads = 8
// The image is published to BOTH registries by .github/workflows/docker-publish.yml (same digest):
// - Docker Hub paururo/bampiro (public -> the cluster pulls it without auth)
// - GHCR ghcr.io/pathogenomics-lab/bampiro (kept private for now; needs a read:packages PAT)
- // Points at the PUBLIC Docker Hub image so Singularity/Apptainer pulls it with no credentials. When you
- // make the GHCR package public you can switch to "docker://ghcr.io/pathogenomics-lab/bampiro:1.0.1".
+ // PINNED BY DIGEST, not by tag: a tag can be repointed at a different image, so two runs a month
+ // apart would silently use different tool versions. This digest is the 1.0.1 image; to move to a new
+ // release, publish it and copy the digest the action prints. `--container` overrides it, and the
+ // resolved value is recorded in the report's provenance footer either way.
// For a PRIVATE image export APPTAINER_DOCKER_USERNAME= + APPTAINER_DOCKER_PASSWORD=.
- // RE-PIN to the exact digest for reproducibility (the action prints it): container = "docker://paururo/bampiro@sha256:".
- // Override with --container to test a different image.
- container = "docker://paururo/bampiro:1.0.1"
+ container = "docker://paururo/bampiro@sha256:c3bc85157c866154490162603ce8cdcffe86e41d3f065064104d6eb875a1cd19"
+ // --- Resource ceilings ---
+ // Each process asks for what its tool actually needs, and Kraken2 asks for 80 GB, which no
+ // laptop has: Nextflow would queue that task forever (local executor) or have the scheduler
+ // reject it. These cap EVERY request to what the machine can give, so one flag makes the whole
+ // pipeline fit rather than editing nine modules. Null means "no cap" (a cluster, where the
+ // per-process numbers are the point). Sizes take Nextflow units: '16.GB', '2.h'.
+ // --max_cpus 4 --max_memory 8.GB
+ max_cpus = null
+ max_memory = null
+ max_time = null
+
// Controls whether sample folders are nested (e.g., MP001 -> MP/00/1)
// Set to 'true' to enable hierarchical output folders.
- nested_output = true
+ nested_output = true
// --- Pathotypr Settings (containerised: bioconda `pathotypr`, marker panels from Zenodo v1.0.0) ---
// Alignment-free (k-mer) MTBC lineage + WHO drug-resistance genotyping. Reference-agnostic: works even
@@ -43,7 +56,9 @@ params {
pathotypr_min_alt = 95
// --- Kraken2 Settings ---
- kraken2_db = "/storage/shared_datasets/kraken2"
+ // No default: point it at your own database (or a site profile's). Without one the
+ // contamination screen is skipped and the report's taxonomy panel hides itself.
+ kraken2_db = null
krakentools_url = "https://raw.githubusercontent.com/jenniferlu717/KrakenTools/master/extract_kraken_reads.py"
// Read the Kraken DB via mmap so parallel tasks SHARE it in RAM instead of each loading
// the full DB. Big peak-memory saving; slightly slower per task on a cold page cache.
@@ -178,22 +193,21 @@ params {
}
// 2. Singularity Configuration
-// Enabled globally for this environment
+// autoMounts binds the work and launch directories. Reads, references and the Kraken database
+// usually live elsewhere on a shared filesystem, so a cluster needs its own --bind list: put it in
+// a site profile (see conf/garnatxa.config) rather than here.
singularity {
enabled = true
autoMounts = true
- // Specific bind paths for your cluster environment
- runOptions = '--bind /scr/ruizro/test_pipes/bampiro/,/storage/shared_datasets/kraken2'
}
// 3. Process Execution Configuration
+// Nothing in this block is site-specific. The executor, the queue/QoS and any `module load` belong
+// to a profile, so a clone of this repository runs correctly anywhere without being edited.
process {
- // Executor settings (SLURM)
- executor = 'slurm'
- clusterOptions = '--qos=short'
-
- // Environment setup before running tasks
- beforeScript = 'module load singularity'
+ executor = 'local'
+
+ // Note: resourceLimits is set at the BOTTOM of this file, after the profiles.
// Container definition (pinned digest via params.container -> reproducible builds)
container = params.container
@@ -202,8 +216,8 @@ process {
// Capture exit codes from upstream processes when piping
shell = ['/bin/bash', '-euo', 'pipefail']
- // Output management: publish mode is user-selectable ('copy' default, 'link' to hardlink).
- publishDirMode = params.publish_mode
+ // Note: there is no `publishDirMode` process directive. The publish mode is user-selectable
+ // via params.publish_mode, which every process passes to its own `publishDir mode:`.
// Retry only on OOM (137/OOM-killer) or signal kills / preemption (SIGTERM 143,
// SIGABRT 134, SIGSEGV 139, 140); genuine errors still fail fast. Processes whose
@@ -262,20 +276,42 @@ workflow.onError = {
System.err.println "BAMpiro ${workflow.runName} FAILED: ${workflow.errorReport ?: workflow.errorMessage}"
}
-// 4. Execution Profiles (Optional - kept standard structure)
+// 4. Execution Profiles
+//
+// WHERE the work runs is a profile, never a default. Pick one executor profile and combine it with
+// as many option profiles as you like, comma-separated and with no spaces:
+//
+// -profile local,docker a laptop, with Docker
+// -profile slurm any SLURM cluster
+// -profile garnatxa the I2SysBio cluster, with its paths already set
+// -profile slurm,generic a SLURM cluster, non-tuberculosis organism
+//
+// With no -profile Nextflow applies `standard`, which runs everything on the current host.
profiles {
+ // 'standard' = the default. Runs on THIS machine, so a fresh clone works anywhere without being
+ // edited. On a cluster login node that means the login node, so choose an executor profile
+ // deliberately for real work. The startup banner prints the executor it resolved.
standard {
- process.executor = 'slurm'
+ process.executor = 'local'
}
- // 'local' = run on THIS machine (laptop / VM / any HPC login node), NOT a SLURM cluster. Overrides the
- // cluster-specific bits of the base config: the SLURM executor, the `module load singularity` beforeScript,
- // and the CSIC-specific singularity --bind paths (autoMounts still binds the work dir). A local Kraken DB
- // outside the launch dir may still need `--bind /path` — see docs/installation.md.
+ // 'local' = an explicit alias for standard, for a command that says what it does.
local {
- process.executor = 'local'
- process.clusterOptions = ''
- process.beforeScript = ''
- singularity.runOptions = ''
+ process.executor = 'local'
+ }
+ // 'slurm' = a generic SLURM cluster: no queue, no account and no module system assumed. Add your
+ // site's QoS/partition and your Singularity --bind paths in a small -c config, or copy
+ // conf/garnatxa.config as a starting point. See docs/tutorials/configuring-a-run.md.
+ slurm {
+ process.executor = 'slurm'
+ executor {
+ queueSize = 100
+ submitRateLimit = '50/1min'
+ pollInterval = '30s'
+ }
+ }
+ // 'garnatxa' = the I2SysBio / University of Valencia cluster, ready to run as-is.
+ garnatxa {
+ includeConfig 'conf/garnatxa.config'
}
// 'docker' = use Docker instead of Singularity. On a laptop combine it with local: `-profile local,docker`.
docker {
@@ -296,15 +332,40 @@ profiles {
params.mask_blindspots = false // H37Rv Illumina blind-spots mask (MTBC-only)
params.annotate_canonical = false // dual H37Rv amino-acid numbering (set your own DB to re-enable)
}
+ // 'test' = the bundled fixture cohort (tests/data), sized for `-stub-run` so CI can validate the whole
+ // DAG in seconds with no container and no reference data. 'test_full' additionally turns ON every
+ // optional branch (Kraken, pathotypr, canonical annotation, liftover, virgin consensus) so a single run
+ // instantiates every process. Launch both from the repository root:
+ // nextflow run main.nf -profile test -stub-run
+ test {
+ includeConfig 'conf/test.config'
+ }
+ test_full {
+ includeConfig 'conf/test.config'
+ includeConfig 'conf/test_full.config'
+ }
// You can add 'debug' or 'local' profiles here if needed in the future
}
-// 5. Pipeline Metadata
+// 5. Resource ceilings
+// Deliberately AFTER the profiles: this reads params.max_*, and a profile that sets them (see
+// conf/test.config) is only merged when its block is parsed. Evaluated higher up, a profile's
+// ceilings would be silently ignored. An entry is dropped when its param is null, so an empty map
+// means every process keeps the size it asked for.
+process {
+ resourceLimits = [
+ cpus: params.max_cpus ? (params.max_cpus as int) : null,
+ memory: params.max_memory ? (params.max_memory as nextflow.util.MemoryUnit) : null,
+ time: params.max_time ? (params.max_time as nextflow.util.Duration) : null,
+ ].findAll { k, v -> v != null }
+}
+
+// 6. Pipeline Metadata
manifest {
name = 'BAMpiro'
author = 'Paula Ruiz-Rodriguez (paula.ruiz.rodriguez@csic.es) / PathoGenOmics-Lab'
description = 'Short Read mapping pipeline with Variant Calling'
- version = '1.0.1'
+ version = '1.1.0'
homePage = 'https://github.com/PathoGenOmics-Lab/BAMpiro'
mainScript = 'main.nf'
nextflowVersion = '!>=24.04.2'
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..75601eb
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,29 @@
+# Tooling configuration for the Python under bin/ and the test suite under tests/.
+# BAMpiro itself is a Nextflow pipeline and is not installed as a package.
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+# Make the pipeline's scripts importable as plain modules from the tests.
+pythonpath = ["bin"]
+addopts = "-ra --strict-markers"
+markers = [
+ "nextflow: needs the `nextflow` CLI on PATH (skipped when absent)",
+]
+
+[tool.ruff]
+line-length = 120
+extend-exclude = [
+ # Vendored verbatim from KrakenTools (see params.krakentools_url); kept
+ # byte-identical to upstream, so it is not ours to restyle.
+ "bin/extract_kraken_reads.py",
+]
+
+[tool.ruff.lint]
+# Deliberately narrow: catch real defects (undefined names, unused imports and
+# variables, broken f-strings, syntax errors, mutable default arguments) without
+# imposing a restyle on a codebase that chose a compact, dense layout.
+select = ["F", "E9", "B006", "B008", "B017", "B904"]
+
+[tool.ruff.lint.per-file-ignores]
+# Tests legitimately import a module purely to assert it imports cleanly.
+"tests/**" = ["F401"]
diff --git a/subworkflows/annotate.nf b/subworkflows/annotate.nf
new file mode 100644
index 0000000..f7318c1
--- /dev/null
+++ b/subworkflows/annotate.nf
@@ -0,0 +1,65 @@
+/*
+===============================================================================
+ Stage 8: Annotation
+===============================================================================
+ snpEff over the legacy split VCFs and over the main per-sample VCF.
+*/
+
+include { ANNOTATE_LEGACY_VCF; ANNOTATE_MAIN_VCF } from '../modules/annotation'
+
+workflow ANNOTATE {
+
+ take:
+ homo_snp // (sId, rId, var.homo.SNPs.vcf)
+ het_snp // (sId, rId, var.het.SNPs.vcf)
+ homo_indel // (sId, rId, var.homo.indel.vcf)
+ raw_fb // (sId, rId, freebayes.raw.vcf.gz, tbi)
+ main_vcf // (sId, rId, vcf.gz, tbi)
+ allpos_vcf // (sId, rId, all.pos.vcf.gz, tbi)
+ snpeff_db // (refId, snpEff.config, data)
+
+ main:
+
+ // 8. Annotation
+ // Annotate Legacy split VCFs (Homo/Het/Indel/Raw)
+ def freebayes_ann = Channel.empty() // annotated freebayes.raw (AD + snpEff ANN) -> SNP dynamics
+ if (params.annotate_legacy_vcfs) {
+ def leg_inputs = Channel.empty()
+ .mix(homo_snp.map { sId, rId, vcf -> tuple(rId, sId, "var.homo.SNPs", vcf) })
+ .mix(het_snp.map { sId, rId, vcf -> tuple(rId, sId, "var.het.SNPs", vcf) })
+ .mix(homo_indel.map { sId, rId, vcf -> tuple(rId, sId, "var.homo.indel", vcf) })
+ .mix(raw_fb.map { sId, rId, vcf, tbi -> tuple(rId, sId, "freebayes.raw", vcf) })
+
+ def ann_leg_in = leg_inputs.combine(snpeff_db, by: 0)
+ .map { rId, sId, lbl, vcf, cfg, dat -> tuple(sId, rId, lbl, vcf, cfg, dat) }
+
+ def ann_leg = ANNOTATE_LEGACY_VCF(ann_leg_in)
+ // Keep the annotated freebayes VCF: it carries per-alt AD (true AF) AND gene/effect (ANN),
+ // so the dynamics panel gets continuous allele frequencies without losing gene annotation.
+ freebayes_ann = ann_leg.out
+ .filter { sId, rId, vcf, tbi -> vcf.name.contains('freebayes.raw') }
+ .map { sId, rId, vcf, tbi -> vcf }
+ }
+
+ // Annotate Main VCF
+ def vcf_for_stats = Channel.empty()
+ def ch_snpeff_stats = Channel.empty()
+
+ if (params.annotate_main_vcf) {
+ def main_pre = main_vcf.map { sId, rId, vcf, tbi -> tuple(rId, sId, vcf, tbi) }
+ def ann_main_in = main_pre.combine(snpeff_db, by: 0)
+ .map { rId, sId, vcf, tbi, cfg, dat -> tuple(sId, rId, vcf, tbi, cfg, dat) }
+
+ def ann_out = ANNOTATE_MAIN_VCF(ann_main_in)
+
+ vcf_for_stats = ann_out.vcf_ann.map { sId, rId, vcf, tbi -> tuple(sId, rId, vcf) }
+ ch_snpeff_stats = ann_out.stats // Collect SnpEff stats CSV for MultiQC
+ } else {
+ vcf_for_stats = allpos_vcf.map { sId, rId, vcf, tbi -> tuple(sId, rId, vcf) }
+ }
+
+ emit:
+ freebayes_vcf = freebayes_ann // annotated freebayes.raw; empty when annotate_legacy_vcfs is off
+ stats_vcf = vcf_for_stats // (sId, rId, vcf) feeding the legacy stats and the cohort report
+ snpeff_stats = ch_snpeff_stats // SnpEff CSV for MultiQC; empty when annotate_main_vcf is off
+}
diff --git a/subworkflows/call_variants.nf b/subworkflows/call_variants.nf
new file mode 100644
index 0000000..4b6c317
--- /dev/null
+++ b/subworkflows/call_variants.nf
@@ -0,0 +1,38 @@
+/*
+===============================================================================
+ Stage 6: Variant Calling
+===============================================================================
+ FreeBayes for the variant sites, the backbone for every other position; the
+ two are merged into the all-positions VCF.
+*/
+
+include { CALL_FREEBAYES; CALL_BACKBONE; MERGE_VCFS } from '../modules/variants'
+
+workflow CALL_VARIANTS {
+
+ take:
+ vbase // (sId, rId, bam, bai, ref_fa, exclude_txt)
+
+ main:
+
+ // 6. Variant Calling
+ // Run FreeBayes and Backbone parallel
+ def fb_out = CALL_FREEBAYES(vbase)
+ def bb_in = vbase.map { sId, rId, bam, bai, ref_fa, exclude_txt -> tuple(sId, rId, bam, bai, ref_fa) }
+ def bb_out = CALL_BACKBONE(bb_in)
+
+ // Merge specific variants with backbone for All-Positions VCF (outLabel "" = masked path)
+ def merge_in = fb_out.snps.join(bb_out.backbone, by: [0,1]).join(bb_out.header, by: [0,1])
+ .map { sId, rId, sv, st, bv, bt, hdr -> tuple(sId, rId, sv, st, bv, bt, hdr, "") }
+ def vcf_ch = MERGE_VCFS(merge_in)
+
+ emit:
+ snps = fb_out.snps // (sId, rId, valid_snps.vcf.gz, tbi)
+ mask_sites = fb_out.mask_sites // (sId, rId, mask_sites.tsv)
+ raw_fb = fb_out.raw_fb // (sId, rId, freebayes.raw.vcf.gz, tbi)
+ homo_snp = fb_out.homo_snp // legacy split VCFs
+ het_snp = fb_out.het_snp
+ homo_indel = fb_out.homo_indel
+ allpos = vcf_ch.allpos // (sId, rId, all.pos.vcf.gz, tbi)
+ main_vcf = vcf_ch.main_vcf // (sId, rId, vcf.gz, tbi)
+}
diff --git a/subworkflows/cohort_report.nf b/subworkflows/cohort_report.nf
new file mode 100644
index 0000000..8578364
--- /dev/null
+++ b/subworkflows/cohort_report.nf
@@ -0,0 +1,88 @@
+/*
+===============================================================================
+ Stage 11: Cohort-level outputs
+===============================================================================
+ The consolidated interactive QC report and the master SNP matrix.
+*/
+
+include { ANNOTATE_CANONICAL } from '../modules/annotation'
+include { COLLECT_SUMMARY; COLLECT_DR; LIFT_VARIANTS; QC_REPORT; SNP_MATRIX } from '../modules/report'
+
+workflow COHORT_REPORT {
+
+ take:
+ legacy_log // per-sample GENERATE_LEGACY_STATS logs
+ masked_consensus // (sId, rId, consensus fasta); empty when make_consensus is off
+ ref_bundle // (refId, ref_fa, indices, exclude_txt)
+ kraken_reports // per-sample Kraken2 reports; empty when Kraken is off
+ patho_dr_results // (sId, dr mutations); empty when pathotypr is off
+ vcf_for_stats // (sId, rId, vcf)
+ freebayes_ann // annotated freebayes.raw VCFs; empty when annotate_legacy_vcfs is off
+ refMap // refId -> refFasta
+ refGffMap // refId -> refGff
+ tsv_name // samplesheet basename, used to name the cohort-level outputs
+
+ main:
+
+ // 11. Cohort-level outputs (QC report + master SNP matrix).
+ // Per-sample annotated VCFs, shared by both: prefer the annotated freebayes VCF (per-alt AD ->
+ // continuous AF + gene/effect); fall back to the annotated main VCF (GT-based AF) if legacy
+ // annotation is off. reference name(s) the samples were mapped against.
+ def report_vcfs = (params.annotate_legacy_vcfs ? freebayes_ann
+ : vcf_for_stats.map { sId, rId, vcf -> vcf })
+ .collect().ifEmpty([])
+ def ref_name = refMap.keySet().join(',')
+
+ // 11a. Consolidated QC report: aggregate every per-sample legacy log into one summary TSV, then
+ // render the self-contained interactive HTML + PASS/WARN/FAIL flags.
+ if (params.make_qc_report) {
+ def all_logs = legacy_log.collect()
+ def summ = COLLECT_SUMMARY(all_logs, tsv_name)
+ def cons_files = masked_consensus.map { sId, rId, fa -> fa }.collect().ifEmpty([])
+ // Reference-level extras (cohort report -> take the reference bundle; single-ref is the norm):
+ // GFF enables the per-gene SNP-density panel, the nucmer/repeat BED the masked-regions panel.
+ def report_ref = refGffMap.keySet().toList().first() // deterministic: the first reference
+ def report_gff = file(refGffMap[report_ref])
+ def report_mask = ref_bundle.filter { rId, fa, idx, excl -> rId == report_ref }
+ .map { rId, fa, idx, excl -> excl }.first()
+ // Provenance footer: pinned container digest + reference(s).
+ def provenance = (["container=${params.container}"] + refMap.keySet().collect { "reference=${it}" })
+ .collect { "\"${it}\"" }.join(' ')
+ // SNP dynamics: the samplesheet is the metadata source (auto-detects time/group columns; the
+ // panel self-hides if absent).
+ def report_meta = file(params.tsv)
+ // Optional QC_REPORT inputs use a per-input placeholder from assets/. They must have DISTINCT
+ // names: Nextflow refuses to stage two inputs of the same task under one filename, and all three
+ // of these are off by default, so a single shared "NO_FILE" fails the run with an input file name
+ // collision. They are real empty files (a dangling symlink breaks under stageInMode 'copy') and
+ // are referenced through projectDir so a stray NO_FILE in the launch directory is never picked up.
+ // QC_REPORT recognises them by the NO_FILE* prefix.
+ // Drug-resistance calls (pathotypr DR run -> one run TSV); placeholder when pathotypr is off.
+ def dr_report = params.run_pathotypr
+ ? COLLECT_DR(patho_dr_results.map { sId, f -> f }.collect().ifEmpty([]), tsv_name).dr
+ : file("${projectDir}/assets/NO_FILE_DR")
+ // Canonical-reference-annotated VCFs for the dual amino-acid numbering (off unless annotate_canonical).
+ // vcf_for_stats is the per-sample (sId,rId,vcf) channel; its positions match report_vcfs, so the
+ // report's sample+position merge finds each variant's canonical amino-acid change.
+ def report_vcfs_h37rv = params.annotate_canonical
+ ? ANNOTATE_CANONICAL(vcf_for_stats, params.canonical_snpeff_db).out
+ .map { sId, vcf -> vcf }.collect().ifEmpty([])
+ : file("${projectDir}/assets/NO_FILE_H37RV")
+ // Kraken2 per-sample reports (deduped) -> Taxonomic composition panel; empty when Kraken is off.
+ def report_kraken = kraken_reports.unique { it.name }.collect().ifEmpty([])
+ // Alignment-free canonical COORDINATE per variant via pathotypr (alternative to the --vcfs-h37rv path):
+ // lift the run's variant positions (mapping-reference coords) onto H37Rv and hand the map to the report.
+ def report_ref_fa = ref_bundle.filter { rId, fa, idx, excl -> rId == report_ref }
+ .map { rId, fa, idx, excl -> fa }.first()
+ def pos_liftover = params.variant_liftover
+ ? LIFT_VARIANTS(report_vcfs, report_ref_fa, report_ref, tsv_name).map
+ : file("${projectDir}/assets/NO_FILE_LIFTOVER")
+ QC_REPORT(summ.summary, summ.gene_burden, cons_files, report_gff, report_mask,
+ report_meta, report_vcfs, report_vcfs_h37rv, pos_liftover, dr_report, report_kraken, provenance, tsv_name)
+ }
+
+ // 11b. Master SNP matrix: rows = SNP sites, columns = reference/annotation + per-sample AF & depth.
+ if (params.make_snp_matrix) {
+ SNP_MATRIX(report_vcfs, ref_name, tsv_name)
+ }
+}
diff --git a/subworkflows/legacy_stats.nf b/subworkflows/legacy_stats.nf
new file mode 100644
index 0000000..2f0b8cc
--- /dev/null
+++ b/subworkflows/legacy_stats.nf
@@ -0,0 +1,79 @@
+/*
+===============================================================================
+ Stage 9: Legacy Stats Generation
+===============================================================================
+ One per-sample log combining the VCF, the BAM stats, the FastP JSONs and the
+ optional pathotypr lineage summary.
+*/
+
+include { GENERATE_LEGACY_STATS } from '../modules/annotation'
+
+workflow LEGACY_STATS {
+
+ take:
+ ref_bundle // (refId, ref_fa, indices, exclude_txt)
+ pe_json // (sId, fastp json) from FASTP_PE
+ se_json // (sId, fastp json) from FASTP_SE
+ // BAM stats already carry (sampleId, refId) as vals -- do NOT re-parse the filename with
+ // tokenize('.'), which truncates any dotted refId (e.g. NC_000962.3) and breaks the join.
+ bam_stats_ch // [sId, rId, stats]
+ vcf_for_stats // (sId, rId, vcf)
+ patho_results // (sId, lineage summary); empty when pathotypr is off
+
+ main:
+
+ // 9. Legacy Stats Generation
+ // Prepare Reference Index channel
+ def ref_fai_ch = ref_bundle.map { rId, fa, indices, excl ->
+ def fai_file = indices.find { it.name.endsWith('.fai') }
+ tuple(rId, fai_file)
+ }
+
+ // Prepare FastP JSONs (grouped by sample). A multi-lane sample has one JSON per lane; pass them ALL
+ // so GENERATE_LEGACY_STATS aggregates read counts (SUM) and quality rates (read/base-weighted) instead
+ // of silently keeping a single lane. Grouped on the real sampleId val (do NOT re-derive it from the
+ // filename: a sampleId containing '__' would be truncated by split('__')). Sorted for a stable -resume.
+ def json_ch = pe_json.mix(se_json) // [sId, json]
+ .groupTuple() // -> [sId, [json_lane1, json_lane2, ...]]
+ .map { sId, jsons -> tuple(sId, jsons.toSorted { it.name }) }
+
+ // Join logic for Stats
+ def vcf_bam_joined = vcf_for_stats.join(bam_stats_ch, by: [0,1]) // [sId, rId, vcf, stats]
+ // combine (not join) on sampleId: one json per sample must fan out to EVERY (sample,ref)
+ // row, else a sample mapped to >1 reference silently loses all but one reference's stats.
+ def vcf_bam_fastp = vcf_bam_joined.combine(json_ch, by: 0) // [sId, rId, vcf, stats, json]
+
+ // Join with Reference Index (Using COMBINE to reuse reference)
+ def ready_no_patho = vcf_bam_fastp
+ .map { sId, rId, vcf, stats, json -> tuple(rId, sId, vcf, stats, json) }
+ .combine(ref_fai_ch)
+ .filter { it[0] == it[5] } // Filter where sample RefID == FAI RefID
+ .map { rId, sId, vcf, stats, json, rId2, fai -> tuple(sId, rId, json, stats, vcf, fai) }
+
+ // Join with Pathotypr (Using remainder to handle missing files safely)
+ def final_stats_input = ready_no_patho
+ .join(patho_results, by: 0, remainder: true)
+ .map { list ->
+ // list size < 7 means missing left-side data (e.g. mapping failed)
+ if (list.size() < 7) return null
+
+ def sId = list[0]
+ def rId = list[1]
+ def json = list[2]
+ def stats = list[3]
+ def vcf = list[4]
+ def fai = list[5]
+ def patho = list[6] // Can be null
+
+ // Fallback placeholder file (see the assets/NO_FILE* note in section 11a)
+ def real_patho = patho ? patho : file("${projectDir}/assets/NO_FILE")
+ tuple(sId, rId, json, stats, vcf, fai, real_patho)
+ }
+ .filter { it != null }
+
+
+ def stats_out = GENERATE_LEGACY_STATS(final_stats_input)
+
+ emit:
+ legacy_log = stats_out.legacy_log // one log per (sample, reference)
+}
diff --git a/subworkflows/lineage_typing.nf b/subworkflows/lineage_typing.nf
new file mode 100644
index 0000000..d2c9bd2
--- /dev/null
+++ b/subworkflows/lineage_typing.nf
@@ -0,0 +1,50 @@
+/*
+===============================================================================
+ Stage 4: Pathotypr typing
+===============================================================================
+*/
+
+include { RUN_PATHOTYPR_PE; RUN_PATHOTYPR_SE } from '../modules/pathotypr'
+
+workflow LINEAGE_TYPING {
+
+ take:
+ pe_reads // (sId, runId, r1, r2, refId, taxId) from FASTP_PE
+ se_reads // (sId, runId, rSE, refId, taxId) from FASTP_SE
+
+ main:
+
+ // 4. Pathotypr (Optional lineage + drug-resistance typing; k-mer, reference-agnostic)
+ def patho_results = Channel.empty() // lineage summary per sample
+ def patho_dr_results = Channel.empty() // drug-resistance summary per sample
+
+ if (params.run_pathotypr) {
+ // Reshape channels for Pathotypr
+ def patho_pe_ch = pe_reads.map { sId, runId, r1, r2, refId, taxId -> tuple(refId, sId, runId, r1, r2) }
+ def patho_se_ch = se_reads.map { sId, runId, rSE, refId, taxId -> tuple(refId, sId, runId, rSE) }
+
+ def run_pe = RUN_PATHOTYPR_PE(
+ patho_pe_ch,
+ file(params.pathotypr_ref),
+ file(params.pathotypr_markers),
+ file(params.pathotypr_dr_markers),
+ params.pathotypr_bin
+ )
+
+ def run_se = RUN_PATHOTYPR_SE(
+ patho_se_ch,
+ file(params.pathotypr_ref),
+ file(params.pathotypr_markers),
+ file(params.pathotypr_dr_markers),
+ params.pathotypr_bin
+ )
+
+ // Combine summaries for later statistics / the QC report
+ patho_results = run_pe.summary.mix(run_se.summary)
+ patho_dr_results = run_pe.dr_mutations.mix(run_se.dr_mutations)
+ }
+
+ emit:
+ summary = patho_results // (sId, lineage summary); empty when pathotypr is off
+ dr_mutations = patho_dr_results // (sId, dr mutations); empty when pathotypr is off
+}
diff --git a/subworkflows/make_consensus.nf b/subworkflows/make_consensus.nf
new file mode 100644
index 0000000..f2f77ec
--- /dev/null
+++ b/subworkflows/make_consensus.nf
@@ -0,0 +1,56 @@
+/*
+===============================================================================
+ Stage 7: Consensus Generation (Optional)
+===============================================================================
+ The masked consensus, plus the parallel virgin (unmasked) path built from the
+ original dedup BAM.
+*/
+
+include { CONSENSUS_FASTA } from '../modules/consensus'
+include { CALL_FREEBAYES_RAW } from '../modules/variants'
+// Aliases for the parallel virgin (unmasked) consensus path (a DSL2 process runs once per name)
+include { CALL_BACKBONE as CALL_BACKBONE_RAW; MERGE_VCFS as MERGE_VCFS_RAW } from '../modules/variants'
+include { CONSENSUS_FASTA as CONSENSUS_FASTA_RAW } from '../modules/consensus'
+
+workflow MAKE_CONSENSUS {
+
+ take:
+ vbase // (sId, rId, bam, bai, ref_fa, exclude_txt): the BAM variants were called on
+ dedup_bam // the ORIGINAL dedup BAM, before any read filtering
+ allpos_vcf // (sId, rId, all.pos.vcf.gz, tbi)
+ fb_snps // (sId, rId, valid_snps.vcf.gz, tbi)
+ fb_mask_sites // (sId, rId, mask_sites.tsv)
+
+ main:
+
+ // 7. Consensus Generation (Optional)
+ def masked_consensus = Channel.empty() // captured for the cohort QC report (section 11)
+ if (params.make_consensus) {
+ // Prepare inputs: VCF + Reference + Mask sites
+ // Note: Script is called from bin/ directly in the module
+ // refmeta from vbase so the masked consensus gets the combined (nucmer + genmap) exclude
+ def refmeta = vbase.map { sId, rId, bam, bai, ref_fa, exclude_txt -> tuple(sId, rId, ref_fa, exclude_txt) }
+ def allpos_mask = allpos_vcf.join(fb_mask_sites, by: [0,1])
+ def cons_in = allpos_mask.join(refmeta, by: [0,1]).map { sId, rId, vcf_gz, tbi, mask, ref_fa, exclude_txt -> tuple(sId, rId, vcf_gz, tbi, mask, ref_fa, exclude_txt, "") }
+
+ masked_consensus = CONSENSUS_FASTA(cons_in).fasta
+
+ // 7b. Virgin (unmasked) consensus from the ORIGINAL dedup BAM, in parallel with the masked one.
+ if (params.keep_virgin_consensus && params.dynamic_read_filter) {
+ def raw_bb_in = dedup_bam.map { sId, rId, bam, bai, ref_fa, excl -> tuple(sId, rId, bam, bai, ref_fa) }
+ def bb_raw = CALL_BACKBONE_RAW(raw_bb_in)
+ // Virgin SNPs: re-call FreeBayes on the raw bam (shows pre-filter variants) or reuse masked SNPs.
+ def raw_snps = params.virgin_full_freebayes ? CALL_FREEBAYES_RAW(dedup_bam).snps : fb_snps
+ def merge_raw = raw_snps.join(bb_raw.backbone, by: [0,1]).join(bb_raw.header, by: [0,1])
+ .map { sId, rId, sv, st, bv, bt, hdr -> tuple(sId, rId, sv, st, bv, bt, hdr, ".raw") }
+ def vcf_raw = MERGE_VCFS_RAW(merge_raw)
+ def refmeta_raw = dedup_bam.map { sId, rId, bam, bai, ref_fa, excl -> tuple(sId, rId, ref_fa, excl) }
+ def cons_raw = vcf_raw.allpos.join(fb_mask_sites, by: [0,1]).join(refmeta_raw, by: [0,1])
+ .map { sId, rId, vcf_gz, tbi, mask, ref_fa, excl -> tuple(sId, rId, vcf_gz, tbi, mask, ref_fa, excl, ".raw") }
+ CONSENSUS_FASTA_RAW(cons_raw)
+ }
+ }
+
+ emit:
+ fasta = masked_consensus // (sId, rId, consensus fasta); empty when make_consensus is off
+}
diff --git a/subworkflows/map_reads.nf b/subworkflows/map_reads.nf
new file mode 100644
index 0000000..2ff19fd
--- /dev/null
+++ b/subworkflows/map_reads.nf
@@ -0,0 +1,72 @@
+/*
+===============================================================================
+ Stage 5: Mapping, per-sample merge and the optional length-aware read filter
+===============================================================================
+ Ends on the BAM that variant calling and the consensus are built from.
+*/
+
+include { MAPPING_PE; MAPPING_SE; MERGE_AND_MARKDUP; FILTER_READS } from '../modules/mapping'
+include { BUILD_MAPPABILITY } from '../modules/reference'
+
+workflow MAP_READS {
+
+ take:
+ pe_reads // (sId, runId, r1, r2, refId, taxId) from FASTP_PE
+ pe_orphans // (sId, runId, rSE, refId, taxId): PE mates orphaned by FastP
+ se_reads // (sId, runId, rSE, refId, taxId) from FASTP_SE
+ ref_bundle // (refId, ref_fa, indices, exclude_txt)
+ expectedMap // "sampleId||refId" -> number of BAMs to expect, for the groupTuple groupKey
+
+ main:
+
+ // 5. Mapping (BWA)
+ def map_pe = pe_reads.map { sId, runId, r1, r2, refId, taxId -> tuple(refId, "PE", sId, runId, r1, r2, taxId) }
+ def map_se1 = pe_orphans.map { sId, runId, rSE, refId, taxId -> tuple(refId, "SE", sId, runId, rSE, null, taxId) }
+ def map_se2 = se_reads.map { sId, runId, rSE, refId, taxId -> tuple(refId, "SE", sId, runId, rSE, null, taxId) }
+ def map_all = map_pe.mix(map_se1).mix(map_se2)
+
+ // Join reads with references (matching RefID)
+ def joined = map_all.combine(ref_bundle).filter { it[0] == it[7] }.map { refId, mode, sId, runId, r1, r2, taxId, refId2, ref_fa, indices, exclude_txt -> tuple(refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt) }
+ def branched = joined.branch { pe: it[1] == "PE"; se: it[1] == "SE" }
+
+ def map_pe_in = branched.pe.map { refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt -> tuple(refId, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt) }
+ def map_se_in = branched.se.map { refId, mode, sId, runId, r1, r2, taxId, ref_fa, indices, exclude_txt -> tuple(refId, sId, runId, r1, taxId, ref_fa, indices, exclude_txt) }
+
+ def mapped_pe = MAPPING_PE(map_pe_in)
+ def mapped_se = MAPPING_SE(map_se_in)
+ def all_bams = mapped_pe.bam.mix(mapped_se.bam)
+
+ // Group BAMs by SampleID + RefID for merging
+ def bams_grouped = all_bams.map { sId, rId, bam, ref_fa, exclude_txt ->
+ def key = "${sId}||${rId}"
+ def nExp = expectedMap[key]
+ tuple(groupKey(key, nExp as int), sId, rId, bam, ref_fa, exclude_txt)
+ }.groupTuple(by: 0).map { gk, sIds, rIds, bams, ref_fas, excls -> tuple(sIds[0], rIds[0], bams, ref_fas[0], excls[0]) }
+
+ def final_bams = MERGE_AND_MARKDUP(bams_grouped)
+
+ // Optional length-aware read filter: drop reads too short to map uniquely at their locus
+ // (short-read false positives in near-repeats), then call/consensus on the filtered BAM.
+ // Feature off -> call on the dedup BAM unchanged.
+ def vbase
+ def filter_stats = Channel.empty()
+ if (params.dynamic_read_filter) {
+ def mapp = BUILD_MAPPABILITY( ref_bundle.map { rId, fa, idx, excl -> tuple(rId, fa) } )
+ def track_bed = mapp.track.join(mapp.repeat_bed, by: 0) // (rId, npz, repeat_bed)
+ def filt_in = final_bams.final_bam
+ .map { sId, rId, bam, bai, fa, excl -> tuple(rId, sId, bam, bai, fa, excl) }
+ .combine(track_bed, by: 0) // fan the per-reference track+bed to each sample
+ .map { rId, sId, bam, bai, fa, excl, npz, bed -> tuple(sId, rId, bam, bai, fa, excl, npz, bed) }
+ def filt = FILTER_READS(filt_in)
+ vbase = filt.filtered_bam // exclude_txt now = nucmer + genmap repeats
+ filter_stats = filt.stats
+ } else {
+ vbase = final_bams.final_bam
+ }
+
+ emit:
+ variant_base = vbase // (sId, rId, bam, bai, ref_fa, exclude_txt): the BAM to call on
+ dedup_bam = final_bams.final_bam // the ORIGINAL dedup BAM, before any read filtering
+ stats = final_bams.stats // (sId, rId, dedup.stats)
+ filter_mqc = filter_stats // empty when dynamic_read_filter is off
+}
diff --git a/subworkflows/multiqc_report.nf b/subworkflows/multiqc_report.nf
new file mode 100644
index 0000000..b218e0c
--- /dev/null
+++ b/subworkflows/multiqc_report.nf
@@ -0,0 +1,41 @@
+/*
+===============================================================================
+ Stage 10: MultiQC Report
+===============================================================================
+*/
+
+include { MULTIQC; DUMP_VERSIONS } from '../modules/qc'
+
+workflow MULTIQC_REPORT {
+
+ take:
+ pe_json // (sId, fastp json) from FASTP_PE
+ se_json // (sId, fastp json) from FASTP_SE
+ bam_stats // (sId, rId, dedup.stats)
+ kraken_reports // per-sample Kraken2 reports; empty when Kraken is off
+ snpeff_stats // SnpEff CSV; empty when annotate_main_vcf is off
+ filter_stats // read-filter drop rates; empty when dynamic_read_filter is off
+ report_filename // "_multiqc_report"
+
+ main:
+
+ // 10. MultiQC Report
+ // Collect all relevant metrics from previous processes.
+ // FastP JSONs and Kraken reports are reference-independent (QC of the raw reads), so a sample
+ // mapped to >1 reference emits identically-named copies from parallel tasks. Deduplicate by
+ // filename before collecting, otherwise MultiQC hits a fatal input-name collision.
+ def fastp_json_mqc = pe_json.mix(se_json).map { sId, json -> json }.unique { it.name }
+ def kraken_mqc = kraken_reports.unique { it.name }
+ // Provenance: dump the container's tool versions once and surface them in the report.
+ def versions_mqc = DUMP_VERSIONS().mqc
+ def qc_collection = Channel.empty()
+ .mix(fastp_json_mqc) // FastP
+ .mix(bam_stats.map { s, r, st -> st }) // Samtools (drop the (sId,rId) key -> bare path)
+ .mix(kraken_mqc) // Kraken
+ .mix(snpeff_stats) // SnpEff
+ .mix(filter_stats) // Length-aware read filter (drop rate)
+ .mix(versions_mqc) // Software versions
+ .collect()
+
+ MULTIQC(qc_collection, report_filename)
+}
diff --git a/subworkflows/prepare_references.nf b/subworkflows/prepare_references.nf
new file mode 100644
index 0000000..18ecafe
--- /dev/null
+++ b/subworkflows/prepare_references.nf
@@ -0,0 +1,25 @@
+/*
+===============================================================================
+ Stage 1: Reference Preparation
+===============================================================================
+ Index every reference in the samplesheet and build its snpEff database.
+*/
+
+include { PREPARE_REFERENCE; SNPEFF_BUILD_DB } from '../modules/reference'
+
+workflow PREPARE_REFERENCES {
+
+ take:
+ ref_in_ch // (refId, refFasta)
+ snpeff_db_in // (refId, refFasta, refGff)
+
+ main:
+
+ // 1. Reference Preparation
+ def ref_out = PREPARE_REFERENCE(ref_in_ch)
+ def snpeff_out = SNPEFF_BUILD_DB(snpeff_db_in)
+
+ emit:
+ bundle = ref_out.bundle // (refId, ref_fa, indices, exclude_txt)
+ snpeff = snpeff_out.db // (refId, snpEff.config, data)
+}
diff --git a/subworkflows/read_qc.nf b/subworkflows/read_qc.nf
new file mode 100644
index 0000000..e7c2057
--- /dev/null
+++ b/subworkflows/read_qc.nf
@@ -0,0 +1,54 @@
+/*
+===============================================================================
+ Stages 2-3: Read Validation, optional Kraken filtering and FastP QC
+===============================================================================
+*/
+
+include { VALIDATE_RAW_READS_PE; VALIDATE_RAW_READS_SE; KRAKEN_FILTER_PE; KRAKEN_FILTER_SE; FASTP_PE; FASTP_SE } from '../modules/qc'
+
+workflow READ_QC {
+
+ take:
+ pe_reads_ch // (sId, runId, r1, r2, refId, taxId)
+ se_reads_ch // (sId, runId, r1, refId, taxId)
+ KRAKEN_ENABLED // boolean: a usable Kraken DB AND at least one taxId in the samplesheet
+
+ main:
+
+ // 2. Read Validation
+ def checked_pe = VALIDATE_RAW_READS_PE(pe_reads_ch)
+ def checked_se = VALIDATE_RAW_READS_SE(se_reads_ch)
+ def pe_final = checked_pe.reads
+ def se_final = checked_se.reads
+
+ // Channel to collect Kraken reports if enabled
+ def ch_kraken_reports = Channel.empty()
+
+ if (KRAKEN_ENABLED) {
+ def pe_with_tax = checked_pe.reads.filter { it[5] != null }
+ def pe_no_tax = checked_pe.reads.filter { it[5] == null }
+ def se_with_tax = checked_se.reads.filter { it[4] != null }
+ def se_no_tax = checked_se.reads.filter { it[4] == null }
+
+ def krk_pe = KRAKEN_FILTER_PE(pe_with_tax, params.kraken2_db)
+ def krk_se = KRAKEN_FILTER_SE(se_with_tax, params.kraken2_db)
+
+ pe_final = krk_pe.reads.mix(pe_no_tax)
+ se_final = krk_se.reads.mix(se_no_tax)
+
+ // Accumulate reports for MultiQC
+ ch_kraken_reports = krk_pe.report.mix(krk_se.report)
+ }
+
+ // 3. FastP QC
+ def fastp_pe = FASTP_PE(pe_final)
+ def fastp_se = FASTP_SE(se_final)
+
+ emit:
+ pe_reads = fastp_pe.pe_reads // (sId, runId, r1, r2, refId, taxId)
+ pe_orphans = fastp_pe.se_reads // (sId, runId, rSE, refId, taxId): PE mates orphaned by FastP
+ se_reads = fastp_se.se_reads // (sId, runId, rSE, refId, taxId)
+ pe_json = fastp_pe.json // (sId, fastp json)
+ se_json = fastp_se.json // (sId, fastp json)
+ kraken_reports = ch_kraken_reports // empty when Kraken is disabled
+}
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 0000000..f0cc91a
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,111 @@
+# Tests
+
+```bash
+tests/run_tests.sh # everything
+tests/run_tests.sh unit # Python unit tests only
+tests/run_tests.sh js # report front-end only
+tests/run_tests.sh pipeline # Nextflow stub runs only
+```
+
+`.github/workflows/ci.yml` runs the same things on every pull request. Nothing here needs a
+container, a reference genome, real reads or a network connection.
+
+## Layout
+
+| Path | What it covers | Needs |
+| :--- | :--- | :--- |
+| `unit/` | Everything under `bin/`: the consensus decision tree, the QC verdict engine, every parser, the k-mer liftover, the read filter, and the awk and filter expressions the process scripts call | `pytest`, `numpy`, `bcftools` for one class |
+| `js/` | The report's hand-written ES5 statistics and pure helpers, plus the integrity of the asset bundle | `node` (>= 18) |
+| `pipeline/` | Samplesheet validation, parameter handling, and a full `-stub-run` of the DAG | `nextflow`, Java 17+ |
+| `data/` | The fixture cohort, generated by `data/make_test_data.py` | - |
+
+Legs whose tool is not installed skip themselves rather than fail, so a partial local setup still
+gets useful signal. The `bcftools` class in `unit/test_vcf_filter_rules.py` is the one place a real
+tool is used: the hom/het expressions have to be ones bcftools accepts *and* that select the records
+they claim to, which no amount of string comparison can establish.
+
+### What the process scripts contribute
+
+A `script:` block is unreachable from any test: `-stub-run` replaces it and nothing can import it.
+So the decisions that used to live inside those blocks are files under `bin/` that the processes
+call, and `unit/` covers them like any other code:
+
+| File | Decides |
+| :--- | :--- |
+| `vcf_filter_rules.py` | What depth, strand support and allele fraction make a call homozygous or heterozygous |
+| `format_snps_for_backbone.awk` | Whether a heterozygote survives re-validation, and the depth the consensus divides by |
+| `backbone_allpos.awk` | Reference and alternate observations from an mpileup base string, and the coverage call |
+
+## The fixture cohort
+
+`tests/data/` holds a 5 kb reference with three genes and three samples (170 kB in total):
+
+| Sample | Layout | Why it is there |
+| :--- | :--- | :--- |
+| `SAMPLE_A` | paired-end, two runs | exercises the per-sample merge |
+| `SAMPLE_B` | paired-end, one run | the ordinary case |
+| `SAMPLE_C` | single-end | the SE branch, which is easy to break unnoticed |
+
+Everything derives from one seed, so `python3 tests/data/make_test_data.py` reproduces the
+committed files byte for byte. CI regenerates them and fails on any diff, which is what keeps the
+fixtures something anyone can rebuild rather than binary blobs of unknown provenance.
+
+`data/invalid/` holds one deliberately broken samplesheet per failure mode. They are generated by
+the same script, because the exact shape of each row decides which error the validator reaches.
+
+## The two test profiles
+
+`-profile test` runs the fixture cohort with BAMpiro's **default** feature set.
+`-profile test_full` additionally turns on every optional branch (Kraken, pathotypr typing,
+canonical annotation, the coordinate liftover, the virgin consensus), so one run instantiates
+every process in the pipeline. `tests/pipeline/test_stub_run.py` asserts exactly that, so an
+optional branch cannot quietly fall out of coverage.
+
+```bash
+nextflow run main.nf -profile test -stub-run # ~3 s, 68 tasks
+nextflow run main.nf -profile test_full -stub-run # ~3 s, 84 tasks
+```
+
+A stub run proves the **wiring**: that every channel joins, every declared output is produced and
+collected, and nothing is starved or duplicated. It proves nothing about the science, which is
+what `unit/` is for.
+
+## Testing the report's front-end from Node
+
+The report ships as one ES5 IIFE that `bin/qcreport/render.py` concatenates from `bin/report_assets/js/*.js`
+and splices into `shell.html`. `01_prelude.js` opens the IIFE and `14_boot.js` closes it, so no
+fragment is a valid script on its own.
+
+`js/harness.mjs` loads the fragments in the order `render.py` itself declares (it parses
+`_JS_MODULES` out of the Python, so the two cannot drift), drops the DOM-wiring fragment, and
+appends its own epilogue to capture the declarations and close the IIFE. Function declarations
+hoist to the top of the IIFE, so every function is reachable regardless of which fragment declared
+it, and nothing depends on line numbers.
+
+The statistics are asserted against **SciPy and statsmodels** reference values rather than against
+a snapshot of our own output, so the tests catch a regression against an independent
+implementation. Current agreement is around 1e-12.
+
+## Both Nextflow parsers
+
+Nextflow 25.10 and later default to a strict parser that rejects statements at the top level of a
+script and requires dynamic process directives to be closures. `main.nf` is written to satisfy it:
+the samplesheet parsing is a function, every statement lives inside `workflow`, and each dynamic
+`publishDir` is a closure. That syntax is also valid for the v1 parser of the minimum supported
+24.04.2, so the CI matrix runs both versions and neither needs `NXF_SYNTAX_PARSER`.
+
+If you add script-level code and the run dies with *"Statements cannot be mixed with script
+declarations"*, that is the strict parser: move the code into the workflow or into a function.
+
+## Adding a test
+
+- Unit tests import a script by path with `load_script("name")` from `conftest.py`; `bin/` is not
+ a package, but `bin/qcreport/` is, and its tests import it normally (`from qcreport import parsers`).
+ Note that importing `qc_report` reads `bin/report_assets/`, so it only works in place.
+- Anything on disk goes in `tmp_path`. Do not add files to `tests/data/`; if a fixture is worth
+ keeping, add it to `make_test_data.py` so it stays reproducible.
+- Tests pin **observed** behaviour. When something looks wrong, the test records what the code
+ does today, with a comment saying so, and the defect gets fixed in its own change. A red suite
+ on `main` teaches everyone to ignore the suite.
+- A new process in `modules/` needs a `stub:` block, or the stub run fails.
+ `test_stub_run.py::test_there_is_a_stub_block_for_every_process` catches that early.
diff --git a/tests/conftest.py b/tests/conftest.py
new file mode 100644
index 0000000..dbce841
--- /dev/null
+++ b/tests/conftest.py
@@ -0,0 +1,95 @@
+"""Shared fixtures for the BAMpiro test suite.
+
+The pipeline's Python lives in `bin/` as standalone scripts, not as an installed
+package, so tests load them by path. Every script guards its entry point behind
+`if __name__ == "__main__"`, so importing one is side-effect free apart from
+`qc_report`, which reads `bin/report_assets/` at import time.
+
+The one exception is `bin/qcreport/`, the package behind `bin/qc_report.py`: it is
+a real package and its tests import it normally (`from qcreport import parsers`),
+which works because pyproject.toml puts `bin` on the pytest pythonpath. Loading
+`qc_report` itself by path still works too, for the same reason.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+BIN = REPO_ROOT / "bin"
+DATA = Path(__file__).resolve().parent / "data"
+
+
+def load_script(name: str):
+ """Import a script from bin/ by name (without the .py extension).
+
+ Registering the module in sys.modules before executing it is required, not
+ cosmetic: WGS_fasta_allpos defines a @dataclass, and dataclasses resolve the
+ module through sys.modules while the class body is being built.
+ """
+ if name in sys.modules:
+ return sys.modules[name]
+ spec = importlib.util.spec_from_file_location(name, BIN / f"{name}.py")
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
+@pytest.fixture(scope="session")
+def repo_root() -> Path:
+ return REPO_ROOT
+
+
+@pytest.fixture(scope="session")
+def data_dir() -> Path:
+ """The committed fixture cohort (see tests/data/make_test_data.py)."""
+ return DATA
+
+
+@pytest.fixture(scope="session")
+def nextflow() -> str:
+ """Path to the nextflow CLI, skipping the test when it is not installed."""
+ exe = shutil.which("nextflow")
+ if not exe:
+ pytest.skip("nextflow is not on PATH")
+ return exe
+
+
+@pytest.fixture(scope="session")
+def nextflow_env() -> dict:
+ """Environment for invoking Nextflow from a test.
+
+ No parser pin: main.nf is accepted by the strict (v2) parser that Nextflow
+ 25.10 and later use by default, as well as by the v1 parser of the minimum
+ supported 24.04.2. CI runs both.
+ """
+ env = dict(os.environ)
+ env["NXF_ANSI_LOG"] = "false"
+ return env
+
+
+def run_pipeline(nextflow_exe, env, workdir: Path, *args, timeout=600):
+ """Run `nextflow run main.nf ...` from the repository root and capture the result."""
+ cmd = [
+ nextflow_exe,
+ "-log", str(workdir / "nextflow.log"),
+ "run", "main.nf",
+ "-work-dir", str(workdir / "work"),
+ *args,
+ ]
+ return subprocess.run(
+ cmd,
+ cwd=REPO_ROOT,
+ env=env,
+ capture_output=True,
+ text=True,
+ timeout=timeout,
+ )
diff --git a/tests/data/invalid/empty_field.tsv b/tests/data/invalid/empty_field.tsv
new file mode 100644
index 0000000..52d878c
--- /dev/null
+++ b/tests/data/invalid/empty_field.tsv
@@ -0,0 +1,2 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+ RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
diff --git a/tests/data/invalid/missing_column.tsv b/tests/data/invalid/missing_column.tsv
new file mode 100644
index 0000000..b460594
--- /dev/null
+++ b/tests/data/invalid/missing_column.tsv
@@ -0,0 +1,2 @@
+sampleId runId r1 r2 refId refFasta taxId
+SAMPLE_A RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta 1773
diff --git a/tests/data/invalid/missing_file.tsv b/tests/data/invalid/missing_file.tsv
new file mode 100644
index 0000000..636a915
--- /dev/null
+++ b/tests/data/invalid/missing_file.tsv
@@ -0,0 +1,2 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+SAMPLE_A RUN1 tests/data/reads/DOES_NOT_EXIST_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
diff --git a/tests/data/invalid/multi_error.tsv b/tests/data/invalid/multi_error.tsv
new file mode 100644
index 0000000..0cdd8fb
--- /dev/null
+++ b/tests/data/invalid/multi_error.tsv
@@ -0,0 +1,5 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+SAMPLE_A RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+ RUN1 tests/data/reads/SAMPLE_C_RUN1.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_B RUN1 tests/data/reads/NOPE_R1.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_C RUN1 tests/data/reads/SAMPLE_C_RUN1.fq.gz TESTREF tests/data/reads/SAMPLE_C_RUN1.fq.gz tests/data/reference.gff3 1773
diff --git a/tests/data/invalid/no_rows.tsv b/tests/data/invalid/no_rows.tsv
new file mode 100644
index 0000000..f77bc7e
--- /dev/null
+++ b/tests/data/invalid/no_rows.tsv
@@ -0,0 +1 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
diff --git a/tests/data/invalid/no_usable_rows.tsv b/tests/data/invalid/no_usable_rows.tsv
new file mode 100644
index 0000000..bc5041e
--- /dev/null
+++ b/tests/data/invalid/no_usable_rows.tsv
@@ -0,0 +1,3 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+#SAMPLE_A RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+
diff --git a/tests/data/invalid/ref_conflict.tsv b/tests/data/invalid/ref_conflict.tsv
new file mode 100644
index 0000000..2734d16
--- /dev/null
+++ b/tests/data/invalid/ref_conflict.tsv
@@ -0,0 +1,3 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+SAMPLE_A RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_B RUN1 tests/data/reads/SAMPLE_C_RUN1.fq.gz TESTREF tests/data/reads/SAMPLE_C_RUN1.fq.gz tests/data/reference.gff3 1773
diff --git a/tests/data/kraken2_db/.gitkeep b/tests/data/kraken2_db/.gitkeep
new file mode 100644
index 0000000..dcfaac2
--- /dev/null
+++ b/tests/data/kraken2_db/.gitkeep
@@ -0,0 +1,3 @@
+Placeholder so git tracks this directory. -profile test_full points
+--kraken2_db here purely so the Kraken branch of the DAG is exercised;
+a stub run never opens the database.
diff --git a/tests/data/make_test_data.py b/tests/data/make_test_data.py
new file mode 100644
index 0000000..f0905e3
--- /dev/null
+++ b/tests/data/make_test_data.py
@@ -0,0 +1,268 @@
+#!/usr/bin/env python3
+"""Regenerate the tiny fixture cohort under tests/data/.
+
+Everything is derived from a fixed seed, so re-running this script reproduces the
+committed files byte for byte. The fixtures are deliberately small (a 5 kb genome,
+three samples) so they can live in the repository and still exercise the pipeline's
+paired-end, single-end and multi-run code paths.
+
+ python3 tests/data/make_test_data.py
+
+The generated cohort is what `-profile test` and the pytest suite read.
+"""
+
+from __future__ import annotations
+
+import gzip
+import random
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+
+SEED = 20260806
+CONTIG = "test_chr"
+TAXID = 1773
+GENOME_LEN = 5000
+READ_LEN = 100
+COVERAGE = 30
+INSERT = 300
+QUAL = "I" * READ_LEN
+
+# Genes are laid out so every one of them is hit by at least one planted variant.
+GENES = [
+ # (locus_tag, gene_name, start, end, strand)
+ ("TEST_0001", "dnaA", 201, 1400, "+"),
+ ("TEST_0002", "rpoB", 1601, 3100, "+"),
+ ("TEST_0003", "katG", 3301, 4700, "-"),
+]
+
+# Planted variants: (position 1-based, alt base, which samples carry it).
+VARIANTS = [
+ (450, "T", {"SAMPLE_A", "SAMPLE_B"}),
+ (1750, "G", {"SAMPLE_B"}),
+ (2100, "A", {"SAMPLE_A", "SAMPLE_C"}),
+ (3500, "C", {"SAMPLE_C"}),
+ (4200, "T", {"SAMPLE_A", "SAMPLE_B", "SAMPLE_C"}),
+]
+
+# (sampleId, runId, layout) - SAMPLE_A has two runs so the merge path is exercised.
+SAMPLES = [
+ ("SAMPLE_A", "RUN1", "PE"),
+ ("SAMPLE_A", "RUN2", "PE"),
+ ("SAMPLE_B", "RUN1", "PE"),
+ ("SAMPLE_C", "RUN1", "SE"),
+]
+
+COMPLEMENT = str.maketrans("ACGT", "TGCA")
+
+
+def revcomp(seq: str) -> str:
+ return seq.translate(COMPLEMENT)[::-1]
+
+
+def build_genome(rng: random.Random) -> str:
+ return "".join(rng.choice("ACGT") for _ in range(GENOME_LEN))
+
+
+def sample_genome(reference: str, sample_id: str) -> str:
+ bases = list(reference)
+ for pos, alt, carriers in VARIANTS:
+ if sample_id in carriers:
+ bases[pos - 1] = alt
+ return "".join(bases)
+
+
+def write_fasta(path: Path, header: str, seq: str, wrap: int = 60) -> None:
+ with path.open("w") as fh:
+ fh.write(f">{header}\n")
+ for i in range(0, len(seq), wrap):
+ fh.write(seq[i : i + wrap] + "\n")
+
+
+def write_gff(path: Path) -> None:
+ with path.open("w") as fh:
+ fh.write("##gff-version 3\n")
+ fh.write(f"##sequence-region {CONTIG} 1 {GENOME_LEN}\n")
+ fh.write(
+ f"{CONTIG}\ttest\tregion\t1\t{GENOME_LEN}\t.\t+\t.\t"
+ f"ID={CONTIG};Name={CONTIG}\n"
+ )
+ for locus, name, start, end, strand in GENES:
+ fh.write(
+ f"{CONTIG}\ttest\tgene\t{start}\t{end}\t.\t{strand}\t.\t"
+ f"ID=gene-{locus};Name={name};locus_tag={locus}\n"
+ )
+ fh.write(
+ f"{CONTIG}\ttest\tCDS\t{start}\t{end}\t.\t{strand}\t0\t"
+ f"ID=cds-{locus};Parent=gene-{locus};Name={name};locus_tag={locus};"
+ f"product={name} test product\n"
+ )
+
+
+def paired_records(seq: str, sample_id: str, run_id: str, rng: random.Random):
+ n_pairs = (GENOME_LEN * COVERAGE) // (READ_LEN * 2)
+ for i in range(n_pairs):
+ start = rng.randint(0, len(seq) - INSERT)
+ fragment = seq[start : start + INSERT]
+ r1 = fragment[:READ_LEN]
+ r2 = revcomp(fragment[-READ_LEN:])
+ name = f"{sample_id}_{run_id}_{i:05d}"
+ yield name, r1, r2
+
+
+def single_records(seq: str, sample_id: str, run_id: str, rng: random.Random):
+ n_reads = (GENOME_LEN * COVERAGE) // READ_LEN
+ for i in range(n_reads):
+ start = rng.randint(0, len(seq) - READ_LEN)
+ read = seq[start : start + READ_LEN]
+ if rng.random() < 0.5:
+ read = revcomp(read)
+ yield f"{sample_id}_{run_id}_{i:05d}", read
+
+
+def write_fastq(path: Path, records) -> None:
+ # mtime=0 keeps the gzip header constant so regenerating gives an identical file.
+ payload = "".join(f"@{name}\n{seq}\n+\n{QUAL}\n" for name, seq in records)
+ with path.open("wb") as raw:
+ with gzip.GzipFile(fileobj=raw, mode="wb", compresslevel=9, mtime=0) as fh:
+ fh.write(payload.encode())
+
+
+def main() -> None:
+ rng = random.Random(SEED)
+ reads_dir = HERE / "reads"
+ reads_dir.mkdir(parents=True, exist_ok=True)
+
+ reference = build_genome(rng)
+ write_fasta(HERE / "reference.fasta", CONTIG, reference)
+ write_gff(HERE / "reference.gff3")
+
+ rows = []
+ for sample_id, run_id, layout in SAMPLES:
+ seq = sample_genome(reference, sample_id)
+ if layout == "PE":
+ pairs = list(paired_records(seq, sample_id, run_id, rng))
+ r1 = reads_dir / f"{sample_id}_{run_id}_R1.fq.gz"
+ r2 = reads_dir / f"{sample_id}_{run_id}_R2.fq.gz"
+ write_fastq(r1, ((f"{n}/1", s) for n, s, _ in pairs))
+ write_fastq(r2, ((f"{n}/2", s) for n, _, s in pairs))
+ rows.append((sample_id, run_id, r1.name, r2.name))
+ else:
+ reads = list(single_records(seq, sample_id, run_id, rng))
+ r1 = reads_dir / f"{sample_id}_{run_id}.fq.gz"
+ write_fastq(r1, reads)
+ rows.append((sample_id, run_id, r1.name, ""))
+
+ header = "sampleId\trunId\tr1\tr2\trefId\trefFasta\trefGff\ttaxId\n"
+ with (HERE / "samplesheet.tsv").open("w") as fh:
+ fh.write(header)
+ for sample_id, run_id, r1, r2 in rows:
+ r1p = f"tests/data/reads/{r1}"
+ r2p = f"tests/data/reads/{r2}" if r2 else ""
+ fh.write(
+ f"{sample_id}\t{run_id}\t{r1p}\t{r2p}\tTESTREF\t"
+ f"tests/data/reference.fasta\ttests/data/reference.gff3\t{TAXID}\n"
+ )
+
+ write_invalid_samplesheets(rows)
+ write_pathotypr_panels(reference)
+
+ # -profile test_full points --kraken2_db here; main.nf only checks that it exists.
+ kraken_dir = HERE / "kraken2_db"
+ kraken_dir.mkdir(exist_ok=True)
+ (kraken_dir / ".gitkeep").write_text(
+ "Placeholder so git tracks this directory. -profile test_full points\n"
+ "--kraken2_db here purely so the Kraken branch of the DAG is exercised;\n"
+ "a stub run never opens the database.\n"
+ )
+
+ print(f"reference : {GENOME_LEN} bp, {len(GENES)} genes")
+ print(f"samples : {len({s for s, _, _ in SAMPLES})} in {len(SAMPLES)} runs")
+ print(f"variants : {len(VARIANTS)} planted")
+
+
+HEADER = ["sampleId", "runId", "r1", "r2", "refId", "refFasta", "refGff", "taxId"]
+REF_FA = "tests/data/reference.fasta"
+REF_GFF = "tests/data/reference.gff3"
+
+
+def write_invalid_samplesheets(rows) -> None:
+ """Samplesheets that main.nf's validator must reject, one per failure mode.
+
+ tests/pipeline/test_samplesheet_validation.py asserts on the exact message each of these
+ produces, so the shape of every row matters. In particular the conflicting-reference case
+ needs a VALID row first: the validator only notices a conflict once a refId has been bound,
+ and rows that fail an earlier check return before binding anything.
+ """
+ out = HERE / "invalid"
+ out.mkdir(exist_ok=True)
+ good = rows[0]
+ ok_row = [good[0], good[1], f"tests/data/reads/{good[2]}",
+ f"tests/data/reads/{good[3]}", "TESTREF", REF_FA, REF_GFF, str(TAXID)]
+ # An existing file that is not the reference, for the conflicting-refFasta case.
+ other_file = f"tests/data/reads/{rows[-1][2]}"
+
+ def write(name, header, data_rows):
+ with (out / name).open("w") as fh:
+ fh.write("\t".join(header) + "\n")
+ for row in data_rows:
+ fh.write("\t".join(row) + "\n")
+
+ write("missing_column.tsv",
+ [c for c in HEADER if c != "refGff"],
+ [[v for i, v in enumerate(ok_row) if HEADER[i] != "refGff"]])
+
+ write("no_rows.tsv", HEADER, [])
+
+ with (out / "no_usable_rows.tsv").open("w") as fh:
+ fh.write("\t".join(HEADER) + "\n")
+ fh.write("#" + "\t".join(ok_row) + "\n")
+ fh.write("\n")
+
+ write("empty_field.tsv", HEADER, [[""] + ok_row[1:]])
+
+ write("missing_file.tsv", HEADER,
+ [[ok_row[0], ok_row[1], "tests/data/reads/DOES_NOT_EXIST_R1.fq.gz"] + ok_row[3:]])
+
+ write("ref_conflict.tsv", HEADER,
+ [ok_row, ["SAMPLE_B", "RUN1", other_file, "", "TESTREF", other_file, REF_GFF, str(TAXID)]])
+
+ # Three DIFFERENT failures at once: the validator must report all of them in one pass.
+ write("multi_error.tsv", HEADER, [
+ ok_row,
+ ["", "RUN1", other_file, "", "TESTREF", REF_FA, REF_GFF, str(TAXID)],
+ ["SAMPLE_B", "RUN1", "tests/data/reads/NOPE_R1.fq.gz", "", "TESTREF", REF_FA, REF_GFF, str(TAXID)],
+ ["SAMPLE_C", "RUN1", other_file, "", "TESTREF", other_file, REF_GFF, str(TAXID)],
+ ])
+
+
+def write_pathotypr_panels(reference: str) -> None:
+ """Marker panels shaped like the real Zenodo v1.0.0 files.
+
+ -profile test_full stages these so the pathotypr branch of the DAG runs; a stub
+ run never parses them, but the column layout matches the real panels so they are
+ also usable as fixtures for the parsing tests.
+ """
+ out = HERE / "pathotypr"
+ out.mkdir(exist_ok=True)
+ write_fasta(out / "reference.fasta", CONTIG, reference)
+
+ with (out / "lineage_markers.tsv").open("w") as fh:
+ fh.write("position\tref_base\talt_base\tlineage1\tlineage2\n")
+ for pos, alt, _ in VARIANTS[:3]:
+ fh.write(f"{pos}\t{reference[pos - 1]}\t{alt}\tL1\tL1.1\n")
+
+ with (out / "dr_markers.tsv").open("w") as fh:
+ fh.write("#pos\tref\talt\tdrug\tresistance\tmarker_name\tgrade\tgene\tmutation\n")
+ fh.write(
+ f"1750\t{reference[1749]}\tG\trifampicin\tR\trpoB_S450L\t"
+ "1) Assoc w R\trpoB\tp.Ser450Leu\n"
+ )
+ fh.write(
+ f"3500\t{reference[3499]}\tC\tisoniazid\tR\tkatG_S315T\t"
+ "1) Assoc w R\tkatG\tp.Ser315Thr\n"
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/data/pathotypr/dr_markers.tsv b/tests/data/pathotypr/dr_markers.tsv
new file mode 100644
index 0000000..ce93a3c
--- /dev/null
+++ b/tests/data/pathotypr/dr_markers.tsv
@@ -0,0 +1,3 @@
+#pos ref alt drug resistance marker_name grade gene mutation
+1750 T G rifampicin R rpoB_S450L 1) Assoc w R rpoB p.Ser450Leu
+3500 C C isoniazid R katG_S315T 1) Assoc w R katG p.Ser315Thr
diff --git a/tests/data/pathotypr/lineage_markers.tsv b/tests/data/pathotypr/lineage_markers.tsv
new file mode 100644
index 0000000..14dddd4
--- /dev/null
+++ b/tests/data/pathotypr/lineage_markers.tsv
@@ -0,0 +1,4 @@
+position ref_base alt_base lineage1 lineage2
+450 G T L1 L1.1
+1750 T G L1 L1.1
+2100 A A L1 L1.1
diff --git a/tests/data/pathotypr/reference.fasta b/tests/data/pathotypr/reference.fasta
new file mode 100644
index 0000000..f9979d4
--- /dev/null
+++ b/tests/data/pathotypr/reference.fasta
@@ -0,0 +1,85 @@
+>test_chr
+GGTGAATGTTGACGCTCAAGAGAGTCCACTCCCCTCTCGCCGGTAAGGTTCCACGCAGCC
+TATTGCTCGCGAGGCCCGGTGGAGTAGGGTATCCGTCAAAGGCCAGAAACTACTATGGGC
+TCGCGCACCAAGGCACGTTCATTGGCAGTGTACCGAGGTGTCATTCCGCTTGCCCTCGTT
+GCGGATTGTCAACCCGTGCCATCCATCCCGCATCAGCACCAGTATTAACCATAGAGGAAG
+AGATCTCACTCAACAGGCCATTTTAACTGCTCTTGGCAGCCTAAGAGATATGTAGCTACT
+CCGTCCGCTCGGCTATTATATTTAGTTATTTCGTGTATGAACGTGCAATACAAATCCCGC
+AATACAAGGACCACCTGTCGAGAACCATCCCATGTTCCATTCATTATGATTCCCTGGCTC
+TTCCGCTGAGCGTATCCTCTGCAAGACCAGGCCGTGACTCCCACCCTTTGAGCCTTGGTA
+ATAAGCAATTAACGCGGTGCAATTGGTCGCTACGAGATAAGACAAATGCCAACAATGGAA
+ACTGCAATTATCTATCTAATAGTTGCCCCCTTCAAACCGCGCACACTTGGCGCGTATATA
+TGTACCTCCTCCTTTTGCGTGCCTGTGATAGGCCGCTGGTCGTGCCCCTCGACCCTGTAC
+CATGACCGGGCGTTGCCTCGAAGCAGCTCGACGGTCACAGACAATAAGAAACGACCTTAT
+GTGCGTGCGGTGGAGAGTTACTATCAAACAACGCTCATGCGGAACACTCCGATGGTGTGT
+AAAGAATACACTGACACTATACGATAGACGGAGCCGAGCTCACATCAATGAGTGTCATCA
+TACCGCACCATTTTTAGACAATAGGTCAGAGTACAGGTTGCGTCGAGTCGTGGGGGCTCG
+GTTAGCGGTACGCGTTAACACGGAAACTCAGCATATGGGCCCTCGGGCCCCATTCATCGT
+GATCATGGTATCTAGCGAACGTGAAACGCAACGTTCAAATATGGCACTGTACCTTCCCCA
+CCTCTCAGTTCCCATGGTTTGCCCGTTACTGAAGATTTGCAAGACTCGTGGCCAATCATG
+GTGTGGAATTAGGGTCGGCTATGTCAGGACATTGTGCTGTAAATGGCTTCTCGCTAAGGT
+CGACTGCTAAGGTAAATGGCCTAGCAACAGCCTATTCGTTGGGCGCACATCAAGGTTCAG
+AGCTAAAGCTCTTTGCGACCCAGCTATGGTAAGGCAAGAGATGGAGACATCTTGCGGAGC
+TGTTATATACCGTGGAAAGCGTCGACCACGAAAGGGCACTGAGACAGAGGATGTAATTGG
+AATCAGTGCATTAGCAACTCGTGTGAGAGGACTTGTGCAAGCTTTTCCTCAATCTCGATA
+AAGATATTTGTCGCTAGGACCTGGGACAGGCCCGGTGGCATGCACTTATGTCATGTATCC
+ATGACGTTGAATTAAAATCTGAGAGCGGATGTGACAGTTATAACAGGGGGTGAGGCTAAA
+CAAACCGGCGTCACGCAATACCGGCGAGGTGGCCTGAGTGGTGCCACTGAGCCTGAAAGA
+GTTGAAGCCGCGAGGACGTTGCTTTCTCAATCACCCATGTGTACGATGAAGGATCGCTAT
+GAAAGAACGCATAGCCTGAACTCTATCTCAGGTGAAGTGGCGTCTACGTAAACATCTTGT
+TCTCCCCAGGTTCTATCCTCGTAGGGGTTGCGGTCCCGAGGGAGCGAGCAGGGTGCTGAA
+CCAACCGTATAAACTGTTGGTATTTTATTCGTATGCTACGGCACAACCCGACCCTAGGAC
+CAGTCTCCCGAACCTGGGGACAGGAGCTCGCTTACCATTCATGTTCGAGCACTTGCATCG
+GTCAGGAGTCTAGCGGACGTTGACAATATGATGTGTTTTACTTTTTCCGATAGGTTCTTG
+TCCAGAAGATTGGCTGGAAAACGAGCCCTACCTCCGCCCTTATAGTGTCGCCGCTCCTCC
+CCCCGGGAACACAAGCTAGGTTCCACTTGGGCAGAGGCAGTTGATGAGGAGCTTAGTAGC
+GGTATAAGATTCTCAGACTAGCATGATCTACTTTGGGTCTGGAAAGAACGATAGTCGAGA
+AAGACTTAGGAGCTGTGTCCTCTATCCGTCCGCGGAGTAGGGATAAATGCATTAGTGCTT
+AGGGATATAATTCCGGCCGTAGGGTCAGTGGTTTCTCTTCGTCGTAGATGTTGTGTTGGG
+CTATCACACCCTTGGAGTGTCGGCCCGTCCTCTAATACATAAGCTTTCACGATTAGGAAC
+CCCCTGCCGCCCCGCGCCGCCGGTTAGTTTTAGGTTGACCCCCATTACATAGTGTTGCCG
+TGTGGAGCCTTAGGGTCTGGAACACAATCCTAGCATTCTATCCGTTTGTCTCTATTTCCA
+GTTACCGTGTACAGCTTACATGGGCCTTTCTCTCATTGCATTTGCATCGATTAGTTGCGC
+GGGCTACGACTTTAGTGTAGTACCAGCTGATAATAGCTTCGTCAATCCGCTATCTTCATG
+TGGGAGGCACAGAGGCCCGCAAGAGATGGTTGCTCGAACTCAGCGTAGGCAGGCGTGGGT
+GTGGACGTTATTTGCGTGTCACGTTTCAAGGCTAAGAAATCTGAAGCGCGATCTAGTGAC
+TTACAGGGTTGGAATGTTCAACCTGCGCAGCAGGGTTGGAACAAGTTCGACCACCTATCA
+AGTCGCAATAGCACTACCGTTCTTAGTTGAAAGAGATCTCAGTTCTCATTAGCACTGGAG
+CGTCGACTTTTCTGGGACCCCTAGCGTCCGACGAGCCGCTCCAAAGAACACACTTCCATC
+GCGAGGACAGATCGCTTTTCTTCAAATGCTGGCTTCTCTTTCGTACAAATGTTACGTGTG
+TATGAACATGGTAGGTCGATCCGTACTGACATCACGGTCACTGGGCCCCTTTGTGACGGC
+CTTGCTCTCTTAACGATAGGATTAACACCACTAAGACATGACATGCGTATAGTCTAGTGA
+TTGAACCGAGCCGAAGTGCATGGGGTCTCAGGCGAACTATCTAATGTTACTTACGACCTA
+GTGTGATATTGTATATTATCGGCGGATCCGGTGTATGTCATGACAGTGATATTGGTCGTA
+CCACGCCTCTGTATCCCTAACGGCTCCAGTCCACAATAAGGCTCAAAACTGTAGCCGGTT
+TATGCGGAGGATAATCGGACGCATTCTCATGGTGAATTGCCTACTATCGGTCCGCTGAGG
+TCTATCACCTGCAATCTTTGTATAGATACATACACTCAATTGGATAACGTTCACCAGTCA
+ATGGGTCGTGACGCGCATTACACACCGTTTGATCTCTGAACCTGGATGCATTGCGCCTCT
+AGGCTTGAGTTCATACAATGTAATTTGCACTCTACCCGCGTTCTTGGGGTATTTAAGGTG
+AATACGGCTCGCCTTACGGTGTGTTGGTTATATGACAGGGGGTGAGACTTGGACACATTT
+TGATGGGACCCGGAACGTGCCGCAGGTGCGCTATAGCAGTCCGCAAGACAAGCAACCGTC
+TATCGGAGATGGCCACACTAAGGCTTACTACTCTCACAGGAGTCTACGGCGATCCAGTAC
+CGACTCCACGTAAAGACGGACATGGAATTGAAGGTCATCATCTTTTAATCGTGTGGGGTC
+TCTTTCACGTTCTCAGTTAGGGCCAGTTTGTTCATAAATAGGCAATCGGCACTGAATACG
+AGACACCTTACCTTCCGAAATATATCACGCGCGGTTCTCCTATGAGCCCTGAGCTGGGGT
+AAGTTTGGACTTGCCAACAAACCTCTTATCATCCAAAGGGGCGTTTCGAGTTGATGCCAC
+ATACACGCTTGCTCAGGTACACCGTTGTCAAGCGGTCTGAACTCTCGGTCAGCCGGGTTC
+GGACCGGATCAAAAACCAGTCAGCATGAGCCCTACCCTGGTGGTTGGGGCAGGTCCTGCC
+TAAATTAGATTGCAATATCAGTCACATGGGCGCATAAATACTAGTCTGCGACTAGGCTGT
+CGCCAACTCACGACCTTGTACTGAGTGGTTAACGTTCATCGCGTGCAGATTACGGTCTTT
+CACCACGGAGCACGCGCTGACGCTGCCATCACGCCTGTGCCAATTTGCGGTGTAAAAATT
+TACGGATTGTCCAACTACGCAGTCGGCAGCTCCCGCAGAAATTAGAACAGATTGGTCACG
+GACGATGCTTCAGGCCCATATCAACAACAAACGAGGTAGTCGGCCGCTAGTTCACCGATA
+GTCTCAACCAACTTTTGCGGGGGTCGTCTGTTGACCAGTGTAGCGGGTCATTTACGACAG
+TTATGGGGACAAGTTGCTTGCCTTCAAACCCCGCTGGGACGCATCGCAGCCACCGAATGT
+TTACTGCCCGGTAATAGATACAACATCTCCGGATCTCCCGTTAGCCACGCCCACCTTCCT
+TCTTAACCCGGGGTTGCTAGAGAAACGATGGCCTTCCGGTTAATTGCACAATGCAGCCGT
+AGCTCTGCAGCTCGGCGTTCACACCCGAATATGAAGTTAACTCACAGCCGCCGTAGCGAC
+AAGTTTGAGATCCGCCGGCTGCCATCCCACCACGAATCAACATTACGCTTGAATTGCGAT
+GCTTGCTACGAGATAATGGTACTGCGTAACGGATGTGCCAATATAGCGGTTGGGCGACGT
+ATTATATTGACCCTCTGCGGTTCCCGGCGTGTTGAGCGTCGTTACCCCCGAACACAGAGT
+GGGGCGTAATTATTGCGATGTCACACCACCCGGGAACGGTCAAACCGTCAGCGGGCTTGT
+GTAGAGCTCAGCCTCTTTGGTTCCGCCACGGACTTCTCATTTTGCACGCGGCTTGCCTTG
+ATTCTATATTTGATTTACCTGAGACGTATGTTCCTGTCCATACTGTGAGATCCGTCGATG
+ACTTATCACATCAGCTTTGCGGAGCTCCGTGCCGGTGGGTCTAACTCGGAAGAAGCCTAG
+GCAAGCTCTCTGGGTTGAAT
diff --git a/tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz b/tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz
new file mode 100644
index 0000000..87a743f
Binary files /dev/null and b/tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz b/tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz
new file mode 100644
index 0000000..f85b112
Binary files /dev/null and b/tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_A_RUN2_R1.fq.gz b/tests/data/reads/SAMPLE_A_RUN2_R1.fq.gz
new file mode 100644
index 0000000..bae43e8
Binary files /dev/null and b/tests/data/reads/SAMPLE_A_RUN2_R1.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_A_RUN2_R2.fq.gz b/tests/data/reads/SAMPLE_A_RUN2_R2.fq.gz
new file mode 100644
index 0000000..c8bd79d
Binary files /dev/null and b/tests/data/reads/SAMPLE_A_RUN2_R2.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_B_RUN1_R1.fq.gz b/tests/data/reads/SAMPLE_B_RUN1_R1.fq.gz
new file mode 100644
index 0000000..0b6184b
Binary files /dev/null and b/tests/data/reads/SAMPLE_B_RUN1_R1.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_B_RUN1_R2.fq.gz b/tests/data/reads/SAMPLE_B_RUN1_R2.fq.gz
new file mode 100644
index 0000000..8566010
Binary files /dev/null and b/tests/data/reads/SAMPLE_B_RUN1_R2.fq.gz differ
diff --git a/tests/data/reads/SAMPLE_C_RUN1.fq.gz b/tests/data/reads/SAMPLE_C_RUN1.fq.gz
new file mode 100644
index 0000000..92b06c8
Binary files /dev/null and b/tests/data/reads/SAMPLE_C_RUN1.fq.gz differ
diff --git a/tests/data/reference.fasta b/tests/data/reference.fasta
new file mode 100644
index 0000000..f9979d4
--- /dev/null
+++ b/tests/data/reference.fasta
@@ -0,0 +1,85 @@
+>test_chr
+GGTGAATGTTGACGCTCAAGAGAGTCCACTCCCCTCTCGCCGGTAAGGTTCCACGCAGCC
+TATTGCTCGCGAGGCCCGGTGGAGTAGGGTATCCGTCAAAGGCCAGAAACTACTATGGGC
+TCGCGCACCAAGGCACGTTCATTGGCAGTGTACCGAGGTGTCATTCCGCTTGCCCTCGTT
+GCGGATTGTCAACCCGTGCCATCCATCCCGCATCAGCACCAGTATTAACCATAGAGGAAG
+AGATCTCACTCAACAGGCCATTTTAACTGCTCTTGGCAGCCTAAGAGATATGTAGCTACT
+CCGTCCGCTCGGCTATTATATTTAGTTATTTCGTGTATGAACGTGCAATACAAATCCCGC
+AATACAAGGACCACCTGTCGAGAACCATCCCATGTTCCATTCATTATGATTCCCTGGCTC
+TTCCGCTGAGCGTATCCTCTGCAAGACCAGGCCGTGACTCCCACCCTTTGAGCCTTGGTA
+ATAAGCAATTAACGCGGTGCAATTGGTCGCTACGAGATAAGACAAATGCCAACAATGGAA
+ACTGCAATTATCTATCTAATAGTTGCCCCCTTCAAACCGCGCACACTTGGCGCGTATATA
+TGTACCTCCTCCTTTTGCGTGCCTGTGATAGGCCGCTGGTCGTGCCCCTCGACCCTGTAC
+CATGACCGGGCGTTGCCTCGAAGCAGCTCGACGGTCACAGACAATAAGAAACGACCTTAT
+GTGCGTGCGGTGGAGAGTTACTATCAAACAACGCTCATGCGGAACACTCCGATGGTGTGT
+AAAGAATACACTGACACTATACGATAGACGGAGCCGAGCTCACATCAATGAGTGTCATCA
+TACCGCACCATTTTTAGACAATAGGTCAGAGTACAGGTTGCGTCGAGTCGTGGGGGCTCG
+GTTAGCGGTACGCGTTAACACGGAAACTCAGCATATGGGCCCTCGGGCCCCATTCATCGT
+GATCATGGTATCTAGCGAACGTGAAACGCAACGTTCAAATATGGCACTGTACCTTCCCCA
+CCTCTCAGTTCCCATGGTTTGCCCGTTACTGAAGATTTGCAAGACTCGTGGCCAATCATG
+GTGTGGAATTAGGGTCGGCTATGTCAGGACATTGTGCTGTAAATGGCTTCTCGCTAAGGT
+CGACTGCTAAGGTAAATGGCCTAGCAACAGCCTATTCGTTGGGCGCACATCAAGGTTCAG
+AGCTAAAGCTCTTTGCGACCCAGCTATGGTAAGGCAAGAGATGGAGACATCTTGCGGAGC
+TGTTATATACCGTGGAAAGCGTCGACCACGAAAGGGCACTGAGACAGAGGATGTAATTGG
+AATCAGTGCATTAGCAACTCGTGTGAGAGGACTTGTGCAAGCTTTTCCTCAATCTCGATA
+AAGATATTTGTCGCTAGGACCTGGGACAGGCCCGGTGGCATGCACTTATGTCATGTATCC
+ATGACGTTGAATTAAAATCTGAGAGCGGATGTGACAGTTATAACAGGGGGTGAGGCTAAA
+CAAACCGGCGTCACGCAATACCGGCGAGGTGGCCTGAGTGGTGCCACTGAGCCTGAAAGA
+GTTGAAGCCGCGAGGACGTTGCTTTCTCAATCACCCATGTGTACGATGAAGGATCGCTAT
+GAAAGAACGCATAGCCTGAACTCTATCTCAGGTGAAGTGGCGTCTACGTAAACATCTTGT
+TCTCCCCAGGTTCTATCCTCGTAGGGGTTGCGGTCCCGAGGGAGCGAGCAGGGTGCTGAA
+CCAACCGTATAAACTGTTGGTATTTTATTCGTATGCTACGGCACAACCCGACCCTAGGAC
+CAGTCTCCCGAACCTGGGGACAGGAGCTCGCTTACCATTCATGTTCGAGCACTTGCATCG
+GTCAGGAGTCTAGCGGACGTTGACAATATGATGTGTTTTACTTTTTCCGATAGGTTCTTG
+TCCAGAAGATTGGCTGGAAAACGAGCCCTACCTCCGCCCTTATAGTGTCGCCGCTCCTCC
+CCCCGGGAACACAAGCTAGGTTCCACTTGGGCAGAGGCAGTTGATGAGGAGCTTAGTAGC
+GGTATAAGATTCTCAGACTAGCATGATCTACTTTGGGTCTGGAAAGAACGATAGTCGAGA
+AAGACTTAGGAGCTGTGTCCTCTATCCGTCCGCGGAGTAGGGATAAATGCATTAGTGCTT
+AGGGATATAATTCCGGCCGTAGGGTCAGTGGTTTCTCTTCGTCGTAGATGTTGTGTTGGG
+CTATCACACCCTTGGAGTGTCGGCCCGTCCTCTAATACATAAGCTTTCACGATTAGGAAC
+CCCCTGCCGCCCCGCGCCGCCGGTTAGTTTTAGGTTGACCCCCATTACATAGTGTTGCCG
+TGTGGAGCCTTAGGGTCTGGAACACAATCCTAGCATTCTATCCGTTTGTCTCTATTTCCA
+GTTACCGTGTACAGCTTACATGGGCCTTTCTCTCATTGCATTTGCATCGATTAGTTGCGC
+GGGCTACGACTTTAGTGTAGTACCAGCTGATAATAGCTTCGTCAATCCGCTATCTTCATG
+TGGGAGGCACAGAGGCCCGCAAGAGATGGTTGCTCGAACTCAGCGTAGGCAGGCGTGGGT
+GTGGACGTTATTTGCGTGTCACGTTTCAAGGCTAAGAAATCTGAAGCGCGATCTAGTGAC
+TTACAGGGTTGGAATGTTCAACCTGCGCAGCAGGGTTGGAACAAGTTCGACCACCTATCA
+AGTCGCAATAGCACTACCGTTCTTAGTTGAAAGAGATCTCAGTTCTCATTAGCACTGGAG
+CGTCGACTTTTCTGGGACCCCTAGCGTCCGACGAGCCGCTCCAAAGAACACACTTCCATC
+GCGAGGACAGATCGCTTTTCTTCAAATGCTGGCTTCTCTTTCGTACAAATGTTACGTGTG
+TATGAACATGGTAGGTCGATCCGTACTGACATCACGGTCACTGGGCCCCTTTGTGACGGC
+CTTGCTCTCTTAACGATAGGATTAACACCACTAAGACATGACATGCGTATAGTCTAGTGA
+TTGAACCGAGCCGAAGTGCATGGGGTCTCAGGCGAACTATCTAATGTTACTTACGACCTA
+GTGTGATATTGTATATTATCGGCGGATCCGGTGTATGTCATGACAGTGATATTGGTCGTA
+CCACGCCTCTGTATCCCTAACGGCTCCAGTCCACAATAAGGCTCAAAACTGTAGCCGGTT
+TATGCGGAGGATAATCGGACGCATTCTCATGGTGAATTGCCTACTATCGGTCCGCTGAGG
+TCTATCACCTGCAATCTTTGTATAGATACATACACTCAATTGGATAACGTTCACCAGTCA
+ATGGGTCGTGACGCGCATTACACACCGTTTGATCTCTGAACCTGGATGCATTGCGCCTCT
+AGGCTTGAGTTCATACAATGTAATTTGCACTCTACCCGCGTTCTTGGGGTATTTAAGGTG
+AATACGGCTCGCCTTACGGTGTGTTGGTTATATGACAGGGGGTGAGACTTGGACACATTT
+TGATGGGACCCGGAACGTGCCGCAGGTGCGCTATAGCAGTCCGCAAGACAAGCAACCGTC
+TATCGGAGATGGCCACACTAAGGCTTACTACTCTCACAGGAGTCTACGGCGATCCAGTAC
+CGACTCCACGTAAAGACGGACATGGAATTGAAGGTCATCATCTTTTAATCGTGTGGGGTC
+TCTTTCACGTTCTCAGTTAGGGCCAGTTTGTTCATAAATAGGCAATCGGCACTGAATACG
+AGACACCTTACCTTCCGAAATATATCACGCGCGGTTCTCCTATGAGCCCTGAGCTGGGGT
+AAGTTTGGACTTGCCAACAAACCTCTTATCATCCAAAGGGGCGTTTCGAGTTGATGCCAC
+ATACACGCTTGCTCAGGTACACCGTTGTCAAGCGGTCTGAACTCTCGGTCAGCCGGGTTC
+GGACCGGATCAAAAACCAGTCAGCATGAGCCCTACCCTGGTGGTTGGGGCAGGTCCTGCC
+TAAATTAGATTGCAATATCAGTCACATGGGCGCATAAATACTAGTCTGCGACTAGGCTGT
+CGCCAACTCACGACCTTGTACTGAGTGGTTAACGTTCATCGCGTGCAGATTACGGTCTTT
+CACCACGGAGCACGCGCTGACGCTGCCATCACGCCTGTGCCAATTTGCGGTGTAAAAATT
+TACGGATTGTCCAACTACGCAGTCGGCAGCTCCCGCAGAAATTAGAACAGATTGGTCACG
+GACGATGCTTCAGGCCCATATCAACAACAAACGAGGTAGTCGGCCGCTAGTTCACCGATA
+GTCTCAACCAACTTTTGCGGGGGTCGTCTGTTGACCAGTGTAGCGGGTCATTTACGACAG
+TTATGGGGACAAGTTGCTTGCCTTCAAACCCCGCTGGGACGCATCGCAGCCACCGAATGT
+TTACTGCCCGGTAATAGATACAACATCTCCGGATCTCCCGTTAGCCACGCCCACCTTCCT
+TCTTAACCCGGGGTTGCTAGAGAAACGATGGCCTTCCGGTTAATTGCACAATGCAGCCGT
+AGCTCTGCAGCTCGGCGTTCACACCCGAATATGAAGTTAACTCACAGCCGCCGTAGCGAC
+AAGTTTGAGATCCGCCGGCTGCCATCCCACCACGAATCAACATTACGCTTGAATTGCGAT
+GCTTGCTACGAGATAATGGTACTGCGTAACGGATGTGCCAATATAGCGGTTGGGCGACGT
+ATTATATTGACCCTCTGCGGTTCCCGGCGTGTTGAGCGTCGTTACCCCCGAACACAGAGT
+GGGGCGTAATTATTGCGATGTCACACCACCCGGGAACGGTCAAACCGTCAGCGGGCTTGT
+GTAGAGCTCAGCCTCTTTGGTTCCGCCACGGACTTCTCATTTTGCACGCGGCTTGCCTTG
+ATTCTATATTTGATTTACCTGAGACGTATGTTCCTGTCCATACTGTGAGATCCGTCGATG
+ACTTATCACATCAGCTTTGCGGAGCTCCGTGCCGGTGGGTCTAACTCGGAAGAAGCCTAG
+GCAAGCTCTCTGGGTTGAAT
diff --git a/tests/data/reference.gff3 b/tests/data/reference.gff3
new file mode 100644
index 0000000..425702e
--- /dev/null
+++ b/tests/data/reference.gff3
@@ -0,0 +1,9 @@
+##gff-version 3
+##sequence-region test_chr 1 5000
+test_chr test region 1 5000 . + . ID=test_chr;Name=test_chr
+test_chr test gene 201 1400 . + . ID=gene-TEST_0001;Name=dnaA;locus_tag=TEST_0001
+test_chr test CDS 201 1400 . + 0 ID=cds-TEST_0001;Parent=gene-TEST_0001;Name=dnaA;locus_tag=TEST_0001;product=dnaA test product
+test_chr test gene 1601 3100 . + . ID=gene-TEST_0002;Name=rpoB;locus_tag=TEST_0002
+test_chr test CDS 1601 3100 . + 0 ID=cds-TEST_0002;Parent=gene-TEST_0002;Name=rpoB;locus_tag=TEST_0002;product=rpoB test product
+test_chr test gene 3301 4700 . - . ID=gene-TEST_0003;Name=katG;locus_tag=TEST_0003
+test_chr test CDS 3301 4700 . - 0 ID=cds-TEST_0003;Parent=gene-TEST_0003;Name=katG;locus_tag=TEST_0003;product=katG test product
diff --git a/tests/data/samplesheet.tsv b/tests/data/samplesheet.tsv
new file mode 100644
index 0000000..6be0455
--- /dev/null
+++ b/tests/data/samplesheet.tsv
@@ -0,0 +1,5 @@
+sampleId runId r1 r2 refId refFasta refGff taxId
+SAMPLE_A RUN1 tests/data/reads/SAMPLE_A_RUN1_R1.fq.gz tests/data/reads/SAMPLE_A_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_A RUN2 tests/data/reads/SAMPLE_A_RUN2_R1.fq.gz tests/data/reads/SAMPLE_A_RUN2_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_B RUN1 tests/data/reads/SAMPLE_B_RUN1_R1.fq.gz tests/data/reads/SAMPLE_B_RUN1_R2.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
+SAMPLE_C RUN1 tests/data/reads/SAMPLE_C_RUN1.fq.gz TESTREF tests/data/reference.fasta tests/data/reference.gff3 1773
diff --git a/tests/js/bundle.test.mjs b/tests/js/bundle.test.mjs
new file mode 100644
index 0000000..87b7f2b
--- /dev/null
+++ b/tests/js/bundle.test.mjs
@@ -0,0 +1,117 @@
+// Integrity of the report bundle itself.
+//
+// bin/qcreport/render.py builds the report by concatenating a HARD-CODED list of CSS and JS fragments
+// and splicing them into shell.html. Nothing checks that list against what is actually on disk, so a
+// new fragment that nobody registers ships as dead code and a syntax error in any one fragment
+// only surfaces when a human opens the HTML. These tests close both gaps.
+
+import { strict as assert } from "node:assert";
+import { readdirSync, readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { describe, it } from "node:test";
+import vm from "node:vm";
+import { bundle, moduleOrder, readModule } from "./harness.mjs";
+
+const ROOT = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
+const ASSETS = join(ROOT, "bin", "report_assets");
+const RENDER_PY = readFileSync(join(ROOT, "bin", "qcreport", "render.py"), "utf8");
+
+function pyList(name) {
+ const block = RENDER_PY.match(new RegExp(`${name}\\s*=\\s*\\[([\\s\\S]*?)\\]`));
+ assert.ok(block, `could not find ${name} in bin/qcreport/render.py`);
+ return [...block[1].matchAll(/"([^"]+)"/g)].map((m) => m[1]);
+}
+
+describe("asset registration", () => {
+ it("ships every .js fragment on disk", () => {
+ const onDisk = readdirSync(join(ASSETS, "js")).filter((f) => f.endsWith(".js")).sort();
+ const registered = moduleOrder().slice().sort();
+ assert.deepEqual(
+ registered,
+ onDisk,
+ "bin/report_assets/js/ and _JS_MODULES in bin/qcreport/render.py disagree; " +
+ "an unregistered fragment is never included in the report",
+ );
+ });
+
+ it("ships every .css fragment on disk", () => {
+ const onDisk = readdirSync(join(ASSETS, "css")).filter((f) => f.endsWith(".css")).sort();
+ const registered = pyList("_CSS_MODULES").map((p) => p.replace(/^css\//, "")).sort();
+ assert.deepEqual(registered, onDisk, "css/ and _CSS_MODULES disagree");
+ });
+
+ it("concatenates the JS fragments in numeric order", () => {
+ const order = moduleOrder();
+ const sorted = order.slice().sort();
+ assert.deepEqual(order, sorted, "the fragments are numbered, so the list must stay sorted");
+ });
+
+ it("ends every fragment with a newline, since they are joined with no separator", () => {
+ // render.py does "".join(...), so a fragment without a trailing newline would splice its
+ // last line into the next fragment's first line.
+ for (const name of moduleOrder()) {
+ const text = readModule(name);
+ assert.ok(text.endsWith("\n"), `${name} does not end with a newline`);
+ }
+ });
+});
+
+describe("bundle syntax", () => {
+ it("parses as the single IIFE the report expects", () => {
+ // 01_prelude.js opens the IIFE and 14_boot.js closes it, so only the whole bundle balances.
+ assert.doesNotThrow(
+ () => new vm.Script(bundle(), { filename: "bundle.js" }),
+ "the concatenated report JS is not syntactically valid",
+ );
+ });
+
+ it("parses inside the function wrapper shell.html puts it in", () => {
+ const wrapped = `function __qcmain__(){\n${bundle()}\n}`;
+ assert.doesNotThrow(() => new vm.Script(wrapped, { filename: "wrapped.js" }));
+ });
+
+ it("stays ES5, with no syntax an old browser would reject", () => {
+ // The report is deliberately ES5: it has to open from a file:// URL on whatever browser a
+ // collaborator happens to have, and past releases have shipped ES5 slips. Node parses ES2015+
+ // happily, so detect the common slips by pattern. Only patterns that cannot plausibly occur in
+ // prose are checked unanchored; `const`/`let` are anchored to a statement position, because
+ // "constant" and words like "first-class" appear all over the comments.
+ const offenders = [
+ [/=>/g, "an arrow function"],
+ [/`/g, "a template literal"],
+ [/(?:^|[;{}()])\s*(?:const|let)\s+[A-Za-z_$]/gm, "a const/let declaration"],
+ ];
+ const problems = [];
+ for (const name of moduleOrder()) {
+ const text = readModule(name);
+ for (const [re, label] of offenders) {
+ for (const hit of text.matchAll(re)) {
+ const line = text.slice(0, hit.index).split("\n").length;
+ problems.push(`${name}:${line} uses ${label}`);
+ }
+ }
+ }
+ assert.deepEqual(problems, [], `non-ES5 syntax in the report bundle:\n ${problems.join("\n ")}`);
+ });
+});
+
+describe("shell.html", () => {
+ const shell = readFileSync(join(ASSETS, "shell.html"), "utf8");
+
+ it("contains every placeholder render.py substitutes", () => {
+ for (const token of ["__CSS__", "__JS__", "__JSON_GZ__", "__LOGO__", "__REPO__", "__TITLE__"]) {
+ assert.ok(shell.includes(token), `shell.html is missing the ${token} placeholder`);
+ }
+ });
+
+ it("does not itself contain a placeholder token inside the JS or CSS fragments", () => {
+ // A fragment containing "__JS__" would be re-substituted and duplicate the whole bundle.
+ for (const name of moduleOrder()) {
+ const text = readModule(name);
+ for (const token of ["__CSS__", "__JS__", "__JSON_GZ__", "__LOGO__"]) {
+ assert.ok(!text.includes(token), `${name} contains the substitution token ${token}`);
+ }
+ }
+ });
+});
diff --git a/tests/js/harness.mjs b/tests/js/harness.mjs
new file mode 100644
index 0000000..d350fd1
--- /dev/null
+++ b/tests/js/harness.mjs
@@ -0,0 +1,135 @@
+// Loads the report's front-end so its pure functions can be unit-tested from Node.
+//
+// The report ships as ONE ES5 IIFE: bin/qcreport/render.py concatenates bin/report_assets/js/*.js
+// in a fixed order and drops the result into shell.html. 01_prelude.js opens the IIFE and
+// 14_boot.js closes it, so no individual fragment (and no prefix of them) is a valid
+// script on its own.
+//
+// To reach the functions without editing the sources we concatenate everything except
+// 14_boot.js - which is pure DOM wiring - and append our own epilogue that captures the
+// declarations and closes the IIFE. Function declarations hoist to the top of the IIFE,
+// so the epilogue sees every function regardless of which fragment declared it. Nothing
+// here depends on line numbers, so the tests survive refactoring of the JS.
+
+import { readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import vm from "node:vm";
+
+const ROOT = join(dirname(fileURLToPath(import.meta.url)), "..", "..");
+const JS_DIR = join(ROOT, "bin", "report_assets", "js");
+
+/** The module order the report uses, read from bin/qcreport/render.py itself so the two cannot drift. */
+export function moduleOrder() {
+ const py = readFileSync(join(ROOT, "bin", "qcreport", "render.py"), "utf8");
+ const block = py.match(/_JS_MODULES\s*=\s*\[([\s\S]*?)\]/);
+ if (!block) throw new Error("could not find _JS_MODULES in bin/qcreport/render.py");
+ return [...block[1].matchAll(/"js\/([^"]+)"/g)].map((m) => m[1]);
+}
+
+export function readModule(name) {
+ return readFileSync(join(JS_DIR, name), "utf8");
+}
+
+/** The exact string render.py injects into shell.html (a bare concatenation, no separator). */
+export function bundle() {
+ return moduleOrder().map(readModule).join("");
+}
+
+// Minimal stand-ins for what the fragments touch while they are being evaluated:
+// 01_prelude.js reads the REPORT global and calls isDark(); 03_state.js reads
+// window.innerWidth and iterates R.samples / R.metrics.
+function stubContext(report) {
+ const classList = { contains: () => false, add() {}, remove() {}, toggle() {} };
+ const element = {
+ classList,
+ style: {},
+ textContent: "",
+ innerHTML: "",
+ appendChild() {},
+ setAttribute() {},
+ getAttribute: () => null,
+ addEventListener() {},
+ querySelectorAll: () => [],
+ };
+ const document = {
+ documentElement: { classList },
+ body: { classList },
+ getElementById: () => null,
+ querySelector: () => null,
+ querySelectorAll: () => [],
+ createElement: () => element,
+ createElementNS: () => element,
+ addEventListener() {},
+ };
+ const window = {
+ innerWidth: 1280,
+ addEventListener() {},
+ matchMedia: () => ({ matches: false, addEventListener() {} }),
+ localStorage: { getItem: () => null, setItem() {}, removeItem() {} },
+ };
+ const ctx = {
+ REPORT: report,
+ document,
+ window,
+ console,
+ Math,
+ JSON,
+ setTimeout,
+ clearTimeout,
+ requestAnimationFrame: (fn) => fn(),
+ };
+ ctx.globalThis = ctx;
+ return ctx;
+}
+
+/**
+ * A REPORT payload just rich enough for every fragment to finish evaluating.
+ *
+ * The fragments run real setup code at load: 01_prelude.js derives metrics off R.samples and
+ * pushes onto R.dist, 03_state.js builds the threshold and metric tables. An empty cohort keeps
+ * all of that a no-op while still exercising the code path.
+ */
+export function emptyReport() {
+ return {
+ samples: [],
+ metrics: [],
+ dist: [],
+ lineages: [],
+ gene_map: {},
+ defs: {},
+ n_ancient: 0,
+ nbins: 0,
+ genome_len: 0,
+ snp_density_ok: false,
+ mask_bins: [],
+ thresholds: {},
+ version: "test",
+ };
+}
+
+/**
+ * Evaluate every fragment except the DOM-wiring one and return the named functions.
+ *
+ * @param {string[]} names function names to pull out of the IIFE scope
+ * @param {object} [report] optional REPORT payload override
+ */
+export function loadFunctions(names, report = emptyReport()) {
+ const order = moduleOrder().filter((m) => m !== "14_boot.js");
+ const epilogue =
+ "\n__CAPTURE__(" +
+ "{" +
+ names.map((n) => `${JSON.stringify(n)}: typeof ${n} === "undefined" ? undefined : ${n}`).join(",") +
+ "});\n})();\n";
+
+ const captured = {};
+ const ctx = stubContext(report);
+ ctx.__CAPTURE__ = (obj) => Object.assign(captured, obj);
+
+ vm.createContext(ctx);
+ new vm.Script(order.map(readModule).join("") + epilogue, { filename: "bampiro-report-bundle.js" }).runInContext(ctx);
+
+ const missing = names.filter((n) => typeof captured[n] !== "function");
+ if (missing.length) throw new Error(`not found in the report bundle: ${missing.join(", ")}`);
+ return captured;
+}
diff --git a/tests/js/helpers.test.mjs b/tests/js/helpers.test.mjs
new file mode 100644
index 0000000..2918050
--- /dev/null
+++ b/tests/js/helpers.test.mjs
@@ -0,0 +1,246 @@
+// Pure helpers behind the report's interactive panels: the column filter grammar, the SNP-dynamics
+// classifier, the QC-space linear algebra, and the small formatters. None of these touch the DOM.
+
+import { strict as assert } from "node:assert";
+import { before, describe, it } from "node:test";
+import { loadFunctions } from "./harness.mjs";
+
+const arr = (x) => Array.from(x);
+
+function closeTo(actual, expected, tol, what) {
+ assert.ok(
+ Math.abs(actual - expected) <= tol,
+ `${what}: ${actual} is not within ${tol} of ${expected}`,
+ );
+}
+
+let fn;
+before(() => {
+ fn = loadFunctions([
+ "colMatchOne", "esc", "fmt", "shortv", "fmtpos",
+ "quant", "med2", "robustMedSig", "robustMS",
+ "dynLogit", "dynSlope", "dynSelCls",
+ "covMatrix", "topEigs", "invRidge", "chi2q975", "spreadEllipse", "yearOf",
+ "epiFmtR", "epiNodeLbl", "drStatus", "drGColor", "snpAfColor", "zeros", "assign",
+ ]);
+});
+
+describe("colMatchOne, the per-column filter grammar", () => {
+ it("treats a blank filter as matching everything", () => {
+ assert.equal(fn.colMatchOne("", 5, "anything"), true);
+ assert.equal(fn.colMatchOne(" ", null, ""), true);
+ });
+
+ it("applies numeric comparisons", () => {
+ assert.equal(fn.colMatchOne(">=10", 10, ""), true);
+ assert.equal(fn.colMatchOne(">=10", 9.9, ""), false);
+ assert.equal(fn.colMatchOne("<5", 4, ""), true);
+ assert.equal(fn.colMatchOne("<5", 5, ""), false);
+ assert.equal(fn.colMatchOne("=42", 42, ""), true);
+ assert.equal(fn.colMatchOne(">-3", -2, ""), true);
+ });
+
+ it("applies ranges and tolerates a reversed one", () => {
+ assert.equal(fn.colMatchOne("10..20", 15, ""), true);
+ assert.equal(fn.colMatchOne("10..20", 21, ""), false);
+ assert.equal(fn.colMatchOne("20..10", 15, ""), true, "a reversed range should be swapped");
+ assert.equal(fn.colMatchOne("1 to 3", 2, ""), true);
+ });
+
+ it("falls back to a case-insensitive substring match on the rendered text", () => {
+ assert.equal(fn.colMatchOne("kat", null, "katG"), true);
+ assert.equal(fn.colMatchOne("KAT", null, "katG"), true);
+ assert.equal(fn.colMatchOne("rpo", null, "katG"), false);
+ });
+
+ it("does not match a numeric filter against a null value", () => {
+ assert.equal(fn.colMatchOne(">=10", null, "10"), false);
+ });
+});
+
+describe("formatters", () => {
+ it("escapes the HTML metacharacters it claims to", () => {
+ assert.equal(fn.esc('&'), "<a href="x">&");
+ });
+
+ it("renders NA rather than a crash for a missing value", () => {
+ assert.equal(fn.fmt(null, "int"), "NA");
+ assert.equal(fn.fmt(null, "pct"), "NA");
+ assert.equal(fn.shortv(null, "int"), "");
+ });
+
+ it("formats by metric kind", () => {
+ assert.equal(fn.fmt(1234.6, "int"), "1,235");
+ assert.equal(fn.fmt(12.345, "pct"), "12.3");
+ assert.equal(fn.fmt(12.345, "float"), "12.35");
+ });
+
+ it("scales genome coordinates", () => {
+ assert.equal(fn.fmtpos(4411532), "4.41 Mb");
+ assert.equal(fn.fmtpos(15000), "15 kb");
+ assert.equal(fn.fmtpos(742), "742 bp");
+ });
+});
+
+describe("quantiles and robust spread", () => {
+ it("interpolates quantiles on a pre-sorted array", () => {
+ const s = [1, 2, 3, 4, 5];
+ assert.equal(fn.quant(s, 0), 1);
+ assert.equal(fn.quant(s, 1), 5);
+ assert.equal(fn.quant(s, 0.5), 3);
+ closeTo(fn.quant(s, 0.25), 2, 1e-12, "q25");
+ assert.equal(fn.quant([], 0.5), null);
+ });
+
+ it("averages the two middle values for an even-length median", () => {
+ assert.equal(fn.med2([1, 2, 3, 4]), 2.5);
+ assert.equal(fn.med2([1, 2, 3]), 2);
+ });
+
+ it("needs four non-null values before reporting a robust spread", () => {
+ assert.equal(fn.robustMedSig([1, 2, 3]), null);
+ assert.equal(fn.robustMedSig([1, null, 2, null]), null);
+ const r = fn.robustMedSig([10, 12, 14, 16]);
+ assert.equal(r.med, 13);
+ assert.ok(r.sig > 0, "sigma must be positive");
+ });
+
+ it("never returns a zero sigma, which would divide by zero downstream", () => {
+ // robustMS has no minimum-n and floors sigma at 1.
+ const [med, sig] = arr(fn.robustMS([7, 7, 7, 7]));
+ assert.equal(med, 7);
+ assert.ok(sig > 0, `sigma was ${sig}`);
+ });
+});
+
+describe("SNP dynamics", () => {
+ it("clamps the logit so a fixed or absent allele stays finite", () => {
+ assert.ok(Number.isFinite(fn.dynLogit(0)), "AF 0 produced a non-finite logit");
+ assert.ok(Number.isFinite(fn.dynLogit(1)), "AF 1 produced a non-finite logit");
+ assert.equal(fn.dynLogit(0.5), 0);
+ });
+
+ it("fits a positive slope to a rising trajectory", () => {
+ const r = fn.dynSlope([0.05, 0.3, 0.7, 0.95], [0, 1, 2, 3]);
+ assert.ok(r.s > 0, `slope was ${r.s}`);
+ assert.ok(r.r2 > 0.9, `r2 was ${r.r2}`);
+ assert.equal(r.n, 4);
+ });
+
+ it("returns null with fewer than two observations", () => {
+ assert.equal(fn.dynSlope([null, 0.4, null], [0, 1, 2]), null);
+ assert.equal(fn.dynSlope([], []), null);
+ });
+
+ it("reports a zero slope when every timepoint is the same", () => {
+ const r = fn.dynSlope([0.2, 0.4, 0.6], [5, 5, 5]);
+ assert.equal(r.s, 0);
+ });
+
+ it("classifies the selection patterns the panel colours by", () => {
+ assert.equal(fn.dynSelCls([0.05, 0.5, 0.95]).cls, "sweep");
+ assert.equal(fn.dynSelCls([0.02, 0.2, 0.4]).cls, "emerge");
+ assert.equal(fn.dynSelCls([0.9, 0.5, 0.05]).cls, "lost");
+ assert.equal(fn.dynSelCls([0.5, 0.52, 0.49]).cls, "stable");
+ assert.equal(fn.dynSelCls([0.4]).cls, "single");
+ });
+});
+
+describe("QC-space linear algebra", () => {
+ it("computes a covariance matrix with the n-1 divisor", () => {
+ // covMatrix does NOT re-center: buildZ has already z-scored the columns, so the input must
+ // be centered. Columns here are [-1,0,1] and [-2,0,2]; var = 1 and 4, cov = 2.
+ const C = fn.covMatrix([[-1, -2], [0, 0], [1, 2]], 2);
+ closeTo(C[0][0], 1, 1e-12, "var x");
+ closeTo(C[1][1], 4, 1e-12, "var y");
+ closeTo(C[0][1], 2, 1e-12, "cov");
+ closeTo(C[0][1], C[1][0], 1e-15, "symmetry");
+ });
+
+ it("recovers the eigenvalues of a diagonal matrix", () => {
+ const e = fn.topEigs([[9, 0], [0, 4]], 2, 2);
+ closeTo(e.vals[0], 9, 1e-6, "first eigenvalue");
+ closeTo(e.vals[1], 4, 1e-6, "second eigenvalue");
+ });
+
+ it("is deterministic, since power iteration is seeded rather than random", () => {
+ const C = [[5, 2], [2, 3]];
+ const a = fn.topEigs(C, 2, 2);
+ const b = fn.topEigs(C, 2, 2);
+ assert.deepEqual(arr(a.vals), arr(b.vals));
+ });
+
+ it("inverts a ridge-regularized matrix", () => {
+ const inv = fn.invRidge([[2, 0], [0, 4]], 2, 0);
+ closeTo(inv[0][0], 0.5, 1e-12, "inv[0][0]");
+ closeTo(inv[1][1], 0.25, 1e-12, "inv[1][1]");
+ });
+
+ it("returns null instead of dividing by a singular pivot", () => {
+ assert.equal(fn.invRidge([[0, 0], [0, 0]], 2, 0), null);
+ });
+
+ it("approximates the chi-square 97.5th percentile", () => {
+ // Wilson-Hilferty, so an approximation. Reference: scipy.stats.chi2.ppf(0.975, k).
+ // The tolerance is the accuracy the approximation actually achieves, which improves with k.
+ closeTo(fn.chi2q975(2), 7.3778, 0.05, "k=2");
+ closeTo(fn.chi2q975(4), 11.1433, 0.03, "k=4");
+ closeTo(fn.chi2q975(10), 20.4832, 0.02, "k=10");
+ });
+
+ it("fits a spread ellipse and needs at least four points", () => {
+ assert.equal(fn.spreadEllipse([[0, 0], [1, 1], [2, 2]]), null);
+ const e = fn.spreadEllipse([[0, 0], [1, 0], [0, 1], [1, 1], [2, 2]]);
+ assert.ok(e.rx >= 0 && e.ry >= 0, "ellipse radii must be non-negative");
+ });
+
+ it("extracts a plausible year and rejects an implausible one", () => {
+ assert.equal(fn.yearOf("collected 2018-04-02"), 2018);
+ assert.equal(fn.yearOf("1998"), 1998);
+ assert.equal(fn.yearOf("sample 12"), null);
+ assert.equal(fn.yearOf(null), null);
+ });
+});
+
+describe("panel colouring and labels", () => {
+ it("marks resistance from the worst WHO grade present", () => {
+ assert.equal(fn.drStatus([1, 4]).t, "R");
+ assert.equal(fn.drStatus([3, 5]).t, "?");
+ assert.equal(fn.drStatus([5]).t, "·");
+ assert.equal(fn.drStatus([]).t, "");
+ });
+
+ it("colours WHO grades 1 and 2 as resistant", () => {
+ assert.equal(fn.drGColor(1), fn.drGColor(2));
+ assert.notEqual(fn.drGColor(1), fn.drGColor(3));
+ });
+
+ it("formats a correlation with an explicit sign and no leading zero", () => {
+ assert.equal(fn.epiFmtR(-0.85), "-.85");
+ assert.equal(fn.epiFmtR(0.85), "+.85");
+ assert.equal(fn.epiFmtR(0), ".00");
+ });
+
+ it("labels an intergenic epistasis node rather than leaving it blank", () => {
+ assert.equal(fn.epiNodeLbl({ gene: "", pos: "chr:761155" }), "(intergenic) 761155");
+ assert.equal(fn.epiNodeLbl({ gene: "rpoB", pos: 761155 }), "rpoB 761155");
+ });
+
+ it("scales the SNP-matrix cell opacity with allele frequency", () => {
+ assert.notEqual(fn.snpAfColor(0), fn.snpAfColor(1));
+ });
+});
+
+describe("ES5 shims", () => {
+ it("zeros builds a zero-filled array", () => {
+ assert.deepEqual(arr(fn.zeros(3)), [0, 0, 0]);
+ assert.deepEqual(arr(fn.zeros(0)), []);
+ });
+
+ it("assign copies only own properties", () => {
+ const target = fn.assign({ a: 1 }, { b: 2 });
+ assert.equal(target.a, 1);
+ assert.equal(target.b, 2);
+ assert.equal(target.toString, Object.prototype.toString, "inherited keys must not be copied");
+ });
+});
diff --git a/tests/js/stats.test.mjs b/tests/js/stats.test.mjs
new file mode 100644
index 0000000..937434d
--- /dev/null
+++ b/tests/js/stats.test.mjs
@@ -0,0 +1,229 @@
+// The report computes its statistics in hand-written ES5 (no library is available inside a
+// self-contained HTML file). These tests check that arithmetic against reference values from
+// SciPy 1.15.2 and statsmodels, so a regression is caught against an independent implementation
+// rather than against a snapshot of our own output.
+//
+// Reference values were produced with:
+// scipy.stats.spearmanr, scipy.stats.kruskal, scipy.stats.chi2.sf, scipy.stats.t.sf,
+// statsmodels.stats.multitest.multipletests(method="fdr_bh")
+
+import { strict as assert } from "node:assert";
+import { before, describe, it } from "node:test";
+import { loadFunctions } from "./harness.mjs";
+
+const TOL = 1e-9;
+
+// Arrays built inside the vm context carry that context's Array.prototype, so deepStrictEqual
+// rejects them as not reference-equal. Copy into a host array before structural comparison.
+const arr = (x) => Array.from(x);
+
+function closeTo(actual, expected, tol = TOL, what = "") {
+ assert.equal(typeof actual, "number", `${what}: expected a number, got ${actual}`);
+ const diff = Math.abs(actual - expected);
+ const scale = Math.max(1, Math.abs(expected));
+ assert.ok(
+ diff / scale <= tol,
+ `${what}: ${actual} is not within ${tol} (relative) of ${expected}`,
+ );
+}
+
+let fn;
+before(() => {
+ fn = loadFunctions([
+ "rankvec", "spearman", "spearmanP", "studentP", "betai",
+ "chiSqP", "kruskalWallis", "bhFDR", "pfmt",
+ ]);
+});
+
+describe("rankvec", () => {
+ it("ranks from 1 and preserves input order", () => {
+ assert.deepEqual(arr(fn.rankvec([3, 1, 2])), [3, 1, 2]);
+ assert.deepEqual(arr(fn.rankvec([10, 20, 30])), [1, 2, 3]);
+ });
+
+ it("averages ranks across ties, like scipy's default", () => {
+ assert.deepEqual(arr(fn.rankvec([5, 5, 5])), [2, 2, 2]);
+ assert.deepEqual(arr(fn.rankvec([1, 2, 2, 3])), [1, 2.5, 2.5, 4]);
+ });
+
+ it("returns an empty array for empty input", () => {
+ assert.deepEqual(arr(fn.rankvec([])), []);
+ });
+});
+
+describe("spearman", () => {
+ // rho values from scipy.stats.spearmanr
+ const cases = [
+ ["perfect positive", [1, 2, 3, 4], [1, 2, 3, 4], 1.0],
+ ["perfect negative", [1, 2, 3, 4], [4, 3, 2, 1], -1.0],
+ ["noisy", [1, 2, 3, 4, 5, 6, 7, 8], [2, 1, 4, 3, 6, 5, 8, 7], 0.9047619047619048],
+ ["with ties", [1, 2, 2, 3, 4, 5], [2, 3, 3, 1, 5, 4], 0.5882352941176471],
+ ];
+
+ for (const [name, x, y, expected] of cases) {
+ it(`matches scipy for ${name}`, () => {
+ closeTo(fn.spearman(x, y), expected, TOL, name);
+ });
+ }
+
+ it("returns null below four points", () => {
+ assert.equal(fn.spearman([1, 2, 3], [1, 2, 3]), null);
+ assert.equal(fn.spearman([], []), null);
+ });
+
+ it("returns null when a vector has no variance", () => {
+ assert.equal(fn.spearman([1, 1, 1, 1], [1, 2, 3, 4]), null);
+ assert.equal(fn.spearman([1, 2, 3, 4], [7, 7, 7, 7]), null);
+ });
+});
+
+describe("spearmanP", () => {
+ // scipy.stats.spearmanr p-values (scipy uses the same Student-t approximation)
+ it("matches scipy for a noisy correlation", () => {
+ const rho = fn.spearman([1, 2, 3, 4, 5, 6, 7, 8], [2, 1, 4, 3, 6, 5, 8, 7]);
+ closeTo(fn.spearmanP(rho, 8), 0.0020082755054294677, 1e-9, "noisy p");
+ });
+
+ it("matches scipy in the presence of ties", () => {
+ const rho = fn.spearman([1, 2, 2, 3, 4, 5], [2, 3, 3, 1, 5, 4]);
+ closeTo(fn.spearmanP(rho, 6), 0.21941787095461004, 1e-9, "tied p");
+ });
+
+ it("returns exactly 0 for a perfect correlation, avoiding a division by zero", () => {
+ assert.equal(fn.spearmanP(1, 10), 0);
+ assert.equal(fn.spearmanP(-1, 10), 0);
+ });
+
+ it("returns null when rho is null or there are too few points", () => {
+ assert.equal(fn.spearmanP(null, 10), null);
+ assert.equal(fn.spearmanP(0.5, 2), null);
+ });
+});
+
+describe("kruskalWallis", () => {
+ // H and p from scipy.stats.kruskal, which applies the same tie correction
+ const cases = [
+ ["two groups", [[1, 2], [3, 4]], 2.3999999999999986, 0.12133525035848222],
+ ["three groups", [[1, 2, 3], [4, 5, 6], [7, 8, 9]], 7.200000000000003, 0.02732372244729252],
+ ["tied values", [[1, 1, 2], [2, 3, 3], [3, 4, 4]], 6.4424778761061905, 0.03990558707075906],
+ [
+ "uneven group sizes",
+ [[2.1, 3.4, 1.9, 4.0], [5.5, 6.1, 4.9], [9.0, 8.2, 7.7, 8.8, 9.4]],
+ 9.692307692307693,
+ 0.0078585446701517,
+ ],
+ ];
+
+ for (const [name, groups, h, p] of cases) {
+ it(`matches scipy for ${name}`, () => {
+ const got = fn.kruskalWallis(groups);
+ closeTo(got.H, h, TOL, `${name} H`);
+ closeTo(got.p, p, 1e-8, `${name} p`);
+ assert.equal(got.k, groups.length);
+ assert.equal(got.df, groups.length - 1);
+ });
+ }
+
+ it("gives H = 0 and p = 1 when every value is identical", () => {
+ const got = fn.kruskalWallis([[1, 1], [1, 1]]);
+ assert.equal(got.H, 0);
+ assert.equal(got.p, 1);
+ });
+
+ it("returns null with fewer than two groups or fewer than three values", () => {
+ assert.equal(fn.kruskalWallis([[1, 2, 3]]), null);
+ assert.equal(fn.kruskalWallis([[1], [2]]), null);
+ assert.equal(fn.kruskalWallis([]), null);
+ });
+});
+
+describe("chiSqP", () => {
+ // scipy.stats.chi2.sf
+ const cases = [
+ [3.84, 1, 0.050043521248705085],
+ [2.4, 1, 0.1213352503584821],
+ [5.99, 2, 0.05003662708658629],
+ [100, 2, 1.9287498479639183e-22],
+ [12.5, 7, 0.0852692751582693],
+ ];
+
+ for (const [x, df, expected] of cases) {
+ it(`matches scipy at x=${x}, df=${df}`, () => {
+ closeTo(fn.chiSqP(x, df), expected, 1e-8, `chi2(${x},${df})`);
+ });
+ }
+
+ it("returns 1 for degenerate input instead of NaN", () => {
+ assert.equal(fn.chiSqP(0, 1), 1);
+ assert.equal(fn.chiSqP(-1, 3), 1);
+ assert.equal(fn.chiSqP(NaN, 3), 1);
+ assert.equal(fn.chiSqP(5, 0), 1);
+ });
+});
+
+describe("studentP", () => {
+ // 2 * scipy.stats.t.sf(|t|, df)
+ it("matches scipy", () => {
+ closeTo(fn.studentP(2.5, 10), 0.03144684423660878, 1e-8, "t=2.5 df=10");
+ closeTo(fn.studentP(0, 5), 1.0, 1e-12, "t=0");
+ closeTo(fn.studentP(1.0, 1), 0.49999999999999956, 1e-8, "t=1 df=1");
+ });
+
+ it("returns 1 for a degenerate degrees of freedom", () => {
+ assert.equal(fn.studentP(2.5, 0), 1);
+ });
+});
+
+describe("betai", () => {
+ it("is pinned at the boundaries", () => {
+ assert.equal(fn.betai(1, 1, 0), 0);
+ assert.equal(fn.betai(1, 1, 1), 1);
+ });
+
+ it("matches the regularized incomplete beta at an interior point", () => {
+ // scipy.special.betainc(2, 3, 0.5)
+ closeTo(fn.betai(2, 3, 0.5), 0.6875, 1e-12, "betai(2,3,0.5)");
+ });
+});
+
+describe("bhFDR", () => {
+ // statsmodels.stats.multitest.multipletests(..., method="fdr_bh")
+ it("matches statsmodels", () => {
+ const q = fn.bhFDR([0.01, 0.02, 0.5]);
+ [0.03, 0.03, 0.5].forEach((e, i) => closeTo(q[i], e, 1e-12, `q[${i}]`));
+ });
+
+ it("returns q values in INPUT order, not sorted order", () => {
+ const q = fn.bhFDR([0.5, 0.01, 0.02]);
+ [0.5, 0.03, 0.03].forEach((e, i) => closeTo(q[i], e, 1e-12, `q[${i}]`));
+ });
+
+ it("matches statsmodels on a ten-hypothesis set", () => {
+ const q = fn.bhFDR([0.001, 0.008, 0.039, 0.041, 0.042, 0.06, 0.074, 0.205, 0.212, 0.216]);
+ const expected = [0.01, 0.04, 0.084, 0.084, 0.084, 0.1, 0.10571428571428572, 0.216, 0.216, 0.216];
+ expected.forEach((e, i) => closeTo(q[i], e, 1e-12, `q[${i}]`));
+ });
+
+ it("stays monotone and never exceeds 1", () => {
+ const ps = [0.9, 0.95, 0.99];
+ const q = fn.bhFDR(ps);
+ assert.ok(q.every((v) => v <= 1), "a q value exceeded 1");
+ const sorted = ps.map((p, i) => [p, q[i]]).sort((a, b) => a[0] - b[0]).map((r) => r[1]);
+ for (let i = 1; i < sorted.length; i++) {
+ assert.ok(sorted[i] >= sorted[i - 1] - 1e-12, "q values are not monotone in p");
+ }
+ });
+
+ it("returns an empty array for empty input", () => {
+ assert.deepEqual(arr(fn.bhFDR([])), []);
+ });
+});
+
+describe("pfmt", () => {
+ it("formats p-values for display", () => {
+ assert.equal(fn.pfmt(null), "n/a");
+ assert.equal(fn.pfmt(1e-9), "< 1e-4");
+ assert.equal(fn.pfmt(5e-4), "5.0e-4");
+ assert.equal(fn.pfmt(0.2), "0.200");
+ });
+});
diff --git a/tests/pipeline/test_params.py b/tests/pipeline/test_params.py
new file mode 100644
index 0000000..4fcc09b
--- /dev/null
+++ b/tests/pipeline/test_params.py
@@ -0,0 +1,164 @@
+"""Parameter declaration, validation and --help.
+
+Nextflow accepts any `--flag` into `params` whether or not the pipeline declares it, so without
+validation a typo runs to completion with the default silently in place. main.nf reads the declared
+set straight out of nextflow.config, which is also what these tests check against.
+"""
+
+from __future__ import annotations
+
+import re
+
+import pytest
+
+from conftest import DATA, REPO_ROOT, run_pipeline
+
+pytestmark = pytest.mark.nextflow
+
+CONFIG = REPO_ROOT / "nextflow.config"
+
+
+def declared_params() -> set[str]:
+ """The names inside the top-level `params { ... }` block of nextflow.config."""
+ text = CONFIG.read_text()
+ start = text.index("params {")
+ depth, end = 0, None
+ for i in range(text.index("{", start), len(text)):
+ if text[i] == "{":
+ depth += 1
+ elif text[i] == "}":
+ depth -= 1
+ if depth == 0:
+ end = i
+ break
+ assert end is not None, "unterminated params block in nextflow.config"
+ body = text[start:end]
+ return set(re.findall(r"^\s*([A-Za-z_][A-Za-z0-9_]*)\s*=", body, re.MULTILINE))
+
+
+def test_every_param_referenced_in_the_pipeline_is_declared():
+ """`params.foo` with no declaration is null at runtime, which usually reads as "feature off"."""
+ declared = declared_params()
+ # Set by Nextflow itself or by a profile rather than by the params block.
+ builtin = {"outdir"}
+
+ used = {}
+ sources = [
+ REPO_ROOT / "main.nf",
+ *sorted((REPO_ROOT / "subworkflows").glob("*.nf")),
+ *sorted((REPO_ROOT / "modules").glob("*.nf")),
+ ]
+ for source in sources:
+ text = source.read_text()
+ for match in re.finditer(r"params\.([A-Za-z_][A-Za-z0-9_]*)", text):
+ # Skip method calls on the params map itself, e.g. params.keySet(). Checked after the
+ # match rather than with a lookahead, which would just backtrack to a shorter name.
+ if text[match.end():].lstrip().startswith("("):
+ continue
+ used.setdefault(match.group(1), source.name)
+
+ undeclared = {name: where for name, where in used.items()
+ if name not in declared and name not in builtin}
+ assert undeclared == {}, f"referenced but never declared in nextflow.config: {undeclared}"
+
+
+def test_help_lists_the_parameters_and_exits_cleanly(nextflow, nextflow_env, tmp_path):
+ result = run_pipeline(nextflow, nextflow_env, tmp_path, "--help", timeout=300)
+ output = result.stdout + result.stderr
+
+ assert result.returncode == 0, output
+ # A representative parameter from three different sections.
+ for expected in ("--tsv", "--kraken2_db", "--freebayes_ploidy"):
+ assert expected in output, f"--help does not mention {expected}:\n{output}"
+ assert "Profiles:" in output
+
+
+def test_help_stays_in_step_with_the_config(nextflow, nextflow_env, tmp_path):
+ """--help is generated from nextflow.config, so a new parameter appears without being listed twice."""
+ result = run_pipeline(nextflow, nextflow_env, tmp_path, "--help", timeout=300)
+ listed = set(re.findall(r"^\s*--(\w+)", result.stdout + result.stderr, re.MULTILINE))
+ missing = declared_params() - listed
+ assert missing == set(), f"declared but absent from --help: {sorted(missing)}"
+
+
+@pytest.mark.parametrize(
+ ("typo", "suggestion"),
+ [
+ ("treads", "--threads"),
+ ("outdirr", "--outdir"),
+ ("kraken_db", "--kraken2_db"),
+ ],
+)
+def test_a_misspelled_parameter_is_rejected_with_a_suggestion(
+ nextflow, nextflow_env, tmp_path, typo, suggestion
+):
+ result = run_pipeline(
+ nextflow, nextflow_env, tmp_path,
+ "-profile", "test", "-stub-run",
+ "--outdir", str(tmp_path / "out"),
+ f"--{typo}", "1",
+ timeout=300,
+ )
+ output = result.stdout + result.stderr
+ assert result.returncode != 0, f"--{typo} was accepted:\n{output}"
+ assert f"--{typo}" in output
+ assert suggestion in output, f"no suggestion for --{typo}:\n{output}"
+
+
+def test_a_declared_parameter_is_accepted(nextflow, nextflow_env, tmp_path):
+ """The validator must not be so eager that it rejects a real flag."""
+ result = run_pipeline(
+ nextflow, nextflow_env, tmp_path,
+ "-profile", "test", "-stub-run",
+ "--outdir", str(tmp_path / "out"),
+ "--mappability_dir", str(tmp_path / "mappability"),
+ "--threads", "2",
+ "--freebayes_ploidy", "1",
+ timeout=900,
+ )
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
+def test_a_missing_samplesheet_parameter_says_what_to_pass(nextflow, nextflow_env, tmp_path):
+ result = run_pipeline(nextflow, nextflow_env, tmp_path, "-stub-run", timeout=300)
+ output = result.stdout + result.stderr
+ assert result.returncode != 0
+ assert "--tsv is required" in output, output
+ assert "sampleId" in output, "the error should name the samplesheet columns"
+
+
+def test_the_startup_banner_reports_where_the_run_will_execute(nextflow, nextflow_env, tmp_path):
+ """Running on a cluster login node by accident is the easiest mistake to make."""
+ result = run_pipeline(
+ nextflow, nextflow_env, tmp_path,
+ "-profile", "test", "-stub-run",
+ "--outdir", str(tmp_path / "out"),
+ "--mappability_dir", str(tmp_path / "mappability"),
+ timeout=900,
+ )
+ output = result.stdout + result.stderr
+ assert result.returncode == 0, output
+ assert "Executor : local" in output, output
+ assert "Profile(s) : test" in output, output
+ assert str(DATA / "samplesheet.tsv") in output
+
+
+def test_resource_caps_let_an_oversized_request_run(nextflow, nextflow_env, tmp_path):
+ """Kraken2 asks for 80 GB, which no laptop has and the local executor will never schedule.
+
+ Only the success direction is asserted: whether an UNCAPPED run fails depends on how much
+ memory the host happens to have, so testing that would pass on a laptop and fail on a big
+ workstation. `-profile test_full` enables Kraken, and 1 GB is available anywhere.
+ """
+ result = run_pipeline(
+ nextflow, nextflow_env, tmp_path,
+ "-profile", "test_full", "-stub-run",
+ "--outdir", str(tmp_path / "out"),
+ "--mappability_dir", str(tmp_path / "mappability"),
+ "--max_cpus", "1",
+ "--max_memory", "1.GB",
+ timeout=900,
+ )
+ output = result.stdout + result.stderr
+ assert result.returncode == 0, output
+ assert "exceeds available" not in output, output
diff --git a/tests/pipeline/test_samplesheet_validation.py b/tests/pipeline/test_samplesheet_validation.py
new file mode 100644
index 0000000..2fe0dc8
--- /dev/null
+++ b/tests/pipeline/test_samplesheet_validation.py
@@ -0,0 +1,87 @@
+"""The samplesheet validator in main.nf.
+
+Validation runs at script level, before any process is scheduled, so these tests are fast:
+Nextflow fails during script evaluation and never touches an executor. Its defining feature is
+that it collects EVERY problem and reports them together, so the tests assert on the full list
+rather than on the first failure.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from conftest import DATA, run_pipeline
+
+pytestmark = pytest.mark.nextflow
+
+INVALID = DATA / "invalid"
+
+
+def validate(nextflow, nextflow_env, tmp_path, sheet):
+ """Run the pipeline far enough to trigger samplesheet validation, and return the result."""
+ return run_pipeline(
+ nextflow,
+ nextflow_env,
+ tmp_path,
+ "-profile", "test",
+ "-stub-run",
+ "--tsv", str(sheet),
+ "--outdir", str(tmp_path / "out"),
+ timeout=300,
+ )
+
+
+def test_the_fixture_samplesheet_is_accepted(nextflow, nextflow_env, tmp_path):
+ result = validate(nextflow, nextflow_env, tmp_path, DATA / "samplesheet.tsv")
+ assert result.returncode == 0, result.stdout + result.stderr
+
+
+@pytest.mark.parametrize(
+ ("sheet", "expected"),
+ [
+ ("missing_column.tsv", "missing the required column 'refGff'"),
+ ("no_rows.tsv", "has a header but no data rows"),
+ ("no_usable_rows.tsv", "produced no usable samples"),
+ ("empty_field.tsv", "empty required field(s): sampleId"),
+ ("missing_file.tsv", "file(s) not found"),
+ ("ref_conflict.tsv", "points to a different refFasta"),
+ ],
+)
+def test_each_kind_of_broken_samplesheet_is_rejected(
+ nextflow, nextflow_env, tmp_path, sheet, expected
+):
+ result = validate(nextflow, nextflow_env, tmp_path, INVALID / sheet)
+ output = result.stdout + result.stderr
+ assert result.returncode != 0, f"{sheet} was accepted:\n{output}"
+ assert expected in output, f"{sheet} did not explain the problem:\n{output}"
+
+
+def test_every_problem_is_reported_in_one_pass(nextflow, nextflow_env, tmp_path):
+ """The whole point of the validator: fix everything in one edit, not one error per rerun."""
+ result = validate(nextflow, nextflow_env, tmp_path, INVALID / "multi_error.tsv")
+ output = result.stdout + result.stderr
+
+ assert result.returncode != 0
+ assert "has 3 problem(s)" in output, f"expected all three problems at once:\n{output}"
+ for expected in ("empty required field(s)", "file(s) not found", "points to a different refFasta"):
+ assert expected in output, f"missing '{expected}' from the report:\n{output}"
+
+
+def test_a_missing_samplesheet_names_the_path(nextflow, nextflow_env, tmp_path):
+ result = validate(nextflow, nextflow_env, tmp_path, tmp_path / "definitely_absent.tsv")
+ output = result.stdout + result.stderr
+ assert result.returncode != 0
+ assert "Samplesheet not found" in output, output
+
+
+def test_row_numbers_refer_to_lines_in_the_file(nextflow, nextflow_env, tmp_path):
+ """A row number that does not match the user's editor makes the message useless."""
+ sheet = tmp_path / "second_row_bad.tsv"
+ good = (DATA / "samplesheet.tsv").read_text().splitlines()
+ sheet.write_text("\n".join([good[0], good[1], "\tRUN9\t\t\t\t\t\t"]) + "\n")
+
+ result = validate(nextflow, nextflow_env, tmp_path, sheet)
+ output = result.stdout + result.stderr
+ assert result.returncode != 0
+ # Header is line 1 and the first data row line 2, so the broken third line is row 3.
+ assert "row 3:" in output, output
diff --git a/tests/pipeline/test_stub_run.py b/tests/pipeline/test_stub_run.py
new file mode 100644
index 0000000..31bf8ac
--- /dev/null
+++ b/tests/pipeline/test_stub_run.py
@@ -0,0 +1,140 @@
+"""End-to-end DAG tests via `-stub-run`.
+
+A stub run executes each process's `stub:` block instead of the real tool, so the entire workflow
+is exercised in seconds with no container and no reference data. What it proves is the wiring:
+that every channel joins, every declared output is produced and collected, and no process is left
+starved or duplicated. It proves nothing about the science, which is what the unit tests are for.
+
+`-profile test` covers the DEFAULT feature set. `-profile test_full` turns on every optional
+branch so a single run instantiates every process in the pipeline.
+"""
+
+from __future__ import annotations
+
+import csv
+import re
+
+import pytest
+
+from conftest import REPO_ROOT, run_pipeline
+
+pytestmark = pytest.mark.nextflow
+
+# Declared in modules/*.nf. CALL_BACKBONE, MERGE_VCFS and CONSENSUS_FASTA are also instantiated
+# a second time under a _RAW alias for the virgin-consensus path.
+ALIASES = {"CALL_BACKBONE_RAW", "MERGE_VCFS_RAW", "CONSENSUS_FASTA_RAW"}
+
+
+def declared_processes():
+ names = set()
+ for module in sorted((REPO_ROOT / "modules").glob("*.nf")):
+ names.update(re.findall(r"^process\s+(\w+)", module.read_text(), re.MULTILINE))
+ return names
+
+
+def stub_run(nextflow, nextflow_env, tmp_path, profile):
+ trace = tmp_path / "trace.txt"
+ result = run_pipeline(
+ nextflow,
+ nextflow_env,
+ tmp_path,
+ "-profile", profile,
+ "-stub-run",
+ "--outdir", str(tmp_path / "out"),
+ # BUILD_MAPPABILITY uses storeDir, so a cache shared between tests makes every run after
+ # the first skip it. Give each test its own so the coverage assertions stay honest.
+ "--mappability_dir", str(tmp_path / "mappability"),
+ "-with-trace", str(trace),
+ timeout=900,
+ )
+ return result, trace
+
+
+def executed_processes(trace):
+ with trace.open() as fh:
+ rows = list(csv.DictReader(fh, delimiter="\t"))
+ # trace "name" is like "MAPPING_PE (SAMPLE_A__RUN1)"; "process" is the fully qualified name.
+ return {row["process"].split(":")[-1].strip() for row in rows}
+
+
+def test_there_is_a_stub_block_for_every_process():
+ """A process without a stub aborts the whole stub run, so this is checked statically too."""
+ missing = []
+ for module in sorted((REPO_ROOT / "modules").glob("*.nf")):
+ text = module.read_text()
+ for block in re.split(r"^process\s+", text, flags=re.MULTILINE)[1:]:
+ name = block.split(None, 1)[0]
+ if "stub:" not in block:
+ missing.append(f"{module.name}:{name}")
+ assert missing == [], f"processes with no stub: block: {missing}"
+
+
+@pytest.mark.parametrize("profile", ["test", "test_full"])
+def test_the_dag_runs_end_to_end(nextflow, nextflow_env, tmp_path, profile):
+ result, trace = stub_run(nextflow, nextflow_env, tmp_path, profile)
+ output = result.stdout + result.stderr
+ assert result.returncode == 0, f"-profile {profile} failed:\n{output}"
+ assert trace.exists(), "no trace file was written"
+
+ with trace.open() as fh:
+ rows = list(csv.DictReader(fh, delimiter="\t"))
+ assert rows, "the run scheduled no tasks at all"
+ failed = [r["name"] for r in rows if r["status"] != "COMPLETED"]
+ assert failed == [], f"tasks did not complete: {failed}"
+
+
+def test_the_full_profile_reaches_every_process(nextflow, nextflow_env, tmp_path):
+ """Guards against an optional branch silently dropping out of the coverage of the stub run."""
+ result, trace = stub_run(nextflow, nextflow_env, tmp_path, "test_full")
+ assert result.returncode == 0, result.stdout + result.stderr
+
+ ran = executed_processes(trace)
+ never_ran = declared_processes() - ran
+ assert never_ran == set(), f"-profile test_full never instantiates: {sorted(never_ran)}"
+ assert ALIASES <= ran, f"the virgin-consensus aliases did not run: {sorted(ALIASES - ran)}"
+
+
+def test_the_default_profile_produces_the_report(nextflow, nextflow_env, tmp_path):
+ """The QC report is the pipeline's headline output, and it is on by default.
+
+ This is a regression test: with pathotypr, canonical annotation and the liftover all off,
+ which is the default, QC_REPORT used to receive three inputs sharing the placeholder name
+ NO_FILE and Nextflow refused to stage them (`input file name collision`).
+ """
+ result, trace = stub_run(nextflow, nextflow_env, tmp_path, "test")
+ assert result.returncode == 0, result.stdout + result.stderr
+ assert "QC_REPORT" in executed_processes(trace), "QC_REPORT never ran on the default profile"
+
+ outdir = tmp_path / "out"
+ reports = list(outdir.glob("*_qc_report.html"))
+ flags = list(outdir.glob("*_qc_flags.tsv"))
+ assert reports, f"no QC report was published into {outdir}"
+ assert flags, f"no qc_flags.tsv was published into {outdir}"
+
+
+def test_multi_run_samples_are_merged_once(nextflow, nextflow_env, tmp_path):
+ """SAMPLE_A has two runs, so it must map twice but merge and call variants only once."""
+ result, trace = stub_run(nextflow, nextflow_env, tmp_path, "test")
+ assert result.returncode == 0, result.stdout + result.stderr
+
+ with trace.open() as fh:
+ rows = list(csv.DictReader(fh, delimiter="\t"))
+
+ def tasks_for(process):
+ return [r["name"] for r in rows if r["process"].split(":")[-1].strip() == process]
+
+ mapping = [t for t in tasks_for("MAPPING_PE") if "SAMPLE_A" in t]
+ merges = [t for t in tasks_for("MERGE_AND_MARKDUP") if "SAMPLE_A" in t]
+ assert len(mapping) == 2, f"expected SAMPLE_A to map once per run, got {mapping}"
+ assert len(merges) == 1, f"expected SAMPLE_A to merge exactly once, got {merges}"
+
+
+def test_single_end_and_paired_end_samples_both_flow_through(nextflow, nextflow_env, tmp_path):
+ """SAMPLE_C is single-end; the SE branch is easy to break without noticing."""
+ result, trace = stub_run(nextflow, nextflow_env, tmp_path, "test")
+ assert result.returncode == 0, result.stdout + result.stderr
+
+ ran = executed_processes(trace)
+ assert "MAPPING_SE" in ran, "the single-end mapping branch never ran"
+ assert "MAPPING_PE" in ran, "the paired-end mapping branch never ran"
+ assert "FASTP_SE" in ran and "FASTP_PE" in ran
diff --git a/tests/run_tests.sh b/tests/run_tests.sh
new file mode 100755
index 0000000..90cdf53
--- /dev/null
+++ b/tests/run_tests.sh
@@ -0,0 +1,67 @@
+#!/usr/bin/env bash
+#
+# Run the whole BAMpiro test suite locally, the same way CI does.
+#
+# tests/run_tests.sh # everything
+# tests/run_tests.sh unit # only the Python unit tests
+# tests/run_tests.sh js # only the report front-end tests
+# tests/run_tests.sh pipeline # only the Nextflow stub runs (needs nextflow + Java 17+)
+# tests/run_tests.sh lint # only ruff
+#
+# The Nextflow leg is skipped automatically when the CLI is not installed.
+
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT"
+
+TARGET="${1:-all}"
+FAILED=()
+
+step() { printf '\n\033[1m== %s\033[0m\n' "$1"; }
+record() { if "$@"; then :; else FAILED+=("$1"); fi; }
+
+# Nextflow writes a storeDir cache and a results tree; start from a clean slate so the stub
+# runs are hermetic and BUILD_MAPPABILITY is actually exercised rather than restored.
+rm -rf .test_cache results_test
+
+if [[ "$TARGET" == "all" || "$TARGET" == "lint" ]]; then
+ step "ruff"
+ if command -v ruff >/dev/null 2>&1; then
+ record ruff check .
+ else
+ echo "ruff not installed, skipping (pip install ruff)"
+ fi
+fi
+
+if [[ "$TARGET" == "all" || "$TARGET" == "unit" ]]; then
+ step "pytest: unit tests for bin/"
+ record python3 -m pytest tests/unit -q
+fi
+
+if [[ "$TARGET" == "all" || "$TARGET" == "js" ]]; then
+ step "node: report front-end"
+ if command -v node >/dev/null 2>&1; then
+ # Pass the files explicitly: `node --test ` is not supported on every Node version.
+ record node --test tests/js/*.test.mjs
+ else
+ echo "node not installed, skipping"
+ fi
+fi
+
+if [[ "$TARGET" == "all" || "$TARGET" == "pipeline" ]]; then
+ step "pytest: Nextflow stub runs"
+ if command -v nextflow >/dev/null 2>&1; then
+ record python3 -m pytest tests/pipeline -q
+ else
+ echo "nextflow not installed, skipping (needs Java 17+)"
+ fi
+fi
+
+echo
+if [[ ${#FAILED[@]} -eq 0 ]]; then
+ printf '\033[32mAll checks passed.\033[0m\n'
+else
+ printf '\033[31mFailed: %s\033[0m\n' "${FAILED[*]}"
+ exit 1
+fi
diff --git a/tests/unit/test_backbone_allpos.py b/tests/unit/test_backbone_allpos.py
new file mode 100644
index 0000000..cd6aad0
--- /dev/null
+++ b/tests/unit/test_backbone_allpos.py
@@ -0,0 +1,187 @@
+"""bin/backbone_allpos.awk - one VCF record per reference position.
+
+The backbone is what makes an all-sites VCF, and therefore a phylogenetic supermatrix, possible.
+Its `count_bases` reads an mpileup base string, which is not one character per read: a read start
+is '^' plus a mapping-quality character that must not be counted, and an indel is a sign, a length,
+and that many bases belonging to the previous position. Miscount any of that and the allelic depth
+the consensus relies on is quietly wrong.
+
+Both branches of CALL_BACKBONE carried their own copy of this until it was pulled out, and neither
+had any coverage: a stub run skips the script block.
+"""
+
+from __future__ import annotations
+
+import subprocess
+
+import pytest
+
+from conftest import BIN
+
+AWK = BIN / "backbone_allpos.awk"
+
+
+def run(rows, mincov=30, gapdp=7, baq=0, tmp_path=None):
+ """Feed mpileup-shaped rows through the awk program and return the VCF fields per line."""
+ src = tmp_path / "pileup.txt"
+ src.write_text("".join(rows))
+ cmd = ["awk", "-v", f"MINCOV={mincov}", "-v", f"BAQ={baq}", "-v", "OFS=\t"]
+ if baq:
+ cmd += ["-v", f"GAPDP={gapdp}"]
+ cmd += ["-f", str(AWK), str(src)]
+ out = subprocess.run(cmd, capture_output=True, text=True, check=True)
+ return [line.split("\t") for line in out.stdout.splitlines()]
+
+
+def pileup(bases, depth=None, pos=100, ref="A", baq_depth=None):
+ """One mpileup row. Column 10 carries the BAQ-adjusted depth when the two pileups are pasted."""
+ dp = len(bases) if depth is None else depth
+ row = f"c\t{pos}\t{ref}\t{dp}\t{bases}\tIIII"
+ if baq_depth is not None:
+ row += f"\tc\t{pos}\t{ref}\t{baq_depth}\t{bases}\tIIII"
+ return row + "\n"
+
+
+def ad_of(fields):
+ """The ref,alt observation counts out of the GT:DP:AD sample column."""
+ return fields[9].split(":")[2]
+
+
+def info_of(fields):
+ return dict(kv.split("=", 1) for kv in fields[7].split(";"))
+
+
+class TestBaseCounting:
+ def test_matches_to_the_reference_count_as_reference_observations(self, tmp_path):
+ [f] = run([pileup("....,,,,")], tmp_path=tmp_path)
+ assert ad_of(f) == "8,0"
+
+ def test_mismatches_count_as_alt_observations_in_either_case(self, tmp_path):
+ [f] = run([pileup("..AAtt")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,4"
+
+ def test_a_read_start_consumes_its_mapping_quality_character(self, tmp_path):
+ """'^I.' is ONE reference observation: the 'I' is a quality score, not a base."""
+ [f] = run([pileup("^I.^I.^I.")], tmp_path=tmp_path)
+ assert ad_of(f) == "3,0", "the mapping-quality character was counted as a base"
+
+ def test_a_read_start_whose_quality_char_looks_like_a_base_is_not_counted(self, tmp_path):
+ """The quality character can legitimately be 'A' or 'T'; it must still be skipped."""
+ [f] = run([pileup("^A.^T.")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0", "a quality character was mistaken for an alt observation"
+
+ def test_a_read_end_marker_is_not_a_base(self, tmp_path):
+ [f] = run([pileup("..$..$")], tmp_path=tmp_path)
+ assert ad_of(f) == "4,0"
+
+ def test_an_insertion_and_its_bases_are_skipped(self, tmp_path):
+ """'+2AG' belongs to the previous position, so neither the length nor the bases count here."""
+ [f] = run([pileup(".+2AG.")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0", "inserted bases were counted at this position"
+
+ def test_a_multi_digit_indel_length_is_parsed_whole(self, tmp_path):
+ """A 12-base insertion must consume 12 bases, not 1 then treat '2' as a length."""
+ [f] = run([pileup(".+12ACGTACGTACGT.")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0"
+
+ def test_a_deletion_and_its_bases_are_skipped(self, tmp_path):
+ [f] = run([pileup(".-3ACG.")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0"
+
+ def test_a_spanning_deletion_placeholder_is_ignored(self, tmp_path):
+ """'*' marks a position inside a deletion: neither reference nor alt evidence."""
+ [f] = run([pileup("..**")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0"
+
+ def test_an_n_base_is_ignored(self, tmp_path):
+ [f] = run([pileup("..NN")], tmp_path=tmp_path)
+ assert ad_of(f) == "2,0"
+
+ def test_a_position_with_no_reads_counts_nothing(self, tmp_path):
+ [f] = run([pileup("*", depth=0)], tmp_path=tmp_path)
+ assert ad_of(f) == "0,0"
+
+
+class TestRecordShape:
+ def test_the_record_is_a_non_variant_site(self, tmp_path):
+ """The backbone asserts coverage, it does not call variants: ALT stays '.'."""
+ [f] = run([pileup("." * 40, pos=4242, ref="G")], tmp_path=tmp_path)
+ assert f[0] == "c" and f[1] == "4242" and f[3] == "G"
+ assert f[4] == "." and f[5] == "."
+ assert f[8] == "GT:DP:AD"
+
+ def test_depth_comes_from_the_pileup_column_not_the_base_string(self, tmp_path):
+ [f] = run([pileup("....", depth=99)], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "99"
+ assert f[9].split(":")[1] == "99"
+
+
+class TestCoverageCall:
+ def test_a_well_covered_position_is_a_confident_reference_call(self, tmp_path):
+ [f] = run([pileup("." * 40, depth=40)], mincov=30, tmp_path=tmp_path)
+ assert f[9].startswith("0:")
+ assert info_of(f)["WT"] == "1" and info_of(f)["NC"] == "0"
+
+ def test_a_shallow_position_is_a_no_call(self, tmp_path):
+ [f] = run([pileup("." * 5, depth=5)], mincov=30, tmp_path=tmp_path)
+ assert f[9].startswith("./.:")
+ assert info_of(f)["WT"] == "0" and info_of(f)["NC"] == "1"
+
+ def test_the_threshold_is_inclusive(self, tmp_path):
+ at = run([pileup(".", depth=30)], mincov=30, tmp_path=tmp_path)[0]
+ below = run([pileup(".", depth=29)], mincov=30, tmp_path=tmp_path)[0]
+ assert at[9].startswith("0:")
+ assert below[9].startswith("./.:")
+
+ def test_the_backbone_never_calls_a_heterozygote(self, tmp_path):
+ [f] = run([pileup("..AATT", depth=40)], tmp_path=tmp_path)
+ assert info_of(f)["HET"] == "0" and info_of(f)["HOM"] == "0"
+
+
+class TestBaqDropout:
+ """A well-covered site whose depth collapses only under BAQ is an indel-adjacent artefact.
+
+ Flagging it lets the consensus mask it with X rather than leave a gap, which is the difference
+ between "we could not read here" and "there is nothing here".
+ """
+
+ def test_a_collapsed_site_is_flagged(self, tmp_path):
+ [f] = run([pileup("." * 40, depth=40, baq_depth=3)], gapdp=7, baq=1, tmp_path=tmp_path)
+ assert f[6] == "baq_dropout"
+
+ def test_a_site_that_holds_up_under_baq_is_not_flagged(self, tmp_path):
+ [f] = run([pileup("." * 40, depth=40, baq_depth=38)], gapdp=7, baq=1, tmp_path=tmp_path)
+ assert f[6] == "."
+
+ def test_a_genuinely_shallow_site_is_not_flagged(self, tmp_path):
+ """Depth at or below the gap threshold to begin with is a gap, not a BAQ artefact."""
+ [f] = run([pileup("..", depth=2, baq_depth=1)], gapdp=7, baq=1, tmp_path=tmp_path)
+ assert f[6] == "."
+
+ def test_without_the_baq_column_nothing_is_ever_flagged(self, tmp_path):
+ """The single-pileup branch has nothing to compare against, so the filter stays empty."""
+ [f] = run([pileup("." * 40, depth=40)], baq=0, tmp_path=tmp_path)
+ assert f[6] == "."
+
+ def test_the_depth_reported_is_the_real_one_not_the_baq_one(self, tmp_path):
+ """ADP must stay the no-BAQ depth; BAQ is only used to decide the flag."""
+ [f] = run([pileup("." * 40, depth=40, baq_depth=3)], gapdp=7, baq=1, tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "40"
+
+
+def test_both_backbone_branches_share_this_one_program():
+ """The BAQ and non-BAQ branches used to carry separate copies of count_bases."""
+ variants = (BIN.parent / "modules" / "variants.nf").read_text()
+ assert variants.count("backbone_allpos.awk") == 2
+ assert "count_bases" not in variants, "the pileup parser is back inline in the module"
+
+
+@pytest.mark.parametrize("bases,expected", [
+ ("^].^].,,", "4,0"), # read starts with ']' as the quality char
+ (".$.+1A.", "3,0"), # read end, then an insertion, then a match
+ ("A^AA", "0,2"), # alt, then a read start whose quality is 'A', then alt
+ ("", "0,0"), # empty base string
+])
+def test_mixed_pileup_strings(bases, expected, tmp_path):
+ [f] = run([pileup(bases, depth=10)], tmp_path=tmp_path)
+ assert ad_of(f) == expected
diff --git a/tests/unit/test_build_min_unique_len.py b/tests/unit/test_build_min_unique_len.py
new file mode 100644
index 0000000..6698e0e
--- /dev/null
+++ b/tests/unit/test_build_min_unique_len.py
@@ -0,0 +1,409 @@
+"""Unit tests for bin/build_min_unique_len.py (genmap k-sweep -> min_unique_len tracks).
+
+The whole script is one `main()` with no importable helpers, so every test drives it as
+a subprocess: a small `.fa.fai`, one bedgraph per probed read length K, and assertions
+on the resulting `.npz` (and on the optional repeat BED).
+
+Conventions under test:
+ * bedgraph coordinates are 0-based half-open and carry through to the arrays;
+ * the array value is the SMALLEST probed K at which the k-mer starting there is
+ genome-unique, with the sentinel (default 65535) meaning "never unique";
+ * the repeat BED is 1-based inclusive, matching the nucmer / bcftools -T contract.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import numpy as np
+import pytest
+
+SENTINEL = 65535
+
+
+# --------------------------------------------------------------------------- helpers
+
+
+def _fai(tmp_path, contigs) -> str:
+ """A minimal samtools .fa.fai; only columns 1 (name) and 2 (length) are read."""
+ path = tmp_path / "ref.fa.fai"
+ path.write_text("".join("%s\t%d\t6\t60\t61\n" % (name, length) for name, length in contigs))
+ return str(path)
+
+
+def _bedgraph(tmp_path, name, rows) -> str:
+ path = tmp_path / name
+ path.write_text("".join("%s\t%d\t%d\t%s\n" % row for row in rows))
+ return str(path)
+
+
+def _build(repo_root, tmp_path, fai, bedgraphs, *extra):
+ """Run the script; return (CompletedProcess, {contig: array})."""
+ prefix = str(tmp_path / "out")
+ cmd = [sys.executable, str(repo_root / "bin" / "build_min_unique_len.py"),
+ "--fai", fai, "--bedgraphs", *bedgraphs, "--out-prefix", prefix, *extra]
+ res = subprocess.run(cmd, capture_output=True, text=True)
+ tracks = {}
+ if res.returncode == 0:
+ with np.load(prefix + ".min_unique_len.npz") as npz:
+ tracks = {key: npz[key] for key in npz.files}
+ return res, tracks
+
+
+# ---------------------------------------------------------------- the k-sweep collapse
+
+
+def test_smallest_unique_k_wins(repo_root, tmp_path):
+ """Bedgraphs are processed in ascending K and only positions still at the sentinel
+ are filled, so a position unique at both 20 and 50 keeps 20."""
+ fai = _fai(tmp_path, [("chr1", 10)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 5, "1"), ("chr1", 5, 10, "0")])
+ b50 = _bedgraph(tmp_path, "k50.bedgraph", [("chr1", 0, 10, "1")])
+
+ res, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20, "50:" + b50])
+
+ assert res.returncode == 0, res.stderr
+ assert list(tracks["chr1"]) == [20] * 5 + [50] * 5
+
+
+def test_command_line_order_of_the_bedgraphs_does_not_matter(repo_root, tmp_path):
+ """The tokens are sorted by K before use, so a caller may list them in any order."""
+ fai = _fai(tmp_path, [("chr1", 10)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 5, "1"), ("chr1", 5, 10, "0")])
+ b50 = _bedgraph(tmp_path, "k50.bedgraph", [("chr1", 0, 10, "1")])
+
+ _, ascending = _build(repo_root, tmp_path, fai, ["20:" + b20, "50:" + b50])
+ _, descending = _build(repo_root, tmp_path, fai, ["50:" + b50, "20:" + b20])
+
+ assert list(descending["chr1"]) == list(ascending["chr1"])
+
+
+def test_never_unique_positions_get_the_sentinel(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 6)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 3, "1"), ("chr1", 3, 6, "0")])
+ b50 = _bedgraph(tmp_path, "k50.bedgraph", [("chr1", 0, 6, "0")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20, "50:" + b50])
+
+ assert list(tracks["chr1"]) == [20, 20, 20, SENTINEL, SENTINEL, SENTINEL]
+
+
+def test_k_is_never_stored_as_the_sentinel(repo_root, tmp_path):
+ """`kv = min(K, sentinel - 1)`: a probed length at or above the sentinel is clamped,
+ so a real (unique) position can never be mistaken for "never unique"."""
+ fai = _fai(tmp_path, [("chr1", 4)])
+ b30 = _bedgraph(tmp_path, "k30.bedgraph", [("chr1", 0, 4, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["30:" + b30], "--sentinel", "30")
+
+ assert list(tracks["chr1"]) == [29, 29, 29, 29]
+
+
+def test_clamping_still_lets_a_smaller_k_win(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 6)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 2, "1"), ("chr1", 2, 6, "0")])
+ b50 = _bedgraph(tmp_path, "k50.bedgraph", [("chr1", 0, 6, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20, "50:" + b50], "--sentinel", "30")
+
+ assert list(tracks["chr1"]) == [20, 20, 29, 29, 29, 29]
+
+
+def test_arrays_are_uint16_and_sized_from_the_fai(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 10), ("chr2", 4)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 10, "1"), ("chr2", 0, 4, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20])
+
+ assert sorted(tracks) == ["chr1", "chr2"]
+ assert tracks["chr1"].dtype == np.uint16 and tracks["chr1"].shape == (10,)
+ assert tracks["chr2"].dtype == np.uint16 and tracks["chr2"].shape == (4,)
+
+
+# ------------------------------------------------------------------ uniqueness cutoff
+
+
+def test_uniqueness_uses_a_greater_or_equal_test(repo_root, tmp_path):
+ """`value >= --unique-eps` counts as unique, so the threshold itself passes."""
+ fai = _fai(tmp_path, [("chr1", 4)])
+ rows = [("chr1", 0, 1, "1"), ("chr1", 1, 2, "0.5"),
+ ("chr1", 2, 3, "0.4999"), ("chr1", 3, 4, "0")]
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", rows)
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20], "--unique-eps", "0.5")
+
+ assert list(tracks["chr1"]) == [20, 20, SENTINEL, SENTINEL]
+
+
+def test_default_epsilon_rejects_almost_unique_positions(repo_root, tmp_path):
+ """The default 0.999999 exists so that genmap's float 1.0 passes while a genuinely
+ multi-mapping 0.99 does not."""
+ fai = _fai(tmp_path, [("chr1", 4)])
+ rows = [("chr1", 0, 1, "1"), ("chr1", 1, 2, "0.999999"),
+ ("chr1", 2, 3, "0.9999989"), ("chr1", 3, 4, "0.99")]
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", rows)
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20])
+
+ assert list(tracks["chr1"]) == [20, 20, SENTINEL, SENTINEL]
+
+
+def test_comment_lines_in_a_bedgraph_are_skipped(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 4)])
+ path = tmp_path / "k20.bedgraph"
+ path.write_text("#track type=bedGraph\nchr1\t0\t4\t1\n")
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + str(path)])
+
+ assert list(tracks["chr1"]) == [20] * 4
+
+
+# ----------------------------------------------------------------------- tail policy
+
+
+def test_tail_policy_keep_zeroes_never_scored_contig_ends(repo_root, tmp_path):
+ """genmap cannot score the last K-1 starts of a contig. Those positions are not
+ repetitive, so the default policy gives them 0 (any span >= 0 keeps the read)."""
+ fai = _fai(tmp_path, [("chr1", 20)])
+ b10 = _bedgraph(tmp_path, "k10.bedgraph", [("chr1", 0, 11, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["10:" + b10])
+
+ assert list(tracks["chr1"]) == [10] * 11 + [0] * 9
+
+
+def test_tail_policy_drop_leaves_the_ends_at_the_sentinel(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 20)])
+ b10 = _bedgraph(tmp_path, "k10.bedgraph", [("chr1", 0, 11, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["10:" + b10], "--tail-policy", "drop")
+
+ assert list(tracks["chr1"]) == [10] * 11 + [SENTINEL] * 9
+
+
+def test_a_repetitive_tail_position_is_not_zeroed(repo_root, tmp_path):
+ """Only positions the smallest K never covered are treated as tail; a scored but
+ non-unique position keeps the sentinel under either policy."""
+ fai = _fai(tmp_path, [("chr1", 6)])
+ b10 = _bedgraph(tmp_path, "k10.bedgraph", [("chr1", 0, 4, "0")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["10:" + b10])
+
+ assert list(tracks["chr1"]) == [SENTINEL] * 4 + [0, 0]
+
+
+def test_coverage_comes_only_from_the_smallest_k(repo_root, tmp_path):
+ """Documented behaviour: `covered` is filled only while processing the smallest K, so a
+ region scored at a larger K but not at the smallest one still counts as an unscored tail
+ and becomes min_unique_len 0, which is fully permissive. Real genmap output is contiguous
+ per K, so the pipeline does not hit it, and deriving coverage from the union of all K
+ would change masking in a way no fixture can validate against real genmap output. It is
+ no longer silent: see test_an_unscored_hole_away_from_the_contig_ends_is_reported."""
+ fai = _fai(tmp_path, [("chr1", 10)])
+ b10 = _bedgraph(tmp_path, "k10.bedgraph", [("chr1", 0, 3, "1"), ("chr1", 7, 10, "1")])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 3, 7, "0")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["10:" + b10, "20:" + b20])
+
+ assert list(tracks["chr1"]) == [10, 10, 10, 0, 0, 0, 0, 10, 10, 10]
+
+
+# --------------------------------------------------------------- fai / bedgraph mismatch
+
+
+def test_a_contig_missing_from_the_fai_is_skipped_silently(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 4)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [
+ ("chr1", 0, 4, "1"),
+ ("plasmid", 0, 100, "1"), # absent from the .fai -> ignored, no warning
+ ])
+
+ res, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20])
+
+ assert res.returncode == 0
+ assert res.stderr == ""
+ assert sorted(tracks) == ["chr1"]
+ assert list(tracks["chr1"]) == [20] * 4
+
+
+def test_a_contig_with_no_bedgraph_rows_is_all_tail(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 4), ("chr2", 3)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 4, "1")])
+
+ _, tracks = _build(repo_root, tmp_path, fai, ["20:" + b20])
+
+ assert list(tracks["chr2"]) == [0, 0, 0] # keep policy
+ _, dropped = _build(repo_root, tmp_path, fai, ["20:" + b20], "--tail-policy", "drop")
+ assert list(dropped["chr2"]) == [SENTINEL] * 3
+
+
+def test_blank_lines_in_the_fai_are_ignored(repo_root, tmp_path):
+ path = tmp_path / "ref.fa.fai"
+ path.write_text("chr1\t4\t6\t60\t61\n\n")
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 4, "1")])
+
+ res, tracks = _build(repo_root, tmp_path, str(path), ["20:" + b20])
+
+ assert res.returncode == 0, res.stderr
+ assert sorted(tracks) == ["chr1"]
+
+
+@pytest.mark.parametrize("body", [
+ "chr1\t0\t4\t1\n\n", # trailing blank line
+ "\nchr1\t0\t4\t1\n", # leading blank line
+ "chr1\t0\t2\t1\n \nchr1\t2\t4\t1\n", # whitespace-only line in the middle
+])
+def test_blank_lines_in_a_bedgraph_are_skipped(repo_root, tmp_path, body):
+ """A blank line is "\\n" (never "") when iterating a file, so it has to be skipped on
+ its content, not on emptiness, or the four-column unpack raises ValueError."""
+ fai = _fai(tmp_path, [("chr1", 4)])
+ path = tmp_path / "k20.bedgraph"
+ path.write_text(body)
+
+ res, tracks = _build(repo_root, tmp_path, fai, ["20:" + str(path)])
+
+ assert res.returncode == 0, res.stderr
+ assert list(tracks["chr1"]) == [20] * 4
+
+
+# -------------------------------------------------------------------------- --mask-bed
+
+
+def _mask_rows(path):
+ lines = path.read_text().splitlines()
+ assert lines[0] == "Chrom\tStart\tEnd\tTag\tComment"
+ return lines[1:]
+
+
+def test_mask_bed_is_one_based_inclusive(repo_root, tmp_path):
+ """chr1 is unique at 3 bp over 0-based 0..2 and 15..19 and repetitive in between.
+ With a 5 bp window a base is masked when no start in the preceding 5 bp can anchor,
+ which is 0-based 7..14, written as 1-based inclusive 8..15."""
+ fai = _fai(tmp_path, [("chr1", 20)])
+ b3 = _bedgraph(tmp_path, "k3.bedgraph", [
+ ("chr1", 0, 3, "1"), ("chr1", 3, 15, "0"), ("chr1", 15, 20, "1")])
+ mask = tmp_path / "repeats.bed"
+
+ res, tracks = _build(repo_root, tmp_path, fai, ["3:" + b3],
+ "--mask-bed", str(mask), "--mask-window", "5")
+
+ assert res.returncode == 0, res.stderr
+ assert list(tracks["chr1"]) == [3, 3, 3] + [SENTINEL] * 12 + [3] * 5
+ assert _mask_rows(mask) == ["chr1\t8\t15\t\t"]
+
+
+def test_mask_bed_is_empty_when_nothing_is_repetitive(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 10)])
+ b3 = _bedgraph(tmp_path, "k3.bedgraph", [("chr1", 0, 10, "1")])
+ mask = tmp_path / "repeats.bed"
+
+ _build(repo_root, tmp_path, fai, ["3:" + b3], "--mask-bed", str(mask), "--mask-window", "5")
+
+ assert _mask_rows(mask) == [] # header written, no intervals
+
+
+def test_mask_bed_covers_a_fully_repetitive_contig(repo_root, tmp_path):
+ """The `-W` initial value (not -1) is what makes a repeat that starts at the contig
+ start mask from base 1."""
+ fai = _fai(tmp_path, [("chr1", 10)])
+ b3 = _bedgraph(tmp_path, "k3.bedgraph", [("chr1", 0, 10, "0")])
+ mask = tmp_path / "repeats.bed"
+
+ _build(repo_root, tmp_path, fai, ["3:" + b3], "--mask-bed", str(mask), "--mask-window", "5")
+
+ assert _mask_rows(mask) == ["chr1\t1\t10\t\t"]
+
+
+def test_mask_bed_lists_every_contig(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 10), ("chr2", 10)])
+ b3 = _bedgraph(tmp_path, "k3.bedgraph", [("chr1", 0, 10, "1"), ("chr2", 0, 10, "0")])
+ mask = tmp_path / "repeats.bed"
+
+ _build(repo_root, tmp_path, fai, ["3:" + b3], "--mask-bed", str(mask), "--mask-window", "5")
+
+ assert _mask_rows(mask) == ["chr2\t1\t10\t\t"]
+
+
+def test_mask_window_widens_the_masked_block(repo_root, tmp_path):
+ """A longer read length considered means more bases count as unmappable."""
+ fai = _fai(tmp_path, [("chr1", 20)])
+ b3 = _bedgraph(tmp_path, "k3.bedgraph", [
+ ("chr1", 0, 3, "1"), ("chr1", 3, 15, "0"), ("chr1", 15, 20, "1")])
+ narrow = tmp_path / "narrow.bed"
+ wide = tmp_path / "wide.bed"
+
+ _build(repo_root, tmp_path, fai, ["3:" + b3], "--mask-bed", str(narrow), "--mask-window", "5")
+ _build(repo_root, tmp_path, fai, ["3:" + b3], "--mask-bed", str(wide), "--mask-window", "9")
+
+ assert _mask_rows(narrow) == ["chr1\t8\t15\t\t"]
+ assert _mask_rows(wide) == ["chr1\t12\t15\t\t"]
+
+
+def test_no_mask_bed_by_default(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 4)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 4, "1")])
+
+ _build(repo_root, tmp_path, fai, ["20:" + b20])
+
+ assert not any(p.suffix == ".bed" for p in tmp_path.iterdir())
+
+
+# -------------------------------------------------------------------------- CLI knobs
+
+
+@pytest.mark.parametrize("missing", ["--fai", "--bedgraphs", "--out-prefix"])
+def test_required_arguments_are_enforced(repo_root, tmp_path, missing):
+ fai = _fai(tmp_path, [("chr1", 4)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 4, "1")])
+ full = {"--fai": [fai], "--bedgraphs": ["20:" + b20],
+ "--out-prefix": [str(tmp_path / "out")]}
+ argv = []
+ for flag, value in full.items():
+ if flag != missing:
+ argv += [flag] + value
+
+ res = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_min_unique_len.py")] + argv,
+ capture_output=True, text=True)
+
+ assert res.returncode != 0
+ assert missing in res.stderr
+
+
+def test_output_file_name_is_the_prefix_plus_a_fixed_suffix(repo_root, tmp_path):
+ fai = _fai(tmp_path, [("chr1", 4)])
+ b20 = _bedgraph(tmp_path, "k20.bedgraph", [("chr1", 0, 4, "1")])
+ prefix = tmp_path / "H37Rv"
+
+ res = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_min_unique_len.py"),
+ "--fai", fai, "--bedgraphs", "20:" + b20, "--out-prefix", str(prefix)],
+ capture_output=True, text=True)
+
+ assert res.returncode == 0, res.stderr
+ assert (tmp_path / "H37Rv.min_unique_len.npz").exists()
+
+
+def test_an_unscored_hole_away_from_the_contig_ends_is_reported(repo_root, tmp_path):
+ """A never-scored position becomes 0, i.e. "placeable at any read length", so the read filter
+ keeps everything there. At a contig end that is deliberate; in the middle it silently disables
+ masking over that stretch, so it has to be visible."""
+ fai = _fai(tmp_path, [("chr1", 20)])
+ # Scored 0-5 and 15-20, nothing in between.
+ bg = _bedgraph(tmp_path, "k35", [("chr1", 0, 5, 1.0), ("chr1", 15, 20, 1.0)])
+ res, tracks = _build(repo_root, tmp_path, fai, [f"35:{bg}"])
+
+ assert res.returncode == 0, res.stderr
+ assert "unscored position(s) away from the contig ends" in res.stderr
+ assert "chr1" in res.stderr
+
+
+def test_unscored_contig_ends_alone_are_not_reported(repo_root, tmp_path):
+ """genmap cannot score a k-mer that runs off the sequence, so a tail is expected and silent."""
+ fai = _fai(tmp_path, [("chr1", 20)])
+ bg = _bedgraph(tmp_path, "k35", [("chr1", 0, 15, 1.0)])
+ res, tracks = _build(repo_root, tmp_path, fai, [f"35:{bg}"])
+
+ assert res.returncode == 0, res.stderr
+ assert "unscored position" not in res.stderr
diff --git a/tests/unit/test_build_snp_matrix.py b/tests/unit/test_build_snp_matrix.py
new file mode 100644
index 0000000..99a7e5b
--- /dev/null
+++ b/tests/unit/test_build_snp_matrix.py
@@ -0,0 +1,335 @@
+"""Unit tests for bin/build_snp_matrix.py.
+
+Aggregates the per-sample annotated VCFs of a run into one wide SNP matrix:
+one row per site, two columns (AF and DP) per sample.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import pytest
+
+from conftest import load_script
+
+bsm = load_script("build_snp_matrix")
+
+
+# --------------------------------------------------------------------------
+# _af_dp: allele frequency
+# --------------------------------------------------------------------------
+
+def test_af_dp_ad_is_preferred_over_ao_ro_and_gt():
+ """AD wins even when AO/RO would give a different answer."""
+ af, dp = bsm._af_dp("GT:AD:AO:RO", "0/1:10,30:1:99")
+ assert af == pytest.approx(0.75)
+ assert dp == 40
+
+
+def test_af_dp_ad_sums_every_alternate_allele():
+ af, dp = bsm._af_dp("GT:AD", "1/2:10,15,25")
+ assert af == pytest.approx(0.8) # (15 + 25) / 50
+ assert dp == 50
+
+
+def test_af_dp_ad_summing_to_zero_falls_through_without_dividing():
+ """`sum(ad) > 0` is what keeps this out of a ZeroDivisionError. The AD branch is skipped
+ and the GT dosage supplies the fraction, but AD still states a depth of zero."""
+ af, dp = bsm._af_dp("GT:AD", "0/1:0,0")
+ assert af == pytest.approx(0.5) # from the GT dosage, not from AD
+ assert dp == 0, "a measured depth of zero is not the same as an unknown depth"
+
+
+def test_af_dp_single_element_ad_is_ignored_for_af_but_still_gives_a_depth():
+ """A one-entry AD (no alternate depth) cannot give an allele fraction, so the GT dosage is
+ used. The depth it does state must not be thrown away: the matrix would emit an empty DP
+ cell for a site whose depth is right there in the record."""
+ af, dp = bsm._af_dp("GT:AD", "1:5")
+ assert af == pytest.approx(1.0)
+ assert dp == 5
+
+
+@pytest.mark.parametrize("ad", [".,.", ".", ""])
+def test_af_dp_ad_of_only_missing_values_is_ignored(ad):
+ af, dp = bsm._af_dp("GT:AD", f"1/1:{ad}")
+ assert af == pytest.approx(1.0)
+ assert dp is None
+
+
+def test_af_dp_unparsable_ad_is_ignored():
+ af, dp = bsm._af_dp("GT:AD", "0/1:a,b")
+ assert af == pytest.approx(0.5)
+ assert dp is None
+
+
+def test_af_dp_ao_ro_when_ad_is_absent():
+ af, dp = bsm._af_dp("GT:AO:RO", "0/1:8:2")
+ assert af == pytest.approx(0.8)
+ assert dp == 10
+
+
+def test_af_dp_ao_ro_sums_multi_allelic_ao():
+ af, dp = bsm._af_dp("GT:AO:RO", "1/2:5,5:10")
+ assert af == pytest.approx(0.5)
+ assert dp == 20
+
+
+def test_af_dp_ro_missing_value_counts_as_zero():
+ af, dp = bsm._af_dp("GT:AO:RO", "1/1:8:.")
+ assert af == pytest.approx(1.0)
+ assert dp == 8
+
+
+def test_af_dp_ao_without_ro_is_ignored():
+ """Both keys are required; AO alone falls through to the GT dosage."""
+ af, dp = bsm._af_dp("GT:AO", "0/1:8")
+ assert af == pytest.approx(0.5)
+ assert dp is None
+
+
+def test_af_dp_ao_plus_ro_of_zero_falls_through():
+ af, dp = bsm._af_dp("GT:AO:RO", "1/1:0:0")
+ assert af == pytest.approx(1.0)
+ assert dp is None
+
+
+def test_af_dp_unparsable_ao_is_ignored():
+ af, dp = bsm._af_dp("GT:AO:RO", "0/1:x:2")
+ assert af == pytest.approx(0.5)
+ assert dp is None
+
+
+@pytest.mark.parametrize("gt,expected", [
+ ("1/1", 1.0),
+ ("1|1", 1.0),
+ ("0/1", 0.5),
+ ("0|1", 0.5),
+ ("1", 1.0),
+ ("0", 0.0),
+ ("1/2", 1.0), # every non-reference allele counts
+ ("0/0", 0.0),
+ ("./1", 1.0), # missing alleles are dropped before the dosage
+ ("0/./1", 0.5),
+])
+def test_af_dp_gt_dosage(gt, expected):
+ af, dp = bsm._af_dp("GT", gt)
+ assert af == pytest.approx(expected)
+ # PIN: the GT branch computes a fraction only - it never fills in a depth.
+ assert dp is None
+
+
+@pytest.mark.parametrize("gt", ["./.", ".", ".|."])
+def test_af_dp_no_call_returns_none(gt):
+ assert bsm._af_dp("GT", gt) == (None, None)
+
+
+def test_af_dp_absent_gt_key_defaults_to_no_call():
+ """No AD, no AO/RO and no GT at all: DP is still read, but AF is None and
+ main() drops the record."""
+ assert bsm._af_dp("DP", "30") == (None, 30)
+
+
+# --------------------------------------------------------------------------
+# _af_dp: depth source order
+# --------------------------------------------------------------------------
+
+def test_af_dp_format_dp_wins_over_ad_sum():
+ _, dp = bsm._af_dp("GT:AD:DP", "0/1:10,30:99")
+ assert dp == 99
+
+
+def test_af_dp_format_dp_wins_over_ao_plus_ro():
+ _, dp = bsm._af_dp("GT:AO:RO:DP", "0/1:8:2:99")
+ assert dp == 99
+
+
+@pytest.mark.parametrize("raw_dp", [".", "", "abc", "1.5"])
+def test_af_dp_unusable_format_dp_falls_back_to_the_ad_sum(raw_dp):
+ _, dp = bsm._af_dp("GT:DP:AD", f"0/1:{raw_dp}:10,30")
+ assert dp == 40
+
+
+def test_af_dp_ad_sum_is_used_when_dp_is_absent():
+ _, dp = bsm._af_dp("GT:AD", "0/1:10,30")
+ assert dp == 40
+
+
+def test_af_dp_extra_format_keys_without_values_are_dropped():
+ """zip() truncates to the shorter of FORMAT and the sample column."""
+ af, dp = bsm._af_dp("GT:AD:DP", "0/1:10,30")
+ assert af == pytest.approx(0.75)
+ assert dp == 40
+
+
+# --------------------------------------------------------------------------
+# _ann
+# --------------------------------------------------------------------------
+
+_FULL_ANN = ("ANN=G|missense_variant|MODERATE|rpoB|GENE_ID|transcript|TX_ID|"
+ "protein_coding|1/1|c.1349C>T|p.Ser450Leu|1349/3519|1349/3519|450/1172||")
+
+
+def test_ann_extracts_gene_effect_and_protein_change():
+ assert bsm._ann(f"DP=30;{_FULL_ANN};MQ=60") == ("rpoB", "missense_variant", "p.Ser450Leu")
+
+
+@pytest.mark.parametrize("info", ["", "DP=30;MQ=60", "AF=0.5", "XANN=a|b|c|d"])
+def test_ann_returns_three_empty_strings_without_ann(info):
+ assert bsm._ann(info) == ("", "", "")
+
+
+def test_ann_uses_the_first_entry_only():
+ info = _FULL_ANN + ",T|synonymous_variant|LOW|katG|GID2|transcript|TX2|protein_coding|1/1|c.1A>G|p.Lys1Lys"
+ assert bsm._ann(info) == ("rpoB", "missense_variant", "p.Ser450Leu")
+
+
+@pytest.mark.parametrize("ann,expected", [
+ ("ANN=G|missense_variant|MODERATE|rpoB", ("rpoB", "missense_variant", "")),
+ ("ANN=G|missense_variant", ("", "missense_variant", "")),
+ ("ANN=G", ("", "", "")),
+ ("ANN=", ("", "", "")),
+])
+def test_ann_tolerates_truncated_entries(ann, expected):
+ assert bsm._ann(ann) == expected
+
+
+# --------------------------------------------------------------------------
+# end to end
+# --------------------------------------------------------------------------
+
+def _vcf(path, sample, records):
+ header = (
+ "##fileformat=VCFv4.2\n"
+ "##contig=\n"
+ f"#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\t{sample}\n"
+ )
+ path.write_text(header + "".join(r + "\n" for r in records))
+ return path
+
+
+def _row(contig, pos, ref, alt, info, fmt, val):
+ return "\t".join([contig, str(pos), ".", ref, alt, "50", "PASS", info, fmt, val])
+
+
+def test_end_to_end_matrix(tmp_path, repo_root):
+ a = _vcf(tmp_path / "SA.vcf", "SA", [
+ _row("chr1", 100, "A", "G", _FULL_ANN, "GT:AD:DP", "1/1:0,30:30"),
+ _row("chr1", 300, "C", "T", ".", "GT:AD:DP", "0/1:10,10:20"),
+ _row("chr1", 500, "AT", "A", ".", "GT:AD:DP", "1/1:0,30:30"), # deletion, dropped
+ _row("chr1", 700, "A", "GG", ".", "GT:AD:DP", "1/1:0,30:30"), # insertion, dropped
+ _row("chr2", 50, "G", "A", ".", "GT:AD:DP", "1/1:0,20:20"),
+ ])
+ b = _vcf(tmp_path / "SB.vcf", "SB", [
+ _row("chr1", 90, "T", "C", ".", "GT:AO:RO", "0/1:6:2"),
+ _row("chr1", 100, "A", "T", ".", "GT:AD:DP", "1/1:0,40:40"),
+ _row("chr1", 300, "C", "T", ".", "GT", "./."), # no call, dropped
+ ])
+ out = tmp_path / "snp_matrix.tsv"
+
+ proc = subprocess.run(
+ [
+ sys.executable, str(repo_root / "bin" / "build_snp_matrix.py"),
+ "--vcfs", str(a), str(b), str(tmp_path / "absent.vcf"),
+ "--reference", "REF1",
+ "-o", str(out),
+ ],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ assert "4 SNP site(s) x 2 sample(s)" in proc.stderr
+
+ rows = [line.split("\t") for line in out.read_text().splitlines()]
+ assert rows[0] == [
+ "reference", "contig", "pos", "ref_allele", "alt_allele", "gene", "effect", "aa_change",
+ "SA|AF", "SA|DP", "SB|AF", "SB|DP",
+ ]
+
+ # Only SNPs survive; sites sort by contig then NUMERIC position (90 < 100 < 300).
+ assert [(r[1], r[2]) for r in rows[1:]] == [
+ ("chr1", "90"), ("chr1", "100"), ("chr1", "300"), ("chr2", "50"),
+ ]
+ assert all(len(r) == len(rows[0]) for r in rows)
+
+ # chr1:90 - called only in SB, so SA gets two EMPTY cells (not zeros).
+ assert rows[1] == ["REF1", "chr1", "90", "T", "C", "", "", "", "", "", "0.7500", "8"]
+ # chr1:100 - both samples; the ALT set is merged and sorted, the annotation
+ # is taken from the first VCF that carried one.
+ assert rows[2] == ["REF1", "chr1", "100", "A", "G,T", "rpoB", "missense_variant",
+ "p.Ser450Leu", "1.0000", "30", "1.0000", "40"]
+ # chr1:300 - SB's './.' record was dropped, so SB is blank here too.
+ assert rows[3] == ["REF1", "chr1", "300", "C", "T", "", "", "", "0.5000", "20", "", ""]
+ assert rows[4] == ["REF1", "chr2", "50", "G", "A", "", "", "", "1.0000", "20", "", ""]
+
+
+def test_end_to_end_reference_defaults_to_the_contig_name(tmp_path, repo_root):
+ a = _vcf(tmp_path / "SA.vcf", "SA", [_row("chr1", 100, "A", "G", ".", "GT:DP", "1/1:30")])
+ out = tmp_path / "snp_matrix.tsv"
+ proc = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_snp_matrix.py"),
+ "--vcfs", str(a), "-o", str(out)],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ row = out.read_text().splitlines()[1].split("\t")
+ assert row[0] == "chr1"
+ assert row[8:] == ["1.0000", "30"]
+
+
+def test_end_to_end_multi_allelic_alt_keeps_only_the_first_allele(tmp_path, repo_root):
+ a = _vcf(tmp_path / "SA.vcf", "SA", [
+ _row("chr1", 100, "A", "G,T", ".", "GT:AD:DP", "1/2:0,20,10:30"),
+ _row("chr1", 200, "A", "GG,T", ".", "GT:AD:DP", "1/2:0,20,10:30"), # first ALT is not a SNP
+ ])
+ out = tmp_path / "snp_matrix.tsv"
+ proc = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_snp_matrix.py"),
+ "--vcfs", str(a), "-o", str(out)],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ rows = [line.split("\t") for line in out.read_text().splitlines()]
+ assert len(rows) == 2
+ assert rows[1][2] == "100"
+ assert rows[1][4] == "G"
+
+
+def test_end_to_end_skips_malformed_records(tmp_path, repo_root):
+ """Records before #CHROM, records with fewer than 8 columns and records
+ with a non-numeric POS are dropped; an unreadable input only warns."""
+ p = tmp_path / "SA.vcf"
+ p.write_text(
+ "##fileformat=VCFv4.2\n"
+ + _row("chr1", 10, "A", "G", ".", "GT:DP", "1/1:30") + "\n" # before #CHROM
+ + "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tSA\n"
+ + "chr1\t20\t.\tA\tG\t50\n" # too few columns
+ + _row("chr1", "NOT_A_POS", "A", "G", ".", "GT:DP", "1/1:30") + "\n"
+ + _row("chr1", 100, "A", "G", ".", "GT:DP", "1/1:30") + "\n"
+ )
+ unreadable = tmp_path / "SB.vcf"
+ unreadable.mkdir()
+ out = tmp_path / "snp_matrix.tsv"
+ proc = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_snp_matrix.py"),
+ "--vcfs", str(p), str(unreadable), "-o", str(out)],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ assert "[snp_matrix] WARN could not read" in proc.stderr
+ rows = out.read_text().splitlines()
+ assert len(rows) == 2
+ assert rows[1].split("\t")[2] == "100"
+
+
+def test_end_to_end_with_no_sites_writes_the_header_only(tmp_path, repo_root):
+ a = _vcf(tmp_path / "SA.vcf", "SA", [])
+ out = tmp_path / "snp_matrix.tsv"
+ proc = subprocess.run(
+ [sys.executable, str(repo_root / "bin" / "build_snp_matrix.py"),
+ "--vcfs", str(a), str(tmp_path / "absent.vcf"), "--reference", "REF1", "-o", str(out)],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ lines = out.read_text().splitlines()
+ assert lines == ["\t".join(["reference", "contig", "pos", "ref_allele", "alt_allele",
+ "gene", "effect", "aa_change", "SA|AF", "SA|DP"])]
diff --git a/tests/unit/test_collect_dr.py b/tests/unit/test_collect_dr.py
new file mode 100644
index 0000000..08f5bef
--- /dev/null
+++ b/tests/unit/test_collect_dr.py
@@ -0,0 +1,243 @@
+"""Unit tests for bin/collect_dr.py.
+
+Aggregates the per-sample pathotypr `*_mutations.tsv` drug-resistance detail
+files into one run TSV. Only `sample_from_name` is importable; the row parsing
+lives inline in `main()`, so it is exercised through the CLI.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import pytest
+
+from conftest import load_script
+
+cdr = load_script("collect_dr")
+
+_OUT_HEADER = ["sample", "drug", "gene", "mutation", "grade", "marker_name", "af", "dp"]
+
+# pathotypr split-fastq column layout for the WHO DR marker panel
+_IN_HEADER = "pos\tref_allele\talt_allele\tref_count\talt_count\talt_fraction\tlineage_path\n"
+
+
+# --------------------------------------------------------------------------
+# sample_from_name
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("path,expected", [
+ # the three suffix forms the regex strips
+ ("S1.dr_mutations.tsv", "S1"),
+ ("S1_mutations.tsv", "S1"),
+ ("S1.tsv", "S1"),
+ # basename is taken first
+ ("/some/deep/path/S1.dr_mutations.tsv", "S1"),
+ ("./rel/S1_mutations.tsv", "S1"),
+ # '__' is the pipeline's sample/run separator: everything after it is dropped
+ ("S1__run2.dr_mutations.tsv", "S1"),
+ ("S1__run2_mutations.tsv", "S1"),
+ ("S1__run2.tsv", "S1"),
+ ("S1__run2__ref3.dr_mutations.tsv", "S1"),
+ ("S1__.tsv", "S1"),
+ # no recognised suffix: the name is returned as-is (minus any '__' tail)
+ ("S1", "S1"),
+ ("S1__run2", "S1"),
+ # only the trailing occurrence is stripped, and only one of the three
+ ("sample.with.dots.tsv", "sample.with.dots"),
+ ("S1.dr_mutations.tsv.gz", "S1.dr_mutations.tsv.gz"),
+ ("S1_mutations.tsv.bak", "S1_mutations.tsv.bak"),
+ ("", ""),
+])
+def test_sample_from_name(path, expected):
+ assert cdr.sample_from_name(path) == expected
+
+
+def test_sample_from_name_leading_double_underscore_yields_empty_sample():
+ """PIN: a name that starts with '__' splits into an empty sample id."""
+ assert cdr.sample_from_name("__run2.dr_mutations.tsv") == ""
+
+
+# --------------------------------------------------------------------------
+# end to end (the row parsing lives inside main())
+# --------------------------------------------------------------------------
+
+def _run(tmp_path, repo_root, files, output="dr.tsv", sample=None):
+ cmd = [sys.executable, str(repo_root / "bin" / "collect_dr.py"), "-o", str(tmp_path / output)]
+ if sample is not None:
+ cmd += ["--sample", sample]
+ cmd += [str(f) for f in files]
+ proc = subprocess.run(cmd, cwd=tmp_path, capture_output=True, text=True, timeout=120)
+ assert proc.returncode == 0, proc.stderr
+ rows = [line.split("\t") for line in (tmp_path / output).read_text().splitlines()]
+ return proc, rows
+
+
+def test_end_to_end_parses_a_mutations_file(tmp_path, repo_root):
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ _IN_HEADER
+ + "761155\tC\tT\t2\t38\t95.0\tRIF;R;marker1;A;rpoB;S450L\n"
+ + "2155168\tG\tC\t5\t45\t90.0\tINH;R;marker2;B;katG;S315T\n"
+ )
+ proc, rows = _run(tmp_path, repo_root, [f])
+
+ assert rows[0] == _OUT_HEADER
+ # sample, drug, gene, mutation, grade, marker_name, af, dp
+ assert rows[1] == ["S1", "RIF", "rpoB", "S450L", "A", "marker1", "0.9500", "40"]
+ assert rows[2] == ["S1", "INH", "katG", "S315T", "B", "marker2", "0.9000", "50"]
+ assert "2 resistance call(s) from 1 file(s)" in proc.stderr
+
+
+def test_end_to_end_alt_fraction_is_a_percent(tmp_path, repo_root):
+ """alt_fraction comes out of pathotypr as a PERCENT and is divided by 100,
+ then formatted to four decimals."""
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ _IN_HEADER
+ + "1\tA\tG\t0\t100\t100.0\tRIF;R;m;A;rpoB;X\n"
+ + "2\tA\tG\t95\t5\t5.0\tRIF;R;m;A;rpoB;Y\n"
+ + "3\tA\tG\t99\t1\t1.234\tRIF;R;m;A;rpoB;Z\n"
+ )
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert [r[6] for r in rows[1:]] == ["1.0000", "0.0500", "0.0123"]
+
+
+def test_end_to_end_dp_is_ref_plus_alt(tmp_path, repo_root):
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ _IN_HEADER
+ + "1\tA\tG\t7\t13\t65.0\tRIF;R;m;A;rpoB;X\n"
+ + "2\tA\tG\t0\t0\t0.0\tRIF;R;m;A;rpoB;Y\n"
+ + "3\tA\tG\t\t9\t90.0\tRIF;R;m;A;rpoB;Z\n" # empty ref_count -> 0
+ )
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert [r[7] for r in rows[1:]] == ["20", "0", "9"]
+
+
+def test_end_to_end_lineage_path_maxsplit_keeps_semicolons_in_the_mutation(tmp_path, repo_root):
+ """split(';', 5) means only the first five separators are consumed, so a
+ ';' inside the trailing mutation descriptor survives intact."""
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ _IN_HEADER
+ + "1\tA\tG\t1\t9\t90.0\tRIF;R;marker1;A;rpoB;S450L;c.1349C>T\n"
+ )
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows[1][3] == "S450L;c.1349C>T"
+ assert rows[1][:3] == ["S1", "RIF", "rpoB"]
+ assert rows[1][4:6] == ["A", "marker1"]
+
+
+def test_end_to_end_short_lineage_path_leaves_trailing_fields_empty(tmp_path, repo_root):
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ _IN_HEADER
+ + "1\tA\tG\t1\t9\t90.0\tRIF;R;marker1\n"
+ + "2\tA\tG\t1\t9\t90.0\t\n"
+ )
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows[1] == ["S1", "RIF", "", "", "", "marker1", "0.9000", "10"]
+ assert rows[2] == ["S1", "", "", "", "", "", "0.9000", "10"]
+
+
+@pytest.mark.parametrize("alt_fraction", ["NA", "", "not_a_number", "95%"])
+def test_end_to_end_non_numeric_alt_fraction_degrades_to_empty(tmp_path, repo_root, alt_fraction):
+ """A bad value produces an empty cell rather than raising."""
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(_IN_HEADER + f"1\tA\tG\t1\t9\t{alt_fraction}\tRIF;R;m;A;rpoB;X\n")
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows[1][6] == ""
+ assert rows[1][7] == "10" # dp is unaffected
+
+
+@pytest.mark.parametrize("counts", [("x", "9"), ("1", "y"), ("1.5", "9")])
+def test_end_to_end_non_numeric_counts_degrade_to_empty(tmp_path, repo_root, counts):
+ ref_count, alt_count = counts
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(_IN_HEADER + f"1\tA\tG\t{ref_count}\t{alt_count}\t90.0\tRIF;R;m;A;rpoB;X\n")
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows[1][7] == ""
+ assert rows[1][6] == "0.9000" # af is unaffected
+
+
+def test_end_to_end_missing_columns_degrade_to_empty(tmp_path, repo_root):
+ """A header without alt_fraction / counts still yields a row, just blank."""
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text("pos\tlineage_path\n1\tRIF;R;marker1;A;rpoB;S450L\n")
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows[1] == ["S1", "RIF", "rpoB", "S450L", "A", "marker1", "", "0"]
+
+
+def test_end_to_end_header_is_case_insensitive_and_blank_lines_are_skipped(tmp_path, repo_root):
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text(
+ "\nPOS\tREF_ALLELE\tALT_ALLELE\tREF_COUNT\tALT_COUNT\t ALT_FRACTION \tLineage_Path\n"
+ "\n1\tA\tG\t1\t9\t90.0\tRIF;R;m;A;rpoB;X\n\n"
+ )
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert len(rows) == 2
+ assert rows[1] == ["S1", "RIF", "rpoB", "X", "A", "m", "0.9000", "10"]
+
+
+def test_end_to_end_infers_the_sample_per_file(tmp_path, repo_root):
+ a = tmp_path / "sampleA.dr_mutations.tsv"
+ b = tmp_path / "sampleB__run7.dr_mutations.tsv"
+ a.write_text(_IN_HEADER + "1\tA\tG\t1\t9\t90.0\tRIF;R;m;A;rpoB;X\n")
+ b.write_text(_IN_HEADER + "2\tA\tG\t1\t9\t90.0\tINH;R;m;A;katG;Y\n")
+ _, rows = _run(tmp_path, repo_root, [a, b])
+ assert [r[0] for r in rows[1:]] == ["sampleA", "sampleB"]
+
+
+def test_end_to_end_sample_flag_overrides_every_file(tmp_path, repo_root):
+ a = tmp_path / "sampleA.dr_mutations.tsv"
+ b = tmp_path / "sampleB.dr_mutations.tsv"
+ a.write_text(_IN_HEADER + "1\tA\tG\t1\t9\t90.0\tRIF;R;m;A;rpoB;X\n")
+ b.write_text(_IN_HEADER + "2\tA\tG\t1\t9\t90.0\tINH;R;m;A;katG;Y\n")
+ _, rows = _run(tmp_path, repo_root, [a, b], sample="OVERRIDE")
+ assert [r[0] for r in rows[1:]] == ["OVERRIDE", "OVERRIDE"]
+
+
+def test_end_to_end_skips_non_existent_inputs_silently(tmp_path, repo_root):
+ good = tmp_path / "S1.dr_mutations.tsv"
+ good.write_text(_IN_HEADER + "1\tA\tG\t1\t9\t90.0\tRIF;R;m;A;rpoB;X\n")
+ proc, rows = _run(tmp_path, repo_root, [tmp_path / "absent.dr_mutations.tsv", good])
+ assert len(rows) == 2
+ assert "absent" not in proc.stderr
+ # the summary line counts the files ASKED for, not the ones actually read
+ assert "1 resistance call(s) from 2 file(s)" in proc.stderr
+
+
+def test_end_to_end_writes_the_header_with_zero_data_rows(tmp_path, repo_root):
+ """The output file is always created, so the downstream report never has a
+ missing input, even when nothing resistant was detected."""
+ empty = tmp_path / "S1.dr_mutations.tsv"
+ empty.write_text(_IN_HEADER) # header only
+ _, rows = _run(tmp_path, repo_root, [empty])
+ assert rows == [_OUT_HEADER]
+
+
+def test_end_to_end_with_no_input_files_at_all(tmp_path, repo_root):
+ proc, rows = _run(tmp_path, repo_root, [])
+ assert rows == [_OUT_HEADER]
+ assert "0 resistance call(s) from 0 file(s)" in proc.stderr
+
+
+def test_end_to_end_completely_empty_file(tmp_path, repo_root):
+ f = tmp_path / "S1.dr_mutations.tsv"
+ f.write_text("")
+ _, rows = _run(tmp_path, repo_root, [f])
+ assert rows == [_OUT_HEADER]
+
+
+def test_end_to_end_unreadable_input_warns_but_does_not_fail(tmp_path, repo_root):
+ """A directory passes the exists() check and then fails to open; the OSError
+ is reported on stderr and the remaining inputs are still processed."""
+ d = tmp_path / "S9.dr_mutations.tsv"
+ d.mkdir()
+ good = tmp_path / "S1.dr_mutations.tsv"
+ good.write_text(_IN_HEADER + "1\tA\tG\t1\t9\t90.0\tRIF;R;m;A;rpoB;X\n")
+ proc, rows = _run(tmp_path, repo_root, [d, good])
+ assert "[collect_dr] WARN could not read" in proc.stderr
+ assert len(rows) == 2
+ assert rows[1][0] == "S1"
diff --git a/tests/unit/test_collect_summary.py b/tests/unit/test_collect_summary.py
new file mode 100644
index 0000000..f25daa9
--- /dev/null
+++ b/tests/unit/test_collect_summary.py
@@ -0,0 +1,372 @@
+"""Unit tests for bin/collect_summary.py.
+
+Phase-B aggregation: turns the per-sample legacy `.log` files written by
+stats_to_legacy.py into one wide summary TSV, plus an optional cohort
+gene-burden table.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import pytest
+
+from conftest import load_script
+
+cs = load_script("collect_summary")
+
+
+# --------------------------------------------------------------------------
+# _to_int
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("raw,expected", [
+ ("5", 5),
+ ("0", 0),
+ ("-3", -3),
+ (" 7 ", 7), # int() tolerates surrounding whitespace
+ (12, 12),
+ (5.9, 5), # truncates towards zero
+])
+def test_to_int_accepts(raw, expected):
+ assert cs._to_int(raw) == expected
+
+
+@pytest.mark.parametrize("raw", [None, "", "NA", "abc", "5.5", "1e3", []])
+def test_to_int_rejects(raw):
+ """Anything unparsable becomes None (never raises, never 0)."""
+ assert cs._to_int(raw) is None
+
+
+# --------------------------------------------------------------------------
+# parse_log
+# --------------------------------------------------------------------------
+
+def test_parse_log_reads_tab_separated_pairs(tmp_path):
+ p = tmp_path / "S1.log"
+ p.write_text("GID_IN\tS1\nSNP_OUT\t42\nTITV_OUT\tNA\n")
+ assert cs.parse_log(str(p)) == {"GID_IN": "S1", "SNP_OUT": "42", "TITV_OUT": "NA"}
+
+
+def test_parse_log_splits_only_on_the_first_tab(tmp_path):
+ """split('\\t', 1) means a value is allowed to contain tabs of its own."""
+ p = tmp_path / "S1.log"
+ p.write_text("KEY\tvalue\twith\ttabs\n")
+ assert cs.parse_log(str(p)) == {"KEY": "value\twith\ttabs"}
+
+
+def test_parse_log_skips_blank_and_tabless_lines(tmp_path):
+ p = tmp_path / "S1.log"
+ p.write_text("GID_IN\tS1\n\n \nJUST_A_KEY\nSNP_OUT\t7\n")
+ assert cs.parse_log(str(p)) == {"GID_IN": "S1", "SNP_OUT": "7"}
+
+
+def test_parse_log_last_duplicate_key_wins(tmp_path):
+ p = tmp_path / "S1.log"
+ p.write_text("GID_IN\tfirst\nGID_IN\tsecond\n")
+ assert cs.parse_log(str(p)) == {"GID_IN": "second"}
+
+
+def test_parse_log_missing_file_returns_empty_dict(tmp_path, capsys):
+ """Never raises: an unreadable log degrades to {} and a warning."""
+ assert cs.parse_log(str(tmp_path / "absent.log")) == {}
+ assert "Could not parse" in capsys.readouterr().err
+
+
+def test_parse_log_empty_file_returns_empty_dict(tmp_path):
+ p = tmp_path / "empty.log"
+ p.write_text("")
+ assert cs.parse_log(str(p)) == {}
+
+
+# --------------------------------------------------------------------------
+# _derive_ann
+# --------------------------------------------------------------------------
+
+def _ann_data(**overrides):
+ data = {
+ "ANN_PRESENT": "yes",
+ "ANN_TOTAL": "100",
+ "ANN_ANNOTATED": "80",
+ "ANN_LOF": "5",
+ "ANN_CODING": "50",
+ "ANN_IMPACT_HIGH": "5",
+ "ANN_IMPACT_MODERATE": "20",
+ "ANN_IMPACT_LOW": "25",
+ "ANN_IMPACT_MODIFIER": "30",
+ }
+ data.update(overrides)
+ return data
+
+
+@pytest.mark.parametrize("present", ["no", "NA", "", "YES", None])
+def test_derive_ann_requires_ann_present_yes(present):
+ data = _ann_data()
+ if present is None:
+ data.pop("ANN_PRESENT")
+ else:
+ data["ANN_PRESENT"] = present
+ assert cs._derive_ann(data) == ("NA", "NA", "NA")
+
+
+def test_derive_ann_returns_formatted_strings():
+ ann_pct, coding_pct, lof_pct = cs._derive_ann(_ann_data())
+ assert (ann_pct, coding_pct, lof_pct) == ("80.00", "62.50", "10.00")
+ assert all(isinstance(x, str) for x in (ann_pct, coding_pct, lof_pct))
+
+
+@pytest.mark.parametrize("total", ["0", "NA", "", None])
+def test_derive_ann_guard_one_zero_or_missing_total(total):
+ """Guard 1: annotated_pct needs a non-zero ANN_TOTAL denominator."""
+ data = _ann_data()
+ if total is None:
+ data.pop("ANN_TOTAL")
+ else:
+ data["ANN_TOTAL"] = total
+ ann_pct, coding_pct, lof_pct = cs._derive_ann(data)
+ assert ann_pct == "NA"
+ # the other two are unaffected by this guard
+ assert (coding_pct, lof_pct) == ("62.50", "10.00")
+
+
+def test_derive_ann_guard_one_also_trips_on_an_unparsable_numerator():
+ ann_pct, _, _ = cs._derive_ann(_ann_data(ANN_ANNOTATED="NA"))
+ assert ann_pct == "NA"
+
+
+def test_derive_ann_guard_two_zero_impact_denominator():
+ """Guard 2: coding_pct needs the four impact buckets to sum above zero."""
+ data = _ann_data(ANN_IMPACT_HIGH="0", ANN_IMPACT_MODERATE="0",
+ ANN_IMPACT_LOW="0", ANN_IMPACT_MODIFIER="0")
+ ann_pct, coding_pct, lof_pct = cs._derive_ann(data)
+ assert coding_pct == "NA"
+ assert (ann_pct, lof_pct) == ("80.00", "10.00")
+
+
+def test_derive_ann_guard_two_all_impacts_unparsable():
+ """sum() over an all-None generator is 0, so the guard still catches it."""
+ data = _ann_data(ANN_IMPACT_HIGH="NA", ANN_IMPACT_MODERATE="NA",
+ ANN_IMPACT_LOW="NA", ANN_IMPACT_MODIFIER="NA")
+ assert cs._derive_ann(data)[1] == "NA"
+
+
+def test_derive_ann_guard_two_treats_unparsable_impacts_as_zero_in_the_numerator():
+ data = _ann_data(ANN_IMPACT_HIGH="NA", ANN_IMPACT_MODERATE="10",
+ ANN_IMPACT_LOW="10", ANN_IMPACT_MODIFIER="80")
+ # numerator = 0 + 10 + 10, denominator = 100
+ assert cs._derive_ann(data)[1] == "20.00"
+
+
+@pytest.mark.parametrize("coding", ["0", "NA", "", None])
+def test_derive_ann_guard_three_zero_or_missing_coding(coding):
+ """Guard 3: lof_pct needs a non-zero ANN_CODING denominator."""
+ data = _ann_data()
+ if coding is None:
+ data.pop("ANN_CODING")
+ else:
+ data["ANN_CODING"] = coding
+ ann_pct, coding_pct, lof_pct = cs._derive_ann(data)
+ assert lof_pct == "NA"
+ assert (ann_pct, coding_pct) == ("80.00", "62.50")
+
+
+def test_derive_ann_guard_three_also_trips_on_an_unparsable_numerator():
+ assert cs._derive_ann(_ann_data(ANN_LOF="NA"))[2] == "NA"
+
+
+def test_derive_ann_lof_pct_stays_within_range():
+ """CODING is the superset of LOF by construction, so the ratio is bounded."""
+ assert cs._derive_ann(_ann_data(ANN_LOF="50", ANN_CODING="50"))[2] == "100.00"
+
+
+# --------------------------------------------------------------------------
+# _accumulate_gene_burden
+# --------------------------------------------------------------------------
+
+@pytest.mark.parametrize("burden", ["NA", "", None])
+def test_accumulate_gene_burden_is_a_noop_for_empty_input(burden):
+ agg = {}
+ cs._accumulate_gene_burden(agg, burden)
+ assert agg == {}
+
+
+def test_accumulate_gene_burden_mutates_the_aggregate_in_place():
+ agg = {}
+ assert cs._accumulate_gene_burden(agg, "rpoB:2:1:missense") is None
+ assert agg == {"rpoB": {"high": 2, "mod": 1, "n_samples": 1, "eff": {"missense": 1}}}
+
+
+def test_accumulate_gene_burden_sums_across_samples():
+ agg = {}
+ cs._accumulate_gene_burden(agg, "rpoB:2:1:missense;katG:1:0:frameshift")
+ cs._accumulate_gene_burden(agg, "rpoB:3:4:stop_gained")
+ assert agg["rpoB"] == {"high": 5, "mod": 5, "n_samples": 2,
+ "eff": {"missense": 1, "stop_gained": 1}}
+ assert agg["katG"] == {"high": 1, "mod": 0, "n_samples": 1, "eff": {"frameshift": 1}}
+
+
+@pytest.mark.parametrize("token", [
+ "rpoB:2:1", # too few fields
+ "rpoB:2:1:missense:x", # too many fields
+ "rpoB",
+ "",
+ "rpoB:x:1:missense", # non-integer HIGH
+ "rpoB:2:y:missense", # non-integer MOD
+])
+def test_accumulate_gene_burden_skips_malformed_tokens(token):
+ agg = {}
+ cs._accumulate_gene_burden(agg, token)
+ assert agg == {}
+
+
+def test_accumulate_gene_burden_skips_only_the_bad_token():
+ agg = {}
+ cs._accumulate_gene_burden(agg, "bad:token;katG:1:0:frameshift;also:bad")
+ assert list(agg) == ["katG"]
+
+
+def test_accumulate_gene_burden_counts_n_samples_once_per_token():
+ """PIN: n_samples is incremented per TOKEN, not per distinct sample. A
+ per-sample burden string never repeats a gene (stats_to_legacy builds it
+ from a dict), but if it did, that one sample would count twice."""
+ agg = {}
+ cs._accumulate_gene_burden(agg, "rpoB:1:0:missense;rpoB:2:0:missense")
+ assert agg["rpoB"]["n_samples"] == 2
+ assert agg["rpoB"]["high"] == 3
+
+
+def test_accumulate_gene_burden_preserves_existing_entries():
+ agg = {"katG": {"high": 9, "mod": 9, "n_samples": 4, "eff": {"missense": 4}}}
+ cs._accumulate_gene_burden(agg, "katG:1:1:frameshift")
+ assert agg["katG"] == {"high": 10, "mod": 10, "n_samples": 5,
+ "eff": {"missense": 4, "frameshift": 1}}
+
+
+# --------------------------------------------------------------------------
+# _write_gene_burden
+# --------------------------------------------------------------------------
+
+def test_write_gene_burden_header_and_sort_order(tmp_path):
+ agg = {
+ "geneLow": {"high": 0, "mod": 1, "n_samples": 1, "eff": {"missense": 1}},
+ "geneTop": {"high": 5, "mod": 0, "n_samples": 3, "eff": {"stop_gained": 2, "frameshift": 1}},
+ "zeta": {"high": 2, "mod": 2, "n_samples": 1, "eff": {"missense": 1}},
+ "alpha": {"high": 2, "mod": 2, "n_samples": 2, "eff": {"missense": 2}},
+ "geneMid": {"high": 2, "mod": 7, "n_samples": 1, "eff": {"splice": 1}},
+ }
+ out = tmp_path / "gene_burden.tsv"
+ cs._write_gene_burden(agg, str(out))
+ rows = [line.split("\t") for line in out.read_text().splitlines()]
+
+ assert rows[0] == ["gene", "high", "moderate", "low", "modifier",
+ "dominant_effect", "n_samples", "total_impactful", "start", "end"]
+ # sorted by -high, then -moderate, then gene name
+ assert [r[0] for r in rows[1:]] == ["geneTop", "geneMid", "alpha", "zeta", "geneLow"]
+ assert rows[1] == ["geneTop", "5", "0", "", "", "stop_gained", "3", "5", "", ""]
+ assert rows[2] == ["geneMid", "2", "7", "", "", "splice", "1", "9", "", ""]
+
+
+def test_write_gene_burden_empty_effect_map_gives_na(tmp_path):
+ out = tmp_path / "gene_burden.tsv"
+ cs._write_gene_burden({"geneX": {"high": 1, "mod": 0, "n_samples": 1, "eff": {}}}, str(out))
+ assert out.read_text().splitlines()[1].split("\t")[5] == "NA"
+
+
+def test_write_gene_burden_empty_aggregate_writes_header_only(tmp_path):
+ out = tmp_path / "gene_burden.tsv"
+ cs._write_gene_burden({}, str(out))
+ assert len(out.read_text().splitlines()) == 1
+
+
+# --------------------------------------------------------------------------
+# end to end
+# --------------------------------------------------------------------------
+
+def _log_text(sample, **overrides):
+ data = {
+ "GID_IN": sample,
+ "TRD_OUT": "1000",
+ "TRM_OUT": "900",
+ "SNP_OUT": "42",
+ "ANN_PRESENT": "yes",
+ "ANN_TOTAL": "100",
+ "ANN_ANNOTATED": "80",
+ "ANN_LOF": "5",
+ "ANN_CODING": "50",
+ "ANN_IMPACT_HIGH": "5",
+ "ANN_IMPACT_MODERATE": "20",
+ "ANN_IMPACT_LOW": "25",
+ "ANN_IMPACT_MODIFIER": "30",
+ "ANN_GENE_BURDEN": "rpoB:2:1:missense",
+ }
+ data.update(overrides)
+ return "".join(f"{k}\t{v}\n" for k, v in data.items())
+
+
+def test_end_to_end_summary(tmp_path, repo_root):
+ """Row order follows the SORTED FILE NAMES, not the sample ids inside."""
+ (tmp_path / "a.log").write_text(_log_text("zeta"))
+ (tmp_path / "b.log").write_text(_log_text("alpha", ANN_PRESENT="no",
+ ANN_GENE_BURDEN="katG:3:0:frameshift"))
+ out = tmp_path / "summary.tsv"
+ burden = tmp_path / "gene_burden.tsv"
+
+ proc = subprocess.run(
+ [
+ sys.executable, str(repo_root / "bin" / "collect_summary.py"),
+ # deliberately passed out of order, and with one path that does not exist
+ str(tmp_path / "b.log"), str(tmp_path / "a.log"), str(tmp_path / "absent.log"),
+ "-o", str(out),
+ "--gene-burden-out", str(burden),
+ ],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ assert "2 samples" in proc.stderr
+
+ rows = [line.split("\t") for line in out.read_text().splitlines()]
+ header = rows[0]
+ assert header == [cs.SUMMARY_COLUMNS[k] for k in cs.COLUMN_ORDER] + cs.DERIVED_COLUMNS
+ assert header[0] == "sample_id"
+ assert header[-3:] == ["annotated_pct", "coding_pct", "lof_pct"]
+
+ assert len(rows) == 3
+ assert [r[0] for r in rows[1:]] == ["zeta", "alpha"] # a.log before b.log
+ assert all(len(r) == len(header) for r in rows)
+
+ first = dict(zip(header, rows[1]))
+ assert first["raw_reads"] == "1000"
+ assert first["snps"] == "42"
+ assert first["annotated_pct"] == "80.00"
+ assert first["coding_pct"] == "62.50"
+ assert first["lof_pct"] == "10.00"
+ # A field absent from the .log becomes 'NA' in the summary.
+ assert first["median_depth"] == "NA"
+
+ second = dict(zip(header, rows[2]))
+ assert second["ann_present"] == "no"
+ assert (second["annotated_pct"], second["coding_pct"], second["lof_pct"]) == ("NA", "NA", "NA")
+
+ # The gene burden is accumulated even for the sample whose ANN_PRESENT is 'no'.
+ burden_rows = [line.split("\t") for line in burden.read_text().splitlines()]
+ assert [r[0] for r in burden_rows[1:]] == ["katG", "rpoB"]
+
+
+def test_end_to_end_with_no_usable_logs_writes_header_only(tmp_path, repo_root):
+ (tmp_path / "empty.log").write_text("")
+ out = tmp_path / "summary.tsv"
+ burden = tmp_path / "gene_burden.tsv"
+ proc = subprocess.run(
+ [
+ sys.executable, str(repo_root / "bin" / "collect_summary.py"),
+ str(tmp_path / "empty.log"), str(tmp_path / "absent.log"),
+ "-o", str(out), "--gene-burden-out", str(burden),
+ ],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ assert out.read_text().splitlines() == ["\t".join(
+ [cs.SUMMARY_COLUMNS[k] for k in cs.COLUMN_ORDER] + cs.DERIVED_COLUMNS)]
+ # The aux table is only written when at least one gene was aggregated.
+ assert not burden.exists()
diff --git a/tests/unit/test_filter_reads_mappability.py b/tests/unit/test_filter_reads_mappability.py
new file mode 100644
index 0000000..21dd374
--- /dev/null
+++ b/tests/unit/test_filter_reads_mappability.py
@@ -0,0 +1,389 @@
+"""Unit tests for bin/filter_reads_mappability.py (length-aware mappability read filter).
+
+`span()` is importable and tested directly. Everything else lives in a closure inside
+`main()`, so the filter is exercised end to end as a subprocess: a hand-written SAM on
+stdin, the filtered SAM on stdout, and a numpy `.npz` mappability track on disk.
+
+Track convention (see bin/build_min_unique_len.py): one uint16 array per contig,
+0-based, value = the shortest read length that is genome-unique at that start, with
+65535 meaning "never unique" (always repetitive).
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import numpy as np
+import pytest
+
+from conftest import load_script
+
+filt = load_script("filter_reads_mappability")
+
+SENTINEL = 65535
+KMIN = 35 # the script default; also what these tests pass explicitly
+
+
+# --------------------------------------------------------------------------- helpers
+
+
+def _track(tmp_path, arrays, name="mul.npz") -> str:
+ """Write a mappability .npz, one uint16 array per contig."""
+ path = tmp_path / name
+ np.savez(path, **{k: np.asarray(v, dtype=np.uint16) for k, v in arrays.items()})
+ return str(path)
+
+
+def _default_track(tmp_path) -> str:
+ """chr1: unique at 30 bp everywhere except 1-based 51..100, which is a repeat."""
+ arr = np.full(200, 30, dtype=np.uint16)
+ arr[50:100] = SENTINEL
+ return _track(tmp_path, {"chr1": arr})
+
+
+def _sam(name, flag, rname="chr1", pos=1, cigar="50M", rnext="*", pnext=0, tags=()):
+ fields = [name, str(flag), rname, str(pos), "60", cigar, rnext, str(pnext), "0",
+ "A" * 10, "I" * 10]
+ return "\t".join(fields + list(tags)) + "\n"
+
+
+HEADER = "@HD\tVN:1.6\tSO:coordinate\n@SQ\tSN:chr1\tLN:200\n@PG\tID:bwa\tPN:bwa\n"
+
+
+def _run(repo_root, track, sam_text, *extra):
+ cmd = [sys.executable, str(repo_root / "bin" / "filter_reads_mappability.py"),
+ "--mul", track, *extra]
+ return subprocess.run(cmd, input=sam_text, capture_output=True, text=True)
+
+
+def _kept(res):
+ """Read names of the alignment records that survived, in output order.
+
+ The exit status is asserted here so that an unnoticed crash can never masquerade as
+ "every read was filtered out".
+ """
+ assert res.returncode == 0, res.stderr
+ return [line.split("\t")[0] for line in res.stdout.splitlines() if not line.startswith("@")]
+
+
+# ------------------------------------------------------------------------------ span
+
+
+@pytest.mark.parametrize("cigar,expected", [
+ ("10M2I5M", 15), # I does not consume the reference
+ ("5S10M5S", 10), # neither does S
+ ("10M5N10M", 25), # N does
+ ("10M2D5M", 17), # D does
+ ("5=3X2M", 10), # = and X do
+ ("5H10M", 10), # hard clip does not
+ ("10P5M", 5), # padding does not
+ ("76M", 76),
+])
+def test_span_sums_only_reference_consuming_ops(cigar, expected):
+ assert filt.span(cigar) == expected
+
+
+def test_span_of_a_missing_cigar_is_zero():
+ assert filt.span("*") == 0
+
+
+@pytest.mark.parametrize("cigar", ["", "10", "not-a-cigar", "MMM", "10m"])
+def test_span_of_a_malformed_cigar_is_zero_without_raising(cigar):
+ # No validation happens: unknown letters are simply not counted, and a run of
+ # digits with no operator contributes nothing. Lowercase 'm' is NOT an operator.
+ assert filt.span(cigar) == 0
+
+
+def test_span_ignores_unknown_operators_but_keeps_the_valid_ones():
+ assert filt.span("5Z10M") == 10
+
+
+# --------------------------------------------------------------------------- headers
+
+
+def test_header_lines_pass_through_unchanged(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ res = _run(repo_root, track, HEADER, "--kmin", str(KMIN))
+ assert res.returncode == 0
+ assert res.stdout == HEADER
+
+
+def test_unmodelled_contig_in_the_header_only_warns(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ header = HEADER + "@SQ\tSN:plasmid\tLN:5000\n"
+ res = _run(repo_root, track, header, "--kmin", str(KMIN))
+ assert res.returncode == 0
+ assert res.stdout == header
+ assert "plasmid" in res.stderr
+ assert "fail-open" in res.stderr
+
+
+def test_strict_contigs_aborts_on_an_unmodelled_contig(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ header = HEADER + "@SQ\tSN:plasmid\tLN:5000\n"
+ res = _run(repo_root, track, header, "--kmin", str(KMIN), "--strict-contigs")
+ assert res.returncode != 0
+ assert "plasmid" in res.stderr
+ assert "--strict-contigs" in res.stderr
+
+
+def test_strict_contigs_is_silent_when_every_contig_is_modelled(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ res = _run(repo_root, track, HEADER, "--strict-contigs")
+ assert res.returncode == 0
+ assert res.stderr == ""
+
+
+# ----------------------------------------------------------------- single-end reads
+
+
+def test_single_end_read_at_a_unique_locus_is_kept(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("unique", 0, pos=11, cigar="50M") # min_unique_len = 30
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["unique"]
+
+
+def test_single_end_read_inside_a_repeat_is_dropped(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("repeat", 0, pos=61, cigar="50M") # min_unique_len = 65535
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_single_end_read_shorter_than_its_min_unique_len_is_dropped(repo_root, tmp_path):
+ """40 bp span at a locus that needs 50 bp: ambiguous, and 40 >= kmin so the
+ soft-clip escape hatch does not apply."""
+ track = _track(tmp_path, {"chr1": np.full(200, 50, dtype=np.uint16)})
+ sam = HEADER + _sam("tooshort", 0, pos=11, cigar="40M")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_clipped_read_below_kmin_at_a_resolvable_locus_is_kept(repo_root, tmp_path):
+ """FIX1: a 20 bp reference span below the smallest probed k is soft clipping, not
+ ambiguity, so a resolvable (non-sentinel) locus keeps the read."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("clipped", 0, pos=11, cigar="20M80S")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["clipped"]
+
+
+def test_clipped_read_below_kmin_inside_a_repeat_is_still_dropped(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("clipped_repeat", 0, pos=61, cigar="20M80S")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_read_starting_past_the_end_of_the_track_is_dropped(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("offtrack", 0, pos=5000, cigar="50M")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+# ------------------------------------------------------------------------ flag drops
+
+
+@pytest.mark.parametrize("flag,label", [
+ (4, "unmapped"),
+ (256, "secondary"),
+ (512, "qcfail"),
+ (1024, "duplicate"),
+ (2048, "supplementary"),
+])
+def test_flag_masked_records_are_dropped(repo_root, tmp_path, flag, label):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam(label, flag, pos=11, cigar="50M")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_keep_unmapped_readmits_only_plain_unmapped_records(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = (HEADER
+ + _sam("unmapped", 4, rname="*", pos=0, cigar="*")
+ + _sam("unmapped_secondary", 4 | 256, pos=11)
+ + _sam("unmapped_supplementary", 4 | 2048, pos=11)
+ + _sam("duplicate", 1024, pos=11))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN), "--keep-unmapped")
+ assert _kept(res) == ["unmapped"]
+
+
+def test_without_keep_unmapped_the_unmapped_record_is_gone(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("unmapped", 4, rname="*", pos=0, cigar="*")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+# ------------------------------------------------------------------ contig fail-open
+
+
+def test_read_on_a_contig_absent_from_the_track_is_kept(repo_root, tmp_path):
+ """Finding2: masking is simply disabled on an unmodelled contig rather than
+ silently deleting every read there."""
+ track = _default_track(tmp_path)
+ sam = HEADER + "@SQ\tSN:plasmid\tLN:5000\n" + _sam("onplasmid", 0, rname="plasmid", pos=1)
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert res.returncode == 0
+ assert _kept(res) == ["onplasmid"]
+
+
+def test_read_whose_mate_is_on_an_unmodelled_contig_is_kept(repo_root, tmp_path):
+ """The self anchor fails (repeat), the mate contig is unknown -> fail open."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("matefaropen", 1 | 2 | 64, pos=61, cigar="50M",
+ rnext="plasmid", pnext=100, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["matefaropen"]
+
+
+# --------------------------------------------------------------------- mate rescue
+
+
+def test_repeat_read_is_rescued_by_an_anchored_mate(repo_root, tmp_path):
+ """Proper pair (0x2), mate mapped, MC:Z gives the mate a 50 bp span at a locus that
+ needs 30 -> the whole fragment is kept."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("rescued", 1 | 2 | 64, pos=61, cigar="50M",
+ rnext="=", pnext=11, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["rescued"]
+
+
+def test_rescue_requires_the_proper_pair_flag(repo_root, tmp_path):
+ """FIX3: a discordant read sitting in the wrong repeat copy must not be re-admitted
+ by its mate, so 0x2 is mandatory."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("discordant", 1 | 64, pos=61, cigar="50M",
+ rnext="=", pnext=11, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_rescue_requires_a_mapped_mate(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("mateunmapped", 1 | 2 | 8 | 64, pos=61, cigar="50M",
+ rnext="=", pnext=11, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_rescue_requires_the_mc_tag(repo_root, tmp_path):
+ """Without MC:Z the mate's reference span is unknown, so the read cannot be rescued."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("nomc", 1 | 2 | 64, pos=61, cigar="50M", rnext="=", pnext=11)
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_rescue_finds_the_mc_tag_among_other_tags(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("taggy", 1 | 2 | 64, pos=61, cigar="50M", rnext="=", pnext=11,
+ tags=("NM:i:0", "MD:Z:50", "MC:Z:50M", "RG:Z:s1"))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["taggy"]
+
+
+def test_rescue_fails_when_the_mate_is_also_in_the_repeat(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("bothrepeat", 1 | 2 | 64, pos=61, cigar="50M",
+ rnext="=", pnext=71, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_rescue_fails_when_the_mate_position_is_off_the_track(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("mateoff", 1 | 2 | 64, pos=61, cigar="50M",
+ rnext="=", pnext=9000, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == []
+
+
+def test_self_anchored_pair_is_kept_without_looking_at_the_mate(repo_root, tmp_path):
+ """The self anchor short-circuits, so neither 0x2 nor MC:Z is needed."""
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("selfok", 1 | 64, pos=11, cigar="50M", rnext="=", pnext=61)
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res) == ["selfok"]
+
+
+def test_mate_rnext_equals_sign_resolves_to_the_read_contig(repo_root, tmp_path):
+ """RNEXT '=' and the explicit contig name must behave identically."""
+ track = _default_track(tmp_path)
+ explicit = HEADER + _sam("explicit", 1 | 2 | 64, pos=61, cigar="50M",
+ rnext="chr1", pnext=11, tags=("MC:Z:50M",))
+ res = _run(repo_root, track, explicit, "--kmin", str(KMIN))
+ assert _kept(res) == ["explicit"]
+
+
+# -------------------------------------------------------------------------- ordering
+
+
+def test_output_preserves_input_order_and_only_drops(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ records = [
+ _sam("a_unique", 0, pos=11),
+ _sam("b_repeat", 0, pos=61),
+ _sam("c_unique", 0, pos=111),
+ _sam("d_secondary", 256, pos=11),
+ _sam("e_unique", 0, pos=151),
+ _sam("f_repeat", 0, pos=99),
+ ]
+ res = _run(repo_root, track, HEADER + "".join(records), "--kmin", str(KMIN))
+ assert _kept(res) == ["a_unique", "c_unique", "e_unique"]
+ # Drops only: every surviving line is byte-identical to its input line.
+ kept_lines = [ln + "\n" for ln in res.stdout.splitlines() if not ln.startswith("@")]
+ assert kept_lines == [records[0], records[2], records[4]]
+
+
+def test_headers_stay_ahead_of_the_records(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ res = _run(repo_root, track, HEADER + _sam("r", 0, pos=11), "--kmin", str(KMIN))
+ lines = res.stdout.splitlines()
+ assert lines[:3] == HEADER.splitlines()
+ assert lines[3].startswith("r\t")
+
+
+# ------------------------------------------------------------------------- CLI knobs
+
+
+def test_custom_sentinel_is_honoured(repo_root, tmp_path):
+ """With --sentinel 999 the value 999 means "never unique", so the soft-clip escape
+ hatch must not fire there."""
+ arr = np.full(200, 999, dtype=np.uint16)
+ track = _track(tmp_path, {"chr1": arr})
+ sam = HEADER + _sam("clipped", 0, pos=11, cigar="20M80S")
+ res = _run(repo_root, track, sam, "--kmin", str(KMIN), "--sentinel", "999")
+ assert _kept(res) == []
+ # Without the override, 999 is an ordinary length and the clipped read is rescued.
+ res2 = _run(repo_root, track, sam, "--kmin", str(KMIN))
+ assert _kept(res2) == ["clipped"]
+
+
+def test_kmin_controls_the_soft_clip_escape_hatch(repo_root, tmp_path):
+ track = _default_track(tmp_path)
+ sam = HEADER + _sam("clipped", 0, pos=11, cigar="20M80S")
+ assert _kept(_run(repo_root, track, sam, "--kmin", "35")) == ["clipped"]
+ assert _kept(_run(repo_root, track, sam, "--kmin", "15")) == []
+
+
+def test_missing_mul_argument_is_an_error(repo_root, tmp_path):
+ cmd = [sys.executable, str(repo_root / "bin" / "filter_reads_mappability.py")]
+ res = subprocess.run(cmd, input="", capture_output=True, text=True)
+ assert res.returncode != 0
+ assert "--mul" in res.stderr
+
+
+def test_a_blank_line_in_the_stream_is_skipped(repo_root, tmp_path):
+ """A blank line has no fields to index. samtools never emits one, but a traceback is a
+ poor way to find that out, and the read after it must still be judged normally."""
+ track = _default_track(tmp_path)
+ res = _run(repo_root, track, HEADER + "\n" + _sam("r", 0, pos=11), "--kmin", str(KMIN))
+ assert res.returncode == 0, res.stderr
+ assert "IndexError" not in res.stderr
+ assert "r" in res.stdout, "the record after the blank line was dropped"
diff --git a/tests/unit/test_format_snps_for_backbone.py b/tests/unit/test_format_snps_for_backbone.py
new file mode 100644
index 0000000..22ccc68
--- /dev/null
+++ b/tests/unit/test_format_snps_for_backbone.py
@@ -0,0 +1,160 @@
+"""bin/format_snps_for_backbone.awk - the last rewrite a SNP gets before the consensus.
+
+It decides the depth the consensus divides by and, more importantly, whether a call stays
+heterozygous. Both FreeBayes paths pipe through it, and until it was pulled out of the process
+script it had no coverage: a stub run skips the script block entirely.
+
+The interesting rule is the het re-validation. `bcftools norm -m -` splits a multiallelic 1/2 into
+biallelic records whose reference index 0 is an artefact with RO near zero. Left alone, that reads
+as a heterozygote and the reference base leaks into the consensus IUPAC code.
+"""
+
+from __future__ import annotations
+
+import subprocess
+
+import pytest
+
+from conftest import BIN
+
+AWK = BIN / "format_snps_for_backbone.awk"
+
+HEADER = (
+ "##fileformat=VCFv4.2\n"
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+)
+
+
+def run(rows, hetmin="0.10", tmp_path=None):
+ """Run the awk program over a small VCF and return the data rows, split into fields."""
+ vcf = tmp_path / "in.vcf"
+ vcf.write_text(HEADER + "".join(rows))
+ out = subprocess.run(["awk", "-v", f"HETMIN={hetmin}", "-f", str(AWK), str(vcf)],
+ capture_output=True, text=True, check=True)
+ return [line.split("\t") for line in out.stdout.splitlines() if not line.startswith("#")]
+
+
+def row(info, gt, dp_field=None, pos=100):
+ fmt_val = gt if dp_field is None else f"{gt}:{dp_field}"
+ fmt = "GT" if dp_field is None else "GT:DP"
+ return f"c\t{pos}\t.\tA\tT\t50\t.\t{info}\t{fmt}\t{fmt_val}\n"
+
+
+def info_of(fields):
+ return dict(kv.split("=", 1) for kv in fields[7].split(";"))
+
+
+class TestHeaders:
+ def test_the_header_passes_through_unchanged(self, tmp_path):
+ vcf = tmp_path / "in.vcf"
+ vcf.write_text(HEADER + row("DP=10;RO=1;AO=9", "1/1", "10"))
+ out = subprocess.run(["awk", "-v", "HETMIN=0.10", "-f", str(AWK), str(vcf)],
+ capture_output=True, text=True, check=True).stdout
+ assert out.startswith("##fileformat=VCFv4.2\n#CHROM\t")
+
+ def test_a_header_only_vcf_produces_no_data_rows(self, tmp_path):
+ assert run([], tmp_path=tmp_path) == []
+
+
+class TestOutputShape:
+ def test_info_is_replaced_with_the_backbone_tags(self, tmp_path):
+ [f] = run([row("DP=30;RO=2;AO=28", "1/1", "30")], tmp_path=tmp_path)
+ assert f[7] == "ADP=30;WT=0;HET=0;HOM=1;NC=0"
+
+ def test_format_is_reduced_to_genotype_and_depth(self, tmp_path):
+ [f] = run([row("DP=30;RO=2;AO=28", "1/1", "30")], tmp_path=tmp_path)
+ assert f[8] == "GT:DP"
+ assert f[9] == "1/1:30"
+
+ def test_the_leading_columns_are_untouched(self, tmp_path):
+ [f] = run([row("DP=30;RO=2;AO=28", "1/1", "30", pos=4411532)], tmp_path=tmp_path)
+ assert f[:7] == ["c", "4411532", ".", "A", "T", "50", "."]
+
+
+class TestDepth:
+ def test_depth_comes_from_info_dp_first(self, tmp_path):
+ [f] = run([row("DP=42;RO=2;AO=28", "1/1", "99")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "42"
+
+ def test_a_lookalike_info_key_is_not_mistaken_for_dp(self, tmp_path):
+ """Keys are anchored, so MQMDP= and ADP= must not be read as the depth."""
+ [f] = run([row("MQMDP=999;ADP=888;RO=4;AO=6", "1/1", "12")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "12", "depth was taken from a look-alike INFO key"
+
+ def test_depth_falls_back_to_the_format_column(self, tmp_path):
+ [f] = run([row("RO=4;AO=6", "1/1", "17")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "17"
+
+ def test_depth_falls_back_to_ro_plus_ao(self, tmp_path):
+ [f] = run([row("RO=4;AO=6", "1/1")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "10"
+
+ def test_depth_never_ends_up_zero(self, tmp_path):
+ """The consensus divides by this, so the last fallback is 1 rather than 0."""
+ [f] = run([row("RO=0;AO=0", "1/1")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "1"
+
+ def test_only_the_first_alt_observation_count_is_used(self, tmp_path):
+ """AO is per-alt; after `bcftools norm -m -` the record is biallelic."""
+ [f] = run([row("RO=4;AO=6,3", "1/1")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "10"
+
+ def test_a_missing_format_depth_still_yields_a_numeric_depth(self, tmp_path):
+ """A '.' in FORMAT/DP must not reach ADP: the consensus reads it as zero depth and gaps."""
+ [f] = run([row("RO=3;AO=7", "0/1", ".")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "10", "ADP must fall through to RO+AO, not carry a '.'"
+
+ def test_an_empty_format_depth_also_falls_through(self, tmp_path):
+ [f] = run([row("RO=0;AO=0", "0/1", ".")], tmp_path=tmp_path)
+ assert info_of(f)["ADP"] == "1"
+
+
+class TestGenotype:
+ def test_a_balanced_heterozygote_stays_heterozygous(self, tmp_path):
+ [f] = run([row("DP=40;RO=20;AO=20", "0/1", "40")], tmp_path=tmp_path)
+ assert f[9].startswith("0/1")
+ assert info_of(f)["HET"] == "1" and info_of(f)["HOM"] == "0"
+
+ def test_a_heterozygote_with_no_reference_support_becomes_homozygous(self, tmp_path):
+ """The multiallelic-split artefact: reference index 0 exists but nothing supports it."""
+ [f] = run([row("DP=30;RO=2;AO=28", "0/1", "30")], tmp_path=tmp_path)
+ assert f[9].startswith("1/1"), "an unsupported reference allele must not read as a het"
+ assert info_of(f)["HET"] == "0" and info_of(f)["HOM"] == "1"
+
+ @pytest.mark.parametrize("gt", ["0/1", "1/0"])
+ def test_both_spellings_of_a_heterozygote_are_re_validated(self, gt, tmp_path):
+ [f] = run([row("DP=30;RO=1;AO=29", gt, "30")], tmp_path=tmp_path)
+ assert f[9].startswith("1/1")
+
+ def test_the_threshold_is_hetmin_and_it_is_exclusive(self, tmp_path):
+ """RO fraction exactly at HETMIN stays a het; below it is rewritten."""
+ at = run([row("DP=100;RO=10;AO=90", "0/1", "100")], tmp_path=tmp_path)[0]
+ below = run([row("DP=100;RO=9;AO=91", "0/1", "100")], tmp_path=tmp_path)[0]
+ assert at[9].startswith("0/1"), "0.10 is not below HETMIN, so it stays heterozygous"
+ assert below[9].startswith("1/1")
+
+ def test_the_threshold_follows_the_hetmin_argument(self, tmp_path):
+ [f] = run([row("DP=100;RO=30;AO=70", "0/1", "100")], hetmin="0.40", tmp_path=tmp_path)
+ assert f[9].startswith("1/1")
+
+ def test_an_already_homozygous_call_is_left_alone(self, tmp_path):
+ [f] = run([row("DP=30;RO=15;AO=15", "1/1", "30")], tmp_path=tmp_path)
+ assert f[9].startswith("1/1")
+ assert info_of(f)["HET"] == "0" and info_of(f)["HOM"] == "1"
+
+ def test_a_genotype_that_is_neither_counts_as_neither(self, tmp_path):
+ """A '.' or a 1/2 that survived is flagged as neither het nor hom rather than guessed at."""
+ [f] = run([row("DP=30;RO=10;AO=20", "1/2", "30")], tmp_path=tmp_path)
+ assert info_of(f)["HET"] == "0" and info_of(f)["HOM"] == "0"
+
+ def test_a_heterozygote_with_no_observations_is_not_rewritten(self, tmp_path):
+ """RO+AO of zero leaves the fraction undefined, so the call is left as it was."""
+ [f] = run([row("DP=30;RO=0;AO=0", "0/1", "30")], tmp_path=tmp_path)
+ assert f[9].startswith("0/1")
+
+
+def test_the_two_freebayes_processes_share_this_one_program():
+ """Both paths must pipe through the same file, or the het rule can drift between them."""
+ variants = (BIN.parent / "modules" / "variants.nf").read_text()
+ assert variants.count("format_snps_for_backbone.awk") == 2
+ assert "gtype=\"1/1\"" not in variants, "the genotype logic is back inline in the module"
diff --git a/tests/unit/test_pathotypr_liftover.py b/tests/unit/test_pathotypr_liftover.py
new file mode 100644
index 0000000..ccbf84d
--- /dev/null
+++ b/tests/unit/test_pathotypr_liftover.py
@@ -0,0 +1,767 @@
+"""Unit tests for bin/pathotypr_liftover.py (alignment-free k-mer coordinate liftover).
+
+The script is a CLI with three subcommands (markers / apply / lift) built on a set of
+pure helpers. The helpers are tested directly; `_lift_chain` is the integration target
+because it takes a plain namespace and RETURNS its result instead of printing it.
+
+Sequences are generated from a seeded `random.Random`, so every test is deterministic
+while still being long and varied enough that a 21-mer is unique by construction.
+"""
+
+from __future__ import annotations
+
+import argparse
+import random
+
+import pytest
+
+from conftest import load_script
+
+lift = load_script("pathotypr_liftover")
+
+COMP = {"A": "T", "C": "G", "G": "C", "T": "A"}
+CODE = {"A": 0, "C": 1, "G": 2, "T": 3}
+
+
+def _rc(seq: str) -> str:
+ """Reverse complement of an ACGT string."""
+ return "".join(COMP[b] for b in reversed(seq))
+
+
+def _random_seq(n: int, seed: int) -> str:
+ rng = random.Random(seed)
+ return "".join(rng.choice("ACGT") for _ in range(n))
+
+
+def _encode(kmer: str) -> int:
+ """The same 2-bit packing `_kmer_unique_map` uses (first base in the high bits)."""
+ code = 0
+ for base in kmer:
+ code = (code << 2) | CODE[base]
+ return code
+
+
+def _write_fasta(path, name: str, seq: str, width: int = 60) -> str:
+ lines = [">%s" % name]
+ lines += [seq[i:i + width] for i in range(0, len(seq), width)]
+ path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+ return str(path)
+
+
+def _chain_args(source_fasta: str, target_fasta: str, **overrides) -> argparse.Namespace:
+ """Every attribute `_lift_chain` reads, with the CLI defaults from `main()`.
+
+ `max_gap` is deliberately None here: the code has an adaptive default
+ (2*k + 2*sample + 2) that only kicks in when the attribute is None.
+ """
+ kwargs = dict(
+ source_fasta=source_fasta,
+ target_fasta=target_fasta,
+ kmer_size=21,
+ sample=1,
+ indel_tol=0,
+ max_gap=None,
+ min_chain=10,
+ min_density=0.2,
+ min_identity=0.9,
+ rd_min=50,
+ rd_out=None,
+ source_contig="source",
+ )
+ kwargs.update(overrides)
+ return argparse.Namespace(**kwargs)
+
+
+# --------------------------------------------------------------------------- _hash64
+
+
+def test_hash64_is_deterministic_and_64_bit():
+ assert lift._hash64(12345) == lift._hash64(12345)
+ assert 0 <= lift._hash64(12345) < (1 << 64)
+ assert lift._hash64(0) != lift._hash64(1)
+
+
+def test_hash64_spreads_small_consecutive_codes():
+ """FracMinHash sampling relies on the hash mixing, not on the raw code order."""
+ values = [lift._hash64(c) % 7 for c in range(200)]
+ assert len(set(values)) == 7
+
+
+# --------------------------------------------------------------------------- _rc_code
+
+
+@pytest.mark.parametrize("k", [1, 2, 3, 4, 8, 21])
+def test_rc_code_is_an_involution(k):
+ """Reverse-complementing twice must return the original 2-bit code."""
+ rng = random.Random(k)
+ codes = [rng.randrange(1 << (2 * k)) for _ in range(50)]
+ for code in codes:
+ assert lift._rc_code(lift._rc_code(code, k), k) == code
+
+
+def test_rc_code_matches_string_reverse_complement():
+ rng = random.Random(3)
+ for _ in range(50):
+ kmer = "".join(rng.choice("ACGT") for _ in range(11))
+ assert lift._rc_code(_encode(kmer), 11) == _encode(_rc(kmer))
+
+
+def test_rc_code_known_values():
+ assert lift._rc_code(_encode("AAAA"), 4) == _encode("TTTT")
+ assert lift._rc_code(_encode("ACGT"), 4) == _encode("ACGT") # self-reverse-complement
+ assert lift._rc_code(_encode("A"), 1) == _encode("T")
+ assert lift._rc_code(_encode("C"), 1) == _encode("G")
+
+
+# -------------------------------------------------------------------- _kmer_unique_map
+
+
+def test_kmer_unique_map_k_longer_than_sequence_is_empty():
+ assert lift._kmer_unique_map("ACGT", 5) == {}
+
+
+def test_kmer_unique_map_all_n_is_empty():
+ assert lift._kmer_unique_map("NNNNNNNNNN", 3) == {}
+
+
+def test_kmer_unique_map_centres_are_one_based():
+ """k=3 over ACGTA: the three k-mers ACG/CGT/GTA have centres 2, 3 and 4."""
+ got = lift._kmer_unique_map("ACGTA", 3)
+ assert got == {_encode("ACG"): 2, _encode("CGT"): 3, _encode("GTA"): 4}
+
+
+def test_kmer_unique_map_drops_a_repeated_kmer():
+ """ACGT occurs twice in ACGTTACGT, so it must not be an anchor."""
+ got = lift._kmer_unique_map("ACGTTACGT", 4)
+ assert _encode("ACGT") not in got
+ assert _encode("CGTT") in got
+
+
+def test_kmer_unique_map_encoding_resets_on_non_acgt():
+ """The rolling code must not bridge an N: ACG and TTT survive, no chimera in between."""
+ got = lift._kmer_unique_map("ACGNTTT", 3)
+ assert got == {_encode("ACG"): 2, _encode("TTT"): 6}
+
+
+def test_kmer_unique_map_lowercase_is_not_recognised():
+ """Callers pass uppercase (via `_read_first_contig`); lowercase resets the code."""
+ assert lift._kmer_unique_map("acgta", 3) == {}
+
+
+def test_kmer_unique_map_sampling_keeps_a_subset():
+ seq = _random_seq(4000, seed=42)
+ full = lift._kmer_unique_map(seq, 21, sample=1)
+ sampled = lift._kmer_unique_map(seq, 21, sample=5)
+ assert 0 < len(sampled) < len(full)
+ assert set(sampled) <= set(full)
+ assert all(full[code] == centre for code, centre in sampled.items())
+
+
+# ------------------------------------------------------------------------------- _lis
+
+
+def test_lis_empty_input():
+ assert lift._lis([]) == ([], [])
+
+
+def test_lis_strictly_decreasing_input_gives_a_length_one_chain():
+ ca, cb = lift._lis([(1, 5), (2, 4), (3, 3)], increasing=True)
+ assert len(ca) == 1 and len(cb) == 1
+
+
+def test_lis_decreasing_mode_takes_the_whole_run():
+ assert lift._lis([(1, 5), (2, 4), (3, 3)], increasing=False) == ([1, 2, 3], [5, 4, 3])
+
+
+def test_lis_worked_example():
+ pairs = [(1, 1), (2, 5), (3, 2), (4, 3), (5, 9)]
+ assert lift._lis(pairs, increasing=True) == ([1, 3, 4, 5], [1, 2, 3, 9])
+
+
+def test_lis_is_strict_so_ties_do_not_extend_a_chain():
+ ca, _ = lift._lis([(1, 5), (2, 5), (3, 5)], increasing=True)
+ assert len(ca) == 1
+
+
+# ------------------------------------------------------------------------ _homologous
+
+
+def test_homologous_identity_context_passes():
+ seq = _random_seq(400, seed=1)
+ assert lift._homologous(seq, seq, 200, 200, 1, 10, 0.9) is True
+
+
+def test_homologous_rejects_a_shifted_coordinate():
+ seq = _random_seq(400, seed=1)
+ assert lift._homologous(seq, seq, 200, 240, 1, 10, 0.9) is False
+
+
+def test_homologous_checks_each_side_independently():
+ """Right half at 0.4 identity, left half at 1.0: the aggregate (0.7) clears 0.6 but
+ the per-side rule must still reject it."""
+ seq = _random_seq(400, seed=2)
+ mutated = list(seq)
+ for i in range(200, 206): # centre p=200 -> right side is 0-based 200..209
+ mutated[i] = "A" if seq[i] != "A" else "C"
+ mutated = "".join(mutated)
+ assert lift._homologous(seq, mutated, 200, 200, 1, 10, 0.6) is False
+ assert lift._homologous(seq, mutated, 200, 200, 1, 10, 0.3) is True
+
+
+def test_homologous_reverse_orientation_uses_the_reverse_complement():
+ seq = _random_seq(400, seed=3)
+ rc_seq = _rc(seq)
+ p = 200
+ t = len(seq) - p + 1 # reflected 1-based coordinate
+ assert lift._homologous(seq, rc_seq, p, t, -1, 10, 0.9) is True
+ assert lift._homologous(seq, rc_seq, p, t, 1, 10, 0.9) is False
+
+
+def test_homologous_refuses_a_trivial_pass_at_a_sequence_edge():
+ """Only 4 in-range comparisons against a half-window of 10 -> too little context."""
+ short = "ACGTA"
+ assert lift._homologous(short, short, 3, 3, 1, 10, 0.9) is False
+
+
+def test_homologous_accepts_a_genome_edge_when_one_full_side_is_in_range():
+ """The width rule is `left + right < w`, so a centre at position 1 of a long sequence
+ still passes on its right side alone."""
+ seq = _random_seq(400, seed=4)
+ assert lift._homologous(seq, seq, 1, 1, 1, 10, 0.9) is True
+
+
+# ---------------------------------------------------------------------------- _chains
+
+
+def test_chains_extracts_successive_blocks():
+ pairs = sorted([(1, 10), (2, 9), (3, 8), (4, 7), (10, 100), (11, 99), (12, 98)])
+ assert lift._chains(pairs, increasing=False, min_anchors=3) == [
+ ([1, 2, 3, 4], [10, 9, 8, 7]),
+ ([10, 11, 12], [100, 99, 98]),
+ ]
+
+
+def test_chains_stops_below_min_anchors():
+ pairs = sorted([(1, 10), (2, 9), (3, 8), (4, 7), (10, 100), (11, 99), (12, 98)])
+ assert lift._chains(pairs, increasing=False, min_anchors=5) == []
+
+
+def test_chains_respects_max_chains():
+ pairs = sorted([(1, 10), (2, 9), (3, 8), (4, 7), (10, 100), (11, 99), (12, 98)])
+ assert lift._chains(pairs, increasing=False, min_anchors=3, max_chains=1) == [
+ ([1, 2, 3, 4], [10, 9, 8, 7])
+ ]
+
+
+def test_chains_on_empty_input():
+ assert lift._chains([], increasing=True, min_anchors=2) == []
+
+
+# --------------------------------------------------------------------- _write_outputs
+
+
+def test_write_outputs_writes_a_sorted_map(tmp_path):
+ good = {20: 500, 5: 100, 6: 101}
+ out_map = tmp_path / "lift.map"
+ lift._write_outputs(good, str(out_map), None, "ctg", "feat")
+ assert out_map.read_text().splitlines() == [
+ "src_pos\ttgt_pos",
+ "5\t100",
+ "6\t101",
+ "20\t500",
+ ]
+
+
+def test_write_outputs_bed_collapses_runs_and_is_zero_based_start(tmp_path):
+ """Targets 100-102 and 500 collapse into two rows. The BED start is `start - 1`
+ (0-based) while the end is the last 1-based position, i.e. an inclusive end that
+ doubles as the 0-based half-open end."""
+ good = {5: 100, 6: 101, 7: 102, 20: 500}
+ out_bed = tmp_path / "lift.bed"
+ lift._write_outputs(good, None, str(out_bed), "ctg", "feat")
+ assert out_bed.read_text() == "ctg\t99\t102\tfeat\nctg\t499\t500\tfeat\n"
+
+
+def test_write_outputs_bed_single_position_run(tmp_path):
+ out_bed = tmp_path / "one.bed"
+ lift._write_outputs({7: 42}, None, str(out_bed), "chrom", "name")
+ assert out_bed.read_text() == "chrom\t41\t42\tname\n"
+
+
+def test_write_outputs_deduplicates_targets(tmp_path):
+ """Two sources landing on the same target give one BED position, not two."""
+ out_bed = tmp_path / "dup.bed"
+ lift._write_outputs({1: 10, 2: 10, 3: 11}, None, str(out_bed), "ctg", "feat")
+ assert out_bed.read_text() == "ctg\t9\t11\tfeat\n"
+
+
+def test_write_outputs_empty_mapping(tmp_path):
+ out_map = tmp_path / "empty.map"
+ out_bed = tmp_path / "empty.bed"
+ lift._write_outputs({}, str(out_map), str(out_bed), "ctg", "feat")
+ assert out_map.read_text() == "src_pos\ttgt_pos\n" # header only
+ assert out_bed.read_text() == "" # no rows at all
+
+
+def test_write_outputs_skips_files_that_were_not_requested(tmp_path):
+ lift._write_outputs({1: 2}, None, None, "ctg", "feat")
+ assert list(tmp_path.iterdir()) == []
+
+
+# ----------------------------------------------------------------- _read_first_contig
+
+
+def test_read_first_contig_reads_only_the_first_record(tmp_path):
+ fasta = tmp_path / "two.fa"
+ fasta.write_text(">one\nacgt\nAACC\n>two\nTTTTTTTT\n")
+ assert lift._read_first_contig(fasta) == "ACGTAACC"
+
+
+def test_read_first_contig_ignores_leading_content_without_a_header(tmp_path):
+ fasta = tmp_path / "noheader.fa"
+ fasta.write_text("ACGT\nACGT\n")
+ assert lift._read_first_contig(fasta) == ""
+
+
+def test_read_first_contig_strips_whitespace(tmp_path):
+ fasta = tmp_path / "ws.fa"
+ fasta.write_text(">c\n ACGT \n\nAC GT\n")
+ # Only leading/trailing whitespace is stripped; an interior space is kept verbatim.
+ assert lift._read_first_contig(fasta) == "ACGTAC GT"
+
+
+# ------------------------------------------------------------------------- _positions
+
+
+def test_positions_plain_list_is_sorted_and_deduplicated(tmp_path):
+ path = tmp_path / "pos.txt"
+ path.write_text("7\n5\n7\n\n1\n")
+ assert lift._positions(path) == [1, 5, 7]
+
+
+def test_positions_expands_a_bed_interval_to_one_based(tmp_path):
+ """BED 0-based half-open [10, 13) becomes 1-based 11, 12, 13."""
+ path = tmp_path / "regions.bed"
+ path.write_text("chr1\t10\t13\n")
+ assert lift._positions(path) == [11, 12, 13]
+
+
+def test_positions_accepts_extra_bed_columns(tmp_path):
+ path = tmp_path / "named.bed"
+ path.write_text("chr1\t10\t13\tname\t0\t+\n")
+ assert lift._positions(path) == [11, 12, 13]
+
+
+def test_positions_treats_commas_as_tabs(tmp_path):
+ path = tmp_path / "csv.bed"
+ path.write_text("chr1,20,22\n")
+ assert lift._positions(path) == [21, 22]
+
+
+def test_positions_skips_header_and_comment_lines(tmp_path):
+ path = tmp_path / "hdr.bed"
+ path.write_text("# a comment\ntrack name=x\nChrom\tStart\tEnd\nchr1\t0\t2\n")
+ assert lift._positions(path) == [1, 2]
+
+
+def test_positions_skips_unparseable_rows(tmp_path):
+ path = tmp_path / "junk.txt"
+ path.write_text("chr1\tfoo\tbar\nnot-a-number\n9\n")
+ assert lift._positions(path) == [9]
+
+
+def test_positions_empty_file(tmp_path):
+ path = tmp_path / "none.txt"
+ path.write_text("")
+ assert lift._positions(path) == []
+
+
+def test_positions_rejects_a_negative_bed_coordinate(tmp_path):
+ """A BED coordinate is never negative. Accepting one expanded to zero and negative
+ positions that no reference has; the row is treated as malformed instead."""
+ path = tmp_path / "neg.bed"
+ path.write_text("chr1\t-2\t1\n")
+ assert lift._positions(path) == []
+
+def test_positions_still_expands_a_valid_bed_interval(tmp_path):
+ """The guard must not reject well-formed rows: BED is 0-based half-open."""
+ path = tmp_path / "ok.bed"
+ path.write_text("chr1\t2\t5\n")
+ assert lift._positions(path) == [3, 4, 5]
+
+
+# ------------------------------------------------------- _lift_chain (integration)
+
+
+def test_lift_chain_identity_maps_positions_onto_themselves(tmp_path):
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ args = _chain_args(src, tgt)
+
+ positions = [50, 100, 200, 300, 500]
+ good, stats = lift._lift_chain(args, positions)
+
+ assert good == {p: p for p in positions}
+ assert stats["fwd"] == len(positions)
+ assert stats["rev"] == 0
+ assert stats["drop"] == 0
+ assert stats["rd"] == 0
+ assert stats["anchors"] > 0
+
+
+def test_lift_chain_drops_positions_outside_the_anchored_span(tmp_path):
+ """No extrapolation: the last k/2 bases have no k-mer centre, so they are past the
+ chain's last anchor and drop instead of receiving a smeared coordinate."""
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ good, stats = lift._lift_chain(_chain_args(src, tgt), [1, 600])
+ assert good == {}
+ assert stats["drop"] == 2
+
+
+def test_lift_chain_tracks_an_insertion_in_the_target(tmp_path):
+ """Positions before the insertion keep their coordinate; positions after it shift by
+ exactly the inserted length."""
+ seq = _random_seq(600, seed=11)
+ insert = "GGATCCTTAAGGCCTTAACCGGTTAACCGGTTAACC"
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq[:300] + insert + seq[300:])
+ args = _chain_args(src, tgt)
+
+ good, stats = lift._lift_chain(args, [50, 100, 250, 350, 400, 500])
+
+ assert good[50] == 50
+ assert good[100] == 100
+ assert good[250] == 250
+ assert good[350] == 350 + len(insert)
+ assert good[400] == 400 + len(insert)
+ assert good[500] == 500 + len(insert)
+ assert stats["fwd"] == 6
+ assert stats["drop"] == 0
+
+
+def test_lift_chain_maps_an_inversion_to_the_reflected_coordinate(tmp_path):
+ """The target carries seq[200:400] reverse-complemented, so a source position p in
+ 201..400 lifts to 601 - p via a reverse chain."""
+ seq = _random_seq(600, seed=11)
+ inverted = seq[:200] + _rc(seq[200:400]) + seq[400:]
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", inverted)
+
+ good, stats = lift._lift_chain(_chain_args(src, tgt), [50, 250, 300, 350, 500])
+
+ assert good[50] == 50 and good[500] == 500 # collinear flanks
+ assert good[250] == 601 - 250
+ assert good[300] == 601 - 300
+ assert good[350] == 601 - 350
+ assert stats["inv_chains"] == 1
+ assert stats["rev"] == 3
+ assert stats["fwd"] == 2
+
+
+def test_lift_chain_reports_an_rd_deletion(tmp_path):
+ """100 bp missing from the target is >= rd_min, so it is written to --rd-out in
+ SOURCE coordinates as the two flanking anchor centres."""
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq[:200] + seq[300:])
+ rd_out = tmp_path / "rd.bed"
+ args = _chain_args(src, tgt, rd_out=str(rd_out), source_contig="H37Rv")
+
+ good, stats = lift._lift_chain(args, [50, 100, 250])
+
+ assert stats["rd"] == 1
+ rows = [line.split("\t") for line in rd_out.read_text().splitlines()]
+ assert len(rows) == 1
+ contig, left, right, tag = rows[0]
+ assert contig == "H37Rv" and tag == "RD_deletion"
+ assert int(left) < 200 < 300 < int(right) # the anchors bracket the lost block
+ assert good[50] == 50 and good[100] == 100 # flank unaffected
+ assert 250 not in good # inside the deleted block -> dropped
+
+
+def test_lift_chain_does_not_write_rd_out_when_unset(tmp_path):
+ seq = _random_seq(400, seed=5)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ lift._lift_chain(_chain_args(src, tgt), [100])
+ assert sorted(p.name for p in tmp_path.iterdir()) == ["src.fa", "tgt.fa"]
+
+
+def test_lift_chain_sampling_keeps_the_same_coordinates(tmp_path):
+ """FracMinHash sampling thins the anchors but must not move a lifted coordinate."""
+ seq = _random_seq(600, seed=11)
+ insert = "GGATCCTTAAGGCCTTAACCGGTTAACCGGTTAACC"
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq[:300] + insert + seq[300:])
+ positions = [50, 100, 250, 400, 500]
+
+ dense, dense_stats = lift._lift_chain(_chain_args(src, tgt), positions)
+ sparse, sparse_stats = lift._lift_chain(_chain_args(src, tgt, sample=4), positions)
+
+ assert sparse == dense
+ assert sparse_stats["anchors"] < dense_stats["anchors"]
+
+
+def test_lift_chain_on_unrelated_sequences_lifts_nothing(tmp_path):
+ src = _write_fasta(tmp_path / "src.fa", "A", _random_seq(600, seed=11))
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", _random_seq(600, seed=99))
+ good, stats = lift._lift_chain(_chain_args(src, tgt), [100, 200, 300])
+ assert good == {}
+ assert stats["drop"] == 3
+
+
+# ------------------------------------------------------------------------- cmd_lift
+
+
+def _lift_args(tmp_path, positions_file, source_fasta, target_fasta, **overrides):
+ """Every attribute `cmd_lift` reads, with the CLI defaults from `main()`."""
+ kwargs = dict(
+ positions=positions_file,
+ source_fasta=source_fasta,
+ target_fasta=target_fasta,
+ out_map=str(tmp_path / "lift.map"),
+ out_bed=str(tmp_path / "lift.bed"),
+ contig="target",
+ name="blindspot",
+ kmer_size=21,
+ max_shift=100,
+ min_margin=20,
+ global_chain=False,
+ )
+ kwargs.update(overrides)
+ return argparse.Namespace(**kwargs)
+
+
+def test_cmd_lift_anchors_unique_context_kmers(tmp_path, capsys):
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ positions = tmp_path / "pos.txt"
+ positions.write_text("100\n101\n300\n")
+
+ args = _lift_args(tmp_path, str(positions), src, tgt)
+ lift.cmd_lift(args)
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n100\t100\n101\t101\n300\t300\n"
+ assert (tmp_path / "lift.bed").read_text() == "target\t99\t101\tblindspot\ntarget\t299\t300\tblindspot\n"
+ assert "3 anchored + 0 recovered-by-synteny" in capsys.readouterr().err
+
+
+def test_cmd_lift_recovers_a_repeated_kmer_by_synteny(tmp_path, capsys):
+ """The target carries a second copy of seq[100:200], so the context k-mer of 150 is
+ no longer unique there. The nearest anchor's offset predicts the syntenic occurrence
+ and the position is recovered instead of dropped.
+
+ Note that 90 and 250 are in the request only to serve as anchors: the anchor set is
+ built from the REQUESTED positions, not from the whole genome (see the next test)."""
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq + seq[100:200])
+ positions = tmp_path / "pos.txt"
+ positions.write_text("90\n150\n250\n")
+
+ lift.cmd_lift(_lift_args(tmp_path, str(positions), src, tgt))
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n90\t90\n150\t150\n250\t250\n"
+ assert "2 anchored + 1 recovered-by-synteny" in capsys.readouterr().err
+
+
+def test_cmd_lift_synteny_needs_other_requested_positions_as_anchors(tmp_path, capsys):
+ """Pinning current behaviour: `predict()` only sees anchors derived from the
+ requested position list, so asking for a single repeated position on its own drops it
+ even though the genome is full of usable anchors. Lifting a whole BED is unaffected;
+ lifting one coordinate at a time is not equivalent to lifting them together."""
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq + seq[100:200])
+ positions = tmp_path / "pos.txt"
+ positions.write_text("150\n")
+
+ lift.cmd_lift(_lift_args(tmp_path, str(positions), src, tgt))
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n"
+ assert "0 anchored + 0 recovered-by-synteny" in capsys.readouterr().err
+
+
+def test_cmd_lift_drops_a_position_whose_kmer_is_absent_from_the_target(tmp_path, capsys):
+ src = _write_fasta(tmp_path / "src.fa", "A", _random_seq(600, seed=11))
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", _random_seq(600, seed=99))
+ positions = tmp_path / "pos.txt"
+ positions.write_text("300\n")
+
+ lift.cmd_lift(_lift_args(tmp_path, str(positions), src, tgt))
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n"
+ assert (tmp_path / "lift.bed").read_text() == ""
+ assert "1 dropped (of 1)" in capsys.readouterr().err
+
+
+def test_cmd_lift_skips_positions_without_a_full_context_window(tmp_path, capsys):
+ """A position closer than k//2 to either end has no centred k-mer, so it never even
+ enters the candidate set (it is not counted as a drop either)."""
+ seq = _random_seq(600, seed=11)
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ positions = tmp_path / "pos.txt"
+ positions.write_text("1\n5\n600\n300\n")
+
+ lift.cmd_lift(_lift_args(tmp_path, str(positions), src, tgt))
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n300\t300\n"
+ assert "1 dropped (of 1)" not in capsys.readouterr().err
+
+
+def test_cmd_lift_global_chain_dispatches_to_lift_chain(tmp_path, capsys):
+ """--global-chain places positions by anchor interpolation, so it also lifts a
+ position whose own context k-mer was destroyed by a nearby insertion."""
+ seq = _random_seq(600, seed=11)
+ insert = "GGATCCTTAAGGCCTTAACCGGTTAACCGGTTAACC"
+ src = _write_fasta(tmp_path / "src.fa", "A", seq)
+ tgt = _write_fasta(tmp_path / "tgt.fa", "B", seq[:300] + insert + seq[300:])
+ positions = tmp_path / "pos.txt"
+ positions.write_text("100\n400\n")
+
+ args = _lift_args(tmp_path, str(positions), src, tgt, global_chain=True,
+ sample=1, indel_tol=0, max_gap=None, min_chain=10, min_density=0.2,
+ min_identity=0.9, rd_min=50, rd_out=None, source_contig="A")
+ lift.cmd_lift(args)
+
+ assert (tmp_path / "lift.map").read_text() == "src_pos\ttgt_pos\n100\t100\n400\t436\n"
+ assert "lift(anchor-chain): 2 fwd + 0 inverted = 2 lifted" in capsys.readouterr().err
+
+
+# --------------------------------------------------------------------- cmd_markers
+
+
+def test_cmd_markers_writes_header_and_reference_bases(tmp_path, capsys):
+ seq = _random_seq(100, seed=7)
+ fasta = _write_fasta(tmp_path / "ref.fa", "REF", seq)
+ positions = tmp_path / "pos.txt"
+ positions.write_text("5\n50\n")
+ out = tmp_path / "markers.tsv"
+
+ lift.cmd_markers(argparse.Namespace(fasta=fasta, positions=str(positions),
+ out=str(out), level="lift"))
+
+ assert out.read_text() == "pos\talt\tlevel\n5\t%s\tlift\n50\t%s\tlift\n" % (seq[4], seq[49])
+ assert "2 written, 0 skipped" in capsys.readouterr().err
+
+
+def test_cmd_markers_skips_out_of_range_and_non_acgt(tmp_path, capsys):
+ seq = "N" + _random_seq(99, seed=7)[1:]
+ fasta = _write_fasta(tmp_path / "ref.fa", "REF", seq)
+ positions = tmp_path / "pos.txt"
+ positions.write_text("0\n1\n5\n1000\n")
+ out = tmp_path / "markers.tsv"
+
+ lift.cmd_markers(argparse.Namespace(fasta=fasta, positions=str(positions),
+ out=str(out), level="lineage4"))
+
+ rows = out.read_text().splitlines()
+ assert rows[0] == "pos\talt\tlevel"
+ assert rows[1:] == ["5\t%s\tlineage4" % seq[4]] # 0 and 1000 out of range, 1 is an N
+ assert "1 written, 3 skipped" in capsys.readouterr().err
+
+
+# ------------------------------------------------------------------------ cmd_apply
+
+
+def _classify_rows(rows) -> str:
+ return "".join("\t".join(r) + "\n" for r in rows)
+
+
+def test_cmd_apply_writes_map_and_bed_with_offset(tmp_path, capsys):
+ classify = tmp_path / "classify.tsv"
+ classify.write_text(_classify_rows([
+ ("genome", "ACGT", "0", "1000", "111", "lin"),
+ ("genome", "ACGT", "0", "1001", "112", "lin"),
+ ("genome", "TTTT", "0", "2000", "500", "lin"),
+ ("truncated", "row"), # < 5 columns -> ignored
+ ("genome", "ACGT", "0", "notanint", "113", "lin"), # unparseable -> ignored
+ ]))
+ out_map = tmp_path / "out.map"
+ out_bed = tmp_path / "out.bed"
+
+ lift.cmd_apply(argparse.Namespace(
+ classify=str(classify), target_fasta=None, source_fasta=None, kmer_size=4,
+ offset=1, out_map=str(out_map), out_bed=str(out_bed), contig="ANCESTOR",
+ name="blindspot"))
+
+ assert out_map.read_text() == "src_pos\ttgt_pos\n111\t1001\n112\t1002\n500\t2001\n"
+ # 1001 and 1002 are consecutive -> one run; 2001 is its own run.
+ assert out_bed.read_text() == "ANCESTOR\t1000\t1002\tblindspot\nANCESTOR\t2000\t2001\tblindspot\n"
+ assert "3 lifted" in capsys.readouterr().err
+
+
+def test_cmd_apply_drops_a_source_mapping_to_two_targets(tmp_path, capsys):
+ classify = tmp_path / "classify.tsv"
+ classify.write_text(_classify_rows([
+ ("genome", "ACGT", "0", "1000", "111", "lin"),
+ ("genome", "ACGT", "0", "2000", "111", "lin"), # same src, different tgt -> ambiguous
+ ("genome", "TTTT", "0", "3000", "222", "lin"),
+ ("genome", "TTTT", "0", "3000", "222", "lin"), # repeated but consistent -> kept
+ ]))
+ out_map = tmp_path / "out.map"
+
+ lift.cmd_apply(argparse.Namespace(
+ classify=str(classify), target_fasta=None, source_fasta=None, kmer_size=4,
+ offset=0, out_map=str(out_map), out_bed=None, contig="ctg", name="feat"))
+
+ assert out_map.read_text() == "src_pos\ttgt_pos\n222\t3000\n"
+ assert "1 ambiguous" in capsys.readouterr().err
+
+
+def test_cmd_apply_drops_kmers_that_recur_in_the_target(tmp_path, capsys):
+ seq = _random_seq(600, seed=13)
+ unique_kmer = seq[100:121]
+ duplicated_kmer = seq[200:221]
+ source = _write_fasta(tmp_path / "src.fa", "A", seq)
+ # The target carries a second copy of `duplicated_kmer`, so its coordinate is ambiguous.
+ target = _write_fasta(tmp_path / "tgt.fa", "B", seq + duplicated_kmer)
+ classify = tmp_path / "classify.tsv"
+ classify.write_text(_classify_rows([
+ ("genome", unique_kmer, "0", "111", "111", "lin"),
+ ("genome", duplicated_kmer, "0", "211", "211", "lin"),
+ ]))
+ out_map = tmp_path / "out.map"
+
+ lift.cmd_apply(argparse.Namespace(
+ classify=str(classify), target_fasta=target, source_fasta=source, kmer_size=21,
+ offset=0, out_map=str(out_map), out_bed=None, contig="ctg", name="feat"))
+
+ assert out_map.read_text() == "src_pos\ttgt_pos\n111\t111\n"
+ assert "1 lifted, 1 dropped" in capsys.readouterr().err
+
+
+def test_cmd_apply_drops_kmers_that_recur_in_the_source(tmp_path, capsys):
+ seq = _random_seq(600, seed=13)
+ duplicated_kmer = seq[300:321]
+ source = _write_fasta(tmp_path / "src.fa", "A", seq + duplicated_kmer)
+ target = _write_fasta(tmp_path / "tgt.fa", "B", seq)
+ classify = tmp_path / "classify.tsv"
+ classify.write_text(_classify_rows([
+ ("genome", duplicated_kmer, "0", "311", "311", "lin"),
+ ]))
+ out_map = tmp_path / "out.map"
+
+ lift.cmd_apply(argparse.Namespace(
+ classify=str(classify), target_fasta=target, source_fasta=source, kmer_size=21,
+ offset=0, out_map=str(out_map), out_bed=None, contig="ctg", name="feat"))
+
+ assert out_map.read_text() == "src_pos\ttgt_pos\n"
+ assert "0 lifted, 1 dropped" in capsys.readouterr().err
+
+
+def test_cmd_apply_with_no_outputs_requested_writes_nothing(tmp_path, capsys):
+ classify = tmp_path / "classify.tsv"
+ classify.write_text(_classify_rows([("genome", "ACGT", "0", "10", "20", "lin")]))
+
+ lift.cmd_apply(argparse.Namespace(
+ classify=str(classify), target_fasta=None, source_fasta=None, kmer_size=4,
+ offset=0, out_map=None, out_bed=None, contig="ctg", name="feat"))
+
+ assert sorted(p.name for p in tmp_path.iterdir()) == ["classify.tsv"]
+ assert "1 lifted" in capsys.readouterr().err
diff --git a/tests/unit/test_qc_report_logic.py b/tests/unit/test_qc_report_logic.py
new file mode 100644
index 0000000..76c133c
--- /dev/null
+++ b/tests/unit/test_qc_report_logic.py
@@ -0,0 +1,1063 @@
+"""Unit tests for the QC verdict engine and the derived statistics of the qcreport package.
+
+The file parsers live in test_qc_report_parsers.py; this module covers the pure
+in-memory logic: flag_sample, the robust statistics, the lineage/het derivations,
+the small VCF field decoders and the three payload builders.
+
+Every name is reached through the module that OWNS it (qcreport.metrics for the
+thresholds and the verdict engine, qcreport.panels for the payload builders,
+qcreport.parsers for the small decoders), so a test says where the code lives.
+
+Every expectation here is pinned against the CURRENT behaviour of the script, and
+the thresholds are read from qc_metrics.DEF / ANC_DEF rather than hardcoded, so a
+deliberate change to a default shows up as a behaviour change and not as a
+spurious failure.
+"""
+
+from __future__ import annotations
+
+import subprocess
+import sys
+
+import pytest
+from conftest import BIN, load_script
+from qcreport import metrics as qc_metrics, panels as qc_panels, parsers as qc_parsers
+
+THR = dict(qc_metrics.DEF)
+ANC = dict(qc_metrics.ANC_DEF)
+
+
+def metrics(**over):
+ """A summary row that PASSes every default check, overridable per test."""
+ m = {
+ "mean_depth": "50",
+ "breadth_pct": "99",
+ "missing_pct": "1",
+ "mapped_pct": "95",
+ "duplication_pct": "5",
+ "iupac_pct": "0.5",
+ "ti_tv": "1.8",
+ "snps": "1000",
+ }
+ m.update(over)
+ return m
+
+
+# --------------------------------------------------------------------------- flag_sample: verdicts
+
+
+def test_flag_sample_clean_row_passes_with_no_flags():
+ assert qc_metrics.flag_sample(metrics(), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_all_missing_metrics_is_a_hard_no_data_fail():
+ assert qc_metrics.flag_sample({}, THR, None, None) == ("FAIL", ["NO_DATA"])
+
+
+def test_flag_sample_na_sentinels_count_as_missing_for_the_no_data_guard():
+ row = {"mean_depth": "NA", "breadth_pct": "", "missing_pct": ".", "mapped_pct": None}
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("FAIL", ["NO_DATA"])
+
+
+def test_flag_sample_no_data_guard_only_looks_at_its_four_keys():
+ # duplication_pct alone is not evidence that the sample produced anything, so the
+ # guard still fires and the HIGH_DUP check is never reached.
+ assert qc_metrics.flag_sample({"duplication_pct": "99"}, THR, None, None) == ("FAIL", ["NO_DATA"])
+
+
+@pytest.mark.parametrize(
+ ("key", "value", "flag"),
+ [
+ ("mean_depth", qc_metrics.DEF["depth_min"] - 1, "LOW_DEPTH"),
+ ("breadth_pct", qc_metrics.DEF["breadth_min"] - 1, "LOW_BREADTH"),
+ ("missing_pct", qc_metrics.DEF["missing_max"] + 1, "HIGH_MISSING"),
+ ],
+)
+def test_flag_sample_fail_flags_drive_a_fail_verdict(key, value, flag):
+ verdict, flags = qc_metrics.flag_sample(metrics(**{key: value}), THR, None, None)
+ assert flags == [flag]
+ assert flag in qc_metrics.FAIL_FLAGS
+ assert verdict == "FAIL"
+
+
+@pytest.mark.parametrize(
+ ("key", "value", "flag"),
+ [
+ ("mapped_pct", qc_metrics.DEF["mapping_min"] - 1, "MAPPING_LOW"),
+ ("duplication_pct", qc_metrics.DEF["dup_max"] + 1, "HIGH_DUP"),
+ ("iupac_pct", qc_metrics.DEF["iupac_max"] + 1, "HIGH_IUPAC"),
+ ],
+)
+def test_flag_sample_soft_flags_only_warn(key, value, flag):
+ verdict, flags = qc_metrics.flag_sample(metrics(**{key: value}), THR, None, None)
+ assert flags == [flag]
+ assert flag not in qc_metrics.FAIL_FLAGS
+ assert verdict == "WARN"
+
+
+@pytest.mark.parametrize(
+ ("key", "value"),
+ [
+ ("mean_depth", qc_metrics.DEF["depth_min"]), # strict <, so exactly at the floor passes
+ ("breadth_pct", qc_metrics.DEF["breadth_min"]), # strict <
+ ("missing_pct", qc_metrics.DEF["missing_max"]), # strict >, so exactly at the ceiling passes
+ ("mapped_pct", qc_metrics.DEF["mapping_min"]), # strict <
+ ("duplication_pct", qc_metrics.DEF["dup_max"]), # strict >
+ ("iupac_pct", qc_metrics.DEF["iupac_max"]), # strict >
+ ],
+)
+def test_flag_sample_threshold_boundaries_are_inclusive_of_the_threshold(key, value):
+ assert qc_metrics.flag_sample(metrics(**{key: value}), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_multiple_problems_accumulate_and_a_single_fail_flag_wins():
+ verdict, flags = qc_metrics.flag_sample(
+ metrics(mean_depth="1", duplication_pct="99", iupac_pct="50"), THR, None, None)
+ assert flags == ["LOW_DEPTH", "HIGH_DUP", "HIGH_IUPAC"]
+ assert verdict == "FAIL"
+
+
+def test_flag_sample_missing_single_metric_is_simply_not_checked():
+ row = metrics()
+ del row["duplication_pct"]
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: Ti/Tv
+
+
+def test_flag_sample_titv_check_is_disabled_at_the_default_threshold_of_zero():
+ assert qc_metrics.DEF["titv_min"] == 0.0
+ assert qc_metrics.flag_sample(metrics(ti_tv="0.3"), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_titv_low_warns_once_the_threshold_is_raised():
+ thr = dict(THR, titv_min=1.5)
+ assert qc_metrics.flag_sample(metrics(ti_tv="1.2"), thr, None, None) == ("WARN", ["TITV_LOW"])
+ assert qc_metrics.flag_sample(metrics(ti_tv="1.5"), thr, None, None) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: SNP outliers
+
+
+def test_flag_sample_snp_low_and_high_are_cohort_z_outliers():
+ med, sig = 1000.0, 10.0
+ span = THR["snp_z"] * sig
+ assert qc_metrics.flag_sample(metrics(snps=med - span - 1), THR, med, sig) == ("WARN", ["SNP_LOW"])
+ assert qc_metrics.flag_sample(metrics(snps=med + span + 1), THR, med, sig) == ("WARN", ["SNP_HIGH"])
+ # both comparisons are strict, so a sample exactly z sigma out is not an outlier
+ assert qc_metrics.flag_sample(metrics(snps=med - span), THR, med, sig) == ("PASS", [])
+ assert qc_metrics.flag_sample(metrics(snps=med + span), THR, med, sig) == ("PASS", [])
+
+
+def test_flag_sample_snp_check_is_skipped_without_a_cohort_scale():
+ assert qc_metrics.flag_sample(metrics(snps="999999"), THR, 1000.0, None) == ("PASS", [])
+ assert qc_metrics.flag_sample(metrics(snps="999999"), THR, 1000.0, 0.0) == ("PASS", [])
+ assert qc_metrics.flag_sample(metrics(snps="999999"), THR, None, 10.0) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: het fraction
+
+
+def test_flag_sample_het_high_uses_the_het_fraction_as_a_percent():
+ # 5 het / 100 variant genotypes = 5%, above the 3% default
+ row = metrics(het_variants="5", homo_variants="95")
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("WARN", ["HET_HIGH"])
+ assert qc_metrics.flag_sample(metrics(het_variants="2", homo_variants="98"), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_het_check_is_skipped_below_the_minimum_variant_count():
+ # 50% het, but only 10 variant genotypes -> het_frac() returns None, so no flag
+ row = metrics(het_variants="5", homo_variants="5")
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: MIXED
+
+
+def test_flag_sample_mixed_needs_two_lineages_over_both_the_fraction_and_the_raw_count():
+ row = metrics(lineage_counts="L4:100;L2:10")
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("WARN", ["MIXED"])
+
+
+def test_flag_sample_mixed_ignores_a_minor_lineage_below_three_markers():
+ # 2 markers is 1.96% of 102 and also below the >= 3 raw-count guard
+ assert qc_metrics.flag_sample(metrics(lineage_counts="L4:100;L2:2"), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_mixed_ignores_a_minor_lineage_below_the_fraction_cutoff():
+ # 3 markers clears the raw count but is 0.03% of the total, below mixed_min_frac
+ assert qc_metrics.flag_sample(metrics(lineage_counts="L4:10000;L2:3"), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_mixed_is_not_raised_by_two_sublineages_of_the_same_lineage():
+ assert qc_metrics.flag_sample(metrics(lineage_counts="L4.1:100;L4.3:100"), THR, None, None) == ("PASS", [])
+
+
+def test_flag_sample_mixed_is_skipped_when_there_are_no_lineage_counts():
+ assert qc_metrics.flag_sample(metrics(lineage_counts="NA"), THR, None, None) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: ancient view
+
+
+def test_flag_sample_ancient_row_is_judged_against_the_adna_thresholds():
+ # 5x depth / 40% breadth / 50% missing FAILs the modern gate but is fine for aDNA
+ row = metrics(mean_depth="5", breadth_pct="40", missing_pct="50", mapped_pct="25",
+ duplication_pct="55", iupac_pct="4")
+ assert qc_metrics.flag_sample(row, THR, None, None)[0] == "FAIL"
+ assert qc_metrics.flag_sample(row, THR, None, None, ancient=True, anc_thr=ANC) == ("PASS", [])
+
+
+def test_flag_sample_ancient_falls_back_to_the_modern_thresholds_without_anc_thr():
+ row = metrics(mean_depth="5")
+ assert qc_metrics.flag_sample(row, THR, None, None, ancient=True, anc_thr=None) == ("FAIL", ["LOW_DEPTH"])
+
+
+def test_flag_sample_ancient_skips_the_snp_outlier_check():
+ row = metrics(snps="1")
+ assert qc_metrics.flag_sample(row, THR, 1000.0, 10.0) == ("WARN", ["SNP_LOW"])
+ assert qc_metrics.flag_sample(row, THR, 1000.0, 10.0, ancient=True, anc_thr=ANC) == ("PASS", [])
+
+
+def test_flag_sample_ancient_skips_the_het_check_because_deamination_inflates_it():
+ row = metrics(het_variants="40", homo_variants="60")
+ assert qc_metrics.flag_sample(row, THR, None, None) == ("WARN", ["HET_HIGH"])
+ assert qc_metrics.flag_sample(row, THR, None, None, ancient=True, anc_thr=ANC) == ("PASS", [])
+
+
+def test_flag_sample_damage_low_flags_an_ancient_sample_under_the_deamination_floor():
+ dmg = {"ct1": ANC["damage_min_ct"] / 2.0}
+ verdict, flags = qc_metrics.flag_sample(metrics(), THR, None, None, ancient=True, anc_thr=ANC, dmg=dmg)
+ assert (verdict, flags) == ("WARN", ["DAMAGE_LOW"])
+
+
+def test_flag_sample_damage_low_is_not_raised_at_or_above_the_floor():
+ for ct1 in (ANC["damage_min_ct"], 0.40):
+ assert qc_metrics.flag_sample(metrics(), THR, None, None,
+ ancient=True, anc_thr=ANC, dmg={"ct1": ct1}) == ("PASS", [])
+
+
+@pytest.mark.parametrize("dmg", [None, {}, {"ct1": None}])
+def test_flag_sample_damage_check_needs_a_measured_ct1(dmg):
+ assert qc_metrics.flag_sample(metrics(), THR, None, None, ancient=True, anc_thr=ANC, dmg=dmg) == ("PASS", [])
+
+
+def test_flag_sample_damage_check_is_not_applied_to_a_modern_sample():
+ dmg = {"ct1": 0.0}
+ assert qc_metrics.flag_sample(metrics(), THR, None, None, ancient=False, anc_thr=ANC, dmg=dmg) == ("PASS", [])
+
+
+# --------------------------------------------------------------------------- flag_sample: threshold dict contract
+
+
+@pytest.mark.parametrize("missing_key", sorted(qc_metrics.DEF))
+def test_flag_sample_raises_key_error_when_the_threshold_dict_is_incomplete(missing_key):
+ """flag_sample does no .get() on thr: a caller-built partial threshold dict is a hard error."""
+ thr = {k: v for k, v in THR.items() if k != missing_key}
+ row = metrics(het_variants="5", homo_variants="95", lineage_counts="L4:100;L2:10")
+ with pytest.raises(KeyError) as exc:
+ qc_metrics.flag_sample(row, thr, 1000.0, 10.0)
+ assert missing_key in str(exc.value)
+
+
+# --------------------------------------------------------------------------- robust
+
+
+def test_robust_of_an_empty_or_all_none_sequence_is_a_pair_of_nones():
+ assert qc_metrics.robust([]) == (None, None)
+ assert qc_metrics.robust([None, None]) == (None, None)
+
+
+def test_robust_of_a_single_value_falls_all_the_way_back_to_sigma_one():
+ assert qc_metrics.robust([7.0]) == (7.0, 1.0)
+
+
+def test_robust_sigma_is_never_zero_for_an_all_ties_cohort():
+ # a zero sigma would make every downstream z-score divide by zero
+ med, sigma = qc_metrics.robust([5.0] * 4)
+ assert med == 5.0
+ assert sigma == 1.0
+
+
+def test_robust_uses_the_mad_scaled_by_the_normal_consistency_constant():
+ med, sigma = qc_metrics.robust([1.0, 2.0, 3.0])
+ assert med == 2.0
+ assert sigma == pytest.approx(1.4826)
+
+
+def test_robust_falls_back_to_the_population_sd_when_the_mad_is_zero():
+ # [1, 1, 1, 5]: median 1, MAD 0 -> population sd of the raw values
+ med, sigma = qc_metrics.robust([1.0, 1.0, 1.0, 5.0])
+ assert med == 1.0
+ assert sigma == pytest.approx(3.0 ** 0.5)
+
+
+def test_robust_uses_the_even_length_midpoint_and_ignores_nones():
+ med, sigma = qc_metrics.robust([3.0, None, 1.0, 2.0, None, 4.0])
+ assert med == 2.5
+ assert sigma == pytest.approx(1.4826)
+
+
+def test_robust_median_is_resistant_to_a_large_outlier():
+ med, _ = qc_metrics.robust([1.0, 2.0, 3.0, 1000.0])
+ assert med == 2.5
+
+
+# --------------------------------------------------------------------------- het_frac
+
+
+def test_het_frac_returns_the_het_share_of_variant_genotypes():
+ assert qc_metrics.het_frac({"het_variants": "10", "homo_variants": "90"}) == pytest.approx(0.1)
+
+
+def test_het_frac_floor_is_inclusive_of_the_minimum_variant_count():
+ n = qc_metrics.MIN_HET_VARIANTS
+ assert qc_metrics.het_frac({"het_variants": "1", "homo_variants": n - 1}) is not None
+ assert qc_metrics.het_frac({"het_variants": "1", "homo_variants": n - 2}) is None
+
+
+@pytest.mark.parametrize("row", [
+ {},
+ {"het_variants": "10"},
+ {"homo_variants": "90"},
+ {"het_variants": "NA", "homo_variants": "90"},
+])
+def test_het_frac_needs_both_counts(row):
+ assert qc_metrics.het_frac(row) is None
+
+
+# --------------------------------------------------------------------------- lineage counts / fractions
+
+
+def test_lineage_counts_parsed_collapses_sublineages_on_the_first_dot():
+ assert qc_metrics.lineage_counts_parsed("L4.1:3200;L4.3:120;L2:40") == {"L4": 3320.0, "L2": 40.0}
+
+
+def test_lineage_counts_parsed_treats_commas_as_semicolons():
+ assert qc_metrics.lineage_counts_parsed("L4:10,L2:20") == {"L4": 10.0, "L2": 20.0}
+
+
+@pytest.mark.parametrize("value", [None, "", "NA", ".", 0])
+def test_lineage_counts_parsed_of_an_absent_value_is_empty(value):
+ assert qc_metrics.lineage_counts_parsed(value) == {}
+
+
+def test_lineage_counts_parsed_drops_malformed_and_non_numeric_entries():
+ assert qc_metrics.lineage_counts_parsed("L4;L2:20;L3:abc") == {"L2": 20.0}
+
+
+def test_lineage_fracs_normalises_the_collapsed_counts():
+ assert qc_metrics.lineage_fracs("L4.1:30;L4.2:10;L2:10") == {"L4": 0.8, "L2": 0.2}
+
+
+def test_lineage_fracs_of_a_zero_total_is_empty_rather_than_a_division_by_zero():
+ assert qc_metrics.lineage_fracs("L4:0") == {}
+ assert qc_metrics.lineage_fracs("NA") == {}
+
+
+# --------------------------------------------------------------------------- is_ancient
+
+
+@pytest.mark.parametrize("value", ["yes", "Yes", " YES ", "y", "true", "True", "1", "ancient"])
+def test_is_ancient_accepts_the_documented_truthy_spellings(value):
+ assert qc_metrics.is_ancient({"ancient": value}) is True
+
+
+@pytest.mark.parametrize("value", ["no", "", "NA", "0", "false", "modern", None])
+def test_is_ancient_is_false_for_anything_else(value):
+ assert qc_metrics.is_ancient({"ancient": value}) is False
+
+
+def test_is_ancient_of_a_row_without_the_column_is_modern():
+ assert qc_metrics.is_ancient({}) is False
+
+
+# --------------------------------------------------------------------------- infer_metric / discover_extra_metrics
+
+
+@pytest.mark.parametrize("key", ["foo_pct", "error_rate", "some_percent", "alt_fraction", "af_frac"])
+def test_infer_metric_recognises_a_percentage_column_by_name(key):
+ _, kind, direction = qc_metrics.infer_metric(key, [1.5])
+ assert kind == "pct"
+ assert direction == "neu"
+
+
+def test_infer_metric_recognises_an_integer_column_by_name_and_by_value():
+ assert qc_metrics.infer_metric("custom_reads", [10.0, 20.0]) == ("Custom reads", "int", "neu")
+ # the same name with a non-integral value is not an int column
+ assert qc_metrics.infer_metric("custom_reads", [10.5]) == ("Custom reads", "float", "neu")
+
+
+def test_infer_metric_defaults_to_float_and_prettifies_the_label():
+ assert qc_metrics.infer_metric("weird_thing", [1.5]) == ("Weird thing", "float", "neu")
+ assert qc_metrics.infer_metric("foo_pct", [1.5])[0] == "Foo %"
+
+
+def test_infer_metric_never_claims_a_direction():
+ # an auto-detected column must not silently start flagging samples
+ for key in ("foo_pct", "custom_reads", "weird_thing"):
+ assert qc_metrics.infer_metric(key, [1.0])[2] == "neu"
+
+
+def test_discover_extra_metrics_of_an_empty_cohort_is_empty():
+ assert qc_metrics.discover_extra_metrics({}) == []
+ assert qc_metrics.discover_extra_metrics(None) == []
+
+
+def test_discover_extra_metrics_picks_up_only_unknown_numeric_columns():
+ summ = {
+ "S1": {"sample_id": "S1", "mean_depth": "30", "lineage": "L4",
+ "custom_reads": "100", "note": "hello"},
+ "S2": {"sample_id": "S2", "mean_depth": "40", "lineage": "L2",
+ "custom_reads": "200", "note": "world"},
+ }
+ assert qc_metrics.discover_extra_metrics(summ) == [
+ {"key": "custom_reads", "label": "Custom reads", "kind": "int", "dir": "neu"}]
+
+
+def test_discover_extra_metrics_needs_only_one_numeric_sample():
+ summ = {"S1": {"sample_id": "S1", "odd_metric": "NA"},
+ "S2": {"sample_id": "S2", "odd_metric": "3.5"}}
+ assert [e["key"] for e in qc_metrics.discover_extra_metrics(summ)] == ["odd_metric"]
+
+
+def test_discover_extra_metrics_is_sorted_by_key():
+ summ = {"S1": {"sample_id": "S1", "zeta_metric": "1", "alpha_metric": "2"}}
+ assert [e["key"] for e in qc_metrics.discover_extra_metrics(summ)] == ["alpha_metric", "zeta_metric"]
+
+
+# --------------------------------------------------------------------------- to_float / clean_str
+
+
+@pytest.mark.parametrize("value", [None, "", "NA", "."])
+def test_to_float_maps_the_missing_sentinels_to_none(value):
+ assert qc_parsers.to_float(value) is None
+
+
+@pytest.mark.parametrize(("value", "expected"), [
+ ("1.5", 1.5), ("0", 0.0), (7, 7.0), ("-3", -3.0), ("1e3", 1000.0), (" 2.5 ", 2.5),
+])
+def test_to_float_parses_a_numeric_value(value, expected):
+ assert qc_parsers.to_float(value) == pytest.approx(expected)
+
+
+@pytest.mark.parametrize("value", ["abc", "1,5", [1], {}])
+def test_to_float_of_an_unparseable_value_is_none_rather_than_a_raise(value):
+ assert qc_parsers.to_float(value) is None
+
+
+@pytest.mark.parametrize("value", [None, "", " ", "NA", ".", "-"])
+def test_clean_str_maps_the_metadata_sentinels_to_none(value):
+ assert qc_parsers.clean_str(value) is None
+
+
+def test_clean_str_strips_a_real_value():
+ assert qc_parsers.clean_str(" L4.3.4 ") == "L4.3.4"
+ assert qc_parsers.clean_str(4) == "4"
+
+
+# --------------------------------------------------------------------------- parse_profile
+
+
+@pytest.mark.parametrize("value", [None, "", "NA", ".", 0])
+def test_parse_profile_of_an_absent_value_is_none(value):
+ assert qc_parsers.parse_profile(value) is None
+
+
+@pytest.mark.parametrize("value", ["a,b,c", "1;2;3"])
+def test_parse_profile_of_an_unparseable_value_is_none(value):
+ assert qc_parsers.parse_profile(value) is None
+
+
+def test_parse_profile_keeps_a_correctly_binned_profile_verbatim():
+ prof = qc_parsers.parse_profile(",".join(str(i) for i in range(qc_parsers.NBINS)))
+ assert prof == list(range(qc_parsers.NBINS))
+
+
+def test_parse_profile_truncates_floats_to_ints():
+ qc_parsers._PROFILE_WARNED.clear()
+ assert qc_parsers.parse_profile("1.7,2.9,0.4")[:3] == [1, 2, 0]
+
+
+def test_parse_profile_pads_a_short_profile_to_nbins():
+ qc_parsers._PROFILE_WARNED.clear()
+ prof = qc_parsers.parse_profile("5,6,7")
+ assert len(prof) == qc_parsers.NBINS
+ assert prof[:3] == [5, 6, 7]
+ assert set(prof[3:]) == {0}
+
+
+def test_parse_profile_truncates_a_long_profile_to_nbins():
+ qc_parsers._PROFILE_WARNED.clear()
+ prof = qc_parsers.parse_profile(",".join("1" for _ in range(qc_parsers.NBINS + 50)))
+ assert prof == [1] * qc_parsers.NBINS
+
+
+def test_parse_profile_warns_once_per_observed_bin_count(capsys):
+ qc_parsers._PROFILE_WARNED.clear()
+ qc_parsers.parse_profile("1,2,3")
+ qc_parsers.parse_profile("4,5,6") # same length -> already warned, stays quiet
+ qc_parsers.parse_profile("1,2,3,4") # a new length warns again
+ err = capsys.readouterr().err
+ assert err.count("variant profile has 3 bins") == 1
+ assert err.count("variant profile has 4 bins") == 1
+ qc_parsers._PROFILE_WARNED.clear()
+
+
+# --------------------------------------------------------------------------- mask_profile
+
+
+def test_mask_profile_without_intervals_or_genome_length_is_none():
+ assert qc_parsers.mask_profile([], 100) == (None, 0.0)
+ assert qc_parsers.mask_profile(None, 100) == (None, 0.0)
+ assert qc_parsers.mask_profile([(0, 10)], 0) == (None, 0.0)
+
+
+def test_mask_profile_marks_the_covered_bins_and_the_total_percent():
+ prof, pct = qc_parsers.mask_profile([(0, 50)], 100, nbins=10)
+ assert prof == [1.0] * 5 + [0.0] * 5
+ assert pct == 50.0
+
+
+def test_mask_profile_clamps_intervals_to_the_genome():
+ prof, pct = qc_parsers.mask_profile([(-10, 200)], 100, nbins=10)
+ assert prof == [1.0] * 10
+ assert pct == 100.0
+
+
+def test_mask_profile_skips_an_empty_or_inverted_interval():
+ prof, pct = qc_parsers.mask_profile([(50, 50), (80, 20)], 100, nbins=10)
+ assert prof == [0.0] * 10
+ assert pct == 0.0
+
+
+def test_mask_profile_reports_a_partially_covered_bin_as_a_fraction():
+ prof, pct = qc_parsers.mask_profile([(0, 5)], 100, nbins=10)
+ assert prof[0] == 0.5
+ assert prof[1] == 0.0
+ assert pct == 5.0
+
+
+def test_mask_profile_defaults_to_the_module_bin_count():
+ prof, _ = qc_parsers.mask_profile([(0, 1000)], 1000)
+ assert len(prof) == qc_parsers.NBINS
+
+
+def test_mask_profile_total_is_the_union_of_overlapping_intervals():
+ # The masked TOTAL is the union, so an un-merged BED with duplicate records cannot report the
+ # same base twice (this used to sum raw lengths and claim 200% of the genome was masked).
+ prof, pct = qc_parsers.mask_profile([(0, 100), (0, 100)], 100, nbins=10)
+ assert prof == [1.0] * 10
+ assert pct == 100.0
+
+
+def test_mask_profile_total_merges_partially_overlapping_intervals():
+ # union of [0,30) and [20,50) is 50 bp, not 30 + 30
+ _, pct = qc_parsers.mask_profile([(0, 30), (20, 50)], 100, nbins=10)
+ assert pct == 50.0
+
+
+def test_mask_profile_total_swallows_a_contained_interval():
+ _, pct = qc_parsers.mask_profile([(10, 60), (20, 30)], 100, nbins=10)
+ assert pct == 50.0
+
+
+def test_mask_profile_total_is_order_independent():
+ # the sweep sorts, so a BED that is not in coordinate order gives the same union
+ assert qc_parsers.mask_profile([(60, 80), (0, 40), (30, 50)], 100, nbins=10)[1] == \
+ qc_parsers.mask_profile([(0, 40), (30, 50), (60, 80)], 100, nbins=10)[1] == 70.0
+
+
+def test_mask_profile_total_still_adds_up_disjoint_intervals():
+ _, pct = qc_parsers.mask_profile([(0, 10), (50, 60), (90, 100)], 100, nbins=10)
+ assert pct == 30.0
+
+
+# --------------------------------------------------------------------------- _pearson
+
+
+def test_pearson_of_a_perfectly_correlated_pair_is_one():
+ assert qc_panels._pearson([1.0, 2.0, 3.0], [2.0, 4.0, 6.0]) == pytest.approx(1.0)
+
+
+def test_pearson_of_a_perfectly_anticorrelated_pair_is_minus_one():
+ assert qc_panels._pearson([1.0, 2.0, 3.0], [6.0, 4.0, 2.0]) == pytest.approx(-1.0)
+
+
+def test_pearson_needs_at_least_three_points():
+ assert qc_panels._pearson([1.0, 2.0], [1.0, 2.0]) is None
+ assert qc_panels._pearson([], []) is None
+
+
+def test_pearson_of_unequal_lengths_is_none():
+ assert qc_panels._pearson([1.0, 2.0, 3.0], [1.0, 2.0, 3.0, 4.0]) is None
+
+
+def test_pearson_of_a_flat_vector_is_none():
+ assert qc_panels._pearson([1.0, 1.0, 1.0], [1.0, 2.0, 3.0]) is None
+ assert qc_panels._pearson([1.0, 2.0, 3.0], [5.0, 5.0, 5.0]) is None
+
+
+# --------------------------------------------------------------------------- _perm_p
+
+
+PAIR = [([0.0, 0.5, 1.0, 0.5], [0.0, 0.5, 1.0, 0.5])]
+
+
+def test_perm_p_is_reproducible_for_a_fixed_seed():
+ assert qc_panels._perm_p(PAIR, 1.0, B=100, seed=7) == qc_panels._perm_p(PAIR, 1.0, B=100, seed=7)
+
+
+def test_perm_p_is_the_plus_one_smoothed_fraction():
+ p = qc_panels._perm_p(PAIR, 1.0, B=100, seed=7)
+ assert p == pytest.approx((round(p * 101) - 1 + 1) / 101)
+ assert 1 / 101 <= p <= 1.0
+
+
+def test_perm_p_bottoms_out_at_one_over_b_plus_one_when_no_null_reaches_the_observed():
+ # |r| can never exceed 1, so an observed mean of 2 is unreachable
+ assert qc_panels._perm_p(PAIR, 2.0, B=100, seed=1) == pytest.approx(1 / 101)
+
+
+def test_perm_p_is_one_when_every_null_reaches_the_observed():
+ p = qc_panels._perm_p([([0.0, 0.5, 1.0, 0.5], [0.1, 0.6, 0.2, 0.9])], 0.0, B=100, seed=1)
+ assert p == pytest.approx(1.0)
+
+
+def test_perm_p_with_undefined_correlations_reports_the_floor():
+ # 2-point trajectories make every _pearson None -> no null ever counts
+ assert qc_panels._perm_p([([0.0, 1.0], [0.0, 1.0])], 0.0, B=100, seed=1) == pytest.approx(1 / 101)
+
+
+# --------------------------------------------------------------------------- _dyn_af / _dyn_dp
+
+
+def test_dyn_af_prefers_ad_over_ao_ro_and_over_the_gt_dosage():
+ assert qc_parsers._dyn_af("GT:AD:AO:RO", "0/1:2,8:99:1") == pytest.approx(0.8)
+
+
+def test_dyn_af_falls_back_to_ao_over_ao_plus_ro():
+ assert qc_parsers._dyn_af("GT:AO:RO", "0/1:3:1") == pytest.approx(0.75)
+
+
+def test_dyn_af_sums_every_alt_allele_of_a_multiallelic_ad():
+ assert qc_parsers._dyn_af("GT:AD", "1/2:2,4,4") == pytest.approx(0.8)
+
+
+@pytest.mark.parametrize(("gt", "expected"), [("1/1", 1.0), ("0/1", 0.5), ("0/0", 0.0), ("1|1", 1.0), ("1", 1.0)])
+def test_dyn_af_falls_back_to_the_gt_dosage(gt, expected):
+ assert qc_parsers._dyn_af("GT", gt) == pytest.approx(expected)
+
+
+def test_dyn_af_of_a_no_call_genotype_is_none():
+ assert qc_parsers._dyn_af("GT", "./.") is None
+ assert qc_parsers._dyn_af("GT", ".|.") is None
+
+
+def test_dyn_af_skips_an_unusable_ad_and_keeps_going():
+ # a zero-depth AD, a single-entry AD and a non-numeric AD all fall through to the GT dosage
+ assert qc_parsers._dyn_af("GT:AD", "1/1:0,0") == pytest.approx(1.0)
+ assert qc_parsers._dyn_af("GT:AD", "0/1:10") == pytest.approx(0.5)
+ assert qc_parsers._dyn_af("GT:AD", "0/1:x,y") == pytest.approx(0.5)
+
+
+def test_dyn_af_of_a_zero_depth_ao_ro_falls_through_to_the_dosage():
+ assert qc_parsers._dyn_af("GT:AO:RO", "0/1:0:0") == pytest.approx(0.5)
+
+
+def test_dyn_af_of_a_non_numeric_ao_ro_falls_through_to_the_dosage():
+ assert qc_parsers._dyn_af("GT:AO:RO", "1/1:x:1") == pytest.approx(1.0)
+
+
+def test_dyn_dp_prefers_the_format_dp_field():
+ assert qc_parsers._dyn_dp("GT:DP:AD", "0/1:37:2,8") == 37
+
+
+def test_dyn_dp_falls_back_to_the_ad_sum_then_to_ao_plus_ro():
+ assert qc_parsers._dyn_dp("GT:AD", "0/1:2,8") == 10
+ assert qc_parsers._dyn_dp("GT:DP:AD", "0/1:.:2,8") == 10
+ assert qc_parsers._dyn_dp("GT:AO:RO", "0/1:3:1") == 4
+
+
+def test_dyn_dp_without_any_depth_field_is_none():
+ assert qc_parsers._dyn_dp("GT", "0/1") is None
+ assert qc_parsers._dyn_dp("GT:DP", "0/1:.") is None
+
+
+def test_dyn_dp_of_a_non_numeric_dp_falls_through():
+ assert qc_parsers._dyn_dp("GT:DP:AD", "0/1:deep:2,8") == 10
+ assert qc_parsers._dyn_dp("GT:DP", "0/1:deep") is None
+
+
+def test_dyn_dp_of_non_numeric_ad_or_ao_ro_is_none():
+ assert qc_parsers._dyn_dp("GT:AD", "0/1:x,y") is None
+ assert qc_parsers._dyn_dp("GT:AO:RO", "0/1:x:1") is None
+
+
+# --------------------------------------------------------------------------- _dyn_ann / _dyn_opos / _dyn_num
+
+
+ANN_INFO = ("DP=40;ANN=T|missense_variant|MODERATE|rpoB|GENE_rpoB|transcript|TX1|protein_coding|"
+ "1/1|c.1349C>T|p.Ser450Leu|450/3519|;AF=0.9")
+
+
+def test_dyn_ann_pulls_the_gene_effect_impact_and_protein_change():
+ assert qc_parsers._dyn_ann(ANN_INFO) == ("rpoB", "missense_variant", "MODERATE", "p.Ser450Leu")
+
+
+def test_dyn_ann_uses_only_the_first_of_several_annotations():
+ info = "ANN=T|synonymous_variant|LOW|geneA|x|x|x|x|x|x|p.Aaa1Aaa|,T|missense_variant|MODERATE|geneB|"
+ assert qc_parsers._dyn_ann(info)[0] == "geneA"
+
+
+def test_dyn_ann_of_an_unannotated_or_truncated_record_is_all_empty_strings():
+ assert qc_parsers._dyn_ann("DP=40;AF=0.9") == ("", "", "", "")
+ assert qc_parsers._dyn_ann(".") == ("", "", "", "")
+ assert qc_parsers._dyn_ann("ANN=T|missense_variant") == ("", "missense_variant", "", "")
+
+
+def test_dyn_opos_reads_the_picard_liftover_original_coordinate():
+ assert qc_parsers._dyn_opos("DP=40;OriginalContig=chrX;OriginalStart=1234") == "chrX:1234"
+
+
+def test_dyn_opos_reads_a_plain_opos_field():
+ assert qc_parsers._dyn_opos("DP=40;OPOS=chrX:1234") == "chrX:1234"
+
+
+@pytest.mark.parametrize("info", [None, "", "DP=40", "OriginalContig=chrX", "OriginalStart=99", "OPOS="])
+def test_dyn_opos_without_a_complete_original_coordinate_is_none(info):
+ assert qc_parsers._dyn_opos(info) is None
+
+
+@pytest.mark.parametrize(("value", "expected"), [
+ ("P3", 3.0), ("passage 12", 12.0), ("-2.5", -2.5), (7, 7.0), ("7.", 7.0), ("2025-01-15", 2025.0),
+])
+def test_dyn_num_extracts_the_first_number_it_finds(value, expected):
+ assert qc_parsers._dyn_num(value) == pytest.approx(expected)
+
+
+@pytest.mark.parametrize("value", [None, "", "abc", "NA"])
+def test_dyn_num_without_a_number_is_none(value):
+ assert qc_parsers._dyn_num(value) is None
+
+
+# --------------------------------------------------------------------------- build_dynamics
+
+
+def variant(af, gene="rpoB", eff="missense_variant", imp="MODERATE", dp=40):
+ return {"ref": "A", "alt": "T", "gene": gene, "eff": eff, "imp": imp,
+ "aa": "p.Ser450Leu", "aa_h37rv": "", "af": af, "dp": dp}
+
+
+def timeseries(afs_by_pos, group="G"):
+ """(metadata, variants) for one connected group with one sample per timepoint."""
+ n = len(next(iter(afs_by_pos.values())))
+ metadata = {f"S{i + 1}": {"time": f"T{i + 1}", "tnum": float(i + 1), "group": group} for i in range(n)}
+ variants = {f"S{i + 1}": {pos: variant(afs[i]) for pos, afs in afs_by_pos.items()} for i in range(n)}
+ return metadata, variants
+
+
+def test_build_dynamics_without_metadata_or_variants_is_none():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ assert qc_panels.build_dynamics({}, variants) is None
+ assert qc_panels.build_dynamics(None, variants) is None
+ assert qc_panels.build_dynamics(metadata, {}) is None
+
+
+def test_build_dynamics_builds_a_trajectory_with_its_event_flags():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ dyn = qc_panels.build_dynamics(metadata, variants)
+ assert dyn["thresholds"] == {"emerge": 0.25, "fix": 0.90, "loss": 0.10}
+ assert len(dyn["groups"]) == 1
+ group = dyn["groups"][0]
+ assert group["group"] == "G"
+ assert group["samples"] == ["S1", "S2"]
+ assert group["times"] == ["T1", "T2"]
+ assert group["tnums"] == [1.0, 2.0]
+ assert group["n_flagged"] == 1
+ series = group["series"][0]
+ assert series["pos"] == "c:10"
+ assert series["traj"] == [0.05, 0.95]
+ assert series["dp"] == [40, 40]
+ assert series["flags"] == ["emergence", "fixation", "nonsyn"]
+
+
+def test_build_dynamics_flags_a_lost_variant():
+ metadata, variants = timeseries({"c:10": [0.95, 0.05]})
+ series = qc_panels.build_dynamics(metadata, variants)["groups"][0]["series"][0]
+ assert "loss" in series["flags"]
+ assert "emergence" not in series["flags"]
+
+
+def test_build_dynamics_flags_a_high_impact_variant():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ for sample in variants:
+ variants[sample]["c:10"]["imp"] = "HIGH"
+ variants[sample]["c:10"]["eff"] = "stop_gained"
+ series = qc_panels.build_dynamics(metadata, variants)["groups"][0]["series"][0]
+ assert "high_impact" in series["flags"]
+
+
+def test_build_dynamics_drops_a_variant_that_does_not_move():
+ metadata, variants = timeseries({"c:10": [0.50, 0.55]})
+ assert qc_panels.build_dynamics(metadata, variants) is None
+
+
+def test_build_dynamics_treats_an_uncalled_site_as_frequency_zero():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ del variants["S1"]["c:10"]
+ series = qc_panels.build_dynamics(metadata, variants)["groups"][0]["series"][0]
+ assert series["traj"] == [0.0, 0.95]
+ assert series["dp"] == [None, 40]
+
+
+def test_build_dynamics_needs_more_than_one_distinct_timepoint():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ metadata["S2"]["time"] = "T1"
+ assert qc_panels.build_dynamics(metadata, variants) is None
+
+
+def test_build_dynamics_ignores_a_sample_without_a_group():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ metadata["S1"]["group"] = None
+ assert qc_panels.build_dynamics(metadata, variants) is None
+
+
+def test_build_dynamics_orders_samples_by_the_numeric_time():
+ metadata, variants = timeseries({"c:10": [0.95, 0.05]})
+ metadata["S1"]["tnum"], metadata["S2"]["tnum"] = 2.0, 1.0
+ group = qc_panels.build_dynamics(metadata, variants)["groups"][0]
+ assert group["samples"] == ["S2", "S1"]
+ assert group["series"][0]["traj"] == [0.05, 0.95]
+
+
+def test_build_dynamics_attaches_the_group_invariant_sample_metadata():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95]})
+ sample_meta = {"fields": ["patient", "arm"],
+ "rows": {"S1": {"patient": "P1", "arm": "A"}, "S2": {"patient": "P1", "arm": "B"}}}
+ group = qc_panels.build_dynamics(metadata, variants, sample_meta)["groups"][0]
+ assert group["meta"] == {"patient": ["P1"], "arm": ["A", "B"]}
+
+
+def test_build_dynamics_honours_custom_event_thresholds():
+ metadata, variants = timeseries({"c:10": [0.30, 0.60]})
+ # a 0.30 move is under a 0.5 min_move, so the variant is not reported at all
+ assert qc_panels.build_dynamics(metadata, variants, min_move=0.5) is None
+ # at the default fixation cut-off 0.60 is not fixed; lowering it makes the same trajectory fix
+ assert qc_panels.build_dynamics(metadata, variants)["groups"][0]["series"][0]["flags"] == ["nonsyn"]
+ series = qc_panels.build_dynamics(metadata, variants, fix=0.55)["groups"][0]["series"][0]
+ assert series["flags"] == ["fixation", "nonsyn"]
+
+
+# --------------------------------------------------------------------------- build_epistasis
+
+
+@pytest.fixture()
+def four_point_dynamics():
+ """One 4-timepoint series with two co-rising variants and one falling one."""
+ rising = [0.05, 0.35, 0.65, 0.95]
+ falling = list(reversed(rising))
+ metadata, variants = timeseries({"c:10": rising, "c:20": rising, "c:30": falling})
+ for sample in variants:
+ variants[sample]["c:30"]["gene"] = "katG"
+ return qc_panels.build_dynamics(metadata, variants)
+
+
+def test_build_epistasis_without_dynamics_is_none():
+ assert qc_panels.build_epistasis(None) is None
+ assert qc_panels.build_epistasis({}) is None
+ assert qc_panels.build_epistasis({"groups": []}) is None
+
+
+def test_build_epistasis_needs_min_points_distinct_timepoints():
+ metadata, variants = timeseries({"c:10": [0.05, 0.95], "c:20": [0.05, 0.95]})
+ two_point = qc_panels.build_dynamics(metadata, variants)
+ assert two_point is not None
+ assert qc_panels.build_epistasis(two_point, perm=10) is None
+
+
+def test_build_epistasis_reports_every_pair_over_the_correlation_threshold(four_point_dynamics):
+ ep = qc_panels.build_epistasis(four_point_dynamics, perm=100)
+ assert ep["n_series"] == 1
+ assert ep["n_candidates"] == 3
+ assert len(ep["pairs"]) == 3
+ assert {(p["posA"], p["posB"]) for p in ep["pairs"]} == {
+ ("c:10", "c:20"), ("c:10", "c:30"), ("c:20", "c:30")}
+ concordant = next(p for p in ep["pairs"] if p["posA"] == "c:10" and p["posB"] == "c:20")
+ assert concordant["r"] == pytest.approx(1.0)
+ assert concordant["direction"] == "concordant"
+ assert concordant["consistent"] is True
+ assert concordant["n"] == 1
+ assert ep["n_concordant"] == 1
+ assert ep["n_discordant"] == 2
+ assert ep["pairs"][0]["geneA"] == "rpoB"
+
+
+def test_build_epistasis_drops_pairs_below_min_r(four_point_dynamics):
+ assert qc_panels.build_epistasis(four_point_dynamics, min_r=1.5, perm=10) is None
+
+
+def test_build_epistasis_q_values_are_bounded_and_monotone_in_p(four_point_dynamics):
+ # the Benjamini-Hochberg loop is inline in build_epistasis, so it is only reachable this way
+ ep = qc_panels.build_epistasis(four_point_dynamics, perm=100)
+ qs = [p["q"] for p in ep["pairs"]]
+ assert all(0.0 < q <= 1.0 for q in qs)
+ by_p = sorted(ep["pairs"], key=lambda p: p["p"])
+ assert [p["q"] for p in by_p] == sorted(p["q"] for p in by_p)
+ # the largest p keeps its own value (rank m: q = p * m / m)
+ assert by_p[-1]["q"] == pytest.approx(by_p[-1]["p"], abs=1e-4)
+ assert all(p["tier"] in ("strong", "moderate", "weak") for p in ep["pairs"])
+ # the payload is sorted by q, then by descending |r|
+ assert qs == sorted(qs)
+
+
+def test_build_epistasis_is_reproducible_across_runs(four_point_dynamics):
+ first = qc_panels.build_epistasis(four_point_dynamics, perm=100)
+ second = qc_panels.build_epistasis(four_point_dynamics, perm=100)
+ assert [(p["posA"], p["posB"], p["p"], p["q"]) for p in first["pairs"]] == \
+ [(p["posA"], p["posB"], p["p"], p["q"]) for p in second["pairs"]]
+
+
+def test_build_epistasis_ignores_a_series_whose_correlation_is_undefined():
+ # a hand-made payload with a flat trajectory: _pearson returns None for every pair it
+ # takes part in, so it contributes no pair and never reaches the correlation matrix
+ dynamics = {"groups": [{"group": "G", "times": ["T1", "T2", "T3"], "series": [
+ {"pos": "c:10", "gene": "a", "aa": "", "eff": "", "traj": [0.1, 0.5, 0.9]},
+ {"pos": "c:20", "gene": "b", "aa": "", "eff": "", "traj": [0.5, 0.5, 0.5]},
+ {"pos": "c:30", "gene": "c", "aa": "", "eff": "", "traj": [0.2, 0.6, 1.0]},
+ ]}]}
+ ep = qc_panels.build_epistasis(dynamics, perm=20)
+ assert [(p["posA"], p["posB"]) for p in ep["pairs"]] == [("c:10", "c:30")]
+ assert ep["matrix"]["total_nodes"] == 2
+
+
+def test_build_epistasis_matrix_holds_every_pairwise_mean_r(four_point_dynamics):
+ matrix = qc_panels.build_epistasis(four_point_dynamics, perm=10)["matrix"]
+ assert matrix["total_nodes"] == 3
+ assert matrix["truncated"] is False
+ assert len(matrix["nodes"]) == 3
+ assert sorted(matrix["cells"].values()) == [-1.0, -1.0, 1.0]
+
+
+def test_build_epistasis_reports_its_scaling_caps(four_point_dynamics):
+ ep = qc_panels.build_epistasis(four_point_dynamics, perm=10)
+ assert ep["vars_capped"] is False
+ assert ep["pairs_capped"] is False
+ assert ep["n_vars_max"] == 3
+ assert ep["max_vars"] == 250
+ assert ep["max_pairs_perm"] == 800
+
+
+def test_build_epistasis_caps_the_variants_tested_per_series(four_point_dynamics):
+ capped = qc_panels.build_epistasis(four_point_dynamics, perm=10, max_vars=2)
+ assert capped["vars_capped"] is True
+ assert capped["n_vars_max"] == 3
+ assert len(capped["pairs"]) == 1 # only one pair survives out of the two kept variants
+
+
+def test_build_epistasis_caps_the_pairs_that_get_a_permutation_test(four_point_dynamics):
+ capped = qc_panels.build_epistasis(four_point_dynamics, perm=10, max_pairs_perm=1)
+ assert capped["pairs_capped"] is True
+ assert capped["n_candidates"] == 3 # all three were candidates, only the strongest was tested
+ assert len(capped["pairs"]) == 1
+
+
+# --------------------------------------------------------------------------- build_snp_matrix
+
+
+def test_build_snp_matrix_without_variants_is_none():
+ assert qc_panels.build_snp_matrix({}) is None
+ assert qc_panels.build_snp_matrix(None) is None
+
+
+def test_build_snp_matrix_of_a_sample_without_calls_is_an_empty_matrix():
+ assert qc_panels.build_snp_matrix({"S1": {}}) == {
+ "samples": ["S1"], "reference": "", "rows": [], "total_sites": 0, "truncated": False}
+
+
+def test_build_snp_matrix_places_one_row_per_site_with_the_called_cells_only():
+ variants = {"S1": {"c:10": variant(0.9), "c:20": variant(0.5, gene="", eff="")},
+ "S2": {"c:10": variant(0.4, dp=12)}}
+ matrix = qc_panels.build_snp_matrix(variants, reference="TESTREF")
+ assert matrix["samples"] == ["S1", "S2"]
+ assert matrix["reference"] == "TESTREF"
+ assert matrix["total_sites"] == 2
+ assert matrix["truncated"] is False
+ shared, private = matrix["rows"]
+ assert (shared["contig"], shared["pos"], shared["ref"], shared["alt"]) == ("c", 10, "A", "T")
+ assert shared["gene"] == "rpoB"
+ assert shared["n"] == 2
+ assert shared["cells"] == {0: [0.9, 40], 1: [0.4, 12]}
+ assert private["pos"] == 20
+ assert private["cells"] == {0: [0.5, 40]}
+
+
+def test_build_snp_matrix_skips_a_site_with_a_non_numeric_position():
+ matrix = qc_panels.build_snp_matrix({"S1": {"c:10": variant(0.9), "c:bad": variant(0.9)}})
+ assert [r["pos"] for r in matrix["rows"]] == [10]
+
+
+def test_build_snp_matrix_merges_alternate_alleles_of_the_same_site():
+ a, b = variant(0.9), variant(0.9)
+ b["alt"] = "G"
+ matrix = qc_panels.build_snp_matrix({"S1": {"c:10": a}, "S2": {"c:10": b}})
+ assert matrix["rows"][0]["alt"] == "G,T"
+
+
+def test_build_snp_matrix_truncation_keeps_the_most_shared_sites_in_coordinate_order():
+ variants = {"S1": {"c:10": variant(0.9), "c:20": variant(0.9), "c:30": variant(0.9)},
+ "S2": {"c:10": variant(0.9), "c:20": variant(0.9)}}
+ matrix = qc_panels.build_snp_matrix(variants, max_sites=2)
+ assert matrix["truncated"] is True
+ assert matrix["total_sites"] == 3
+ assert [r["pos"] for r in matrix["rows"]] == [10, 20]
+
+
+# --------------------------------------------------------------------------- build_gene_map
+
+
+def test_build_gene_map_without_a_gff_is_the_curated_h37rv_map():
+ assert qc_metrics.build_gene_map(None) == qc_metrics.GENE_RV
+ assert qc_metrics.build_gene_map("/nonexistent/reference.gff3") == qc_metrics.GENE_RV
+
+
+def test_build_gene_map_ignores_a_non_h37rv_reference(data_dir):
+ # the committed fixture GFF uses TEST_000n locus tags, which are not H37Rv-style
+ assert qc_metrics.build_gene_map(str(data_dir / "reference.gff3")) == qc_metrics.GENE_RV
+
+
+def test_build_gene_map_overlays_h37rv_style_locus_tags_from_the_run_gff(tmp_path):
+ gff = tmp_path / "ref.gff3"
+ gff.write_text(
+ "##gff-version 3\n"
+ "chr\ttest\tgene\t1\t100\t.\t+\t.\tID=g1;gene=rpoB;locus_tag=Rv9999\n"
+ "chr\ttest\tgene\t200\t300\t.\t+\t.\tID=g2;gene=newGene;locus_tag=Rv1234\n"
+ "chr\ttest\tgene\t400\t500\t.\t+\t.\tID=g3;gene=strainGene;locus_tag=ABC_0001\n",
+ encoding="utf-8")
+ gene_map = qc_metrics.build_gene_map(str(gff))
+ assert gene_map["rpoB"] == "Rv9999" # overrides the curated Rv0667
+ assert gene_map["newGene"] == "Rv1234" # a gene the curated map does not carry
+ assert "strainGene" not in gene_map # a strain-specific tag is ignored
+ assert gene_map["katG"] == qc_metrics.GENE_RV["katG"]
+
+
+# --------------------------------------------------------------------------- the CLI entry point
+#
+# bin/qc_report.py is a thin CLI over the qcreport package, and modules/report.nf runs it
+# BY PATH (`python3 ${projectDir}/bin/qc_report.py ...`) from a Nextflow work directory.
+# That only resolves `import qcreport` because Python puts the script's own directory,
+# bin/, on sys.path[0]. Nothing else in the suite exercises that, so pin it here.
+
+
+def test_qc_report_module_imports_cleanly():
+ """The CLI's import list stays in step with what the package actually exports."""
+ cli = load_script("qc_report")
+ assert callable(cli.main)
+
+
+def test_qc_report_runs_by_path_from_an_unrelated_cwd(tmp_path):
+ proc = subprocess.run([sys.executable, str(BIN / "qc_report.py"), "--help"],
+ cwd=str(tmp_path), capture_output=True, text=True, timeout=120)
+ assert proc.returncode == 0, proc.stderr
+ assert "--out-flags" in proc.stdout
diff --git a/tests/unit/test_qc_report_parsers.py b/tests/unit/test_qc_report_parsers.py
new file mode 100644
index 0000000..773f67c
--- /dev/null
+++ b/tests/unit/test_qc_report_parsers.py
@@ -0,0 +1,906 @@
+"""Unit tests for the file parsers of bin/qcreport/parsers.py.
+
+Each parser gets a happy path built from a small inline fixture in tmp_path, plus
+the empty / missing-file behaviour, which differs per function: most return [], {}
+or None, but parse_summary and consensus_stats deliberately raise (the pipeline
+always hands them a file it just produced). The GFF and samplesheet parsers are
+additionally exercised against the committed fixture cohort in tests/data/, so a
+change to those files breaks a test instead of silently changing a report.
+"""
+
+from __future__ import annotations
+
+import gzip
+
+import pytest
+from qcreport import metrics as qc_metrics # build_gene_map: a metric helper over a parsed GFF
+from qcreport import parsers as qc_parsers
+
+
+def write(path, text):
+ """Write a fixture file and return its path as a string (the parsers take str paths)."""
+ path.write_text(text, encoding="utf-8")
+ return str(path)
+
+
+def write_gz(path, text):
+ with gzip.open(path, "wt", encoding="utf-8") as fh:
+ fh.write(text)
+ return str(path)
+
+
+MISSING = "/nonexistent/directory/definitely-not-here.tsv"
+
+
+# --------------------------------------------------------------------------- parse_summary
+
+
+SUMMARY = (
+ "sample_id\tmean_depth\tbreadth_pct\tsnps\tlineage\n"
+ "S1\t42.5\t99.1\t1200\tL4\n"
+ "S2\t8.0\t70.0\t900\tL2\n"
+)
+
+
+def test_parse_summary_reads_one_dict_per_sample(tmp_path):
+ rows = qc_parsers.parse_summary(write(tmp_path / "summary.tsv", SUMMARY))
+ assert sorted(rows) == ["S1", "S2"]
+ assert rows["S1"] == {"sample_id": "S1", "mean_depth": "42.5", "breadth_pct": "99.1",
+ "snps": "1200", "lineage": "L4"}
+
+
+def test_parse_summary_pads_a_short_row_with_empty_strings(tmp_path):
+ path = write(tmp_path / "summary.tsv", SUMMARY + "S3\t10.0\n")
+ rows = qc_parsers.parse_summary(path)
+ assert rows["S3"] == {"sample_id": "S3", "mean_depth": "10.0", "breadth_pct": "",
+ "snps": "", "lineage": ""}
+
+
+def test_parse_summary_skips_blank_lines_and_rows_without_a_sample_id(tmp_path):
+ path = write(tmp_path / "summary.tsv", SUMMARY + "\n \n\t5.0\t5.0\t5\tL1\n")
+ assert sorted(qc_parsers.parse_summary(path)) == ["S1", "S2"]
+
+
+def test_parse_summary_falls_back_to_the_first_column_without_a_sample_id_header(tmp_path):
+ path = write(tmp_path / "summary.tsv", "name\tmean_depth\nS9\t30\n")
+ rows = qc_parsers.parse_summary(path)
+ assert list(rows) == ["S9"]
+ assert rows["S9"]["mean_depth"] == "30"
+
+
+def test_parse_summary_keeps_the_last_row_of_a_duplicated_sample(tmp_path):
+ path = write(tmp_path / "summary.tsv", SUMMARY + "S1\t99.9\t50.0\t1\tL9\n")
+ assert qc_parsers.parse_summary(path)["S1"]["mean_depth"] == "99.9"
+
+
+def test_parse_summary_of_an_empty_file_is_an_empty_cohort(tmp_path):
+ assert qc_parsers.parse_summary(write(tmp_path / "summary.tsv", "")) == {}
+
+
+def test_parse_summary_of_a_missing_file_raises(tmp_path):
+ # the summary is a required input the pipeline just produced: absence is a bug, not a state
+ with pytest.raises(OSError):
+ qc_parsers.parse_summary(str(tmp_path / "nope.tsv"))
+
+
+# --------------------------------------------------------------------------- consensus_stats
+
+
+# 80 callable bases, 10 N, 5 gap, 5 IUPAC -> callable 80%, missing 15%, IUPAC 5%
+CONSENSUS_SEQ = "ACGT" * 20 + "N" * 10 + "-" * 5 + "R" * 5
+CONSENSUS_FASTA = ">S1 some description\n" + "\n".join(
+ CONSENSUS_SEQ[i:i + 60] for i in range(0, len(CONSENSUS_SEQ), 60)) + "\n"
+
+
+def assert_consensus_fixture(stats):
+ assert stats["consensus_id"] == "S1"
+ assert stats["length"] == 100
+ assert stats["callable_pct"] == pytest.approx(80.0)
+ assert stats["missing_pct"] == pytest.approx(15.0)
+ assert stats["iupac_pct"] == pytest.approx(5.0)
+ assert stats["longest_n_run"] == 15 # the N run and the gap run are contiguous
+ assert stats["n_gaps"] == 1
+ assert len(stats["miss_profile"]) == qc_parsers.NBINS
+
+
+def test_consensus_stats_measures_a_plain_fasta(tmp_path):
+ assert_consensus_fixture(qc_parsers.consensus_stats(write(tmp_path / "S1.fa", CONSENSUS_FASTA)))
+
+
+def test_consensus_stats_reads_a_gzipped_fasta_identically(tmp_path):
+ assert_consensus_fixture(qc_parsers.consensus_stats(write_gz(tmp_path / "S1.fa.gz", CONSENSUS_FASTA)))
+
+
+def test_consensus_stats_is_case_insensitive(tmp_path):
+ lower = qc_parsers.consensus_stats(write(tmp_path / "S1.fa", CONSENSUS_FASTA.lower()))
+ assert lower["callable_pct"] == pytest.approx(80.0)
+ assert lower["missing_pct"] == pytest.approx(15.0)
+
+
+def test_consensus_stats_measures_every_record_but_keeps_the_first_id(tmp_path):
+ path = write(tmp_path / "multi.fa", ">contig1\nACGT\n>contig2\nNNNN\n")
+ stats = qc_parsers.consensus_stats(path)
+ assert stats["consensus_id"] == "contig1"
+ assert stats["length"] == 8
+ assert stats["callable_pct"] == pytest.approx(50.0)
+ assert stats["missing_pct"] == pytest.approx(50.0)
+
+
+def test_consensus_stats_counts_separate_gap_runs(tmp_path):
+ path = write(tmp_path / "S1.fa", ">S1\nACNNGT--AC\n")
+ stats = qc_parsers.consensus_stats(path)
+ assert stats["n_gaps"] == 2
+ assert stats["longest_n_run"] == 2
+
+
+def test_consensus_stats_of_a_zero_length_sequence_does_not_divide_by_zero(tmp_path):
+ # a header with no bases: the `len(seq) or 1` guard keeps every percentage at 0
+ stats = qc_parsers.consensus_stats(write(tmp_path / "empty.fa", ">S1 empty\n"))
+ assert stats["length"] == 0
+ assert stats["missing_pct"] == 0.0
+ assert stats["iupac_pct"] == 0.0
+ assert stats["callable_pct"] == 0.0
+ assert stats["longest_n_run"] == 0
+ assert stats["n_gaps"] == 0
+ assert stats["miss_profile"] == [0.0] * qc_parsers.NBINS
+
+
+def test_consensus_stats_of_a_completely_empty_file_has_no_id(tmp_path):
+ stats = qc_parsers.consensus_stats(write(tmp_path / "empty.fa", ""))
+ assert stats["consensus_id"] is None
+ assert stats["length"] == 0
+
+
+def test_consensus_stats_of_a_missing_file_raises(tmp_path):
+ with pytest.raises(OSError):
+ qc_parsers.consensus_stats(str(tmp_path / "nope.fa"))
+
+
+# --------------------------------------------------------------------------- parse_lineage_colors
+
+
+def test_parse_lineage_colors_reads_a_palette_tsv(tmp_path):
+ path = write(tmp_path / "colors.tsv",
+ "# lineage palette\n"
+ "L1\t#ff0000\tIndo-Oceanic\n"
+ "L2\t#0f0\tEast-Asian\n"
+ "\n"
+ "L3\t#00ff00\n")
+ assert qc_parsers.parse_lineage_colors(path) == {"L1": "#ff0000", "L2": "#0f0", "L3": "#00ff00"}
+
+
+def test_parse_lineage_colors_rejects_anything_that_is_not_a_hex_colour(tmp_path):
+ path = write(tmp_path / "colors.tsv",
+ "L1\tred\n" # not a hex colour
+ "L2\t#ff00\n" # wrong length
+ "L3\t#abcdef\n"
+ "L4\n") # no colour column
+ assert qc_parsers.parse_lineage_colors(path) == {"L3": "#abcdef"}
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_lineage_colors_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_lineage_colors(path) == {}
+
+
+# --------------------------------------------------------------------------- parse_bed
+
+
+BED = ("# a masking BED\n"
+ "track name=mask\n"
+ "browser hide all\n"
+ "chr\t0\t100\tPE_PGRS\n"
+ "chr\t500\t900\n"
+ "\n"
+ "chr\tbad\t900\n"
+ "chr\t950\n")
+
+
+def test_parse_bed_reads_the_half_open_intervals(tmp_path):
+ assert qc_parsers.parse_bed(write(tmp_path / "mask.bed", BED)) == [(0, 100), (500, 900)]
+
+
+def test_parse_bed_reads_a_gzipped_bed(tmp_path):
+ assert qc_parsers.parse_bed(write_gz(tmp_path / "mask.bed.gz", BED)) == [(0, 100), (500, 900)]
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_bed_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_bed(path) == []
+
+
+def test_parse_bed_of_an_empty_file_is_empty(tmp_path):
+ assert qc_parsers.parse_bed(write(tmp_path / "mask.bed", "")) == []
+
+
+# --------------------------------------------------------------------------- parse_gene_burden
+
+
+GENE_BURDEN = (
+ "gene\thigh\tmoderate\tlow\tmodifier\tdominant_effect\tn_samples\ttotal_impactful\tstart\tend\n"
+ "katG\t3\t1\t0\t5\tstop_gained\t4\t4\t3301\t4700\n"
+ "\n"
+ "rpoB\t1\t5\t2\t1\tmissense_variant\t3\t6\t1601\t3100\n"
+)
+
+
+def test_parse_gene_burden_sorts_by_descending_impactful_count(tmp_path):
+ rows = qc_parsers.parse_gene_burden(write(tmp_path / "burden.tsv", GENE_BURDEN))
+ assert [r["gene"] for r in rows] == ["rpoB", "katG"]
+ assert rows[0] == {"gene": "rpoB", "high": 1, "moderate": 5, "low": 2, "modifier": 1,
+ "dominant_effect": "missense_variant", "n_samples": 3,
+ "total_impactful": 6, "start": 1601, "end": 3100}
+
+
+def test_parse_gene_burden_derives_a_missing_total_from_high_plus_moderate(tmp_path):
+ path = write(tmp_path / "burden.tsv",
+ "gene\thigh\tmoderate\n"
+ "katG\t3\t4\n"
+ "rpoB\t\t\n")
+ rows = qc_parsers.parse_gene_burden(path)
+ assert rows[0]["total_impactful"] == 7
+ assert rows[1] == {"gene": "rpoB", "high": 0, "moderate": 0, "low": None, "modifier": None,
+ "dominant_effect": "NA", "n_samples": None, "total_impactful": 0,
+ "start": None, "end": None}
+
+
+def test_parse_gene_burden_skips_rows_without_a_gene(tmp_path):
+ path = write(tmp_path / "burden.tsv", GENE_BURDEN + "\t9\t9\t0\t0\tx\t1\t18\t1\t2\n")
+ assert [r["gene"] for r in qc_parsers.parse_gene_burden(path)] == ["rpoB", "katG"]
+
+
+def test_parse_gene_burden_is_capped_at_two_hundred_genes(tmp_path):
+ body = "".join(f"gene{i:04d}\t1\t1\t0\t0\teff\t1\t{i}\t1\t2\n" for i in range(250))
+ path = write(tmp_path / "burden.tsv", GENE_BURDEN.splitlines()[0] + "\n" + body)
+ rows = qc_parsers.parse_gene_burden(path)
+ assert len(rows) == 200
+ assert rows[0]["gene"] == "gene0249" # the highest total_impactful survives the cap
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_gene_burden_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_gene_burden(path) == []
+
+
+def test_parse_gene_burden_of_an_empty_file_is_empty(tmp_path):
+ assert qc_parsers.parse_gene_burden(write(tmp_path / "burden.tsv", "")) == []
+
+
+# --------------------------------------------------------------------------- parse_pnps
+
+
+PNPS = (
+ "gene\tgene_name\tn_samples\tn_pairs\tmean_dN\tmean_dS\tdNdS\teffect_dominant\n"
+ "TEST_0001\tdnaA\t4\t6\t0.0012\t0.0009\t1.33\tmissense\n"
+ "\n"
+ "TEST_0002\trpoB\t4\t6\t0.0000\t0.0000\tNA\tsynonymous\n"
+ "TEST_0003\tkatG\t4\t6\t0.0020\t0.0000\tinf\tmissense\n"
+)
+
+
+def test_parse_pnps_reads_the_eskaks_aggregate_columns(tmp_path):
+ rows = qc_parsers.parse_pnps(write(tmp_path / "pnps.tsv", PNPS))
+ assert [r["gene"] for r in rows] == ["dnaA", "rpoB", "katG"]
+ assert rows[0] == {"gene": "dnaA", "pn": pytest.approx(0.0012), "ps": pytest.approx(0.0009),
+ "pnps": pytest.approx(1.33), "n": 6.0, "effect": "missense"}
+
+
+@pytest.mark.parametrize("row_index", [1, 2])
+def test_parse_pnps_reads_the_non_finite_sentinels_as_none(tmp_path, row_index):
+ # NA and inf must never be blindly float()-ed: an inf dN/dS would poison every downstream axis
+ rows = qc_parsers.parse_pnps(write(tmp_path / "pnps.tsv", PNPS))
+ assert rows[row_index]["pnps"] is None
+
+
+def test_parse_pnps_falls_back_to_the_gene_column(tmp_path):
+ path = write(tmp_path / "pnps.tsv", "gene\tdNdS\nTEST_0001\t0.5\n")
+ assert qc_parsers.parse_pnps(path) == [{"gene": "TEST_0001", "pn": None, "ps": None,
+ "pnps": 0.5, "n": None, "effect": None}]
+
+
+def test_parse_pnps_skips_rows_without_a_gene(tmp_path):
+ path = write(tmp_path / "pnps.tsv", "gene_name\tdNdS\n\t0.5\ndnaA\t0.7\n")
+ assert [r["gene"] for r in qc_parsers.parse_pnps(path)] == ["dnaA"]
+
+
+def test_parse_pnps_is_capped_at_three_hundred_genes(tmp_path):
+ body = "".join(f"g{i:04d}\t0.5\n" for i in range(320))
+ assert len(qc_parsers.parse_pnps(write(tmp_path / "pnps.tsv", "gene\tdNdS\n" + body))) == 300
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_pnps_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_pnps(path) == []
+
+
+def test_parse_pnps_of_an_empty_file_is_empty(tmp_path):
+ assert qc_parsers.parse_pnps(write(tmp_path / "pnps.tsv", "")) == []
+
+
+# --------------------------------------------------------------------------- parse_dr
+
+
+DR = (
+ "# pathotypr drug-resistance calls\n"
+ "sample\tdrug\tgene\tmutation\tgrade\tmarker_name\taf\tdp\n"
+ "S1\tINH\tkatG\tS315T\t1) Assoc w R\tkatG_S315T\t0.98\t40\n"
+ "S1\tRIF\trpoB\tS450L\t2) Assoc w R - Interim\trpoB_S450L\t0.55\t22\n"
+ "S2\tEMB\tembB\tM306V\tUncertain\t\tNA\t.\n"
+)
+
+
+def test_parse_dr_reads_the_calls_samples_and_drug_order(tmp_path):
+ dr = qc_parsers.parse_dr(write(tmp_path / "dr.tsv", DR))
+ assert dr["samples"] == ["S1", "S2"]
+ # drugs follow the curated first/second-line order, not alphabetical order
+ assert dr["drugs"] == ["RIF", "INH", "EMB"]
+ assert len(dr["calls"]) == 3
+ assert dr["calls"][0] == {"s": "S1", "drug": "INH", "gene": "katG", "mutation": "S315T",
+ "grade": "1) Assoc w R", "gn": 1, "marker": "katG_S315T",
+ "af": pytest.approx(0.98), "dp": 40}
+
+
+def test_parse_dr_extracts_the_leading_who_grade_number(tmp_path):
+ calls = qc_parsers.parse_dr(write(tmp_path / "dr.tsv", DR))["calls"]
+ assert [c["gn"] for c in calls] == [1, 2, None]
+
+
+def test_parse_dr_reads_the_missing_sentinels_as_none(tmp_path):
+ last = qc_parsers.parse_dr(write(tmp_path / "dr.tsv", DR))["calls"][-1]
+ assert last["af"] is None
+ assert last["dp"] is None
+ assert last["marker"] == ""
+
+
+def test_parse_dr_falls_back_to_the_marker_column(tmp_path):
+ path = write(tmp_path / "dr.tsv", "sample\tdrug\tmarker\nS1\tRIF\tm1\n")
+ assert qc_parsers.parse_dr(path)["calls"][0]["marker"] == "m1"
+
+
+def test_parse_dr_puts_an_unknown_drug_last(tmp_path):
+ path = write(tmp_path / "dr.tsv", "sample\tdrug\nS1\tZZZ\nS1\tRIF\n")
+ assert qc_parsers.parse_dr(path)["drugs"] == ["RIF", "ZZZ"]
+
+
+def test_parse_dr_skips_blank_lines_and_rows_without_a_sample(tmp_path):
+ path = write(tmp_path / "dr.tsv", "sample\tdrug\n\nS1\tRIF\n\t INH\n")
+ dr = qc_parsers.parse_dr(path)
+ assert dr["samples"] == ["S1"]
+ assert len(dr["calls"]) == 1
+
+
+def test_parse_dr_reads_a_float_formatted_depth(tmp_path):
+ """A depth written as '40.0' is still a depth of 40, not a parse failure."""
+ path = write(tmp_path / "dr.tsv", "sample\tdrug\tdp\nS1\tRIF\t40.0\n")
+ assert qc_parsers.parse_dr(path)["calls"][0]["dp"] == 40
+
+
+def test_parse_dr_of_a_non_numeric_depth_is_none(tmp_path):
+ path = write(tmp_path / "dr.tsv", "sample\tdrug\tdp\nS1\tRIF\tdeep\n")
+ assert qc_parsers.parse_dr(path)["calls"][0]["dp"] is None
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_dr_of_an_absent_file_is_none(path):
+ assert qc_parsers.parse_dr(path) is None
+
+
+def test_parse_dr_without_any_call_is_none(tmp_path):
+ assert qc_parsers.parse_dr(write(tmp_path / "dr.tsv", "")) is None
+ assert qc_parsers.parse_dr(write(tmp_path / "dr2.tsv", "sample\tdrug\n")) is None
+
+
+# --------------------------------------------------------------------------- parse_kraken
+
+
+KRAKEN = (
+ " 12.00\t1200\t1200\tU\t0\tunclassified\n"
+ " 88.00\t8800\t100\tR\t1\troot\n"
+ " 70.00\t7000\t7000\tS\t1773\t Mycobacterium tuberculosis\n"
+ " 10.00\t1000\t1000\tS\t1280\t Staphylococcus aureus\n"
+ "short line\n"
+ "not-a-number\t1\t1\tS\t9\t Bogus\n"
+)
+
+
+def test_parse_kraken_summarises_the_species_composition(tmp_path):
+ path = write(tmp_path / "S1__RUN1.kraken.report", KRAKEN)
+ sample = qc_parsers.parse_kraken([path])["samples"][0]
+ assert sample["s"] == "S1"
+ assert sample["unclassified"] == 12.0
+ assert sample["classified"] == 88.0
+ assert sample["primary"] == {"name": "Mycobacterium tuberculosis", "pct": 70.0}
+ assert sample["secondary"] == {"name": "Staphylococcus aureus", "pct": 10.0}
+ assert len(sample["top"]) == 2
+
+
+def test_parse_kraken_derives_the_sample_id_from_the_file_name(tmp_path):
+ a = write(tmp_path / "S1__RUN1.kraken.report", KRAKEN)
+ b = write(tmp_path / "S2.kraken.report", KRAKEN)
+ assert [s["s"] for s in qc_parsers.parse_kraken([a, b])["samples"]] == ["S1", "S2"]
+
+
+def test_parse_kraken_keeps_only_the_top_six_species(tmp_path):
+ body = "".join(f"{i}.00\t10\t10\tS\t{i}\ttaxon{i}\n" for i in range(1, 12))
+ path = write(tmp_path / "S1.kraken.report", body)
+ sample = qc_parsers.parse_kraken([path])["samples"][0]
+ assert len(sample["top"]) == 6
+ assert sample["primary"]["name"] == "taxon11" # sorted by descending percent
+
+
+def test_parse_kraken_keeps_a_fully_unclassified_sample(tmp_path):
+ path = write(tmp_path / "S1.kraken.report", "100.00\t10\t10\tU\t0\tunclassified\n")
+ sample = qc_parsers.parse_kraken([path])["samples"][0]
+ assert sample["unclassified"] == 100.0
+ assert sample["primary"] is None
+ assert sample["secondary"] is None
+
+
+def test_parse_kraken_skips_an_uninformative_report(tmp_path):
+ path = write(tmp_path / "S1.kraken.report", "88.00\t8800\t100\tR\t1\troot\n")
+ assert qc_parsers.parse_kraken([path]) is None
+
+
+@pytest.mark.parametrize("paths", [None, [], [MISSING], ["", None]])
+def test_parse_kraken_without_any_readable_report_is_none(paths):
+ assert qc_parsers.parse_kraken(paths) is None
+
+
+# --------------------------------------------------------------------------- parse_gff / parse_gene_locus
+
+
+def test_parse_gff_reads_the_committed_reference_annotation(data_dir):
+ genes = qc_parsers.parse_gff(str(data_dir / "reference.gff3"))
+ assert genes == [{"name": "dnaA", "start": 201, "end": 1400},
+ {"name": "rpoB", "start": 1601, "end": 3100},
+ {"name": "katG", "start": 3301, "end": 4700}]
+
+
+def test_parse_gff_reads_a_gzipped_gff(tmp_path, data_dir):
+ text = (data_dir / "reference.gff3").read_text(encoding="utf-8")
+ assert qc_parsers.parse_gff(write_gz(tmp_path / "ref.gff3.gz", text)) == \
+ qc_parsers.parse_gff(str(data_dir / "reference.gff3"))
+
+
+def test_parse_gff_prefers_a_gene_feature_over_a_cds_of_the_same_name(tmp_path):
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tCDS\t50\t60\t.\t+\t0\tName=abc\n"
+ "chr\tt\tgene\t10\t100\t.\t+\t.\tName=abc\n")
+ assert qc_parsers.parse_gff(path) == [{"name": "abc", "start": 10, "end": 100}]
+
+
+def test_parse_gff_falls_back_through_the_attribute_keys_then_to_the_coordinates(tmp_path):
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t10\t20\t.\t+\t.\tlocus_tag=LT1\n"
+ "chr\tt\tgene\t30\t40\t.\t+\t.\tID=only-an-id\n"
+ "chr\tt\tgene\t50\t60\t.\t+\t.\t.\n")
+ assert [g["name"] for g in qc_parsers.parse_gff(path)] == ["LT1", "only-an-id", "50-60"]
+
+
+def test_parse_gff_skips_non_gene_features_headers_and_bad_coordinates(tmp_path):
+ path = write(tmp_path / "ref.gff3",
+ "##gff-version 3\n"
+ "chr\tt\tregion\t1\t5000\t.\t+\t.\tName=chr\n"
+ "chr\tt\tgene\tx\t20\t.\t+\t.\tName=bad\n"
+ "a line with no tabs\n"
+ "chr\tt\tgene\t10\t20\t.\t+\t.\tName=ok\n")
+ assert [g["name"] for g in qc_parsers.parse_gff(path)] == ["ok"]
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_gff_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_gff(path) == []
+
+
+def test_parse_gff_attribute_lookup_matches_the_whole_key_not_a_suffix(tmp_path):
+ """The attribute key is matched whole, so 'locus_tag' is not read out of 'old_locus_tag'
+ and 'gene' is not read out of 'pseudogene'. Both attributes are legal GFF3 and routine in
+ RefSeq / Prokka output, so a substring search named features after an obsolete tag or after
+ a pseudogene TYPE ('unitary')."""
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t1\t100\t.\t+\t.\tID=g1;old_locus_tag=OLDTAG;locus_tag=NEWTAG\n"
+ "chr\tt\tgene\t200\t300\t.\t+\t.\tID=g2;pseudogene=unitary;locus_tag=LT2\n")
+ assert [g["name"] for g in qc_parsers.parse_gff(path)] == ["NEWTAG", "LT2"]
+
+
+def test_parse_gff_tolerates_spaces_around_the_attribute_separator(tmp_path):
+ path = write(tmp_path / "ref.gff3", "chr\tt\tgene\t1\t100\t.\t+\t.\tID=g1; locus_tag=LT1 ; note=x\n")
+ assert [g["name"] for g in qc_parsers.parse_gff(path)] == ["LT1"]
+
+
+def test_parse_gff_keeps_an_equals_sign_inside_an_attribute_value(tmp_path):
+ """Only the FIRST '=' splits key from value, so a value containing '=' survives intact."""
+ path = write(tmp_path / "ref.gff3", "chr\tt\tgene\t1\t100\t.\t+\t.\tName=a=b;locus_tag=LT1\n")
+ assert [g["name"] for g in qc_parsers.parse_gff(path)] == ["a=b"]
+
+
+def test_parse_gene_locus_maps_the_committed_gene_names_to_locus_tags(data_dir):
+ assert qc_parsers.parse_gene_locus(str(data_dir / "reference.gff3")) == {
+ "dnaA": "TEST_0001", "rpoB": "TEST_0002", "katG": "TEST_0003"}
+
+
+def test_parse_gene_locus_keeps_the_first_locus_tag_seen_for_a_gene(tmp_path):
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t10\t20\t.\t+\t.\tgene=rpoB;locus_tag=FIRST\n"
+ "chr\tt\tCDS\t10\t20\t.\t+\t0\tgene=rpoB;locus_tag=SECOND\n")
+ assert qc_parsers.parse_gene_locus(path) == {"rpoB": "FIRST"}
+
+
+def test_parse_gene_locus_needs_both_a_name_and_a_locus_tag(tmp_path):
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t10\t20\t.\t+\t.\tlocus_tag=NO_NAME\n"
+ "chr\tt\tgene\t30\t40\t.\t+\t.\tName=no_locus\n"
+ "chr\tt\tgene\t50\t60\t.\t+\t.\tName=both;locus_tag=LT9\n")
+ assert qc_parsers.parse_gene_locus(path) == {"both": "LT9"}
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_gene_locus_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_gene_locus(path) == {}
+
+
+def test_parse_gene_locus_attribute_lookup_matches_the_whole_key_not_a_suffix(tmp_path):
+ """Same whole-key match as parse_gff: 'old_locus_tag' must not answer for 'locus_tag'
+ (that mapped the gene to its OBSOLETE tag) and 'pseudogene' must not answer for 'gene'
+ (that turned the pseudogene TYPE into a gene name)."""
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t1\t100\t.\t+\t.\tID=g1;old_locus_tag=MTV_OLD;locus_tag=Rv0667;gene=rpoB\n"
+ "chr\tt\tgene\t200\t300\t.\t+\t.\tID=g2;pseudogene=unitary;locus_tag=LT2\n")
+ assert qc_parsers.parse_gene_locus(path) == {"rpoB": "Rv0667"}
+
+
+def test_build_gene_map_takes_the_real_locus_tag_not_the_obsolete_one(tmp_path):
+ """build_gene_map overlays H37Rv-style tags from the GFF onto the curated map. An Rv-style
+ 'old_locus_tag' read in place of 'locus_tag' passed the _RV_LOCUS_RE filter and silently
+ overrode the curated locus with a retired one."""
+ path = write(tmp_path / "ref.gff3",
+ "chr\tt\tgene\t1\t100\t.\t+\t.\tgene=rpoB;old_locus_tag=Rv9999;locus_tag=Rv0667\n")
+ assert qc_metrics.build_gene_map(path)["rpoB"] == "Rv0667"
+
+
+# --------------------------------------------------------------------------- samplesheet parsers
+
+
+ANNOTATED_SHEET = (
+ "sampleId\trunId\tr1\tr2\tpassage\tpatient\ttreatment\tdose\n"
+ "S1\tRUN1\ta_R1.fq.gz\ta_R2.fq.gz\tP1\tPAT1\tdrugA\t10\n"
+ "S2\tRUN1\tb_R1.fq.gz\tb_R2.fq.gz\tP2\tPAT1\tdrugA\t20\n"
+ "S3\tRUN1\tc_R1.fq.gz\tc_R2.fq.gz\tP1\tPAT2\tcontrol\t\n"
+)
+
+
+def test_parse_sample_meta_keeps_only_the_user_annotation_columns(tmp_path):
+ meta = qc_parsers.parse_sample_meta(write(tmp_path / "sheet.tsv", ANNOTATED_SHEET))
+ # the sample id, the pipeline's file/reference columns and the numeric dose are all excluded
+ assert meta["fields"] == ["passage", "patient", "treatment"]
+ assert meta["rows"]["S1"] == {"passage": "P1", "patient": "PAT1", "treatment": "drugA"}
+ assert sorted(meta["rows"]) == ["S1", "S2", "S3"]
+
+
+def test_parse_sample_meta_tags_the_time_group_and_treatment_columns(tmp_path):
+ meta = qc_parsers.parse_sample_meta(write(tmp_path / "sheet.tsv", ANNOTATED_SHEET))
+ assert meta["time_field"] == "passage"
+ assert meta["group_field"] == "patient"
+ assert meta["tx_field"] == "treatment"
+
+
+def test_parse_sample_meta_keeps_the_first_row_of_a_repeated_sample(tmp_path):
+ path = write(tmp_path / "sheet.tsv", ANNOTATED_SHEET +
+ "S1\tRUN2\td_R1.fq.gz\td_R2.fq.gz\tP9\tPAT9\tother\t99\n")
+ assert qc_parsers.parse_sample_meta(path)["rows"]["S1"]["patient"] == "PAT1"
+
+
+def test_parse_sample_meta_of_the_committed_samplesheet_is_none(data_dir):
+ # the fixture samplesheet carries only pipeline columns (runId / reads / reference / taxId),
+ # so there is no user annotation to show and the panel self-hides
+ assert qc_parsers.parse_sample_meta(str(data_dir / "samplesheet.tsv")) is None
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_sample_meta_of_an_absent_file_is_none(path):
+ assert qc_parsers.parse_sample_meta(path) is None
+
+
+def test_parse_sample_meta_without_rows_or_header_is_none(tmp_path):
+ assert qc_parsers.parse_sample_meta(write(tmp_path / "sheet.tsv", "")) is None
+ assert qc_parsers.parse_sample_meta(write(tmp_path / "h.tsv", "sampleId\tpassage\n")) is None
+
+
+def test_parse_dose_reads_the_numeric_dose_column(tmp_path):
+ # S3 has an empty dose and is simply absent from the map
+ assert qc_parsers.parse_dose(write(tmp_path / "sheet.tsv", ANNOTATED_SHEET)) == {"S1": 10.0, "S2": 20.0}
+
+
+def test_parse_dose_accepts_the_spanish_column_name(tmp_path):
+ path = write(tmp_path / "sheet.tsv", "sampleId\tdosis\nS1\t2.5\n")
+ assert qc_parsers.parse_dose(path) == {"S1": 2.5}
+
+
+def test_parse_dose_without_a_dose_column_is_empty(tmp_path, data_dir):
+ assert qc_parsers.parse_dose(write(tmp_path / "sheet.tsv", "sampleId\tpassage\nS1\tP1\n")) == {}
+ assert qc_parsers.parse_dose(str(data_dir / "samplesheet.tsv")) == {}
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_dose_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_dose(path) == {}
+
+
+def test_parse_metadata_auto_detects_the_time_and_group_columns(tmp_path):
+ meta = qc_parsers.parse_metadata(write(tmp_path / "sheet.tsv", ANNOTATED_SHEET))
+ assert meta == {"S1": {"time": "P1", "tnum": 1.0, "group": "PAT1"},
+ "S2": {"time": "P2", "tnum": 2.0, "group": "PAT1"},
+ "S3": {"time": "P1", "tnum": 1.0, "group": "PAT2"}}
+
+
+def test_parse_metadata_of_the_committed_samplesheet_has_no_time_or_group(data_dir):
+ # one entry per sample (the two SAMPLE_A runs collapse to the first), but nothing to plot
+ meta = qc_parsers.parse_metadata(str(data_dir / "samplesheet.tsv"))
+ assert sorted(meta) == ["SAMPLE_A", "SAMPLE_B", "SAMPLE_C"]
+ assert all(v == {"time": None, "tnum": None, "group": None} for v in meta.values())
+
+
+def test_parse_metadata_skips_comments_and_blank_lines(tmp_path):
+ path = write(tmp_path / "sheet.tsv",
+ "# a comment\n\nsample\ttimepoint\tpatient\nS1\tday 3\tPAT1\n")
+ assert qc_parsers.parse_metadata(path) == {"S1": {"time": "day 3", "tnum": 3.0, "group": "PAT1"}}
+
+
+@pytest.mark.parametrize("path", [None, "", MISSING])
+def test_parse_metadata_of_an_absent_file_is_empty(path):
+ assert qc_parsers.parse_metadata(path) == {}
+
+
+def test_parse_metadata_of_an_empty_file_is_empty(tmp_path):
+ assert qc_parsers.parse_metadata(write(tmp_path / "sheet.tsv", "")) == {}
+
+
+@pytest.mark.parametrize(("parser", "empty"), [
+ (qc_parsers.parse_sample_meta, None), (qc_parsers.parse_dose, {}), (qc_parsers.parse_metadata, {})])
+def test_samplesheet_parsers_skip_a_row_shorter_than_the_sample_column(tmp_path, parser, empty):
+ path = write(tmp_path / "sheet.tsv", "# header comment\nrunId\tsampleId\tdose\n\nRUN1\n")
+ assert parser(path) == empty
+
+
+def test_parse_sample_meta_skips_a_row_shorter_than_the_sample_column(tmp_path):
+ path = write(tmp_path / "sheet.tsv", "runId\tsampleId\tpassage\nRUN1\nRUN2\tS2\tP2\n")
+ assert list(qc_parsers.parse_sample_meta(path)["rows"]) == ["S2"]
+
+
+def test_parse_dose_of_an_empty_file_is_empty(tmp_path):
+ assert qc_parsers.parse_dose(write(tmp_path / "sheet.tsv", "")) == {}
+
+
+def test_parse_dose_keeps_the_first_row_of_a_repeated_sample(tmp_path):
+ # the samplesheet has one row per run, so a multi-run sample appears more than once
+ path = write(tmp_path / "sheet.tsv",
+ "sampleId\trunId\tdose\nS1\tRUN1\t10\nS1\tRUN2\t99\n\tRUN3\t5\n")
+ assert qc_parsers.parse_dose(path) == {"S1": 10.0}
+
+
+def test_parse_dose_skips_a_row_without_a_dose_cell(tmp_path):
+ path = write(tmp_path / "sheet.tsv", "sampleId\tpassage\tdose\nS1\tP1\nS2\tP2\t5\n")
+ assert qc_parsers.parse_dose(path) == {"S2": 5.0}
+
+
+# --------------------------------------------------------------------------- parse_vcfs
+
+
+ANN = ("ANN=T|missense_variant|MODERATE|rpoB|GENE_rpoB|transcript|TX1|protein_coding|"
+ "1/1|c.1349C>T|p.Ser450Leu|450/3519|")
+VCF = (
+ "##fileformat=VCFv4.2\n"
+ "##contig=\n"
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+ f"chr\t100\t.\tC\tT\t60\tPASS\t{ANN}\tGT:AD\t0/1:2,8\n"
+ "chr\t200\t.\tA\tAT\t60\tPASS\t.\tGT:AD\t1/1:0,10\n" # an insertion: not a SNP
+ "chr\t300\t.\tAC\tA\t60\tPASS\t.\tGT:AD\t1/1:0,10\n" # a deletion: not a SNP
+ "chr\t400\t.\tA\t.\t60\tPASS\t.\tGT:AD\t0/0:10,0\n" # no alt allele
+ "chr\t500\t.\tG\tA\t60\tPASS\t.\tGT:DP\t./.:30\n" # no genotype -> no frequency
+ "chr\t600\t.\tG\tA\t60\tPASS\t.\tGT:DP:AD\t1/1:33:1,32\n"
+)
+
+
+def test_parse_vcfs_keeps_only_snps_with_a_usable_frequency(tmp_path):
+ variants = qc_parsers.parse_vcfs([write(tmp_path / "S1.vcf", VCF)])
+ assert list(variants) == ["S1"]
+ assert sorted(variants["S1"]) == ["chr:100", "chr:600"]
+
+
+def test_parse_vcfs_decodes_the_frequency_depth_and_annotation(tmp_path):
+ variants = qc_parsers.parse_vcfs([write(tmp_path / "S1.vcf", VCF)])
+ assert variants["S1"]["chr:100"] == {
+ "ref": "C", "alt": "T", "gene": "rpoB", "eff": "missense_variant", "imp": "MODERATE",
+ "aa": "p.Ser450Leu", "aa_h37rv": "", "af": 0.8, "dp": 10, "opos": None}
+ # AD wins over the 1/1 genotype dosage, and the explicit DP wins over sum(AD)
+ assert variants["S1"]["chr:600"]["af"] == 0.9697
+ assert variants["S1"]["chr:600"]["dp"] == 33
+ assert variants["S1"]["chr:600"]["gene"] == ""
+
+
+def test_parse_vcfs_takes_the_sample_from_the_chrom_header_not_the_file_name(tmp_path):
+ path = write(tmp_path / "unrelated_name.vcf", VCF.replace("\tS1\n", "\tREAL_ID\n"))
+ assert list(qc_parsers.parse_vcfs([path])) == ["REAL_ID"]
+
+
+def test_parse_vcfs_falls_back_to_the_file_stem_for_a_sampleless_vcf(tmp_path):
+ path = write(tmp_path / "SAMPLE_Z.vcf",
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\n"
+ "chr\t100\t.\tC\tT\t60\tPASS\t.\n")
+ variants = qc_parsers.parse_vcfs([path])
+ assert list(variants) == ["SAMPLE_Z"]
+ # with no FORMAT/sample column the record is taken at face value: fixed, unknown depth
+ assert variants["SAMPLE_Z"]["chr:100"]["af"] == 1.0
+ assert variants["SAMPLE_Z"]["chr:100"]["dp"] is None
+
+
+def test_parse_vcfs_reads_a_gzipped_vcf(tmp_path):
+ assert qc_parsers.parse_vcfs([write_gz(tmp_path / "S1.vcf.gz", VCF)]) == \
+ qc_parsers.parse_vcfs([write(tmp_path / "S1.vcf", VCF)])
+
+
+def test_parse_vcfs_records_the_liftover_original_position(tmp_path):
+ path = write(tmp_path / "S1.vcf",
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+ "chr\t100\t.\tC\tT\t60\tPASS\tOriginalContig=used;OriginalStart=77\tGT\t1/1\n")
+ assert qc_parsers.parse_vcfs([path])["S1"]["chr:100"]["opos"] == "used:77"
+
+
+def test_parse_vcfs_keeps_the_first_alt_of_a_multiallelic_snp(tmp_path):
+ path = write(tmp_path / "S1.vcf",
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+ "chr\t100\t.\tC\tT,G\t60\tPASS\t.\tGT:AD\t1/2:1,5,4\n")
+ record = qc_parsers.parse_vcfs([path])["S1"]["chr:100"]
+ assert record["alt"] == "T"
+ assert record["af"] == 0.9
+
+
+def test_parse_vcfs_merges_several_files(tmp_path):
+ a = write(tmp_path / "a.vcf", VCF)
+ b = write(tmp_path / "b.vcf", VCF.replace("\tS1\n", "\tS2\n"))
+ assert sorted(qc_parsers.parse_vcfs([a, b])) == ["S1", "S2"]
+
+
+def test_parse_vcfs_without_a_chrom_header_yields_nothing(tmp_path):
+ path = write(tmp_path / "S1.vcf", "##fileformat=VCFv4.2\nchr\t100\t.\tC\tT\t60\tPASS\t.\n")
+ assert qc_parsers.parse_vcfs([path]) == {}
+
+
+def test_parse_vcfs_skips_a_truncated_data_line(tmp_path):
+ path = write(tmp_path / "S1.vcf",
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+ "chr\t100\t.\tC\n"
+ "chr\t200\t.\tC\tT\t60\tPASS\t.\tGT\t1/1\n")
+ assert list(qc_parsers.parse_vcfs([path])["S1"]) == ["chr:200"]
+
+
+@pytest.mark.parametrize("paths", [None, [], [MISSING], ["", None]])
+def test_parse_vcfs_without_any_readable_file_is_empty(paths):
+ assert qc_parsers.parse_vcfs(paths) == {}
+
+
+# --------------------------------------------------------------------------- mapDamage readers
+
+
+FREQ = "pos\t5pC>T\n1\t0.25\n2\t0.12\n3\t0.05\n"
+LGDIST = "# table produced by mapDamage\nStd\tLength\tOccurences\n-\t50\t10\n-\t100\t10\n"
+
+
+def test_read_freq_file_reads_the_terminal_damage_profile(tmp_path):
+ freq = qc_parsers._read_freq_file(write(tmp_path / "5pCtoT_freq.txt", FREQ))
+ assert freq["pos1"] == pytest.approx(0.25)
+ assert freq["profile"] == [[1, 0.25], [2, 0.12], [3, 0.05]]
+
+
+def test_read_freq_file_accepts_comma_and_space_separated_rows(tmp_path):
+ comma = qc_parsers._read_freq_file(write(tmp_path / "a.txt", "1,0.25\n2,0.12\n"))
+ spaced = qc_parsers._read_freq_file(write(tmp_path / "b.txt", "1 0.25\n2 0.12\n"))
+ assert comma == spaced
+ assert comma["pos1"] == pytest.approx(0.25)
+
+
+def test_read_freq_file_sorts_by_position_and_caps_the_profile(tmp_path):
+ body = "".join(f"{p}\t0.1\n" for p in range(30, 0, -1))
+ freq = qc_parsers._read_freq_file(write(tmp_path / "freq.txt", body))
+ assert len(freq["profile"]) == 25
+ assert freq["profile"][0][0] == 1
+
+
+def test_read_freq_file_honours_max_pos(tmp_path):
+ freq = qc_parsers._read_freq_file(write(tmp_path / "freq.txt", FREQ), max_pos=2)
+ assert len(freq["profile"]) == 2
+
+
+def test_read_freq_file_falls_back_to_the_first_row_without_a_position_one(tmp_path):
+ freq = qc_parsers._read_freq_file(write(tmp_path / "freq.txt", "3\t0.05\n2\t0.12\n"))
+ assert freq["pos1"] == pytest.approx(0.12)
+
+
+def test_read_freq_file_skips_malformed_rows(tmp_path):
+ freq = qc_parsers._read_freq_file(write(tmp_path / "freq.txt",
+ "1\t0.25\nonlyonefield\nx\ty\n2\tNA\n2\t0.12\n"))
+ assert freq["profile"] == [[1, 0.25], [2, 0.12]]
+
+
+def test_read_freq_file_of_an_empty_or_missing_file_is_none(tmp_path):
+ assert qc_parsers._read_freq_file(write(tmp_path / "freq.txt", "")) is None
+ assert qc_parsers._read_freq_file(write(tmp_path / "h.txt", "pos\tfreq\n")) is None
+ assert qc_parsers._read_freq_file(str(tmp_path / "nope.txt")) is None
+
+
+def test_mean_fraglen_is_the_occurrence_weighted_mean_length(tmp_path):
+ assert qc_parsers._mean_fraglen(write(tmp_path / "lgdistribution.txt", LGDIST)) == pytest.approx(75.0)
+
+
+def test_mean_fraglen_skips_comments_headers_and_short_rows(tmp_path):
+ path = write(tmp_path / "lg.txt", LGDIST + "-\t999\n" + "# trailing comment\n")
+ assert qc_parsers._mean_fraglen(path) == pytest.approx(75.0)
+
+
+def test_mean_fraglen_skips_a_row_with_a_non_numeric_length_or_count(tmp_path):
+ path = write(tmp_path / "lg.txt", LGDIST + "-\tlong\tmany\n-\t50\tNA\n")
+ assert qc_parsers._mean_fraglen(path) == pytest.approx(75.0)
+
+
+def test_mean_fraglen_without_any_occurrence_is_none(tmp_path):
+ assert qc_parsers._mean_fraglen(write(tmp_path / "lg.txt", "-\t50\t0\n")) is None
+ assert qc_parsers._mean_fraglen(write(tmp_path / "empty.txt", "")) is None
+ assert qc_parsers._mean_fraglen(str(tmp_path / "nope.txt")) is None
+
+
+def test_mapdamage_stats_reads_a_complete_sample_directory(tmp_path):
+ d = tmp_path / "S1"
+ d.mkdir()
+ write(d / "5pCtoT_freq.txt", FREQ)
+ write(d / "3pGtoA_freq.txt", "pos\t3pG>A\n1\t0.18\n2\t0.09\n")
+ write(d / "lgdistribution.txt", LGDIST)
+ stats = qc_parsers.mapdamage_stats(str(d))
+ assert stats["ct1"] == pytest.approx(0.25)
+ assert stats["ga1"] == pytest.approx(0.18)
+ assert stats["fraglen"] == pytest.approx(75.0)
+ assert stats["ct_profile"] == [[1, 0.25], [2, 0.12], [3, 0.05]]
+
+
+def test_mapdamage_stats_tolerates_a_partial_directory(tmp_path):
+ d = tmp_path / "S1"
+ d.mkdir()
+ write(d / "lgdistribution.txt", LGDIST)
+ assert qc_parsers.mapdamage_stats(str(d)) == {"ct1": None, "ga1": None,
+ "fraglen": pytest.approx(75.0), "ct_profile": None}
+
+
+def test_mapdamage_stats_of_an_empty_or_absent_directory_is_none(tmp_path):
+ empty = tmp_path / "empty"
+ empty.mkdir()
+ assert qc_parsers.mapdamage_stats(str(empty)) is None
+ assert qc_parsers.mapdamage_stats(str(tmp_path / "nope")) is None
+ assert qc_parsers.mapdamage_stats(None) is None
+ # a file where a directory is expected is not a mapDamage output
+ assert qc_parsers.mapdamage_stats(write(tmp_path / "a-file.txt", FREQ)) is None
+
+
+# --------------------------------------------------------------------------- unreadable optional inputs
+
+
+@pytest.mark.parametrize(("parser", "empty"), [
+ (qc_parsers.parse_lineage_colors, {}),
+ (qc_parsers.parse_bed, []),
+ (qc_parsers.parse_gene_burden, []),
+ (qc_parsers.parse_pnps, []),
+ (qc_parsers.parse_dr, None),
+ (qc_parsers.parse_gff, []),
+ (qc_parsers.parse_gene_locus, {}),
+ (qc_parsers.parse_sample_meta, None),
+ (qc_parsers.parse_dose, {}),
+ (qc_parsers.parse_metadata, {}),
+])
+def test_optional_parsers_degrade_quietly_on_an_unreadable_path(tmp_path, parser, empty):
+ """A path that exists but cannot be opened (here a directory) hides its panel rather
+ than crashing the whole report. Only the two required inputs, parse_summary and
+ consensus_stats, are allowed to raise."""
+ assert parser(str(tmp_path)) == empty
+
+
+@pytest.mark.parametrize(("parser", "empty"), [(qc_parsers.parse_kraken, None), (qc_parsers.parse_vcfs, {})])
+def test_multi_file_parsers_skip_an_unreadable_path(tmp_path, parser, empty):
+ assert parser([str(tmp_path)]) == empty
diff --git a/tests/unit/test_stats_to_legacy.py b/tests/unit/test_stats_to_legacy.py
new file mode 100644
index 0000000..77f5eb6
--- /dev/null
+++ b/tests/unit/test_stats_to_legacy.py
@@ -0,0 +1,1037 @@
+"""Unit tests for bin/stats_to_legacy.py.
+
+The script turns fastp JSON, samtools stats, a VCF and an optional pathotypr
+report into the flat legacy `.log`. Almost every parser here is written
+to swallow exceptions and hand back a zeroed default, so a large part of these
+tests exists to PIN that "no input file" never propagates an exception.
+"""
+
+from __future__ import annotations
+
+import gzip
+import subprocess
+import sys
+
+import pytest
+
+from conftest import load_script
+
+stl = load_script("stats_to_legacy")
+
+
+# --------------------------------------------------------------------------
+# helpers
+# --------------------------------------------------------------------------
+
+# snpEff ANN has a fixed 16-subfield layout; index 15 is ERRORS/WARNINGS/INFO.
+_ANN_TEMPLATE = [
+ "G", "missense_variant", "MODERATE", "geneA", "GENE_ID", "transcript",
+ "TX_ID", "protein_coding", "1/1", "c.100A>G", "p.Lys34Arg", "100/1000",
+ "100/900", "34/300", "", "",
+]
+
+
+def _ann_entry(allele="G", effect="missense_variant", impact="MODERATE", gene="geneA", errors=""):
+ sub = list(_ANN_TEMPLATE)
+ sub[0], sub[1], sub[2], sub[3], sub[15] = allele, effect, impact, gene, errors
+ return "|".join(sub)
+
+
+def _write_fai(tmp_path, rows, name="ref.fa.fai"):
+ p = tmp_path / name
+ p.write_text("".join(f"{contig}\t{length}\t0\t60\t61\n" for contig, length in rows))
+ return p
+
+
+_VCF_HEADER = (
+ "##fileformat=VCFv4.2\n"
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1\n"
+)
+
+
+def _rec(pos, ref, alt, gt, info="DP=10", contig="chr1", fmt="GT:DP", sample_extra="10"):
+ sample = gt if fmt == "GT" else f"{gt}:{sample_extra}"
+ return "\t".join([contig, str(pos), ".", ref, alt, "50", "PASS", info, fmt, sample])
+
+
+def _write_vcf(tmp_path, records, name="calls.vcf", header=_VCF_HEADER):
+ p = tmp_path / name
+ p.write_text(header + "".join(r + "\n" for r in records))
+ return p
+
+
+# --------------------------------------------------------------------------
+# _effect_class
+# --------------------------------------------------------------------------
+
+_EFFECT_CASES = [
+ ("frameshift_variant", "frameshift"),
+ # ORDER MATTERS: 'frameshift' is the first test in the cascade, so a term
+ # that also mentions stop_gained still resolves to the frameshift class.
+ ("frameshift_variant&stop_gained", "frameshift"),
+ ("stop_gained", "stop_gained"),
+ # stop_gained is checked before splice, so the compound term is a stop_gained.
+ ("stop_gained&splice_region_variant", "stop_gained"),
+ ("stop_lost", "stop_lost"),
+ ("stop_lost&splice_region_variant", "stop_lost"),
+ ("start_lost", "start_lost"),
+ ("missense_variant", "missense"),
+ # missense is checked before splice for the same reason.
+ ("missense_variant&splice_region_variant", "missense"),
+ ("stop_retained_variant", "synonymous"),
+ ("start_retained_variant", "synonymous"),
+ ("synonymous_variant", "synonymous"),
+ ("inframe_insertion", "inframe_indel"),
+ ("inframe_deletion", "inframe_indel"),
+ ("disruptive_inframe_deletion", "inframe_indel"),
+ ("conservative_inframe_insertion", "inframe_indel"),
+ ("splice_acceptor_variant", "splice"),
+ ("splice_donor_variant", "splice"),
+ ("splice_region_variant", "splice"),
+ ("intergenic_region", "intergenic"),
+ ("intragenic_variant", "intergenic"),
+ ("upstream_gene_variant", "regulatory"),
+ ("downstream_gene_variant", "regulatory"),
+ ("5_prime_utr_variant", "regulatory"),
+ ("3_prime_utr_premature_start_codon_gain_variant", "regulatory"),
+ ("regulatory_region_variant", "regulatory"),
+ ("tf_binding_site_variant", "regulatory"),
+ ("non_coding_transcript_exon_variant", "other"),
+ ("sequence_feature", "other"),
+ ("", "other"),
+]
+
+
+@pytest.mark.parametrize("effect,expected", _EFFECT_CASES)
+def test_effect_class(effect, expected):
+ assert stl._effect_class(effect) == expected
+
+
+def test_effect_class_cases_cover_every_declared_class():
+ """The parametrised table above must exercise all of _EFF_CLASSES."""
+ seen = {expected for _, expected in _EFFECT_CASES}
+ assert seen == set(stl._EFF_CLASSES)
+
+
+def test_effect_class_is_substring_based_not_exact():
+ """The cascade uses `in`, so any term containing the token is classified."""
+ assert stl._effect_class("some_prefix_missense_thing") == "missense"
+
+
+# --------------------------------------------------------------------------
+# _parse_ann_field
+# --------------------------------------------------------------------------
+
+def test_parse_ann_field_no_ann_returns_none():
+ assert stl._parse_ann_field("DP=10;AF=0.5;MQ=60") is None
+
+
+def test_parse_ann_field_empty_info_returns_none():
+ assert stl._parse_ann_field("") is None
+
+
+def test_parse_ann_field_empty_ann_value_returns_none():
+ """`ANN=` with nothing after it is falsy and treated as un-annotated."""
+ assert stl._parse_ann_field("DP=10;ANN=") is None
+
+
+def test_parse_ann_field_too_few_subfields_returns_none():
+ """Fewer subfields than the impact index (2) means malformed -> unannotated."""
+ assert stl._parse_ann_field("ANN=G|missense_variant") is None
+ assert stl._parse_ann_field("ANN=G") is None
+
+
+def test_parse_ann_field_exactly_three_subfields_is_usable():
+ """Three subfields is the minimum; gene and errors default to empty."""
+ cls, impact, gene, has_error, db_error = stl._parse_ann_field("ANN=G|missense_variant|HIGH")
+ assert (cls, impact, gene, has_error, db_error) == ("missense", "HIGH", "", False, False)
+
+
+def test_parse_ann_field_full_entry():
+ info = f"DP=10;ANN={_ann_entry(effect='stop_gained', impact='HIGH', gene='rpoB')};MQ=60"
+ assert stl._parse_ann_field(info) == ("stop_gained", "HIGH", "rpoB", False, False)
+
+
+def test_parse_ann_field_uses_first_ann_entry_only():
+ """snpEff pre-sorts entries by severity, so only the first one is read."""
+ info = "ANN=" + ",".join([
+ _ann_entry(effect="stop_gained", impact="HIGH", gene="geneA"),
+ _ann_entry(effect="synonymous_variant", impact="LOW", gene="geneB"),
+ ])
+ assert stl._parse_ann_field(info) == ("stop_gained", "HIGH", "geneA", False, False)
+
+
+def test_parse_ann_field_splits_effect_on_ampersand():
+ info = f"ANN={_ann_entry(effect='frameshift_variant&stop_gained', impact='HIGH', gene='katG')}"
+ assert stl._parse_ann_field(info)[0] == "frameshift"
+
+
+@pytest.mark.parametrize("raw", ["high", " HIGH ", "HiGh"])
+def test_parse_ann_field_normalises_impact_case_and_space(raw):
+ assert stl._parse_ann_field(f"ANN={_ann_entry(impact=raw)}")[1] == "HIGH"
+
+
+@pytest.mark.parametrize("raw", ["BOGUS", "", "moderate_ish"])
+def test_parse_ann_field_unknown_impact_is_coerced_to_modifier(raw):
+ """Anything outside HIGH/MODERATE/LOW/MODIFIER falls back to MODIFIER."""
+ assert stl._parse_ann_field(f"ANN={_ann_entry(impact=raw)}")[1] == "MODIFIER"
+
+
+def test_parse_ann_field_flags_warnings():
+ info = f"ANN={_ann_entry(errors='WARNING_TRANSCRIPT_NO_START_CODON')}"
+ _, _, _, has_error, db_error = stl._parse_ann_field(info)
+ assert has_error is True
+ assert db_error is False
+
+
+@pytest.mark.parametrize("token", ["ERROR_CHROMOSOME_NOT_FOUND", "ERROR_OUT_OF_CHROMOSOME_RANGE"])
+def test_parse_ann_field_flags_db_errors(token):
+ _, _, _, has_error, db_error = stl._parse_ann_field(f"ANN={_ann_entry(errors=token)}")
+ assert has_error is True
+ assert db_error is True
+
+
+def test_parse_ann_field_whitespace_only_errors_is_not_an_error():
+ _, _, _, has_error, _ = stl._parse_ann_field(f"ANN={_ann_entry(errors=' ')}")
+ assert has_error is False
+
+
+# --------------------------------------------------------------------------
+# _format_gene_burden
+# --------------------------------------------------------------------------
+
+def test_format_gene_burden_empty_is_na():
+ assert stl._format_gene_burden({}) == "NA"
+
+
+def test_format_gene_burden_no_high_or_moderate_is_na():
+ """Genes carrying only LOW/MODIFIER variants are dropped, leaving nothing."""
+ agg = {"geneA": [0, 0, {"synonymous": 5}], "geneB": [0, 0, {"regulatory": 3}]}
+ assert stl._format_gene_burden(agg) == "NA"
+
+
+def test_format_gene_burden_format_and_sort_order():
+ agg = {
+ "geneLow": [0, 1, {"missense": 1}],
+ "geneTop": [3, 0, {"stop_gained": 2, "frameshift": 1}],
+ "geneMid": [1, 9, {"missense": 4}],
+ "geneSkip": [0, 0, {"synonymous": 7}],
+ }
+ out = stl._format_gene_burden(agg)
+ assert out == "geneTop:3:0:stop_gained;geneMid:1:9:missense;geneLow:0:1:missense"
+
+
+def test_format_gene_burden_ties_break_on_gene_name():
+ agg = {"zeta": [2, 1, {"missense": 1}], "alpha": [2, 1, {"missense": 1}]}
+ assert stl._format_gene_burden(agg) == "alpha:2:1:missense;zeta:2:1:missense"
+
+
+def test_format_gene_burden_empty_effect_map_falls_back_to_other():
+ assert stl._format_gene_burden({"geneA": [1, 0, {}]}) == "geneA:1:0:other"
+
+
+def test_format_gene_burden_sanitises_delimiters_in_gene_names():
+ agg = {"we;ird:gene\tname": [1, 0, {"missense": 1}]}
+ assert stl._format_gene_burden(agg) == "we_ird_gene_name:1:0:missense"
+
+
+def test_format_gene_burden_is_capped_at_40_genes():
+ # Distinct HIGH counts make the ordering fully deterministic.
+ agg = {f"gene{i:03d}": [100 - i, 0, {"missense": 1}] for i in range(45)}
+ out = stl._format_gene_burden(agg)
+ items = out.split(";")
+ assert stl._GENE_BURDEN_CAP == 40
+ assert len(items) == 40
+ assert items[0] == "gene000:100:0:missense"
+ assert items[-1] == "gene039:61:0:missense"
+
+
+# --------------------------------------------------------------------------
+# get_genome_size / get_contig_offsets
+# --------------------------------------------------------------------------
+
+def test_get_genome_size_sums_second_column(tmp_path):
+ fai = _write_fai(tmp_path, [("chr1", 1000), ("chr2", 500)])
+ assert stl.get_genome_size(str(fai)) == 1500
+
+
+def test_get_genome_size_missing_file_returns_zero(tmp_path):
+ """Never raises: a missing .fai degrades to an unknown (0) genome size."""
+ assert stl.get_genome_size(str(tmp_path / "absent.fai")) == 0
+
+
+def test_get_genome_size_skips_short_lines(tmp_path):
+ fai = tmp_path / "ref.fai"
+ fai.write_text("chr1\t1000\t0\t60\t61\nnotabhere\n\nchr2\t7\t0\t60\t61\n")
+ assert stl.get_genome_size(str(fai)) == 1007
+
+
+def test_get_genome_size_skips_a_non_numeric_length_and_keeps_reading(tmp_path, capsys):
+ """The guard is per ROW: a bad length is skipped with a warning and every later
+ contig still counts. A truncated genome length would silently propagate into mean
+ depth, breadth, evenness and every density bin."""
+ fai = tmp_path / "ref.fai"
+ fai.write_text("chr1\t1000\t0\t60\t61\nchr2\tNOT_A_NUMBER\t0\t60\t61\nchr3\t99\t0\t60\t61\n")
+ assert stl.get_genome_size(str(fai)) == 1099
+ assert "WARN get_genome_size" in capsys.readouterr().err
+
+
+def test_get_contig_offsets_is_cumulative(tmp_path):
+ fai = _write_fai(tmp_path, [("chr1", 1000), ("chr2", 500), ("chr3", 250)])
+ assert stl.get_contig_offsets(str(fai)) == {"chr1": 0, "chr2": 1000, "chr3": 1500}
+
+
+def test_get_contig_offsets_skips_a_non_numeric_length_and_keeps_reading(tmp_path, capsys):
+ """A bad row drops only its own contig; the offsets after it are still built (and
+ still cumulative over the contigs that did parse)."""
+ fai = tmp_path / "ref.fai"
+ fai.write_text("chr1\t1000\t0\t60\t61\nchr2\tNOT_A_NUMBER\t0\t60\t61\nchr3\t250\t0\t60\t61\n")
+ assert stl.get_contig_offsets(str(fai)) == {"chr1": 0, "chr3": 1000}
+ assert "WARN get_contig_offsets" in capsys.readouterr().err
+
+
+def test_get_contig_offsets_missing_file_returns_empty(tmp_path):
+ assert stl.get_contig_offsets(str(tmp_path / "absent.fai")) == {}
+
+
+# --------------------------------------------------------------------------
+# get_bam_stats
+# --------------------------------------------------------------------------
+
+_SN_BLOCK = """# This file was produced by samtools stats
+CHK\t0000\t0000\t0000
+SN\traw total sequences:\t1000\t# excluding supplementary
+SN\treads mapped:\t900
+SN\treads mapped and paired:\t880
+SN\treads properly paired:\t850
+SN\treads duplicated:\t50
+SN\treads unmapped:\t100
+SN\treads MQ0:\t10
+SN\taverage quality:\t35.5
+SN\tinsert size average:\t300.0
+SN\tinsert size standard deviation:\t50.0
+SN\terror rate:\t0.001\t# mismatches / bases mapped
+SN\tbases mapped (cigar):\t90000
+SN\tsome unknown key:\t7
+SN\ttruncated line
+"""
+
+
+def test_get_bam_stats_parses_sn_block(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text(_SN_BLOCK)
+ got = stl.get_bam_stats(str(p))
+ assert got == {
+ "total": 1000,
+ "mapped": 900,
+ "mapped_paired": 880,
+ "properly_paired": 850,
+ "duplicates": 50,
+ "avg_quality": 35.5,
+ "insert_size_avg": 300.0,
+ "insert_size_sd": 50.0,
+ "bases_mapped": 90000,
+ "reads_unmapped": 100,
+ "reads_mq0": 10,
+ "error_rate": 0.001,
+ }
+
+
+def test_get_bam_stats_missing_file_returns_zeroed_defaults(tmp_path):
+ got = stl.get_bam_stats(str(tmp_path / "absent.txt"))
+ assert got["total"] == 0
+ assert got["mapped"] == 0
+ assert got["error_rate"] == 0.0
+ assert set(got) == {
+ "total", "mapped", "mapped_paired", "properly_paired", "duplicates",
+ "avg_quality", "insert_size_avg", "insert_size_sd", "bases_mapped",
+ "reads_unmapped", "reads_mq0", "error_rate",
+ }
+
+
+def test_get_bam_stats_ignores_non_sn_lines(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text("FFQ\t1\t2\t3\nCOV\t[1-1]\t1\t500\nSN\treads mapped:\t7\n")
+ assert stl.get_bam_stats(str(p))["mapped"] == 7
+
+
+def test_get_bam_stats_skips_an_unparsable_value_and_keeps_reading(tmp_path, capsys):
+ """The guard is per LINE: only the offending metric is lost, every SN line after
+ it is still parsed instead of being discarded with it."""
+ p = tmp_path / "stats.txt"
+ p.write_text("SN\traw total sequences:\t1000\n"
+ "SN\treads mapped:\tNOT_A_NUMBER\n"
+ "SN\treads properly paired:\t850\n"
+ "SN\terror rate:\t0.001\n")
+ got = stl.get_bam_stats(str(p))
+ assert got["total"] == 1000
+ assert got["mapped"] == 0 # the bad line keeps its zeroed default
+ assert got["properly_paired"] == 850 # ...and the rest of the block survives
+ assert got["error_rate"] == 0.001
+ assert "WARN get_bam_stats" in capsys.readouterr().err
+
+
+# --------------------------------------------------------------------------
+# calc_breadth_of_coverage
+# --------------------------------------------------------------------------
+
+_COV_BLOCK = """SN\traw total sequences:\t1000
+COV\t[1-1]\t1\t200
+COV\t[2-2]\t2\t200
+COV\t[6-6]\t6\t100
+COV\t[11-11]\t11\t100
+"""
+
+
+def test_calc_breadth_of_coverage_fraction(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text(_COV_BLOCK)
+ assert stl.calc_breadth_of_coverage(str(p), 1000, min_dp=1) == pytest.approx(0.6)
+
+
+def test_calc_breadth_of_coverage_respects_min_dp(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text(_COV_BLOCK)
+ # Only the >=5x bins (100 + 100 bases) qualify.
+ assert stl.calc_breadth_of_coverage(str(p), 1000, min_dp=5) == pytest.approx(0.2)
+
+
+def test_calc_breadth_of_coverage_counts_the_overflow_bin(tmp_path):
+ """samtools emits a final '[1000<]' bin; the '<' is stripped so those deep
+ bases are counted rather than dropped as an unparsable range."""
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[1-1]\t1\t100\nCOV\t[1000<]\t1000\t400\n")
+ assert stl.calc_breadth_of_coverage(str(p), 1000, min_dp=1) == pytest.approx(0.5)
+ assert stl.calc_breadth_of_coverage(str(p), 1000, min_dp=500) == pytest.approx(0.4)
+
+
+def test_calc_breadth_of_coverage_zero_genome_short_circuits(tmp_path, capsys):
+ """genome_size <= 0 returns before the file is even opened, so a missing
+ file produces no warning and no division."""
+ assert stl.calc_breadth_of_coverage(str(tmp_path / "absent.txt"), 0) == 0.0
+ assert stl.calc_breadth_of_coverage(str(tmp_path / "absent.txt"), -5) == 0.0
+ assert capsys.readouterr().err == ""
+
+
+def test_calc_breadth_of_coverage_missing_file_returns_zero(tmp_path, capsys):
+ assert stl.calc_breadth_of_coverage(str(tmp_path / "absent.txt"), 1000) == 0.0
+ assert "WARN calc_breadth_of_coverage" in capsys.readouterr().err
+
+
+def test_calc_breadth_of_coverage_skips_unparsable_ranges(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[junk]\tx\t900\nCOV\t[1-1]\t1\t100\nCOV\t[2-2]\n")
+ assert stl.calc_breadth_of_coverage(str(p), 1000, min_dp=1) == pytest.approx(0.1)
+
+
+# --------------------------------------------------------------------------
+# cov_histogram_stats
+# --------------------------------------------------------------------------
+
+def test_cov_histogram_stats_zero_genome_short_circuits(tmp_path, capsys):
+ """A genome size of 0 returns the zeroed default before any file read or
+ division by the (zero) denominator."""
+ out = stl.cov_histogram_stats(str(tmp_path / "absent.txt"), 0)
+ assert out == {"median_depth": 0.0, "breadth5": 0.0, "breadth10": 0.0, "evenness": 0.0}
+ assert capsys.readouterr().err == ""
+
+
+def test_cov_histogram_stats_missing_file_returns_zeroed_defaults(tmp_path, capsys):
+ out = stl.cov_histogram_stats(str(tmp_path / "absent.txt"), 1000)
+ assert out == {"median_depth": 0.0, "breadth5": 0.0, "breadth10": 0.0, "evenness": 0.0}
+ assert "WARN cov_histogram_stats" in capsys.readouterr().err
+
+
+def test_cov_histogram_stats_full_calculation(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text(_COV_BLOCK)
+ out = stl.cov_histogram_stats(str(p), 1000)
+ # covered = 600 bases, zero-depth = 400, total = 1000
+ assert out["breadth5"] == pytest.approx(20.0) # 200 bases at depth >= 5
+ assert out["breadth10"] == pytest.approx(10.0) # 100 bases at depth >= 10
+ # half = 500; the running count starts at 400 (zero bin) and crosses at [1-1]
+ assert out["median_depth"] == pytest.approx(1.0)
+ # mean = 2300/1000 = 2.3; var = 16700/1000 - 2.3^2 = 11.41
+ assert out["evenness"] == pytest.approx((11.41 ** 0.5) / 2.3)
+
+
+def test_cov_histogram_stats_uses_bin_midpoint(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[10-20]\t10\t1000\n")
+ out = stl.cov_histogram_stats(str(p), 1000)
+ assert out["median_depth"] == pytest.approx(15.0)
+ assert out["breadth10"] == pytest.approx(100.0)
+ assert out["evenness"] == pytest.approx(0.0)
+
+
+def test_cov_histogram_stats_counts_the_overflow_bin(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[1000<]\t1000\t1000\n")
+ out = stl.cov_histogram_stats(str(p), 1000)
+ assert out["median_depth"] == pytest.approx(1000.0)
+ assert out["breadth10"] == pytest.approx(100.0)
+
+
+def test_cov_histogram_stats_median_stays_zero_when_half_the_genome_is_uncovered(tmp_path):
+ """The zero-depth bin is entered into the running count first. When it
+ already reaches half the genome the median loop never runs and the median
+ is reported as 0.0 (the true median depth IS 0 here)."""
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[5-5]\t5\t400\n")
+ out = stl.cov_histogram_stats(str(p), 1000) # 600 of 1000 bases at depth 0
+ assert out["median_depth"] == 0.0
+ assert out["breadth5"] == pytest.approx(40.0)
+
+
+def test_cov_histogram_stats_exactly_half_uncovered_also_stays_zero(tmp_path):
+ """Boundary: run == half is not < half, so the loop is skipped."""
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[5-5]\t5\t500\n")
+ out = stl.cov_histogram_stats(str(p), 1000)
+ assert out["median_depth"] == 0.0
+
+
+def test_cov_histogram_stats_skips_unparsable_and_truncated_bins(tmp_path):
+ p = tmp_path / "stats.txt"
+ p.write_text("COV\t[junk-bin]\tx\t900\nCOV\t[9-9]\n\nCOV\t[4-4]\t4\t1000\n")
+ out = stl.cov_histogram_stats(str(p), 1000)
+ assert out["median_depth"] == pytest.approx(4.0)
+
+
+# --------------------------------------------------------------------------
+# analyze_vcf
+# --------------------------------------------------------------------------
+
+def test_analyze_vcf_missing_file_returns_zeroed_defaults(tmp_path, capsys):
+ v = stl.analyze_vcf(str(tmp_path / "absent.vcf"), 1000, 10, {})
+ assert v["snps"] == 0
+ assert v["indels"] == 0
+ assert v["ann_present"] is False
+ assert v["snp_prof"] == [0] * 10
+ assert "WARN analyze_vcf" in capsys.readouterr().err
+
+
+def test_analyze_vcf_haploid_one_counts_as_homozygous(tmp_path):
+ """Deliberate fix: homozygosity is `len(set(alleles)) == 1` over non-'.'
+ alleles, so a haploid '1' (freebayes ploidy=1) is homozygous, not het.
+ The old 'len == 2 and equal' test miscounted every haploid call."""
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", "1")])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["homo_total"] == 1
+ assert v["het_total"] == 0
+ assert v["snps"] == 1
+
+
+@pytest.mark.parametrize("gt,homo", [
+ ("1", True),
+ ("1/1", True),
+ ("1|1", True),
+ ("2/2", True),
+ ("1/2", False),
+ ("0/1", False),
+ ("0|1", False),
+ ("./1", True), # the missing allele is dropped, leaving a single '1'
+ ("1/.", True),
+])
+def test_analyze_vcf_zygosity(tmp_path, gt, homo):
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", gt)], name=f"z{gt.replace('/', '_').replace('|', 'p')}.vcf")
+ v = stl.analyze_vcf(str(vcf))
+ assert v["homo_total"] == (1 if homo else 0)
+ assert v["het_total"] == (0 if homo else 1)
+
+
+def test_analyze_vcf_skips_a_haploid_reference_call(tmp_path):
+ """The mirror image of the haploid '1' fix: a HAPLOID reference call ('0') is not a
+ variant. A diploid-only skip list let it through, and len(set(['0'])) == 1 then made it
+ a homozygous SNP, inflating SNP_OUT, HOM_OUT and the SNP density profile on every
+ `-profile generic` run (freebayes_ploidy = 1)."""
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", "0")])
+ v = stl.analyze_vcf(str(vcf), g_size=1000, nbins=10, contig_offsets={"chr1": 0})
+ assert v["snps"] == 0
+ assert v["homo_total"] == 0
+ assert v["ann_total"] == 0
+ assert v["snp_prof"] == [0] * 10
+
+
+def test_analyze_vcf_skips_a_haploid_no_call(tmp_path):
+ """A haploid no-call ('.') has no called allele at all, so it is neither a variant nor
+ heterozygous; it used to land in the het bucket and inflate HET_OUT."""
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", ".")])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["snps"] == 0
+ assert v["het_total"] == 0
+ assert v["ann_total"] == 0
+
+
+@pytest.mark.parametrize("gt", ["./.", "0/0", "0|0", "0", ".", "0/.", "0/0/0"])
+def test_analyze_vcf_skips_non_calls_and_ref_calls(tmp_path, gt):
+ """Ploidy-agnostic: no spelling of "no called ALT allele" is a variant."""
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", gt)],
+ name=f"s{gt.replace('/', '_').replace('|', 'p').replace('.', 'd')}.vcf")
+ v = stl.analyze_vcf(str(vcf))
+ assert v["snps"] == 0
+ assert v["ann_total"] == 0
+
+
+def test_analyze_vcf_skips_a_bad_record_and_keeps_reading(tmp_path, capsys, monkeypatch):
+ """The guard is per RECORD: an exception on one line must not discard the rest of the
+ VCF. Faulted through the annotation parser, the only place a record can realistically
+ blow up after the field-count checks."""
+ real = stl._parse_ann_field
+
+ def boom(info_str):
+ if "BOOM" in info_str:
+ raise ValueError("bad ANN")
+ return real(info_str)
+
+ monkeypatch.setattr(stl, "_parse_ann_field", boom)
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1/1", info="DP=10"),
+ _rec(200, "C", "T", "1/1", info="BOOM=1"),
+ _rec(300, "G", "A", "1/1", info="DP=10"),
+ ])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["snps"] == 2 # the first and the LAST record, not just the first
+ assert "skipping record" in capsys.readouterr().err
+
+
+@pytest.mark.parametrize("alt", [".", "*"])
+def test_analyze_vcf_skips_placeholder_alt(tmp_path, alt):
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", alt, "1/1")], name=f"alt{'dot' if alt == '.' else 'star'}.vcf")
+ assert stl.analyze_vcf(str(vcf))["snps"] == 0
+
+
+def test_analyze_vcf_skips_records_with_fewer_than_ten_columns(tmp_path):
+ p = tmp_path / "short.vcf"
+ p.write_text(_VCF_HEADER + "chr1\t100\t.\tA\tG\t50\tPASS\tDP=10\tGT:DP\n")
+ assert stl.analyze_vcf(str(p))["snps"] == 0
+
+
+def test_analyze_vcf_skips_records_without_gt_in_format(tmp_path):
+ p = tmp_path / "nogt.vcf"
+ p.write_text(_VCF_HEADER + "chr1\t100\t.\tA\tG\t50\tPASS\tDP=10\tDP:AD\t10:5,5\n")
+ assert stl.analyze_vcf(str(p))["snps"] == 0
+
+
+def test_analyze_vcf_skips_when_sample_column_is_shorter_than_the_gt_index(tmp_path):
+ p = tmp_path / "shortsample.vcf"
+ p.write_text(_VCF_HEADER + "chr1\t100\t.\tA\tG\t50\tPASS\tDP=10\tDP:GT\t10\n")
+ assert stl.analyze_vcf(str(p))["snps"] == 0
+
+
+def test_analyze_vcf_snp_indel_split_and_titv(tmp_path):
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1/1"), # transition
+ _rec(200, "C", "T", "1/1"), # transition
+ _rec(300, "G", "T", "0/1"), # transversion, heterozygous
+ _rec(400, "AT", "A", "1/1"), # homozygous deletion
+ _rec(500, "A", "ATT", "0/1"), # heterozygous insertion
+ ])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["snps"] == 3
+ assert v["indels"] == 2
+ assert v["homo_total"] == 3
+ assert v["het_total"] == 2
+ assert v["homo_indels"] == 1
+ assert v["ti"] == 2
+ assert v["tv"] == 1
+
+
+def test_analyze_vcf_titv_ignores_non_acgt_and_identical_alleles(tmp_path):
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "N", "G", "1/1"),
+ _rec(200, "A", "N", "1/1"),
+ _rec(300, "A", "A", "1/1"),
+ ])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["snps"] == 3
+ assert (v["ti"], v["tv"]) == (0, 0)
+
+
+def test_analyze_vcf_binning_formula(tmp_path):
+ """b = min(nbins - 1, (contig_offset + POS) * nbins // genome_size)."""
+ vcf = _write_vcf(tmp_path, [
+ _rec(1, "A", "G", "1/1"), # 1*10//1000 -> bin 0
+ _rec(500, "A", "G", "1/1"), # 500*10//1000 -> bin 5
+ _rec(1000, "A", "G", "1/1"), # 1000*10//1000 = 10 -> clamped to 9
+ ])
+ v = stl.analyze_vcf(str(vcf), g_size=1000, nbins=10, contig_offsets={"chr1": 0})
+ assert v["snp_prof"] == [1, 0, 0, 0, 0, 1, 0, 0, 0, 1]
+
+
+def test_analyze_vcf_binning_applies_contig_offsets(tmp_path):
+ """VCF POS restarts at 1 per contig, so the cumulative offset places the
+ variant on a single whole-genome axis."""
+ vcf = _write_vcf(tmp_path, [
+ _rec(500, "A", "G", "1/1", contig="chr1"), # gpos 500 -> 500*10//2000 = 2
+ _rec(500, "A", "G", "1/1", contig="chr2"), # gpos 1500 -> 1500*10//2000 = 7
+ ])
+ v = stl.analyze_vcf(str(vcf), g_size=2000, nbins=10, contig_offsets={"chr1": 0, "chr2": 1000})
+ assert v["snp_prof"] == [0, 0, 1, 0, 0, 0, 0, 1, 0, 0]
+
+
+def test_analyze_vcf_unknown_contig_offsets_to_zero(tmp_path):
+ vcf = _write_vcf(tmp_path, [_rec(500, "A", "G", "1/1", contig="chrUnknown")])
+ v = stl.analyze_vcf(str(vcf), g_size=1000, nbins=10, contig_offsets={"chr1": 0})
+ assert v["snp_prof"][5] == 1
+
+
+def test_analyze_vcf_no_binning_when_genome_size_is_unknown(tmp_path):
+ vcf = _write_vcf(tmp_path, [_rec(500, "A", "G", "1/1"), _rec(600, "AT", "A", "0/1")])
+ v = stl.analyze_vcf(str(vcf), g_size=0, nbins=10)
+ assert v["snps"] == 1
+ assert v["snp_prof"] == [0] * 10
+ assert v["het_prof"] == [0] * 10
+ assert v["indel_prof"] == [0] * 10
+
+
+def test_analyze_vcf_non_numeric_position_is_not_binned(tmp_path):
+ p = tmp_path / "badpos.vcf"
+ p.write_text(_VCF_HEADER + "chr1\tNOT_A_POS\t.\tA\tG\t50\tPASS\tDP=10\tGT:DP\t1/1:10\n")
+ v = stl.analyze_vcf(str(p), g_size=1000, nbins=10)
+ assert v["snps"] == 1
+ assert v["snp_prof"] == [0] * 10
+
+
+def test_analyze_vcf_only_homozygous_snps_reach_the_snp_profile(tmp_path):
+ """Heterozygous SNPs go to het_prof only; indels go to indel_prof
+ regardless of zygosity, and a het indel is additionally in het_prof."""
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1/1"), # homozygous SNP -> snp_prof[1]
+ _rec(300, "A", "G", "0/1"), # het SNP -> het_prof[3] only
+ _rec(500, "AT", "A", "1/1"), # homozygous indel-> indel_prof[5] only
+ _rec(700, "AT", "A", "0/1"), # het indel -> indel_prof[7] + het_prof[7]
+ ])
+ v = stl.analyze_vcf(str(vcf), g_size=1000, nbins=10, contig_offsets={"chr1": 0})
+ assert v["snp_prof"] == [0, 1, 0, 0, 0, 0, 0, 0, 0, 0]
+ assert v["het_prof"] == [0, 0, 0, 1, 0, 0, 0, 1, 0, 0]
+ assert v["indel_prof"] == [0, 0, 0, 0, 0, 1, 0, 1, 0, 0]
+
+
+def test_analyze_vcf_reads_gzipped_input(tmp_path):
+ p = tmp_path / "calls.vcf.gz"
+ with gzip.open(p, "wt") as fh:
+ fh.write(_VCF_HEADER + _rec(100, "A", "G", "1/1") + "\n")
+ assert stl.analyze_vcf(str(p))["snps"] == 1
+
+
+def test_analyze_vcf_without_ann_leaves_annotation_absent(tmp_path):
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", "1/1", info="DP=10")])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["ann_present"] is False
+ assert v["ann_total"] == 1 # counted as a candidate record...
+ assert v["ann_records"] == 0 # ...but not as an annotated one
+
+
+def test_analyze_vcf_accumulates_annotation_counts(tmp_path):
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1/1", info="ANN=" + _ann_entry(effect="missense_variant", impact="MODERATE", gene="geneA")),
+ _rec(200, "C", "T", "1/1", info="ANN=" + _ann_entry(effect="synonymous_variant", impact="LOW", gene="geneA")),
+ _rec(300, "G", "T", "0/1", info="ANN=" + _ann_entry(effect="stop_gained", impact="HIGH", gene="geneB",
+ errors="WARNING_TRANSCRIPT_INCOMPLETE")),
+ _rec(400, "AT", "A", "1/1", info="DP=10"),
+ ])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["ann_present"] is True
+ assert v["ann_total"] == 4
+ assert v["ann_records"] == 3
+ assert v["ann_impact"] == {"HIGH": 1, "MODERATE": 1, "LOW": 1, "MODIFIER": 0}
+ assert v["ann_class"]["missense"] == 1
+ assert v["ann_class"]["synonymous"] == 1
+ assert v["ann_class"]["stop_gained"] == 1
+ assert v["ann_warn"] == 1
+ assert v["ann_db_error"] is False
+ assert v["ann_gene"] == {
+ "geneA": [0, 1, {"missense": 1, "synonymous": 1}],
+ "geneB": [1, 0, {"stop_gained": 1}],
+ }
+
+
+def test_analyze_vcf_flags_db_error(tmp_path):
+ vcf = _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1/1", info="ANN=" + _ann_entry(errors="ERROR_CHROMOSOME_NOT_FOUND")),
+ ])
+ assert stl.analyze_vcf(str(vcf))["ann_db_error"] is True
+
+
+def test_analyze_vcf_ignores_genes_with_an_empty_name(tmp_path):
+ vcf = _write_vcf(tmp_path, [_rec(100, "A", "G", "1/1", info="ANN=G|intergenic_region|MODIFIER|")])
+ v = stl.analyze_vcf(str(vcf))
+ assert v["ann_records"] == 1
+ assert v["ann_gene"] == {}
+
+
+# --------------------------------------------------------------------------
+# get_pathotypr_data
+# --------------------------------------------------------------------------
+
+_PATHO_TSV = "genome\tlineage:count\tmajor_lineage\nS1\tL4:10;L4.3:5\tL4.3\n"
+
+
+def test_get_pathotypr_data_parses_report(tmp_path):
+ p = tmp_path / "lineage.tsv"
+ p.write_text(_PATHO_TSV)
+ assert stl.get_pathotypr_data(str(p)) == {"major": "L4.3", "counts": "L4:10;L4.3:5"}
+
+
+@pytest.mark.parametrize("path", [None, "", "some/NO_LINEAGE_FILE/report.tsv"])
+def test_get_pathotypr_data_sentinel_paths_return_na(path):
+ assert stl.get_pathotypr_data(path) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_missing_file_returns_na(tmp_path):
+ assert stl.get_pathotypr_data(str(tmp_path / "absent.tsv")) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_empty_file_returns_na(tmp_path):
+ p = tmp_path / "empty.tsv"
+ p.write_text("")
+ assert stl.get_pathotypr_data(str(p)) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_header_only_returns_na(tmp_path):
+ p = tmp_path / "hdr.tsv"
+ p.write_text("genome\tlineage:count\tmajor_lineage\n")
+ assert stl.get_pathotypr_data(str(p)) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_unknown_columns_return_na(tmp_path):
+ p = tmp_path / "other.tsv"
+ p.write_text("genome\tsomething\tsomething_else\nS1\ta\tb\n")
+ assert stl.get_pathotypr_data(str(p)) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_partial_columns(tmp_path):
+ """Only the columns actually present in the header are filled in."""
+ p = tmp_path / "partial.tsv"
+ p.write_text("genome\tmajor_lineage\nS1\tL2.2.1\n")
+ assert stl.get_pathotypr_data(str(p)) == {"major": "L2.2.1", "counts": "NA"}
+
+
+def test_get_pathotypr_data_short_data_row_returns_na(tmp_path):
+ p = tmp_path / "short.tsv"
+ p.write_text("genome\tlineage:count\tmajor_lineage\nS1\n")
+ assert stl.get_pathotypr_data(str(p)) == {"major": "NA", "counts": "NA"}
+
+
+def test_get_pathotypr_data_unreadable_path_returns_na(tmp_path, capsys):
+ """A directory passes the os.path.exists() check and then fails to open;
+ the exception is swallowed and the defaults are returned."""
+ d = tmp_path / "a_directory"
+ d.mkdir()
+ assert stl.get_pathotypr_data(str(d)) == {"major": "NA", "counts": "NA"}
+ assert "Error parsing Pathotypr file" in capsys.readouterr().err
+
+
+# --------------------------------------------------------------------------
+# end to end
+# --------------------------------------------------------------------------
+
+def _fastp_json():
+ return (
+ '{"summary": {"before_filtering": {"total_reads": 1000},'
+ ' "after_filtering": {"total_reads": 900, "total_bases": 90000,'
+ ' "read1_mean_length": 100, "q20_rate": 0.98, "q30_rate": 0.95,'
+ ' "gc_content": 0.65}}, "duplication": {"rate": 0.1}}'
+ )
+
+
+def test_end_to_end_writes_the_legacy_log(tmp_path, repo_root):
+ """main() has NO output flag: it writes '.log' into the CURRENT
+ WORKING DIRECTORY, hence cwd=tmp_path."""
+ (tmp_path / "fastp.json").write_text(_fastp_json())
+ (tmp_path / "stats.txt").write_text(_SN_BLOCK + _COV_BLOCK.split("\n", 1)[1])
+ _write_fai(tmp_path, [("chr1", 1000)])
+ _write_vcf(tmp_path, [
+ _rec(100, "A", "G", "1", info="ANN=" + _ann_entry(effect="missense_variant", impact="MODERATE", gene="geneA")),
+ _rec(200, "C", "T", "1/1", info="ANN=" + _ann_entry(effect="synonymous_variant", impact="LOW", gene="geneA")),
+ _rec(300, "G", "T", "0/1", info="ANN=" + _ann_entry(effect="stop_gained", impact="HIGH", gene="geneB")),
+ _rec(400, "AT", "A", "1/1", info="ANN=" + _ann_entry(effect="frameshift_variant", impact="HIGH", gene="geneB")),
+ ])
+ (tmp_path / "lineage.tsv").write_text(_PATHO_TSV)
+ (tmp_path / "dr.tsv").write_text("genome\tlineage:count\tmajor_lineage\nS1\trpoB:3\tRIF\n")
+
+ proc = subprocess.run(
+ [
+ sys.executable, str(repo_root / "bin" / "stats_to_legacy.py"),
+ "--sample", "S1",
+ "--fastp-json", str(tmp_path / "fastp.json"),
+ "--bam-stats", str(tmp_path / "stats.txt"),
+ "--vcf", str(tmp_path / "calls.vcf"),
+ "--ref-fai", str(tmp_path / "ref.fa.fai"),
+ "--pathotypr-report", str(tmp_path / "lineage.tsv"),
+ "--pathotypr-dr-report", str(tmp_path / "dr.tsv"),
+ ],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+
+ lines = (tmp_path / "S1.log").read_text().splitlines()
+ keys = [ln.split("\t", 1)[0] for ln in lines]
+ assert keys == stl.ORDERED_KEYS
+ data = dict(ln.split("\t", 1) for ln in lines)
+
+ # DAT_OUT is today's date, so it is excluded from the golden comparison.
+ assert data.pop("DAT_OUT")
+ profiles = {k: data.pop(k) for k in ("SNP_PROFILE", "HET_PROFILE", "INDEL_PROFILE")}
+
+ golden = {
+ "GID_IN": "S1",
+ "FQ_IN": "S1_R1/R2",
+ "ANC_OUT": "no",
+ "GENOME_LEN": "1000",
+ # fastp
+ "PHE_OUT": "33",
+ "FV_OUT": "FASTQ_SUCCESS",
+ "TRD_OUT": "1000",
+ "TRM_OUT": "900",
+ "RDL_OUT": "100.0",
+ "DUP_OUT": "10.00",
+ "Q20_PCT": "98.00",
+ "Q30_PCT": "95.00",
+ "GC_PCT": "65.00",
+ "LNG_OUT": "90.00", # 90000 after-filter bases / 1000 bp genome
+ # samtools stats
+ "MAP_OUT": "900",
+ "MAP_PCT": "90.00",
+ "PP_OUT": "85.00",
+ "UNMAP_PCT": "10.00",
+ "SINGLE_PCT": "2.00", # (900 mapped - 880 mapped-and-paired) / 1000
+ "MQ0_PCT": "1.00",
+ "ERR_RATE": "0.100", # 0.001 emitted as a percent
+ "ISIZE_MEAN": "300.0",
+ "ISIZE_SD": "50.0",
+ "STD_OUT": "35.5",
+ "COV_OUT": "90.00", # 90000 cigar bases / 1000 bp
+ "COV_BREADTH": "60.00",
+ "COV_MEDIAN": "1.00",
+ "COV_BREADTH5": "20.00",
+ "COV_BREADTH10": "10.00",
+ "COV_EVENNESS": "1.469",
+ # VCF
+ "VAR_OUT": "4",
+ "SNP_OUT": "3",
+ "HOM_OUT": "3",
+ "HET_OUT": "1",
+ "HOMO_IND_OUT": "1",
+ "TITV_OUT": "2.000",
+ # snpEff ANN
+ "ANN_PRESENT": "yes",
+ "ANN_TOTAL": "4",
+ "ANN_ANNOTATED": "4",
+ "ANN_ANNOTATED_FRAC": "1.0000",
+ "ANN_IMPACT_HIGH": "2",
+ "ANN_IMPACT_MODERATE": "1",
+ "ANN_IMPACT_LOW": "1",
+ "ANN_IMPACT_MODIFIER": "0",
+ "ANN_MISSENSE": "1",
+ "ANN_SYNON": "1",
+ "ANN_STOP_GAINED": "1",
+ "ANN_STOP_LOST": "0",
+ "ANN_START_LOST": "0",
+ "ANN_FRAMESHIFT": "1",
+ "ANN_INFRAME_INDEL": "0",
+ "ANN_SPLICE": "0",
+ "ANN_INTERGENIC": "0",
+ "ANN_REGULATORY": "0",
+ "ANN_LOF": "2",
+ "ANN_CODING": "4",
+ "ANN_MISSENSE_SILENT": "1.000",
+ "ANN_WARN": "0",
+ "ANN_DB_ERROR": "no",
+ "ANN_GENE_BURDEN": "geneB:2:0:stop_gained;geneA:0:1:missense",
+ # pathotypr
+ "LIN_OUT": "L4.3",
+ "LIN_COUNTS": "L4:10;L4.3:5",
+ "DR_OUT": "RIF",
+ "DR_COUNTS": "rpoB:3",
+ }
+ for key, expected in golden.items():
+ assert data.pop(key) == expected, key
+ # Every remaining legacy field is untouched.
+ assert set(data.values()) == {"NA"}
+
+ for name, profile in profiles.items():
+ assert len(profile.split(",")) == stl.BIN_N, name
+ # 200 bins over 1000 bp -> bin = POS // 5
+ assert profiles["SNP_PROFILE"].split(",")[20] == "1" # homozygous SNP at 100
+ assert profiles["SNP_PROFILE"].split(",")[40] == "1" # homozygous SNP at 200
+ assert profiles["HET_PROFILE"].split(",")[60] == "1" # het SNP at 300
+ assert profiles["INDEL_PROFILE"].split(",")[80] == "1" # homozygous indel at 400
+
+
+def _run_legacy(tmp_path, repo_root, sample="S1"):
+ """Run the script over the inputs already written into tmp_path and return the parsed log."""
+ proc = subprocess.run(
+ [
+ sys.executable, str(repo_root / "bin" / "stats_to_legacy.py"),
+ "--sample", sample,
+ "--fastp-json", str(tmp_path / "fastp.json"),
+ "--bam-stats", str(tmp_path / "stats.txt"),
+ "--vcf", str(tmp_path / "calls.vcf"),
+ "--ref-fai", str(tmp_path / "ref.fa.fai"),
+ ],
+ cwd=tmp_path, capture_output=True, text=True, timeout=120,
+ )
+ assert proc.returncode == 0, proc.stderr
+ return dict(ln.split("\t", 1) for ln in (tmp_path / f"{sample}.log").read_text().splitlines())
+
+
+def test_a_measured_zero_is_reported_as_zero_not_na(tmp_path, repo_root):
+ """A sample that FAILED must not be indistinguishable from one that was never measured.
+ Every value below was genuinely computed as 0 and used to be erased to NA by an `if x > 0`."""
+ (tmp_path / "fastp.json").write_text(
+ '{"summary": {"before_filtering": {"total_reads": 0},'
+ ' "after_filtering": {"total_reads": 0, "total_bases": 0, "read1_mean_length": 0}},'
+ ' "duplication": {"rate": 0}}')
+ # 1000 reads sequenced but nothing covered (no COV block), a perfectly uniform 300 bp insert
+ # (sd 0) and no mismatch over 5000 mapped bases.
+ (tmp_path / "stats.txt").write_text(
+ "SN\traw total sequences:\t1000\n"
+ "SN\treads mapped:\t0\n"
+ "SN\taverage quality:\t0.0\n"
+ "SN\tinsert size average:\t300.0\n"
+ "SN\tinsert size standard deviation:\t0.0\n"
+ "SN\terror rate:\t0.0\n"
+ "SN\tbases mapped (cigar):\t5000\n")
+ _write_fai(tmp_path, [("chr1", 1000)])
+ _write_vcf(tmp_path, [])
+
+ data = _run_legacy(tmp_path, repo_root)
+ assert data["TRD_OUT"] == "0" # fastp WAS read: zero reads is a measurement
+ assert data["TRM_OUT"] == "0"
+ assert data["STD_OUT"] == "0.0" # reads present, so a 0 mean base quality is real
+ assert data["ERR_RATE"] == "0.000" # mapped bases present, so 0 mismatches is real
+ assert data["ISIZE_SD"] == "0.0" # a real mean, so an identical-insert library is real
+ assert data["COV_BREADTH"] == "0.00"
+ assert data["COV_MEDIAN"] == "0.00" # nothing covered -> the median depth IS 0
+ assert data["COV_BREADTH5"] == "0.00"
+ assert data["COV_BREADTH10"] == "0.00"
+ # The one exception: evenness (CoV) is UNDEFINED at zero mean depth and cov_histogram_stats
+ # returns the same 0.0 for that and for a perfectly flat genome, so the > 0 guard is the only
+ # thing separating them and stays. 0.000 here would read as ideal evenness.
+ assert data["COV_EVENNESS"] == "NA"
+
+
+def test_an_unmeasured_metric_is_still_na(tmp_path, repo_root):
+ """The other half of the contract: absent input still yields NA, never a fabricated 0."""
+ (tmp_path / "fastp.json").write_text("not json at all") # no lane is parsed
+ (tmp_path / "stats.txt").write_text(
+ "SN\traw total sequences:\t0\n"
+ "SN\tinsert size average:\t0.0\n"
+ "SN\tinsert size standard deviation:\t0.0\n"
+ "SN\tbases mapped (cigar):\t0\n")
+ (tmp_path / "ref.fa.fai").write_text("") # unknown genome length
+ _write_vcf(tmp_path, [])
+
+ data = _run_legacy(tmp_path, repo_root)
+ assert data["TRD_OUT"] == "NA"
+ assert data["TRM_OUT"] == "NA"
+ assert data["STD_OUT"] == "NA" # no reads -> no measurable mean base quality
+ assert data["ERR_RATE"] == "NA" # no mapped bases -> the rate has no denominator
+ assert data["ISIZE_MEAN"] == "NA" # no paired insert at all -> not applicable, not 0 bp
+ assert data["ISIZE_SD"] == "NA"
+ assert data["GENOME_LEN"] == "NA"
+ assert data["COV_BREADTH"] == "NA" # unknown genome length -> no denominator
+ assert data["COV_MEDIAN"] == "NA"
+ assert data["COV_EVENNESS"] == "NA"
diff --git a/tests/unit/test_vcf_filter_rules.py b/tests/unit/test_vcf_filter_rules.py
new file mode 100644
index 0000000..827306c
--- /dev/null
+++ b/tests/unit/test_vcf_filter_rules.py
@@ -0,0 +1,149 @@
+"""The bcftools expressions that define a homozygous or heterozygous SNP call.
+
+These are the scientific definition of a call, not plumbing. They lived inline in two FreeBayes
+processes, written out twice, and had no coverage at all: a stub run skips the script block and
+nothing else could reach them. The expressions are also checked against a real bcftools here, so a
+change that produces a syntactically valid but wrong filter still fails.
+"""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+import sys
+import textwrap
+
+import pytest
+
+from conftest import BIN, load_script
+
+rules = load_script("vcf_filter_rules")
+
+# Production defaults, from nextflow.config.
+DEFAULTS = dict(min_dp=30, min_alt_fwd=2, min_alt_rev=2, hom_threshold=0.90, het_min_frac=0.10)
+
+
+def hom(**kw):
+ a = {**DEFAULTS, **kw}
+ return rules.hom_rule(a["min_dp"], a["min_alt_fwd"], a["min_alt_rev"], a["hom_threshold"])
+
+
+def het(**kw):
+ a = {**DEFAULTS, **kw}
+ return rules.het_rule(a["min_dp"], a["min_alt_fwd"], a["min_alt_rev"], a["hom_threshold"],
+ a["het_min_frac"])
+
+
+class TestExpressionShape:
+ def test_both_rules_require_depth_from_either_source(self):
+ # A caller may record depth per-sample or per-site; either satisfies the threshold.
+ for expr in (hom(), het()):
+ assert "FMT/DP >= 30" in expr
+ assert "INFO/DP >= 30" in expr
+
+ def test_both_rules_require_support_on_both_strands(self):
+ """A variant seen on one strand only is the classic short-read artefact."""
+ for expr in (hom(), het()):
+ assert "INFO/SAF[0] >= 2" in expr
+ assert "INFO/SAR[0] >= 2" in expr
+
+ def test_both_rules_guard_the_allele_fraction_division(self):
+ for expr in (hom(), het()):
+ assert "(INFO/AO[0] + INFO/RO) > 0" in expr
+
+ def test_the_het_band_is_closed_below_and_open_above(self):
+ """The boundary must belong to exactly one rule, or a call at 0.90 is both or neither."""
+ assert ">= 0.1" in het() and "< 0.9" in het()
+ assert ">= 0.9" in hom()
+
+ def test_the_thresholds_come_from_the_arguments(self):
+ assert ">= 0.75" in hom(hom_threshold=0.75)
+ assert "FMT/DP >= 5" in het(min_dp=5)
+
+
+@pytest.mark.skipif(not shutil.which("bcftools"), reason="bcftools is not installed")
+class TestAgainstRealBcftools:
+ """The expression has to be one bcftools accepts AND one that selects the right records."""
+
+ @staticmethod
+ def _vcf(tmp_path, rows):
+ header = textwrap.dedent("""\
+ ##fileformat=VCFv4.2
+ ##contig=
+ ##INFO=
+ ##INFO=
+ ##INFO=
+ ##INFO=
+ ##INFO=
+ ##FORMAT=
+ ##FORMAT=
+ #CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tS1
+ """)
+ path = tmp_path / "in.vcf"
+ path.write_text(header + "".join(rows))
+ return path
+
+ @staticmethod
+ def _row(pos, dp, ro, ao, saf, sar, gt="0/1"):
+ return (f"c\t{pos}\t.\tA\tT\t50\t.\t"
+ f"DP={dp};RO={ro};AO={ao};SAF={saf};SAR={sar}\tGT:DP\t{gt}:{dp}\n")
+
+ def _select(self, vcf, expr):
+ out = subprocess.run(["bcftools", "view", "-H", "-i", expr, str(vcf)],
+ capture_output=True, text=True, check=True)
+ return [line.split("\t")[1] for line in out.stdout.splitlines()]
+
+ def test_the_rules_partition_calls_by_allele_fraction(self, tmp_path):
+ vcf = self._vcf(tmp_path, [
+ self._row(100, dp=50, ro=1, ao=49, saf=25, sar=24), # AF 0.98 -> hom
+ self._row(200, dp=50, ro=25, ao=25, saf=13, sar=12), # AF 0.50 -> het
+ self._row(300, dp=50, ro=48, ao=2, saf=1, sar=1), # AF 0.04 -> neither
+ ])
+ assert self._select(vcf, hom()) == ["100"]
+ assert self._select(vcf, het()) == ["200"]
+
+ def test_a_shallow_site_passes_neither_rule(self, tmp_path):
+ vcf = self._vcf(tmp_path, [self._row(100, dp=5, ro=0, ao=5, saf=3, sar=2)])
+ assert self._select(vcf, hom()) == []
+ assert self._select(vcf, het()) == []
+
+ def test_a_single_strand_variant_passes_neither_rule(self, tmp_path):
+ """Full depth and a clean allele fraction, but every alt read on the forward strand."""
+ vcf = self._vcf(tmp_path, [self._row(100, dp=50, ro=1, ao=49, saf=49, sar=0)])
+ assert self._select(vcf, hom()) == []
+ assert self._select(vcf, het()) == []
+
+ def test_a_site_with_no_observations_does_not_divide_by_zero(self, tmp_path):
+ vcf = self._vcf(tmp_path, [self._row(100, dp=40, ro=0, ao=0, saf=0, sar=0)])
+ assert self._select(vcf, hom()) == []
+ assert self._select(vcf, het()) == []
+
+ def test_the_boundary_fraction_is_homozygous_and_not_heterozygous(self, tmp_path):
+ """Exactly 0.90: hom is inclusive, het is exclusive, so it must land in hom only."""
+ vcf = self._vcf(tmp_path, [self._row(100, dp=50, ro=5, ao=45, saf=23, sar=22)])
+ assert self._select(vcf, hom()) == ["100"]
+ assert self._select(vcf, het()) == []
+
+
+class TestCli:
+ """The processes call this as a command and substitute the result into bcftools."""
+
+ def _run(self, *args):
+ cmd = [sys.executable, str(BIN / "vcf_filter_rules.py"), *args,
+ "--min-dp", "30", "--min-alt-fwd", "2", "--min-alt-rev", "2",
+ "--hom-threshold", "0.90", "--het-min-frac", "0.10"]
+ return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout.strip()
+
+ def test_each_rule_is_printed_on_one_line(self):
+ for rule in ("hom", "het", "either", "base"):
+ out = self._run(rule)
+ assert out and "\n" not in out, f"{rule} did not print a single line"
+
+ def test_either_is_the_disjunction_the_processes_use(self):
+ assert self._run("either") == f"{self._run('hom')} || {self._run('het')}"
+
+ def test_an_unknown_rule_is_rejected(self):
+ cmd = [sys.executable, str(BIN / "vcf_filter_rules.py"), "nonsense",
+ "--min-dp", "1", "--min-alt-fwd", "1", "--min-alt-rev", "1",
+ "--hom-threshold", "0.9", "--het-min-frac", "0.1"]
+ assert subprocess.run(cmd, capture_output=True).returncode != 0
diff --git a/tests/unit/test_wgs_fasta_allpos.py b/tests/unit/test_wgs_fasta_allpos.py
new file mode 100644
index 0000000..c921061
--- /dev/null
+++ b/tests/unit/test_wgs_fasta_allpos.py
@@ -0,0 +1,1190 @@
+"""Unit tests for bin/WGS_fasta_allpos.py, the all.pos consensus builder.
+
+The script is a standalone file under bin/, so it is loaded through the shared
+`load_script` helper in tests/conftest.py rather than imported as a package.
+
+Every assertion here pins the CURRENT, observed behaviour of the script. Where
+that behaviour looks questionable the test carries a `QUIRK:` comment naming the
+issue; those tests are deliberately written to pass against the code as it is.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import gzip
+
+import pytest
+
+from conftest import load_script
+
+wgs = load_script("WGS_fasta_allpos")
+
+
+# --------------------------------------------------------------------------- #
+# Helpers
+# --------------------------------------------------------------------------- #
+
+# The argparse defaults from main(), so a test only states the knob it exercises.
+DEFAULT_OPTS = {
+ "mask_char": "X",
+ "nocall_char": "-",
+ "min_dp": 0,
+ "ref_min_dp": 10,
+ "max_ref_altfrac": 0.10,
+ "max_ref_min_alt": 2,
+ "max_ref_del_frac": 0.5,
+}
+
+BUILD_DEFAULTS = {
+ "exclude_path": None,
+ "mask_sites_path": None,
+ "special_csv_path": None,
+ "wrap": 60,
+ "strict": False,
+ **DEFAULT_OPTS,
+}
+
+VCF_HEADER = (
+ "##fileformat=VCFv4.2\n"
+ "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tSAMPLE\n"
+)
+
+
+def make_rec(chrom="chr1", pos=1, ref="A", alt=".", qual="60", flt="PASS",
+ info=".", fmt="GT:DP:AD", sample="0/0:30:30,0"):
+ """A VcfRecord that, with the default options, yields a confident reference call."""
+ return wgs.VcfRecord(chrom=chrom, pos=pos, ref=ref, alt=alt, qual=qual,
+ flt=flt, info=info, fmt=fmt, sample=sample)
+
+
+def call_consensus(recs, ref_seqs=None, interval_mask=None, point_mask=None,
+ csv_mask=None, **overrides):
+ opts = dict(DEFAULT_OPTS)
+ opts.update(overrides)
+ return wgs.consensus_for_position(
+ recs,
+ {} if ref_seqs is None else ref_seqs,
+ interval_mask if interval_mask is not None else wgs.IntervalMasker(None),
+ point_mask if point_mask is not None else wgs.PointMasker(None),
+ csv_mask if csv_mask is not None else wgs.CsvMasker(None),
+ **opts,
+ )
+
+
+def vcf_line(chrom, pos, ref, alt, flt="PASS", info=".", fmt="GT:DP:AD",
+ sample="0/0:30:30,0"):
+ return "\t".join([chrom, str(pos), ".", ref, alt, "60", flt, info, fmt, sample])
+
+
+def write_text(path, text, gz=False):
+ if gz:
+ with gzip.open(path, "wt", encoding="utf-8") as fh:
+ fh.write(text)
+ else:
+ path.write_text(text, encoding="utf-8")
+ return str(path)
+
+
+def write_vcf(path, lines, gz=False):
+ return write_text(path, VCF_HEADER + "".join(f"{ln}\n" for ln in lines), gz=gz)
+
+
+def write_fasta(path, seqs, wrap=60, gz=False):
+ out = []
+ for name, seq in seqs.items():
+ out.append(f">{name}\n")
+ for i in range(0, max(len(seq), 1), wrap):
+ out.append(seq[i:i + wrap] + "\n")
+ return write_text(path, "".join(out), gz=gz)
+
+
+def read_fasta(path):
+ """Minimal FASTA parser for the produced output (independent of FastaReader)."""
+ seqs = {}
+ name = None
+ for line in path.read_text(encoding="utf-8").splitlines():
+ if line.startswith(">"):
+ name = line[1:]
+ seqs[name] = ""
+ elif name is not None:
+ seqs[name] += line
+ return seqs
+
+
+def run_build(tmp_path, vcf_lines, ref_seqs, **overrides):
+ kwargs = dict(BUILD_DEFAULTS)
+ kwargs.update(overrides)
+ out = tmp_path / "consensus.fa"
+ wgs.build_consensus(
+ vcf_path=write_vcf(tmp_path / "in.vcf", vcf_lines),
+ reference_path=write_fasta(tmp_path / "ref.fa", ref_seqs),
+ out_path=str(out),
+ **kwargs,
+ )
+ return out
+
+
+# --------------------------------------------------------------------------- #
+# iupac_for_bases
+# --------------------------------------------------------------------------- #
+
+class TestIupacForBases:
+ @pytest.mark.parametrize("bases,expected", [
+ (["A", "C", "G"], "V"),
+ (["A", "C", "T"], "H"),
+ (["A", "G", "T"], "D"),
+ (["C", "G", "T"], "B"),
+ (["A", "G"], "R"),
+ (["C", "T"], "Y"),
+ (["G", "C"], "S"),
+ (["A", "T"], "W"),
+ (["G", "T"], "K"),
+ (["A", "C"], "M"),
+ ])
+ def test_ambiguity_codes(self, bases, expected):
+ assert wgs.iupac_for_bases(bases) == expected
+
+ def test_all_four_bases_collapse_to_n(self):
+ assert wgs.iupac_for_bases(["A", "C", "G", "T"]) == "N"
+
+ def test_empty_input_is_n(self):
+ assert wgs.iupac_for_bases([]) == "N"
+
+ def test_only_unknown_characters_is_n(self):
+ assert wgs.iupac_for_bases(["X", "Z", "-", "N"]) == "N", \
+ "characters outside ACGT contribute nothing, so the set is empty -> N"
+
+ @pytest.mark.parametrize("bases,expected", [
+ (["A"], "A"),
+ (["t"], "T"),
+ (["G", "G", "G"], "G"),
+ ])
+ def test_single_distinct_base_returns_itself(self, bases, expected):
+ assert wgs.iupac_for_bases(bases) == expected
+
+ def test_non_dna_members_are_dropped_before_lookup(self):
+ # 'N' and '' are discarded, leaving {A, G} -> R rather than N.
+ assert wgs.iupac_for_bases(["A", "", "N", "g"]) == "R"
+
+ def test_accepts_any_iterable_of_characters(self):
+ assert wgs.iupac_for_bases("ACG") == "V"
+
+
+# --------------------------------------------------------------------------- #
+# parse_info_field
+# --------------------------------------------------------------------------- #
+
+class TestParseInfoField:
+ @pytest.mark.parametrize("info", [".", ""])
+ def test_missing_info_is_empty_dict(self, info):
+ assert wgs.parse_info_field(info) == {}
+
+ def test_key_value_pairs(self):
+ assert wgs.parse_info_field("DP=30;ADP=28") == {"DP": "30", "ADP": "28"}
+
+ def test_flag_without_value_becomes_true(self):
+ assert wgs.parse_info_field("INDEL;DP=30") == {"INDEL": "true", "DP": "30"}
+
+ def test_items_are_stripped_and_blanks_skipped(self):
+ assert wgs.parse_info_field(" DP=30 ;; INDEL ") == {"DP": "30", "INDEL": "true"}
+
+ def test_only_the_first_equals_sign_splits(self):
+ assert wgs.parse_info_field("ANN=a=b=c") == {"ANN": "a=b=c"}
+
+
+# --------------------------------------------------------------------------- #
+# parse_format_sample
+# --------------------------------------------------------------------------- #
+
+class TestParseFormatSample:
+ @pytest.mark.parametrize("fmt", ["", "."])
+ def test_missing_format_is_empty_dict(self, fmt):
+ assert wgs.parse_format_sample(fmt, "0/1:30") == {}
+
+ def test_zips_keys_to_values(self):
+ assert wgs.parse_format_sample("GT:DP:AD", "0/1:30:15,15") == {
+ "GT": "0/1", "DP": "30", "AD": "15,15",
+ }
+
+ def test_short_sample_column_pads_with_dot(self):
+ assert wgs.parse_format_sample("GT:DP:AD:RO", "0/1:30") == {
+ "GT": "0/1", "DP": "30", "AD": ".", "RO": ".",
+ }, "keys past the end of the sample column must be filled with '.'"
+
+ def test_empty_sample_column_pads_every_key(self):
+ assert wgs.parse_format_sample("GT:DP", "") == {"GT": ".", "DP": "."}
+
+ def test_extra_sample_values_are_ignored(self):
+ assert wgs.parse_format_sample("GT", "0/1:30:99") == {"GT": "0/1"}
+
+
+# --------------------------------------------------------------------------- #
+# get_dp
+# --------------------------------------------------------------------------- #
+
+class TestGetDp:
+ def test_format_dp_wins_over_info(self):
+ assert wgs.get_dp("GT:DP", "0/0:42", "DP=7;ADP=8") == 42
+
+ def test_falls_back_to_info_dp_when_format_dp_is_missing(self):
+ assert wgs.get_dp("GT", "0/0", "DP=7") == 7
+
+ @pytest.mark.parametrize("dp_value", [".", ""])
+ def test_falls_back_to_info_dp_when_format_dp_is_blank(self, dp_value):
+ assert wgs.get_dp("GT:DP", f"0/0:{dp_value}", "DP=7") == 7
+
+ def test_falls_back_to_info_dp_when_format_dp_is_not_numeric(self):
+ assert wgs.get_dp("GT:DP", "0/0:high", "DP=7") == 7
+
+ def test_falls_back_to_info_adp_when_info_dp_is_absent(self):
+ assert wgs.get_dp("GT", "0/0", "ADP=9") == 9
+
+ def test_falls_back_to_info_adp_when_info_dp_is_not_numeric(self):
+ assert wgs.get_dp("GT", "0/0", "DP=deep;ADP=9") == 9
+
+ def test_zero_when_no_depth_anywhere(self):
+ assert wgs.get_dp("GT", "0/0", ".") == 0
+
+ def test_float_depth_is_truncated_to_int(self):
+ assert wgs.get_dp("GT:DP", "0/0:12.9", ".") == 12
+
+
+# --------------------------------------------------------------------------- #
+# get_gt
+# --------------------------------------------------------------------------- #
+
+class TestGetGt:
+ @pytest.mark.parametrize("gt", [".", "./.", ".|.", "./", ".|", ""])
+ def test_missing_genotypes_are_none(self, gt):
+ assert wgs.get_gt("GT:DP", f"{gt}:30") is None
+
+ def test_absent_gt_key_is_none(self):
+ assert wgs.get_gt("DP:AD", "30:30,0") is None
+
+ @pytest.mark.parametrize("gt", ["0/0", "0/1", "1|2", "1"])
+ def test_present_genotypes_are_returned_verbatim(self, gt):
+ assert wgs.get_gt("GT:DP", f"{gt}:30") == gt
+
+ def test_half_missing_genotype_is_not_filtered_here(self):
+ # QUIRK: '0/.' is not in the missing-genotype set, so get_gt passes it
+ # through; parse_gt_alleles is what ultimately rejects it.
+ assert wgs.get_gt("GT", "0/.") == "0/."
+ assert wgs.parse_gt_alleles("0/.") is None
+
+
+# --------------------------------------------------------------------------- #
+# get_ro_ao
+# --------------------------------------------------------------------------- #
+
+class TestGetRoAo:
+ def test_reads_ro_and_ao(self):
+ assert wgs.get_ro_ao("GT:RO:AO", "0/1:80:20") == (80, 20)
+
+ def test_ao_with_multiple_alts_uses_only_the_first_value(self):
+ assert wgs.get_ro_ao("GT:RO:AO", "1/2:5:7,9") == (5, 7)
+
+ def test_falls_back_to_ad_when_ro_and_ao_are_absent(self):
+ assert wgs.get_ro_ao("GT:AD", "0/1:12,8") == (12, 8)
+
+ def test_ad_alt_counts_are_summed(self):
+ assert wgs.get_ro_ao("GT:AD", "1/2:4,3,2") == (4, 5), \
+ "ro is AD[0] and ao is the sum of every remaining AD field"
+
+ def test_ad_with_a_single_field_gives_zero_alt_reads(self):
+ assert wgs.get_ro_ao("GT:AD", "0/0:30") == (30, 0)
+
+ @pytest.mark.parametrize("fmt,sample", [
+ ("GT:DP", "0/0:30"),
+ ("GT:RO:AO:AD", "0/0:.:.:."),
+ ])
+ def test_returns_none_none_when_nothing_is_available(self, fmt, sample):
+ assert wgs.get_ro_ao(fmt, sample) == (None, None)
+
+ def test_ad_fallback_overrides_a_present_ro_when_ao_is_missing(self):
+ # QUIRK: the AD fallback triggers when EITHER RO or AO is missing and it
+ # reassigns BOTH, so the explicit RO=99 is silently discarded.
+ assert wgs.get_ro_ao("GT:RO:AD", "0/1:99:12,8") == (12, 8)
+
+ def test_ad_fallback_rejects_float_formatted_counts(self):
+ # QUIRK: RO/AO go through int(float(...)) but the AD fallback uses a bare
+ # int(), so a float-formatted AD is dropped entirely.
+ assert wgs.get_ro_ao("GT:AD", "0/0:30.0,0") == (None, None)
+
+ def test_a_malformed_ad_yields_neither_count(self):
+ """Both or neither: a half-assigned (5, None) is a trap for any future caller that
+ checks only one of the two."""
+ assert wgs.get_ro_ao("GT:AD", "0/1:5,x") == (None, None)
+
+
+# --------------------------------------------------------------------------- #
+# parse_gt_alleles
+# --------------------------------------------------------------------------- #
+
+class TestParseGtAlleles:
+ @pytest.mark.parametrize("gt,expected", [
+ ("1", [1]),
+ ("0", [0]),
+ ("0/0", [0, 0]),
+ ("0/1", [0, 1]),
+ ("0|1", [0, 1]),
+ ("1/2/3", [1, 2, 3]),
+ (" 0 / 1 ", [0, 1]),
+ ])
+ def test_parses_allele_indices(self, gt, expected):
+ assert wgs.parse_gt_alleles(gt) == expected
+
+ @pytest.mark.parametrize("gt", ["", ".", "./.", ".|.", "0/.", "./1", "A/T", "0/1x"])
+ def test_missing_or_non_integer_genotypes_are_none(self, gt):
+ assert wgs.parse_gt_alleles(gt) is None
+
+
+# --------------------------------------------------------------------------- #
+# parse_vcf_record
+# --------------------------------------------------------------------------- #
+
+class TestParseVcfRecord:
+ def test_parses_a_ten_column_record(self):
+ rec = wgs.parse_vcf_record(
+ "chr1\t42\trs1\tA\tG\t60\tPASS\tDP=30\tGT:DP\t0/1:30\n"
+ )
+ assert (rec.chrom, rec.pos, rec.ref, rec.alt) == ("chr1", 42, "A", "G")
+ assert (rec.qual, rec.flt, rec.info) == ("60", "PASS", "DP=30")
+ assert (rec.fmt, rec.sample) == ("GT:DP", "0/1:30")
+
+ def test_fewer_than_ten_columns_is_none(self):
+ assert wgs.parse_vcf_record("chr1\t42\trs1\tA\tG\t60\tPASS\tDP=30\tGT:DP") is None
+
+ def test_non_integer_position_is_none(self):
+ assert wgs.parse_vcf_record(
+ "chr1\tNA\trs1\tA\tG\t60\tPASS\t.\tGT\t0/1"
+ ) is None
+
+ def test_space_separated_line_is_none(self):
+ # The split is tab-only, so a space-delimited line collapses to one field.
+ assert wgs.parse_vcf_record("chr1 42 rs1 A G 60 PASS . GT 0/1") is None
+
+ def test_extra_sample_columns_are_ignored(self):
+ rec = wgs.parse_vcf_record(
+ "chr1\t7\t.\tA\tG\t60\tPASS\t.\tGT\t0/1\t1/1\t0/0\n"
+ )
+ assert rec.sample == "0/1", "only the first sample column is read"
+
+ def test_record_is_frozen(self):
+ rec = wgs.parse_vcf_record("chr1\t7\t.\tA\tG\t60\tPASS\t.\tGT\t0/1")
+ with pytest.raises(dataclasses.FrozenInstanceError):
+ rec.pos = 8
+
+
+# --------------------------------------------------------------------------- #
+# allele_bases_from_record
+# --------------------------------------------------------------------------- #
+
+class TestAlleleBasesFromRecord:
+ def test_homozygous_alt_snp(self):
+ rec = make_rec(ref="A", alt="T", sample="1/1:30:0,30")
+ assert wgs.allele_bases_from_record(rec, "A") == {"T"}
+
+ def test_heterozygous_snp_includes_the_reference_base(self):
+ rec = make_rec(ref="A", alt="T", sample="0/1:30:15,15")
+ assert wgs.allele_bases_from_record(rec, "A") == {"A", "T"}
+
+ def test_ignore_ref_drops_the_reference_allele(self):
+ rec = make_rec(ref="A", alt="T", sample="0/1:30:15,15")
+ assert wgs.allele_bases_from_record(rec, "A", ignore_ref=True) == {"T"}
+
+ def test_multiallelic_genotype_collects_both_alts(self):
+ rec = make_rec(ref="A", alt="C,G", sample="1/2:30:0,15,15")
+ assert wgs.allele_bases_from_record(rec, "A") == {"C", "G"}
+
+ def test_monomorphic_record_yields_the_reference_base(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:30:30,0")
+ assert wgs.allele_bases_from_record(rec, "A") == {"A"}
+
+ def test_monomorphic_record_with_ignore_ref_signals_mask(self):
+ # Nothing is left to report once the reference allele is dropped.
+ rec = make_rec(ref="A", alt=".", sample="0/0:30:30,0")
+ assert wgs.allele_bases_from_record(rec, "A", ignore_ref=True) is None
+
+ def test_monomorphic_record_with_non_dna_reference_signals_mask(self):
+ rec = make_rec(ref="N", alt=".", sample="0/0:30:30,0")
+ assert wgs.allele_bases_from_record(rec, "N") is None
+
+ def test_reference_allele_is_skipped_when_ref_base_is_not_dna(self):
+ rec = make_rec(ref="N", alt="T", sample="0/1:30:15,15")
+ assert wgs.allele_bases_from_record(rec, "N") == {"T"}
+
+ @pytest.mark.parametrize("alt,sample", [
+ ("AT", "1/1:30:0,30"), # insertion
+ ("ATG", "1/1:30:0,30"), # MNP
+ ("", "1/1:30:0,30"), # symbolic allele
+ ])
+ def test_non_snp_allele_signals_mask(self, alt, sample):
+ rec = make_rec(ref="A", alt=alt, sample=sample)
+ assert wgs.allele_bases_from_record(rec, "A") is None, \
+ "a non-SNP allele must return None, the mask signal"
+
+ def test_allele_index_out_of_range_signals_mask(self):
+ rec = make_rec(ref="A", alt="T", sample="2/2:30:0,30")
+ assert wgs.allele_bases_from_record(rec, "A") is None
+
+ def test_negative_allele_index_signals_mask(self):
+ rec = make_rec(ref="A", alt="T", sample="-1/-1:30:0,30")
+ assert wgs.allele_bases_from_record(rec, "A") is None
+
+ def test_missing_genotype_signals_mask(self):
+ rec = make_rec(ref="A", alt="T", sample="./.:30:15,15")
+ assert wgs.allele_bases_from_record(rec, "A") is None
+
+ def test_spanning_deletion_star_allele_is_skipped(self):
+ rec = make_rec(ref="A", alt="*,T", sample="1/2:30:0,15,15")
+ assert wgs.allele_bases_from_record(rec, "A") == {"T"}
+
+ def test_only_star_allele_signals_mask(self):
+ rec = make_rec(ref="A", alt="*", sample="1/1:30:0,30")
+ assert wgs.allele_bases_from_record(rec, "A") is None
+
+ def test_alt_alleles_are_uppercased(self):
+ rec = make_rec(ref="A", alt="t", sample="1/1:30:0,30")
+ assert wgs.allele_bases_from_record(rec, "A") == {"T"}
+
+
+# --------------------------------------------------------------------------- #
+# consensus_for_position: decision order and every branch
+# --------------------------------------------------------------------------- #
+
+def interval_masker(tmp_path, body):
+ return wgs.IntervalMasker(write_text(tmp_path / "exclude.tsv", body))
+
+
+def point_masker(tmp_path, body):
+ return wgs.PointMasker(write_text(tmp_path / "mask_sites.tsv", body))
+
+
+def csv_masker(tmp_path, body, contigs=None):
+ return wgs.CsvMasker(write_text(tmp_path / "special.csv", body), contigs)
+
+
+class TestConsensusMasking:
+ def test_interval_mask_returns_mask_char(self, tmp_path):
+ rec = make_rec(pos=10)
+ assert call_consensus([rec], interval_mask=interval_masker(tmp_path, "chr1\t5\t15\n")) == "X"
+
+ def test_point_mask_returns_mask_char(self, tmp_path):
+ rec = make_rec(pos=10)
+ assert call_consensus([rec], point_mask=point_masker(tmp_path, "chr1\t10\n")) == "X"
+
+ def test_csv_mask_returns_mask_char(self, tmp_path):
+ body = "rv_position,pal,homopolymer,GCrich,repetitive,blindspot\n10,0,0,0,1,0\n"
+ rec = make_rec(pos=10)
+ assert call_consensus([rec], csv_mask=csv_masker(tmp_path, body, ["chr1"])) == "X"
+
+ def test_mask_char_is_configurable(self, tmp_path):
+ rec = make_rec(pos=10)
+ masker = point_masker(tmp_path, "chr1\t10\n")
+ assert call_consensus([rec], point_mask=masker, mask_char="?") == "?"
+
+ def test_masking_beats_the_depth_gate(self, tmp_path):
+ # Masks are checked before the DP gate, so a zero-depth masked position is
+ # X and not the no-call char. (Mask vs FILTER ordering is not observable:
+ # both branches return mask_char.)
+ rec = make_rec(pos=10, fmt="GT:DP", sample="0/0:0")
+ masker = point_masker(tmp_path, "chr1\t10\n")
+ assert call_consensus([rec], point_mask=masker, nocall_char="-") == "X"
+
+ def test_position_outside_the_mask_is_not_masked(self, tmp_path):
+ rec = make_rec(pos=20, ref="A")
+ assert call_consensus([rec], interval_mask=interval_masker(tmp_path, "chr1\t5\t15\n")) == "A"
+
+
+class TestConsensusFilterAndDepth:
+ @pytest.mark.parametrize("flt", ["str10", "baq_dropout", "LowQual", "q10;str10"])
+ def test_non_pass_filter_is_masked(self, flt):
+ assert call_consensus([make_rec(flt=flt)]) == "X"
+
+ @pytest.mark.parametrize("flt", ["PASS", ".", ""])
+ def test_pass_like_filters_are_accepted(self, flt):
+ assert call_consensus([make_rec(ref="A", flt=flt)]) == "A"
+
+ def test_a_single_failing_record_masks_the_whole_position(self):
+ recs = [make_rec(pos=5, alt="C", sample="1/0:30:15,15"),
+ make_rec(pos=5, alt="T", flt="str10", sample="0/1:30:15,15")]
+ assert call_consensus(recs) == "X"
+
+ def test_filter_is_checked_before_depth(self):
+ # DP=0 alone would give the no-call char; the failing FILTER wins.
+ rec = make_rec(flt="str10", fmt="GT:DP", sample="0/0:0")
+ assert call_consensus([rec]) == "X", "FILTER is evaluated before the depth gate"
+
+ def test_depth_at_or_below_min_dp_is_a_no_call(self):
+ rec = make_rec(fmt="GT:DP", sample="0/0:5")
+ assert call_consensus([rec], min_dp=5) == "-", "the gate is `dp <= min_dp`, not `<`"
+
+ def test_depth_just_above_min_dp_proceeds(self):
+ rec = make_rec(ref="A", fmt="GT:DP:AD", sample="0/0:6:6,0")
+ assert call_consensus([rec], min_dp=5, ref_min_dp=0) == "A"
+
+ def test_zero_depth_is_a_no_call_with_default_min_dp(self):
+ assert call_consensus([make_rec(fmt="GT:DP", sample="0/0:0")]) == "-"
+
+ def test_nocall_char_is_configurable(self):
+ rec = make_rec(fmt="GT:DP", sample="0/0:0")
+ assert call_consensus([rec], nocall_char="?") == "?"
+
+ def test_depth_is_the_maximum_across_records_at_the_position(self):
+ recs = [make_rec(pos=5, alt="C", fmt="GT:DP:AD", sample="1/0:0:0,0"),
+ make_rec(pos=5, alt="T", fmt="GT:DP:AD", sample="0/1:50:25,25")]
+ assert call_consensus(recs, min_dp=10) == "Y", \
+ "dp_pos is max() over the records, so one deep record rescues the site"
+
+
+class TestConsensusReferenceBaseSource:
+ def test_reference_fasta_wins_over_the_record_ref_column(self):
+ rec = make_rec(pos=2, ref="A", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec], ref_seqs={"chr1": "GGGG"}) == "G"
+
+ def test_record_ref_is_used_when_the_contig_is_absent(self):
+ rec = make_rec(pos=2, ref="C", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec], ref_seqs={"other": "GGGG"}) == "C"
+
+ def test_record_ref_is_used_when_the_position_is_past_the_contig_end(self):
+ rec = make_rec(pos=99, ref="T", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec], ref_seqs={"chr1": "GG"}) == "T"
+
+ def test_only_the_first_character_of_a_multi_base_ref_is_used(self):
+ rec = make_rec(pos=1, ref="CTT", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec]) == "C"
+
+ def test_reference_base_is_uppercased(self):
+ rec = make_rec(pos=1, ref="a", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec], ref_seqs={"chr1": "acgt"}) == "A"
+
+ def test_falls_back_to_n_when_no_reference_is_available(self):
+ rec = make_rec(pos=1, ref="", alt=".", sample="0/0:100:100,0")
+ assert call_consensus([rec]) == "N", \
+ "with no FASTA and an empty REF the reference base defaults to N"
+
+ def test_non_dna_reference_base_is_n_at_a_monomorphic_site(self):
+ rec = make_rec(pos=1, ref="N", alt=".", sample="0/0:100:100,0")
+ assert call_consensus([rec]) == "N"
+
+
+class TestConsensusMonomorphicGates:
+ """The confidence gates that guard a monomorphic-reference (ALT '.') call."""
+
+ def test_confident_reference_call_emits_the_reference_base(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:30:30,0")
+ assert call_consensus([rec]) == "A"
+
+ def test_empty_alt_column_is_treated_as_monomorphic(self):
+ rec = make_rec(ref="A", alt="", sample="0/0:30:30,0")
+ assert call_consensus([rec]) == "A"
+
+ def test_missing_genotype_is_n(self):
+ rec = make_rec(ref="A", alt=".", sample="./.:30:30,0")
+ assert call_consensus([rec]) == "N"
+
+ def test_non_homozygous_reference_genotype_is_n(self):
+ rec = make_rec(ref="A", alt=".", sample="0/1:30:30,0")
+ assert call_consensus([rec]) == "N", \
+ "a non-ref allele index with no ALT to resolve it cannot be trusted"
+
+ def test_haploid_reference_genotype_is_accepted(self):
+ rec = make_rec(ref="A", alt=".", sample="0:30:30,0")
+ assert call_consensus([rec]) == "A"
+
+ def test_base_depth_below_ref_min_dp_is_n(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:30:5,0")
+ assert call_consensus([rec], max_ref_del_frac=1.0) == "N", \
+ "RO+AO (5) is below ref_min_dp (10) even though DP (30) is not"
+
+ def test_base_depth_uses_ro_plus_ao_not_dp(self):
+ rec = make_rec(ref="A", alt=".", fmt="GT:DP:RO:AO", sample="0/0:100:8:1")
+ assert call_consensus([rec], max_ref_del_frac=1.0) == "N", \
+ "the gate must read base-calling depth (RO+AO=9), not raw DP=100"
+
+ def test_base_depth_falls_back_to_dp_without_ro_ao_or_ad(self):
+ rec = make_rec(ref="A", alt=".", fmt="GT:DP", sample="0/0:30")
+ assert call_consensus([rec]) == "A", \
+ "with no AD/RO/AO the raw DP stands in for base depth"
+
+ def test_ref_min_dp_zero_disables_the_depth_gate(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:3:3,0")
+ assert call_consensus([rec], ref_min_dp=0) == "A"
+
+ def test_deletion_fraction_at_or_above_the_threshold_is_n(self):
+ # DP counts 100 reads but only 40 carry a base: 60% span a deletion.
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:40,0")
+ assert call_consensus([rec]) == "N"
+
+ def test_deletion_fraction_below_the_threshold_emits_the_reference(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:60,0")
+ assert call_consensus([rec]) == "A", "(100-60)/100 = 0.4 is below the 0.5 default"
+
+ def test_deletion_gate_boundary_is_inclusive(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:50,0")
+ assert call_consensus([rec]) == "N", "the comparison is `>=`, so exactly 0.5 masks"
+
+ def test_max_ref_del_frac_one_disables_the_deletion_gate(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:10,0")
+ assert call_consensus([rec], max_ref_del_frac=1.0) == "A"
+
+ def test_alt_fraction_at_or_above_the_threshold_is_n(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:90,10")
+ assert call_consensus([rec]) == "N", "10/100 = 0.10 reaches max_ref_altfrac"
+
+ def test_alt_fraction_below_the_threshold_emits_the_reference(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:91,9")
+ assert call_consensus([rec]) == "A"
+
+ def test_alt_fraction_gate_needs_enough_alt_reads(self):
+ # 1/10 = 0.10 clears max_ref_altfrac but only one alt read is below
+ # max_ref_min_alt, so a single stray read cannot N a reference call.
+ rec = make_rec(ref="A", alt=".", sample="0/0:10:9,1")
+ assert call_consensus([rec]) == "A"
+
+ def test_alt_fraction_gate_fires_at_exactly_max_ref_min_alt(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:20:18,2")
+ assert call_consensus([rec]) == "N"
+
+ def test_alt_fraction_gate_works_from_ro_and_ao(self):
+ rec = make_rec(ref="A", alt=".", fmt="GT:DP:RO:AO", sample="0/0:100:90:10")
+ assert call_consensus([rec]) == "N"
+
+ def test_max_ref_altfrac_one_disables_the_alt_fraction_gate(self):
+ rec = make_rec(ref="A", alt=".", sample="0/0:100:50,50")
+ assert call_consensus([rec], max_ref_altfrac=1.0, max_ref_del_frac=1.0) == "A"
+
+ def test_alt_fraction_gate_is_skipped_without_ro_ao_evidence(self):
+ # QUIRK: with no AD/RO/AO the alt-fraction gate cannot run at all, so a
+ # DP-only record always passes it.
+ rec = make_rec(ref="A", alt=".", fmt="GT:DP", sample="0/0:100")
+ assert call_consensus([rec]) == "A"
+
+
+class TestConsensusVariantSites:
+ def test_homozygous_alt_snp(self):
+ rec = make_rec(ref="A", alt="T", sample="1/1:30:0,30")
+ assert call_consensus([rec]) == "T"
+
+ def test_heterozygous_snp_becomes_an_iupac_code(self):
+ rec = make_rec(ref="A", alt="G", sample="0/1:30:15,15")
+ assert call_consensus([rec]) == "R"
+
+ def test_three_way_multiallelic_in_one_record(self):
+ rec = make_rec(ref="A", alt="C,G", sample="1/2:30:0,15,15")
+ assert call_consensus([rec], ref_seqs={"chr1": "A"}) == "S", \
+ "{C,G} without the reference allele is S"
+
+ def test_indel_allele_is_masked(self):
+ rec = make_rec(ref="A", alt="AGG", sample="1/1:30:0,30")
+ assert call_consensus([rec]) == "X"
+
+ def test_out_of_range_allele_index_is_masked(self):
+ rec = make_rec(ref="A", alt="T", sample="3/3:30:0,30")
+ assert call_consensus([rec]) == "X"
+
+ def test_missing_genotype_at_a_variant_site_is_masked(self):
+ rec = make_rec(ref="A", alt="T", sample="./.:30:15,15")
+ assert call_consensus([rec]) == "X"
+
+ def test_variant_sites_skip_the_monomorphic_confidence_gates(self):
+ # A shallow ALT call is emitted as-is; ref_min_dp only guards ALT '.' sites.
+ rec = make_rec(ref="A", alt="T", fmt="GT:DP:AD", sample="1/1:2:0,2")
+ assert call_consensus([rec], ref_min_dp=100) == "T"
+
+
+class TestConsensusMultiRecordUnion:
+ def test_norm_split_multiallelic_unions_both_alts(self):
+ recs = [make_rec(pos=5, ref="A", alt="C", sample="1/0:30:15,15"),
+ make_rec(pos=5, ref="A", alt="G", sample="0/1:30:15,15")]
+ assert call_consensus(recs, ref_seqs={"chr1": "AAAAA"}) == "S", \
+ "the per-record 0 is a split artifact, so the reference base is dropped"
+
+ def test_single_record_keeps_the_reference_allele(self):
+ # Same genotype shape, one record: here 0 is a real reference call.
+ rec = make_rec(pos=5, ref="A", alt="C", sample="0/1:30:15,15")
+ assert call_consensus([rec], ref_seqs={"chr1": "AAAAA"}) == "M"
+
+ def test_three_records_union_to_a_three_base_code(self):
+ recs = [make_rec(pos=5, ref="T", alt="A", sample="1/0:30:10,10"),
+ make_rec(pos=5, ref="T", alt="C", sample="0/1:30:10,10"),
+ make_rec(pos=5, ref="T", alt="G", sample="0/1:30:10,10")]
+ assert call_consensus(recs, ref_seqs={"chr1": "TTTTT"}) == "V"
+
+ def test_union_of_all_four_bases_is_n(self):
+ recs = [make_rec(pos=5, ref="T", alt="A", sample="1/0:30:10,10"),
+ make_rec(pos=5, ref="T", alt="C", sample="0/1:30:10,10"),
+ make_rec(pos=5, ref="T", alt="G,T", sample="1/2:30:5,5,5")]
+ assert call_consensus(recs, ref_seqs={"chr1": "TTTTT"}) == "N"
+
+ def test_one_non_snp_sibling_masks_the_position(self):
+ recs = [make_rec(pos=5, ref="A", alt="C", sample="1/0:30:15,15"),
+ make_rec(pos=5, ref="A", alt="ACC", sample="0/1:30:15,15")]
+ assert call_consensus(recs, ref_seqs={"chr1": "AAAAA"}) == "X"
+
+ def test_a_monomorphic_sibling_masks_the_position(self):
+ # QUIRK: with ignore_ref the ALT '.' record contributes nothing, which
+ # allele_bases_from_record reports as None, i.e. the mask signal. A VCF
+ # that carries both an ALT '.' row and a variant row at one position
+ # therefore masks rather than reporting the variant.
+ recs = [make_rec(pos=5, ref="A", alt=".", sample="0/0:30:30,0"),
+ make_rec(pos=5, ref="A", alt="C", sample="0/1:30:15,15")]
+ assert call_consensus(recs, ref_seqs={"chr1": "AAAAA"}) == "X"
+
+ def test_a_star_only_sibling_masks_the_position(self):
+ # QUIRK: the '*' allele is skipped and, with ignore_ref, the sibling's 0 is
+ # dropped too, so that record contributes an empty set. allele_bases_from_record
+ # reports empty as None, i.e. mask, so the site is X rather than C.
+ recs = [make_rec(pos=5, ref="A", alt="*", sample="1/0:30:15,15"),
+ make_rec(pos=5, ref="A", alt="C", sample="0/1:30:15,15")]
+ assert call_consensus(recs, ref_seqs={"chr1": "AAAAA"}) == "X"
+
+
+# --------------------------------------------------------------------------- #
+# IntervalMasker
+# --------------------------------------------------------------------------- #
+
+class TestIntervalMasker:
+ def test_no_file_masks_nothing(self):
+ assert wgs.IntervalMasker(None).masked("chr1", 10) is False
+
+ def test_missing_file_warns_without_raising(self, tmp_path, capsys):
+ masker = wgs.IntervalMasker(str(tmp_path / "nope.tsv"))
+ assert masker.masked("chr1", 10) is False
+ assert "Exclude file not found" in capsys.readouterr().err
+
+ @pytest.mark.parametrize("pos,expected", [
+ (4, False), (5, True), (10, True), (15, True), (16, False),
+ ])
+ def test_interval_bounds_are_inclusive(self, tmp_path, pos, expected):
+ masker = interval_masker(tmp_path, "chr1\t5\t15\n")
+ assert masker.masked("chr1", pos) is expected
+
+ def test_overlapping_intervals_are_merged(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t1\t10\nchr1\t5\t20\n")
+ assert masker.intervals["chr1"] == [(1, 20)]
+
+ def test_adjacent_intervals_are_merged(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t1\t5\nchr1\t6\t10\n")
+ assert masker.intervals["chr1"] == [(1, 10)], \
+ "intervals touching end+1 are joined into one"
+
+ def test_intervals_with_a_one_base_gap_stay_separate(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t1\t5\nchr1\t7\t10\n")
+ assert masker.intervals["chr1"] == [(1, 5), (7, 10)]
+
+ def test_reversed_interval_is_swapped(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t20\t10\n")
+ assert masker.intervals["chr1"] == [(10, 20)]
+ assert masker.masked("chr1", 15) is True
+
+ def test_contained_interval_is_absorbed(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t1\t100\nchr1\t20\t30\n")
+ assert masker.intervals["chr1"] == [(1, 100)]
+
+ def test_comment_header_and_short_lines_are_skipped(self, tmp_path):
+ body = (
+ "# a comment\n"
+ "chrom\tstart\tend\n"
+ "\n"
+ "chr1\t5\n"
+ "chr1\tstart\tend\n"
+ "chr1\t5\t15\n"
+ )
+ masker = interval_masker(tmp_path, body)
+ assert masker.intervals == {"chr1": [(5, 15)]}
+
+ def test_a_contig_named_like_the_header_is_still_masked(self, tmp_path):
+ """A header row is one whose coordinate columns are not numbers. Recognising it by the
+ contig name instead silently discarded every interval on a contig called chrom*."""
+ masker = interval_masker(tmp_path, "chromosome_1\t5\t15\n")
+ assert masker.intervals == {"chromosome_1": [(5, 15)]}
+ assert masker.masked("chromosome_1", 10) is True
+
+ def test_a_real_header_row_is_still_skipped(self, tmp_path):
+ masker = interval_masker(tmp_path, "Chrom\tStart\tEnd\nchr1\t5\t15\n")
+ assert masker.intervals == {"chr1": [(5, 15)]}
+
+ def test_intervals_are_tracked_per_contig(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t5\t15\nchr2\t100\t200\n")
+ assert masker.masked("chr1", 10) is True
+ assert masker.masked("chr2", 150) is True
+ assert masker.masked("chr2", 10) is False
+
+ def test_ascending_queries_across_several_intervals(self, tmp_path):
+ masker = interval_masker(tmp_path, "chr1\t5\t10\nchr1\t20\t25\n")
+ assert [masker.masked("chr1", p) for p in (1, 7, 12, 22, 30)] == \
+ [False, True, False, True, False]
+
+ def test_forward_only_pointer_misses_a_descending_query(self, tmp_path):
+ # DOCUMENTED LIMITATION (pinned): masked() advances a per-contig pointer
+ # and never rewinds, so it is only correct for ascending positions. That
+ # holds for build_consensus, which walks the VCF in order. This test pins
+ # the limitation rather than asserting the "right" answer.
+ masker = interval_masker(tmp_path, "chr1\t10\t20\n")
+ assert masker.masked("chr1", 15) is True
+ assert masker.masked("chr1", 25) is False
+ assert masker.masked("chr1", 15) is False, \
+ "after querying past the interval the pointer cannot go back"
+
+
+# --------------------------------------------------------------------------- #
+# PointMasker
+# --------------------------------------------------------------------------- #
+
+class TestPointMasker:
+ def test_no_file_masks_nothing(self):
+ assert wgs.PointMasker(None).masked("chr1", 10) is False
+
+ def test_missing_file_warns_without_raising(self, tmp_path, capsys):
+ masker = wgs.PointMasker(str(tmp_path / "nope.tsv"))
+ assert masker.masked("chr1", 10) is False
+ assert "Mask-sites file not found" in capsys.readouterr().err
+
+ def test_masks_listed_positions_only(self, tmp_path):
+ masker = point_masker(tmp_path, "chr1\t10\nchr1\t12\nchr2\t10\n")
+ assert masker.masked("chr1", 10) is True
+ assert masker.masked("chr1", 11) is False
+ assert masker.masked("chr2", 10) is True
+ assert masker.masked("chr3", 10) is False
+
+ def test_lookup_is_stateless_and_order_independent(self, tmp_path):
+ masker = point_masker(tmp_path, "chr1\t10\nchr1\t20\n")
+ assert masker.masked("chr1", 20) is True
+ assert masker.masked("chr1", 10) is True, \
+ "unlike IntervalMasker this is a plain set lookup, so order is irrelevant"
+
+ def test_comments_blanks_short_and_non_numeric_lines_are_skipped(self, tmp_path):
+ body = "# comment\n\nchr1\nchr1\tNA\nchr1\t10\textra\n"
+ masker = point_masker(tmp_path, body)
+ assert masker.points == {"chr1": {10}}, "extra columns past POS are ignored"
+
+
+# --------------------------------------------------------------------------- #
+# CsvMasker
+# --------------------------------------------------------------------------- #
+
+CSV_HEADER = "rv_position,pal,homopolymer,GCrich,repetitive,blindspot\n"
+
+
+class TestCsvMasker:
+ def test_no_file_masks_nothing(self):
+ assert wgs.CsvMasker(None).masked("chr1", 10) is False
+
+ def test_missing_file_warns_without_raising(self, tmp_path, capsys):
+ masker = wgs.CsvMasker(str(tmp_path / "nope.csv"), ["chr1"])
+ assert masker.masked("chr1", 10) is False
+ assert "Special Mask CSV file not found" in capsys.readouterr().err
+
+ def test_masks_on_repetitive_flag(self, tmp_path):
+ masker = csv_masker(tmp_path, CSV_HEADER + "10,0,0,0,1,0\n", ["chr1"])
+ assert masker.masked("chr1", 10) is True
+
+ def test_masks_on_blindspot_flag(self, tmp_path):
+ masker = csv_masker(tmp_path, CSV_HEADER + "10,0,0,0,0,1\n", ["chr1"])
+ assert masker.masked("chr1", 10) is True
+
+ def test_other_flags_do_not_mask(self, tmp_path):
+ masker = csv_masker(tmp_path, CSV_HEADER + "10,1,1,1,0,0\n", ["chr1"])
+ assert masker.masked("chr1", 10) is False, \
+ "pal / homopolymer / GCrich are read but never mask"
+
+ @pytest.mark.parametrize("flag,masked", [
+ ("1", True), ("1.0", True), ("0", False), ("", False),
+ ("2", False), ("yes", False),
+ ])
+ def test_flag_parsing(self, tmp_path, flag, masked):
+ masker = csv_masker(tmp_path, CSV_HEADER + f"10,0,0,0,{flag},0\n", ["chr1"])
+ assert masker.masked("chr1", 10) is masked, \
+ "only a flag that parses to exactly 1 masks"
+
+ def test_rows_with_a_bad_position_are_skipped_individually(self, tmp_path):
+ body = CSV_HEADER + "NA,0,0,0,1,0\n11,0,0,0,1,0\n"
+ masker = csv_masker(tmp_path, body, ["chr1"])
+ assert masker.points == {11}, "a malformed row must not drop the rest of the file"
+
+ def test_missing_rv_position_header_warns_and_masks_nothing(self, tmp_path, capsys):
+ body = "position,repetitive,blindspot\n10,1,0\n"
+ masker = csv_masker(tmp_path, body, ["chr1"])
+ assert masker.points == set()
+ assert "missing 'rv_position' header" in capsys.readouterr().err
+
+ def test_mask_is_bound_to_the_single_reference_contig(self, tmp_path):
+ masker = csv_masker(tmp_path, CSV_HEADER + "10,0,0,0,1,0\n", ["chr1"])
+ assert masker.contig == "chr1"
+ assert masker.masked("chr2", 10) is False, \
+ "single-contig rv_position coordinates must not leak onto another contig"
+
+ def test_without_contigs_the_mask_applies_to_any_contig(self, tmp_path):
+ masker = csv_masker(tmp_path, CSV_HEADER + "10,0,0,0,1,0\n", None)
+ assert masker.contig is None
+ assert masker.masked("anything", 10) is True
+
+ def test_multiple_contigs_raise(self, tmp_path):
+ with pytest.raises(RuntimeError, match="single-contig"):
+ csv_masker(tmp_path, CSV_HEADER + "10,0,0,0,1,0\n", ["chr1", "chr2"])
+
+ def test_multiple_contigs_without_a_csv_do_not_raise(self):
+ assert wgs.CsvMasker(None, ["chr1", "chr2"]).masked("chr1", 10) is False
+
+
+# --------------------------------------------------------------------------- #
+# FastaReader
+# --------------------------------------------------------------------------- #
+
+class TestFastaReader:
+ def test_reads_multiple_contigs(self, tmp_path):
+ path = write_fasta(tmp_path / "ref.fa", {"chr1": "ACGT", "chr2": "TTTT"}, wrap=2)
+ assert wgs.FastaReader(path).seqs == {"chr1": "ACGT", "chr2": "TTTT"}
+
+ def test_sequence_is_uppercased_and_unwrapped(self, tmp_path):
+ path = write_text(tmp_path / "ref.fa", ">chr1\nac\n gt \nAC\n")
+ assert wgs.FastaReader(path).seqs == {"chr1": "ACGTAC"}, \
+ "line whitespace is stripped and the sequence is joined and uppercased"
+
+ def test_header_is_truncated_at_the_first_whitespace(self, tmp_path):
+ path = write_text(tmp_path / "ref.fa", ">chr1 Mycobacterium tuberculosis\nACGT\n")
+ assert list(wgs.FastaReader(path).seqs) == ["chr1"]
+
+ def test_reads_a_gzipped_reference(self, tmp_path):
+ path = write_text(tmp_path / "ref.fa.gz", ">chr1\nACGT\n", gz=True)
+ assert wgs.FastaReader(path).seqs == {"chr1": "ACGT"}
+
+ def test_empty_file_yields_no_sequences(self, tmp_path):
+ assert wgs.FastaReader(write_text(tmp_path / "ref.fa", "")).seqs == {}
+
+ def test_a_bare_greater_than_header_is_reported_clearly(self, tmp_path):
+ """A malformed FASTA should name the file and the problem, not raise a bare IndexError."""
+ path = write_text(tmp_path / "ref.fa", ">\nACGT\n")
+ with pytest.raises(RuntimeError, match="no sequence name"):
+ wgs.FastaReader(path)
+
+
+# --------------------------------------------------------------------------- #
+# FastaWriter
+# --------------------------------------------------------------------------- #
+
+class TestFastaWriter:
+ def test_writes_header_and_bases(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=0)
+ w.start("chr1")
+ for b in "ACGT":
+ w.write_base(b)
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nACGT\n"
+
+ def test_wraps_at_the_requested_width(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=3)
+ w.start("chr1")
+ for b in "ABCDEFG":
+ w.write_base(b)
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nABC\nDEF\nG\n"
+
+ @pytest.mark.parametrize("wrap", [0, -5])
+ def test_non_positive_wrap_disables_wrapping(self, tmp_path, wrap):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=wrap)
+ w.start("chr1")
+ for b in "ABCDEFG":
+ w.write_base(b)
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nABCDEFG\n"
+
+ def test_empty_base_is_written_as_n(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=0)
+ w.start("chr1")
+ w.write_base("")
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nN\n"
+
+ def test_only_the_first_character_is_written_and_uppercased(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=0)
+ w.start("chr1")
+ w.write_base("acgt")
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nA\n"
+
+ def test_counts_bases_per_contig(self, tmp_path):
+ w = wgs.FastaWriter(str(tmp_path / "o.fa"), wrap=0)
+ w.start("chr1")
+ for _ in range(3):
+ w.write_base("A")
+ w.start("chr2")
+ w.write_base("C")
+ w.close()
+ assert w.counts == {"chr1": 3, "chr2": 1}
+
+ def test_a_new_contig_closes_the_previous_line(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=0)
+ w.start("chr1")
+ w.write_base("A")
+ w.start("chr2")
+ w.write_base("C")
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nA\n>chr2\nC\n"
+
+ def test_no_blank_line_when_the_wrap_already_closed_the_line(self, tmp_path):
+ out = tmp_path / "o.fa"
+ w = wgs.FastaWriter(str(out), wrap=2)
+ w.start("chr1")
+ w.write_base("A")
+ w.write_base("C")
+ w.start("chr2")
+ w.write_base("G")
+ w.close()
+ assert out.read_text(encoding="utf-8") == ">chr1\nAC\n>chr2\nG\n"
+
+
+# --------------------------------------------------------------------------- #
+# build_consensus: end to end
+# --------------------------------------------------------------------------- #
+
+# A 13 bp reference exercised by every branch of the writer in one pass.
+E2E_REF = "ACGTACGTACGTA"
+
+E2E_VCF = [
+ # pos 1: confident monomorphic reference call -> A
+ vcf_line("chr1", 1, "A", ".", sample="0/0:30:30,0"),
+ # pos 2: absent from the VCF and unmasked -> gap filled with the no-call char
+ # pos 3: absent from the VCF but inside an exclude interval -> gap filled with X
+ # pos 4: base depth 5 < ref_min_dp 10 -> N
+ vcf_line("chr1", 4, "T", ".", sample="0/0:5:5,0"),
+ # pos 5: non-PASS FILTER -> X
+ vcf_line("chr1", 5, "A", "G", flt="str10", sample="1/1:30:0,30"),
+ # pos 6: DP 0 <= min_dp 0 -> no-call char
+ vcf_line("chr1", 6, "C", ".", fmt="GT:DP", sample="0/0:0"),
+ # pos 7: heterozygous G/A -> R (and present in the CSV with both flags 0)
+ vcf_line("chr1", 7, "G", "A", sample="0/1:30:15,15"),
+ # pos 8: a confident reference call that --mask-sites overrides -> X
+ vcf_line("chr1", 8, "T", ".", sample="0/0:40:40,0"),
+ # pos 9: homozygous alt -> C
+ vcf_line("chr1", 9, "A", "C", sample="1/1:40:0,40"),
+ # pos 10: norm-split multiallelic A + T with the artifact ref allele dropped -> W
+ vcf_line("chr1", 10, "C", "A", sample="1/0:40:20,20"),
+ vcf_line("chr1", 10, "C", "T", sample="0/1:40:20,20"),
+ # pos 11: monomorphic but 20% alt reads -> N
+ vcf_line("chr1", 11, "G", ".", sample="0/0:100:80,20"),
+ # pos 12: confident monomorphic reference call -> T
+ vcf_line("chr1", 12, "T", ".", sample="0/0:30:30,0"),
+ # pos 13: a confident reference call that the special CSV mask overrides -> X
+ vcf_line("chr1", 13, "A", ".", sample="0/0:30:30,0"),
+]
+
+E2E_EXPECTED = "A-XNX-RXCWNTX"
+
+
+class TestBuildConsensusEndToEnd:
+ @pytest.fixture()
+ def e2e_out(self, tmp_path):
+ exclude = write_text(
+ tmp_path / "exclude.tsv",
+ "# regions to drop\nchrom\tstart\tend\nchr1\t3\t3\n",
+ )
+ mask_sites = write_text(tmp_path / "mask_sites.tsv", "# sites\nchr1\t8\n")
+ special = write_text(
+ tmp_path / "special.csv",
+ CSV_HEADER + "13,0,0,0,0,1\n7,0,0,0,0,0\n",
+ )
+ return run_build(
+ tmp_path, E2E_VCF, {"chr1": E2E_REF},
+ exclude_path=exclude, mask_sites_path=mask_sites,
+ special_csv_path=special, wrap=5,
+ )
+
+ def test_consensus_sequence(self, e2e_out):
+ assert read_fasta(e2e_out) == {"chr1": E2E_EXPECTED}
+
+ def test_consensus_length_matches_the_reference(self, e2e_out):
+ assert len(read_fasta(e2e_out)["chr1"]) == len(E2E_REF)
+
+ def test_output_is_wrapped(self, e2e_out):
+ lines = e2e_out.read_text(encoding="utf-8").splitlines()
+ assert lines[0] == ">chr1"
+ assert lines[1:] == ["A-XNX", "-RXCW", "NTX"]
+
+ def test_two_contigs_are_written_in_vcf_order(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 2, "C", "."),
+ vcf_line("chr2", 1, "G", "."), vcf_line("chr2", 2, "T", ".")]
+ out = run_build(tmp_path, vcf, {"chr1": "AC", "chr2": "GT"})
+ assert read_fasta(out) == {"chr1": "AC", "chr2": "GT"}
+
+ def test_reads_gzipped_inputs(self, tmp_path):
+ vcf = write_vcf(tmp_path / "in.vcf.gz",
+ [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 2, "C", ".")],
+ gz=True)
+ ref = write_fasta(tmp_path / "ref.fa.gz", {"chr1": "AC"}, gz=True)
+ out = tmp_path / "consensus.fa"
+ wgs.build_consensus(vcf_path=vcf, reference_path=ref, out_path=str(out),
+ **BUILD_DEFAULTS)
+ assert read_fasta(out) == {"chr1": "AC"}
+
+ def test_leading_gap_is_filled_from_position_one(self, tmp_path):
+ out = run_build(tmp_path, [vcf_line("chr1", 3, "G", ".")], {"chr1": "ACG"})
+ assert read_fasta(out) == {"chr1": "--G"}
+
+ def test_malformed_and_header_lines_are_skipped(self, tmp_path):
+ vcf = ["##contig=", "not\ta\tvcf\tline",
+ vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 2, "C", ".")]
+ out = run_build(tmp_path, vcf, {"chr1": "AC"})
+ assert read_fasta(out) == {"chr1": "AC"}
+
+ def test_custom_mask_and_nocall_characters(self, tmp_path):
+ mask_sites = write_text(tmp_path / "mask_sites.tsv", "chr1\t1\n")
+ vcf = [vcf_line("chr1", 1, "A", "."),
+ vcf_line("chr1", 3, "G", ".", fmt="GT:DP", sample="0/0:0")]
+ out = run_build(tmp_path, vcf, {"chr1": "ACG"},
+ mask_sites_path=mask_sites, mask_char="?", nocall_char="~")
+ assert read_fasta(out) == {"chr1": "?~~"}, \
+ "position 2 is a filled gap and position 3 is a zero-depth record"
+
+
+class TestBuildConsensusErrors:
+ def test_contig_missing_from_the_reference_raises_under_strict(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chrZ", 1, "G", ".")]
+ with pytest.raises(RuntimeError, match="Contig 'chrZ' not found in reference"):
+ run_build(tmp_path, vcf, {"chr1": "A"}, strict=True)
+
+ def test_contig_missing_from_the_reference_is_skipped_without_strict(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chrZ", 1, "G", ".")]
+ out = run_build(tmp_path, vcf, {"chr1": "A"}, strict=False)
+ assert read_fasta(out) == {"chr1": "A"}
+
+ def test_short_output_raises_a_length_mismatch(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 2, "C", ".")]
+ with pytest.raises(RuntimeError, match=r"consensus length mismatch for chr1: wrote 2 != reference 5"):
+ run_build(tmp_path, vcf, {"chr1": "ACGTA"})
+
+ def test_a_reference_contig_with_no_records_raises(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 2, "C", ".")]
+ with pytest.raises(RuntimeError, match=r"missing contig\(s\) entirely: \['chr2'\]"):
+ run_build(tmp_path, vcf, {"chr1": "AC", "chr2": "GT"})
+
+ def test_empty_vcf_raises(self, tmp_path):
+ with pytest.raises(RuntimeError, match=r"missing contig\(s\) entirely"):
+ run_build(tmp_path, [], {"chr1": "AC"})
+
+ def test_out_of_order_position_raises_under_strict(self, tmp_path):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 3, "G", "."),
+ vcf_line("chr1", 2, "C", ".")]
+ with pytest.raises(RuntimeError, match=r"Unexpected duplicate/out-of-order chr1:2"):
+ run_build(tmp_path, vcf, {"chr1": "ACG"}, strict=True)
+
+ def test_out_of_order_position_is_dropped_without_strict(self, tmp_path):
+ # The late record is discarded and position 2 keeps the gap-filled no-call.
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr1", 3, "G", "."),
+ vcf_line("chr1", 2, "C", ".")]
+ out = run_build(tmp_path, vcf, {"chr1": "ACG"}, strict=False)
+ assert read_fasta(out) == {"chr1": "A-G"}
+
+ @pytest.mark.parametrize("strict", [True, False])
+ def test_a_contig_that_reappears_raises_regardless_of_strict(self, tmp_path, strict):
+ vcf = [vcf_line("chr1", 1, "A", "."), vcf_line("chr2", 1, "G", "."),
+ vcf_line("chr1", 2, "C", ".")]
+ with pytest.raises(RuntimeError, match=r"Contig 'chr1' reappears out of order"):
+ run_build(tmp_path, vcf, {"chr1": "AC", "chr2": "G"}, strict=strict)