Test suite, CI, and a pipeline that runs off our cluster

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,151 @@
+name: Bug report
+description: A run fails, or BAMpiro produces a result that looks wrong
+title: "[bug] "
+labels: ["bug"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for the report. Most first-run failures are already covered in
+        [Troubleshooting](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/main/docs/troubleshooting.md);
+        a common one is a request that exceeds the machine ("Process requirement exceeds
+        available memory"), which `--max_cpus` and `--max_memory` cap.
+
+        When a task fails, Nextflow prints its **work directory**. `cd` there before filing:
+        `.command.err`, `.command.sh` and `.command.log` are what make the bug reproducible for us.
+
+  - type: textarea
+    id: what-happened
+    attributes:
+      label: What happened
+      description: What you ran into, and at which point of the run.
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected
+    attributes:
+      label: What you expected instead
+      description: For a wrong result rather than a crash, say what the right answer looks like and how you know.
+    validations:
+      required: true
+
+  - type: textarea
+    id: command
+    attributes:
+      label: The exact command
+      description: The full `nextflow run` line, including every `--` parameter and `-` option.
+      render: shell
+      placeholder: |
+        nextflow run main.nf --tsv samples.tsv --outdir results_bampiro -profile local,docker
+    validations:
+      required: true
+
+  - type: dropdown
+    id: profile
+    attributes:
+      label: Profile(s) used with -profile
+      multiple: true
+      options:
+        - standard (SLURM, the default)
+        - local
+        - docker
+        - generic (non-tuberculosis organism)
+        - test / test_full
+        - a custom profile or -c config of my own
+    validations:
+      required: true
+
+  - type: dropdown
+    id: engine
+    attributes:
+      label: Container engine
+      options:
+        - Docker
+        - Singularity
+        - Apptainer
+        - None (-stub-run or tools on PATH)
+    validations:
+      required: true
+
+  - type: dropdown
+    id: where
+    attributes:
+      label: Where it runs
+      options:
+        - Laptop or single workstation
+        - SLURM cluster
+        - Another scheduler
+        - Cloud VM
+    validations:
+      required: true
+
+  - type: input
+    id: nextflow-version
+    attributes:
+      label: Nextflow version
+      description: The output of `nextflow -version` (the version line is enough).
+      placeholder: "24.04.2"
+    validations:
+      required: true
+
+  - type: input
+    id: bampiro-version
+    attributes:
+      label: BAMpiro version
+      description: >-
+        The release tag or `git rev-parse --short HEAD` if you cloned. It is also `manifest.version`
+        in `nextflow.config` and appears in the QC report header.
+      placeholder: "1.0.1"
+    validations:
+      required: true
+
+  - type: input
+    id: failing-process
+    attributes:
+      label: Failing process
+      description: The process name Nextflow reports, e.g. `MAP_READS`, `FREEBAYES_CALL`, `QC_REPORT`.
+      placeholder: "CONSENSUS_FASTA (Consensus: T00001)"
+
+  - type: textarea
+    id: command-err
+    attributes:
+      label: .command.err from the task's work directory
+      description: The end of the file is usually the useful part. Trim it if it is huge.
+      render: shell
+
+  - type: textarea
+    id: command-sh
+    attributes:
+      label: .command.sh from the task's work directory
+      description: The resolved script Nextflow actually ran, with your parameters substituted in.
+      render: shell
+
+  - type: textarea
+    id: nextflow-log
+    attributes:
+      label: Nextflow's error block, or .command.log
+      description: >-
+        The `ERROR ~ Error executing process ...` block Nextflow prints, or the relevant part of
+        `.nextflow.log` / the task's `.command.log`.
+      render: shell
+
+  - type: textarea
+    id: samplesheet
+    attributes:
+      label: The samplesheet row(s) involved
+      description: >-
+        Header plus the affected rows. Replace real paths and identifiers if the cohort is not
+        public. Please do not paste patient data.
+      render: text
+
+  - type: checkboxes
+    id: checks
+    attributes:
+      label: Before submitting
+      options:
+        - label: I read [Troubleshooting](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/main/docs/troubleshooting.md) and my problem is not one of the entries there
+          required: true
+        - label: I searched the existing issues
+          required: true
+        - label: I re-ran with `-resume` and it fails the same way

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,11 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Troubleshooting guide
+    url: https://github.com/PathoGenOmics-Lab/BAMpiro/blob/main/docs/troubleshooting.md
+    about: Common first-run errors and their fixes. Check this before filing a bug.
+  - name: Documentation
+    url: https://github.com/PathoGenOmics-Lab/BAMpiro/tree/main/docs
+    about: Installation, quick start, the samplesheet format, the full parameter reference, the QC report and the tutorial series.
+  - name: Questions about running BAMpiro
+    url: https://github.com/PathoGenOmics-Lab/BAMpiro/issues
+    about: Discussions are not enabled on this repository, so please open an issue and label it as a question.

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -0,0 +1,74 @@
+name: Feature request
+description: Suggest a new option, process, output or report panel
+title: "[feature] "
+labels: ["enhancement"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        BAMpiro is a small lab tool, so the useful proposals are the ones tied to an analysis you
+        are actually trying to run. Describe that first; the implementation is the easy part to
+        discuss afterwards.
+
+        If the thing you need is already possible through a parameter, the
+        [configuration reference](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/main/docs/configuration.md)
+        lists every one of them.
+
+  - type: textarea
+    id: problem
+    attributes:
+      label: What are you trying to do
+      description: The analysis or workflow that BAMpiro does not currently support, and why the current behaviour gets in the way.
+    validations:
+      required: true
+
+  - type: textarea
+    id: proposal
+    attributes:
+      label: What you would like it to do
+      description: A new parameter, a new process, an extra output file, another report panel. Be as concrete as you can.
+    validations:
+      required: true
+
+  - type: dropdown
+    id: area
+    attributes:
+      label: Which part of the pipeline
+      multiple: true
+      options:
+        - Samplesheet, parameters or profiles
+        - Reference preparation, repeat and mappability masking
+        - Read QC, trimming, Kraken2 contamination screening
+        - Mapping and deduplication
+        - Variant calling and filtering
+        - Annotation, canonical numbering or the coordinate liftover
+        - Consensus sequences
+        - Pathotypr lineage and drug-resistance typing
+        - The interactive QC report
+        - Documentation
+        - Tests or CI
+        - Something else
+    validations:
+      required: true
+
+  - type: input
+    id: organism
+    attributes:
+      label: Organism and reference
+      description: >-
+        BAMpiro's defaults are tuned for MTBC but it is organism-agnostic, and that often decides
+        whether a feature belongs behind a flag.
+      placeholder: "M. tuberculosis / H37Rv, or Legionella pneumophila with my own reference + GFF"
+
+  - type: textarea
+    id: alternatives
+    attributes:
+      label: What you tried instead
+      description: Existing parameters, a custom `-c` config, or post-processing outside the pipeline, and why that was not enough.
+
+  - type: checkboxes
+    id: contribute
+    attributes:
+      label: Would you like to implement it
+      options:
+        - label: I would be willing to open a pull request for this (see [CONTRIBUTING.md](https://github.com/PathoGenOmics-Lab/BAMpiro/blob/main/CONTRIBUTING.md))

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,22 @@
+## What changed and why
+
+<!-- A couple of sentences. Link the issue if there is one. If this changes a default or an output
+     file, say so explicitly: that is what reviewers most need to notice. -->
+
+## How it was verified
+
+<!-- Paste the result, or say which legs you ran. Note anything you could not run locally, e.g. the
+     stub run without Nextflow installed. -->
+
+```bash
+tests/run_tests.sh
+```
+
+## Checklist
+
+- [ ] `tests/run_tests.sh` passes locally (or the legs I could run do, and I said which)
+- [ ] `ruff check .` is clean, and I did not restyle code the change does not touch
+- [ ] New or changed behaviour has a test, and tests pin what the code does today
+- [ ] A new process in `modules/` has a `stub:` block, and `-profile test_full` reaches it
+- [ ] `tests/data/` was not edited by hand (fixtures come from `make_test_data.py`)
+- [ ] User-visible changes have a `CHANGELOG.md` entry and, where relevant, a docs update

.github/workflows/ci.yml

@@ -0,0 +1,106 @@
+name: CI
+
+# The gate for every change: linting, unit tests for the Python under bin/, tests for the report's
+# hand-written front-end statistics, and a full stub run of the DAG. (The docs build lives in
+# docs.yml, which runs on pull requests too.)
+#
+# Nothing here needs a container, a reference genome or real reads: the fixture cohort under
+# tests/data/ is 170 kB and `-stub-run` executes each process's stub: block instead of the tool.
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+# A new push supersedes the previous run of the same branch. The repository is private on a free
+# plan, so Actions minutes are metered; do not burn them on results nobody will read.
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  lint:
+    name: Lint (ruff)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install ruff
+      - run: ruff check .
+
+  unit:
+    name: Unit tests (Python ${{ matrix.python }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # The oldest version the scripts must still run on, and the one in the container.
+        python: ["3.10", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install pytest numpy
+      - run: python -m pytest tests/unit -q
+
+  fixtures:
+    name: Fixtures are reproducible
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      # The committed cohort must be exactly what the generator produces, or the seed has drifted
+      # and every test that reads tests/data/ is measuring something nobody can regenerate.
+      - name: Regenerate and diff
+        run: |
+          python tests/data/make_test_data.py
+          git diff --exit-code --stat tests/data/
+
+  report-frontend:
+    name: Report front-end (Node)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+      # No dependencies: the tests use node:test and load the report bundle through node:vm.
+      - run: node --test tests/js/*.test.mjs
+
+  pipeline:
+    name: Stub run (Nextflow ${{ matrix.nxf_ver }})
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # The minimum the manifest declares (v1 parser) and the current release (strict v2
+        # parser). Both must pass: main.nf is written to satisfy the stricter of the two.
+        nxf_ver: ["24.04.2", "latest-stable"]
+    env:
+      NXF_ANSI_LOG: "false"
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: "17"
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install pytest
+      - uses: nf-core/setup-nextflow@v2
+        with:
+          version: ${{ matrix.nxf_ver }}
+      - run: python -m pytest tests/pipeline -q
+
+# The docs are validated by docs.yml, which already installs the cairo/pango libraries the
+# social plugin needs and now runs on pull requests too. Nothing to duplicate here.

.github/workflows/docs.yml

@@ -15,7 +15,14 @@ name: docs
 
 on:
   push:
-    branches: [main, indel-mask]
+    branches: [main]
+    paths:
+      - docs/**
+      - mkdocs.yml
+      - .github/workflows/docs.yml
+  # Validate the site on pull requests too, so a broken link fails review rather than main.
+  # Only `build` runs there: `deploy` is gated on a push below.
+  pull_request:
     paths:
       - docs/**
       - mkdocs.yml

.gitignore

@@ -15,6 +15,11 @@ dag*.dot
 results_bampiro/
 mappability_cache/
 
+# Test-suite scratch (tests/run_tests.sh clears these)
+results_test/
+.test_cache/
+.pytest_cache/
+
 # Python
 __pycache__/
 *.pyc