JD2112 · Oct 30, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026 · Oct 13, 2026
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -12,9 +12,27 @@ on:
   workflow_dispatch:
 
 jobs:
+  security_audit:
+    name: 'Security, Secrets & PHI Audit'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Run Pre-Commit Security Scanner
+        run: |
+          chmod +x security_check.sh bin/security_audit.py
+          ./security_check.sh --strict
+
   pytest:
     name: 'Python Unit & Integrity Tests'
     runs-on: ubuntu-latest
+    needs: security_audit
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@v4
@@ -30,13 +48,31 @@ jobs:
           pip install pytest
           pytest tests/unit_tests/ -v
 
+  negative_tests:
+    name: 'Fail-Fast Negative Tests'
+    runs-on: ubuntu-latest
+    needs: security_audit
+    steps:
+      - name: Check out pipeline code
+        uses: actions/checkout@v4
+
+      - name: Set up Nextflow
+        uses: nf-core/setup-nextflow@v2
+        with:
+          version: 'latest'
+
+      - name: Run negative regression tests
+        run: |
+          chmod +x tests/run_negative_tests.sh
+          ./tests/run_negative_tests.sh
+
   pipeline_stub:
     name: 'Nextflow Pipeline Stub Tests'
     runs-on: ubuntu-latest
-    needs: pytest
+    needs: [pytest, negative_tests]
     strategy:
       matrix:
-        profile: ['test_ont', 'test_pacbio']
+        profile: ['test_ont', 'test_pacbio', 'test_giab']
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@v4
@@ -50,3 +86,4 @@ jobs:
         run: |
           nextflow run main.nf -profile ${{ matrix.profile }} -stub
 
+
diff --git a/.gitignore b/.gitignore
@@ -44,8 +44,25 @@ db/
 *.tar.gz
 *.fa.fai
 *.fasta.fai
+!tests/data/**/*.fai
 notes.md
 debugging.md
 quindecagon_test.md
 UPDATES.md
-TODO.md
+TODO.md
+review/
+
+# Credentials, Tokens & Security Secrets
+*.pem
+*.key
+*.p12
+*.pfx
+*.pkcs12
+id_rsa*
+id_ed25519*
+.env
+.env.*
+!.env.example
+credentials*
+*.secret
+ROADMAP*
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,16 @@
+# .pre-commit-config.yaml
+# LongVarMeth Git Pre-Commit Hook Configuration
+# Automatically runs security, credential, and PHI audit before every commit.
+# Install:
+#     pip install pre-commit && pre-commit install
+
+repos:
+  - repo: local
+    hooks:
+      - id: security-audit
+        name: LongVarMeth Security, Secret & PHI Audit
+        entry: ./security_check.sh --staged-only --strict
+        language: script
+        pass_filenames: false
+        always_run: true
+
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [0.4.0] - 2026-10-30
+
+### Added
+- **Modular SNV/Indel Caller Dispatch Matrix**:
+  - Added **`modules/nf-core/deepvariant`** supporting platform-specific neural models (`PACBIO` and `ONT_R104`).
+  - Added **`modules/nf-core/clair3`** supporting long-read neural models for ONT and PacBio HiFi.
+  - Upgraded `subworkflows/local/variant_calling/main.nf` with dynamic caller selection via `--snv_caller` (`bcftools`, `deepvariant`, `clair3`).
+- **Epigenomics Specialization**:
+  - Added **`modules/nf-core/pbcpgtools`** for native PacBio HiFi CpG aggregation (`aligned_bam_to_cpg_scores`).
+  - Implemented platform-split epigenomics in `subworkflows/local/methylation/main.nf`: native `pbcpgtools` for PacBio HiFi vs `modkit pileup` for Oxford Nanopore.
+  - Added **`CHECK_MOD_TAGS`** validation process ensuring SAM auxiliary kinetic tags (`MM`/`ML`) are verified prior to pileup.
+- **Joint Multi-Variant Co-Phasing**:
+  - Upgraded **`modules/nf-core/hiphase`** to jointly co-phase small variants (SNVs/indels), structural variants (`Sniffles2`), and tandem repeat expansions (`TRGT`) into unified phase sets (`PS`).
+  - Added **`modules/nf-core/whatshap/stats`** for phase block metric calculation and N50 reporting.
+- **Empirical GIAB HG002 Truth-Set Benchmarking**:
+  - Added **`modules/nf-core/happy`** for small variant (SNV/Indel) concordance evaluation against NIST GIAB truth sets.
+  - Added **`modules/nf-core/truvari`** for structural variant ($\ge 50$ bp) precision and recall benchmarking against GIAB Tier 1 benchmarks.
+  - Created **`subworkflows/local/benchmarking/main.nf`** integrated into all pipeline entrypoints.
+  - Added dedicated benchmark profile **`-profile test_giab`** (`conf/tests/test_giab.config`) and mock truth assets in `tests/data/giab/`.
+  - Authored comprehensive validation guide and baseline expectations in **`docs/validation.md`**.
+- **Transparent MultiQC Analytical Stage Execution Matrix**:
+  - Implemented **`modules/local/execution_status_matrix/main.nf`** generating `pipeline_execution_status_mqc.tsv`.
+  - Wired into MultiQC (`assets/multiqc_config.yml`) to transparently report the status (`COMPLETED`, `SKIPPED (Pre-aligned Input)`, `SKIPPED (Parameter-Disabled)`, `FAILED`) across all 12 pipeline analytical stages.
+- **Automated Security, Secret & PHI Auditing**:
+  - Added **`bin/security_audit.py`** and root-level runner **`security_check.sh`** for zero-dependency scanning of credentials, tokens, cloud keys, private keys, local user paths, and PHI.
+  - Added **`.pre-commit-config.yaml`** for local git pre-commit hook integration.
+  - Integrated fail-fast security gate into **`.github/workflows/ci.yml`** to block pull requests and commits containing sensitive data.
+- **Automated Negative Test Regression Suite**:
+  - Added **`tests/run_negative_tests.sh`** and **`tests/pipeline/negative_tests.nf.test`** testing fail-fast exit codes (exit 1) on missing references, platform mismatches, and invalid samplesheets.
+- **On-Demand Auto-Indexing**:
+  - Added **`modules/nf-core/samtools/index`** and **`modules/nf-core/samtools/faidx`** for automatic indexing of pre-aligned BAMs and reference FASTA files.
+
+### Changed
+- **Regulatory Scope & Intended-Use Reframing**:
+  - Re-scoped `README.md` and `docs/validation.md` strictly to **Research Use Only (RUO) / Clinical Evaluation Platform**, clarifying that clinical diagnostic deployment requires site-specific analytical validation (e.g. CAP/CLIA or ISO 15189).
+  - Documented physical chemistry boundaries (Oxford Nanopore simplex homopolymer indel resolution, 5mC vs 5hmC depth requirements, PacBio HiFi coverage and kinetic tags, phase block continuity limits).
+- **Native Streaming Tag Preservation**:
+  - Deprecated in-memory dictionary-based `transfer_mod_tags.py`, replacing it with native streaming alignment (`minimap2 -ax map-ont -y` and `pbmm2`), guaranteeing coordinate-accurate reverse-strand `MM`/`ML` tag propagation with minimal memory overhead.
+- **Full-Chain PHI De-Identification & Forensic Audit Manifest**:
+  - Upgraded `subworkflows/local/governance`: Dedicated `SANITIZE_BAM` and `SANITIZE_VCF` generating deterministic `anon_id`, stripping patient identifiers, hospital run barcodes, and tool command lines.
+  - Re-routed downstream `VISUALIZATION` (JBrowse 2, Methylartist, Dashboard) and `REPORTING` across all workflows to consume exclusively sanitized artifacts.
+  - Expanded `manifest.json` into a cryptographic full-chain provenance record binding raw input SHAs, reference FASTA checksums, workflow commit ID, container digests, execution parameters, and output checksums.
+  - Pinned container image digests in `conf/version_lock.config`.
+
+### Fixed
+- **Groovy Boolean Channel Evaluation Bug**:
+  - Replaced Groovy boolean channel checks (`if (channel)`) with idiomatic Nextflow DSL2 `.branch { ... }` operators in `nanopore.nf`, `pacbio.nf`, and `longvarmeth.nf`.
+  - Fixed dropped FASTQ inputs and prevented pre-aligned BAMs from incorrectly entering `MERGE_BAMS`.
+- **Silent Parameter Bypass on Missing Reference**:
+  - Updated `lib/Utils.groovy` with fatal fail-fast validation: immediately terminates execution (`exit 1`) with an explicit error banner when `--run_type reference_based` lacks a valid `--ref_nuc`.
+- **Platform & Chemistry Cross-Validation**:
+  - Added strict platform validation in `lib/Utils.groovy`, instantly rejecting mismatched platform inputs (e.g. PacBio BAMs passed to `--platform ont`).
+  - Added duplicate sample ID detection and required column header validation.
+- **Channel Closure Arity**:
+  - Resolved variable tuple arity exceptions in multi-channel joins within `phasing/main.nf` and `governance/main.nf`.
+
+### Security
+- Sanitized remaining host-specific filesystem paths (`/data/Jyotirmoy/...`) from `conf/fraka.config`, `conf/human.config`, `conf/eukaryote.config`, and template `samplesheet.csv`.
+- Hardened `.gitignore` to strictly ignore private keys (`*.key`, `*.pem`, `*.p12`, `*.pfx`, `id_rsa*`, `id_ed25519*`), environment files (`.env*`), and sensitive credentials.
+- Added automated fail-fast security scanning in CI (`.github/workflows/ci.yml`).
+
+---
+
 ## [0.3.0] - 2026-10-12
 
 ### Added

diff --git a/README.md b/README.md
@@ -7,7 +7,11 @@
 [![code style](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/JD2112/longvarmeth/badges/black.json)](https://github.com/psf/black)
 [![flake8](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/JD2112/longvarmeth/badges/flake8.json)](https://flake8.pycqa.org)
 
-A modular, clinical-grade Nextflow DSL2 platform for single-molecule long-read genomics and epigenomics (**Oxford Nanopore & PacBio HiFi**). It features an explicit 12-stage analytical architecture with host GPU-accelerated basecalling/demux, pre- and post-alignment QC, *de novo* assembly & polishing, kinetic tag preservation (`MM`/`ML`), small and structural variant calling (SNV/InDel/SV/CNV/STR), haplotype co-phasing, strict clinical governance & PHI sanitization (inspired by `milou`), and interactive reporting.
+A modular Nextflow DSL2 evaluation and research platform with forensic clinical governance controls for single-molecule long-read genomics and epigenomics (**Oxford Nanopore & PacBio HiFi**). It features an explicit 12-stage analytical architecture with host GPU-accelerated basecalling/demux, pre- and post-alignment QC, *de novo* assembly & polishing, native kinetic tag streaming (`MM`/`ML`), modular small variant calling (`DeepVariant`, `Clair3`, `BCFtools`), structural variant calling (`Sniffles2`, `TRGT`, `CNVkit`), joint multi-variant haplotype co-phasing (`HiPhase`, `WhatsHap`), automated truth-set benchmarking (`hap.py`, `Truvari`), full-chain PHI de-identification and SHA-256 provenance manifests (inspired by `milou`), and a transparent MultiQC execution status matrix.
+
+> [!IMPORTANT]
+> **Intended Use & Regulatory Statement**:  
+> LongVarMeth is designed for **Research Use Only (RUO)** and clinical evaluation. It is not an FDA-cleared or CE-IVD diagnostic device. While the architecture enforces strict clinical governance mechanisms (PHI scrub, deterministic anonymization, cryptographic audit trail), clinical laboratories must establish analytical validity independently in compliance with local regulatory standards (CAP/CLIA, ISO 15189) before reporting clinical findings. See [docs/validation.md](docs/validation.md).
 
 ## Pipeline Architecture
 
@@ -18,19 +22,18 @@ A modular, clinical-grade Nextflow DSL2 platform for single-molecule long-read g
 
 | Stage | Oxford Nanopore <br> (`--platform ont`) | PacBio HiFi <br> (`--platform pacbio`) | Inputs $\rightarrow$ Outputs |
 | :--- | :--- | :--- | :--- |
-| **Input Checksum** | Ingestion SHA-256 verification | Ingestion SHA-256 verification | Raw files $\rightarrow$<br>Audit manifest |
-| **Read QC** | `FastQC` + `NanoPlot`<br>`Cramino` + `samtools` | `Cramino` + `FastQC`<br>`NanoPlot` | Reads / BAM $\rightarrow$<br>Yield & Q-scores |
-| **Basecalling**<br>*(ONT raw)* | Host GPU `Dorado`<br>+ `pod5 convert` demux | *On-instrument demux*<br>(Revio / Sequel IIe) | `pod5` / `fast5` $\rightarrow$<br>Basecalled BAM |
-| **Assembly**<br>*(de novo)* | `Flye --nano-hq`<br>+ `Medaka` polishing | `Hifiasm` (diploid)<br>+ `Racon` polishing | Long reads $\rightarrow$<br>Assembly FASTA + BUSCO |
-| **Alignment** | `minimap2 -ax map-ont`<br>+ `MM`/`ML` tag restoration | `pbmm2 align --preset HIFI`<br>*(native kinetic preservation)* | Reads + Ref $\rightarrow$<br>Sorted BAM (`MM`/`ML`) |
-| **Small Variants** | `Clair3` (ONT model)<br>`DeepVariant` / `BCFtools` | `DeepVariant` (HiFi model)<br>`Clair3` / `BCFtools` | Sorted BAM $\rightarrow$<br>SNV / InDel VCF |
-| **SVs, CNVs & STRs** | `Sniffles2` (SVs)<br>`CNVkit` (CNVs) | `Sniffles2` (SVs) + `HiFiCNV`<br>`TRGT` (tandem repeats) | Sorted BAM $\rightarrow$<br>SV, CNV & STR calls |
-| **Epigenomics** | `modkit pileup`<br>(5mC / 5hmC / 6mA motifs) | `pbcpgtools` + `methbat`<br>+ `modkit pileup` | Sorted BAM $\rightarrow$<br>BedMethyl & CpG scores |
-| **Phasing** | `WhatsHap phase`<br>+ `haplotag` (`HP:i:1/2`) | `HiPhase`<br>*(co-phases SNV + SV + STR)* | VCFs + BAM $\rightarrow$<br>Phased VCF & Haplotagged BAM |
-| **Annotation** | `Ensembl VEP`<br>`SnpEff` + ClinVar | `Ensembl VEP`<br>`SnpEff` + ClinVar | Phased VCF $\rightarrow$<br>Annotated clinical VCF |
-| **Governance** | `SANITIZE_METADATA` (PHI scrub)<br>+ cryptographic SHA-256 manifest | `SANITIZE_METADATA` (PHI scrub)<br>+ cryptographic SHA-256 manifest | Artifacts $\rightarrow$<br>Sanitized BAM & manifest |
-| **Visualization** | `JBrowse 2` web workspace<br>`Methylartist` locus plots<br>Interactive Methylation Dashboard | `JBrowse 2` web workspace<br>`Methylartist` locus plots<br>Interactive Methylation Dashboard | BAMs, VCFs $\rightarrow$<br>Web workspace & HTML reports |
-| **Reporting** | `MultiQC` consolidated dashboard<br>+ Executive clinical summary | `MultiQC` consolidated dashboard<br>+ Executive clinical summary | Metrics $\rightarrow$<br>MultiQC HTML report |
+| **1. Input Checksum** | Ingestion SHA-256 verification | Ingestion SHA-256 verification | Raw files $\rightarrow$ Audit manifest |
+| **2. Basecalling & Demux**<br>*(ONT raw)* | Host GPU `Dorado`<br>+ `pod5 convert` demux | *On-instrument demux*<br>(Revio / Sequel IIe) | `pod5` / `fast5` $\rightarrow$ Basecalled BAM |
+| **3. Read QC & Trimming** | `FastQC` + `NanoPlot`<br>`Cramino` + `Filtlong` | `Cramino` + `FastQC`<br>`NanoPlot` | Reads / BAM $\rightarrow$ Yield & Q-scores |
+| **4. Alignment & Tag Streaming** | `minimap2 -ax map-ont -y`<br>*(streaming MM/ML tag preservation)* | `pbmm2 align --preset HIFI`<br>*(native kinetic preservation)* | Reads + Ref $\rightarrow$ Sorted BAM (`MM`/`ML`) |
+| **5. Mod Tag Verification** | `CHECK_MOD_TAGS`<br>*(samtools view aux check)* | `CHECK_MOD_TAGS`<br>*(kinetics validation)* | Sorted BAM $\rightarrow$ Kinetic Tag Audit Log |
+| **6. Small Variants** | `DeepVariant` (`ONT_R104`)<br>`Clair3` / `BCFtools` | `DeepVariant` (`PACBIO`)<br>`Clair3` / `BCFtools` | Sorted BAM $\rightarrow$ SNV / InDel VCF |
+| **7. SVs, CNVs & STRs** | `Sniffles2` (SVs)<br>`CNVkit` (CNVs) | `Sniffles2` (SVs) + `HiFiCNV`<br>`TRGT` (tandem repeats) | Sorted BAM $\rightarrow$ SV, CNV & STR calls |
+| **8. Epigenomics** | `modkit pileup`<br>(5mC / 5hmC / 6mA motifs) | `pbcpgtools` (`aligned_bam_to_cpg_scores`)<br>+ `methbat` | Sorted BAM $\rightarrow$ BedMethyl & CpG scores |
+| **9. Joint Co-Phasing** | `WhatsHap phase`<br>+ `haplotag` (`HP:i:1/2`) | `HiPhase`<br>*(joint SNV + SV + STR co-phasing)* | VCFs + BAM $\rightarrow$ Phased VCF & Haplotagged BAM |
+| **10. Truth-Set Benchmark** | `hap.py` (SNVs/Indels)<br>+ `Truvari` (SVs) | `hap.py` (SNVs/Indels)<br>+ `Truvari` (SVs) | VCFs + GIAB HG002 $\rightarrow$ Precision/Recall/F1 TSVs |
+| **11. Clinical Governance** | `SANITIZE_BAM` + `SANITIZE_VCF`<br>+ cryptographic SHA-256 manifest | `SANITIZE_BAM` + `SANITIZE_VCF`<br>+ cryptographic SHA-256 manifest | Artifacts $\rightarrow$ Sanitized BAM/VCF & manifest.json |
+| **12. Visualization & Reporting**| `JBrowse 2` + `Methylartist`<br>`MultiQC` Execution Status Matrix | `JBrowse 2` + `Methylation Dashboard`<br>`MultiQC` Execution Status Matrix | BAMs, VCFs $\rightarrow$ MultiQC Report & Tracks |
 
 ### Multi-Organism Tailoring Matrix
 
@@ -111,7 +114,47 @@ You can configure execution parameters from the command line or via custom confi
 | `--enable_vep` | boolean | `false` | Enable Ensembl VEP functional variant annotation |
 | `--enable_snpeff` | boolean | `false` | Enable SnpEff variant annotation |
 | `--enable_methylation_dashboard` | boolean | `false` (auto in `human`) | Generate interactive AG-Grid & IGV-Web clinical methylation report (adapted from `gms-nallo`) |
-| `--dorado_bin` | string | `/path/to/dorado`| Host path to Dorado binary for host GPU-accelerated basecalling |
+| `--run_benchmarking` | boolean | `false` | Enable empirical truth-set concordance evaluation (`hap.py` + `Truvari`) |
+| `--benchmark_truth_vcf` | string | `""` | Path to NIST GIAB HG002 small variant truth VCF |
+| `--benchmark_truth_bed` | string | `""` | Path to NIST GIAB HG002 small variant benchmark region BED |
+| `--benchmark_sv_truth_vcf` | string | `""` | Path to NIST GIAB HG002 structural variant truth VCF |
+| `--benchmark_sv_truth_bed` | string | `""` | Path to NIST GIAB HG002 structural variant benchmark region BED |
+| `--dorado_bin` | string | `dorado` | Host path to Dorado binary for host GPU-accelerated basecalling |
+
+---
+
+## Biological Scope & Technology Limitations
+
+Understanding the physical and algorithmic boundaries of single-molecule sequencing is critical for valid result interpretation:
+
+1. **Oxford Nanopore R10.4.1 Chemistry**:
+   - **Simplex vs Duplex**: Simplex R10.4.1 reads achieve modal Q20+ raw accuracy. However, small insertion/deletion (indel) errors persist in homopolymer tracts ($N \ge 6$) due to ionic current dwell-time saturation. Indel findings in homopolymers should be interpreted conservatively.
+   - **Base Modifications**: Oxford Nanopore chemistry natively distinguishes 5mC, 5hmC, and unmethylated cytosines using `modkit pileup`. High-confidence quantification requires $\ge 15\times$ read depth per CpG site.
+2. **PacBio HiFi Chemistry (Revio / Sequel IIe)**:
+   - **Consensus Accuracy**: HiFi circular consensus sequencing achieves Q30+ base quality, yielding exceptional SNV and indel accuracy ($F_1 > 0.99$).
+   - **Kinetics**: 5mC detection relies on inter-pulse duration (IPD) kinetics aggregated via `pbcpgtools` (`aligned_bam_to_cpg_scores`). BAM inputs must preserve kinetics tags (`MM`, `ML` or pulse tags).
+3. **Joint Haplotype Co-Phasing (`HiPhase` / `WhatsHap`)**:
+   - **Phase Block Continuity**: Long-range phase blocks depend on read length N50 and sample heterozygosity. In extended runs of homozygosity (ROH) or low-complexity segmental duplications, phase blocks break deterministically.
+   - **Multi-Variant Resolution**: HiPhase jointly co-phases small variants, SVs (`Sniffles2`), and STR repeat expansions (`TRGT`) into unified phase sets (`PS`).
+
+---
+
+## Truth-Set Benchmarking & MultiQC Execution Matrix
+
+LongVarMeth introduces automated truth-set benchmarking using the NIST / GIAB HG002 reference standard:
+
+```bash
+# Run automated GIAB benchmark check in stub mode:
+nextflow run main.nf -profile test_giab -stub
+```
+
+The resulting `multiqc_report.html` features:
+- **Pipeline Execution Status Matrix**: Explicit table classifying each of the 12 pipeline stages as `COMPLETED`, `SKIPPED (Pre-aligned Input)`, `SKIPPED (Parameter-Disabled)`, or `FAILED`, ensuring incomplete runs are never mistaken for completed analyses.
+- **Hap.py Concordance Table**: Precision, recall, and F1 scores stratified by variant class.
+- **Truvari SV Benchmark Table**: TP, FP, FN, recall, and F1 scores for structural variants $\ge 50$ bp.
+- **Cryptographic Audit Manifest**: Full-chain SHA-256 provenance tracking.
+
+For full validation procedures and benchmark calibrations, see **[docs/validation.md](docs/validation.md)**.