From b55908fa58629cf061874e301dc2d985c6b46f01 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 12:24:46 +0200 Subject: [PATCH 01/50] docs(design): add design document for graph merge command --- dev/design/merge.md | 423 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 423 insertions(+) create mode 100644 dev/design/merge.md diff --git a/dev/design/merge.md b/dev/design/merge.md new file mode 100644 index 00000000..990c879a --- /dev/null +++ b/dev/design/merge.md @@ -0,0 +1,423 @@ +# Graph merging — design manifesto + +Status: **design accepted, implementation not started** +Integration branch: `feat/merge` (merged into `master` last, after all phase branches below) + +This document describes the design for a new `pangraph merge` command, which combines two +pre-existing pangenome graphs into one. It covers the motivation, the identifier model, the +required changes to existing code, and the phased roadmap. + +An accompanying `dev/design/merge-implementation-notes.md` will record as-built decisions and +gotchas as the phases land; this file describes the *intent*. + +--- + +## 1. Motivation + +Today `pangraph build` can only construct a graph from scratch, out of a set of FASTA sequences. +There is no way to add new genomes to an existing graph: the only option is to rebuild everything +from the full sequence set. + +The proposed workflow for extending a graph is a two-step composition of existing pieces: + +```bash +# 1. turn the additional genomes into their own (possibly singleton) graph +pangraph build new_genomes.fa -o new.json + +# 2. merge it into the existing graph +pangraph merge base.json new.json -o extended.json +``` + +Step 2 is exactly the operation that already happens at every internal node of the guide tree +during `build`. The command is a thin CLI wrapper around it. + +### What this buys us + +- Adding genomes to a large graph without redoing the whole build. +- Merging graphs built from different sources or with different parameters. +- A directly testable, user-visible entry point into the merge machinery, which is currently only + reachable through the full build pipeline (or the undocumented `bin/merge_two_graphs.rs`). + +### What it does not buy us + +`merge(build(A), build(B))` is **not** expected to be identical to `build(A ∪ B)`. The guide tree +and therefore the order in which homologous regions are discovered differ, so the resulting block +decompositions will differ in detail. Both are valid pangenome graphs over the same sequences. + +--- + +## 2. Current state + +### 2.1 What already exists + +The core operation is complete and generic. `merge_graphs(left, right, args)` +(`packages/pangraph/src/pangraph/graph_merging.rs:26`) joins two graphs, iteratively self-merges +homologous blocks until convergence, and removes transitive edges. It makes **no assumption** that +its inputs are singletons or that they come from the same build. `bin/merge_two_graphs.rs` is a +development binary that already tries to call it directly. + +Distance computation also already works on arbitrary graphs: `mash_distance` takes `&[Pangraph]` +and sketches all block consensuses (`distance/mash/mash_distance.rs:9`). + +So this is primarily a plumbing and identifier problem, not an algorithmic one. + +### 2.2 What is missing + +1. **Identifier collisions** between independently built graphs (§3). This is the hard blocker. +2. `build` **panics on a single input sequence**, so the "wrap my new genome in a singleton graph" + half of the workflow does not work (§4.1). +3. The merge parameters are entangled with `PangraphBuildArgs`, which carries build-only fields + (§4.2). +4. **Path names are not guaranteed unique**, although the rest of the codebase treats them as the + de-facto identity of a genome (§4.3). +5. Sequence verification is keyed on input order / FASTA index, which is meaningless for a merged + graph (§4.4). +6. There is no `merge` subcommand (§4.5). + +--- + +## 3. The identifier model + +### 3.1 The problem + +`Pangraph::singleton` (`pangraph/pangraph.rs:29`) assigns + +```rust +let node_id = NodeId(fasta.index); +let block_id = BlockId(fasta.index); +let path_id = PathId(fasta.index); +``` + +i.e. small sequential integers taken from the position of the record in the input FASTA. Every +later identifier is content-derived, **but only for entities that are actually touched by a +merge**. Blocks that never find a homologous partner keep their original small integer id all the +way to the final graph. + +Measured on two graphs built independently from disjoint subsets of `data/ges-1.fa` (6 + 6 +sequences): + +``` +paths A: 6 B: 6 overlap: 6 → {0,1,2,3,4,5} +blocks A: 43 B: 33 overlap: 0 +nodes A: 94 B: 85 overlap: 0 +``` + +and on a graph built from two unrelated sequences (influenza + mpox, no homology found at all): + +``` +blocks: {0, 1} nodes: {0, 1} paths: {0, 1} +``` + +So: **path ids always collide**, and **block and node ids collide whenever a block never merged** — +which covers every singleton graph, every accessory block, and every divergent genome. +`graph_join` (`graph_merging.rs:74`) panics on any duplicate key, so `merge` fails immediately +today. + +### 3.2 Two properties of the current model + +**Identifiers are labels, not content hashes.** `circularize::merge_blocks` deliberately *reuses* +an existing `BlockId` for a newly concatenated block (`circularize/merge_blocks.rs:110-142`). +Nothing in the codebase relies on a block id being a hash of its content, so we are free to +relabel entities as long as all references are rewritten consistently. + +**Identifiers are re-hashed during the merge**, and those hashes consume the ids we relabel: + +| Site | New id derived from | +|---|---| +| `pangraph_node.rs:45` | `(block_id, path_id, strand, position)` | +| `reweave.rs:132` | `(qry.name, qry.interval, reff.name, reff.interval)` | +| `pangraph_interval.rs:129` | `(block_id, interval)` | +| `detach_unaligned.rs:99` | `(node_id, seq)` | +| `pangraph_path.rs:44` | `(nodes, tot_len, circular, desc, name)` | + +This is safe **provided the inputs are made disjoint first**. It also means a half-measure — for +instance deduplicating keys only inside `graph_join` — would not work: colliding inputs would +produce colliding *derived* ids later in the same merge. + +### 3.3 `PathId` is semantically load-bearing + +Unlike the other two, `PathId` currently carries meaning beyond identity: `reconstruct` sorts paths +by id and uses `path_id.0` as the output `FastaRecord.index` +(`commands/reconstruct/reconstruct_run.rs:56-76`), which `build --verify` then uses to index back +into the input FASTA records (`commands/build/build_run.rs:44`). For a graph produced by `build`, +`PathId` == input FASTA index, so reconstruction order == input order. + +That correspondence cannot survive a merge (§4.4), but path ids should still be small, contiguous +and ordered rather than becoming arbitrary hashes. + +### 3.4 Decision: relabel everything after the first graph + +Before joining, the right-hand graph is relabeled so that its identifier space is disjoint from +the left-hand one. The **left graph is never modified**. + +```rust +/// Re-derives every block, node and path id of the graph. Block and node ids become +/// `id((salt, old_id))`; path ids are renumbered contiguously from `path_id_offset`, +/// preserving their relative order. Consensuses, edits, names, descriptions, strands +/// and positions are unchanged. +fn relabel_in_place(&mut self, salt: usize, path_id_offset: usize) + +/// Makes `right` id-disjoint from `left`. `left` is left untouched. +pub fn make_disjoint(left: &Pangraph, right: &mut Pangraph) -> Result<(), Report> +``` + +Rationale for relabeling *all* of the right graph's block and node ids, rather than only the ones +that actually clash: it is uniform, trivial to reason about, and trivial to test. The alternative +(remap-on-collision) would preserve more identifiers across an append, but block ids of an appended +graph are of limited interest to users compared with those of the base graph — and those are +preserved either way. + +**Consequences, to be documented in the user docs:** + +- Put the **base graph first**. Its block, node and path ids survive the merge unchanged (except + where blocks are genuinely reweaved); the second graph's ids always change. +- `merge` is therefore *not* symmetric in identifiers, even though it is essentially symmetric in + content. + +### 3.5 Implementation notes + +- **Relabel in place.** Block consensuses are the bulk of a graph's memory; building a relabeled + copy would double peak usage on a large base graph. Use `std::mem::take` on the three maps and + rebuild them in place. +- No new constructors are needed: `PangraphBlock::new`, `PangraphNode::new(Some(id), …)` and + `PangraphPath::new(Some(id), …)` all accept an explicit id. +- References to rewrite: the three map keys, `PangraphBlock::alignments` keys, + `PangraphNode::{block_id, path_id}`, and `PangraphPath::nodes`. +- **Assert injectivity.** After relabeling, the three maps must have the same lengths as before (a + hash collision inside one graph would silently drop entities) and the resulting id sets must be + disjoint from the left graph's. `make_disjoint` bumps the salt and retries if not; this is + astronomically unlikely with `XxHash64`, but the check is cheap. +- Path ids are assigned as `path_id_offset + rank`, where `rank` is the position in the graph's + existing (sorted) path key order, and `path_id_offset` is `max(left path ids) + 1`. This keeps + ids contiguous and keeps the right graph's genomes in their original relative order, after all of + the left graph's. +- Relabeling is deterministic: `utils::id::id` uses `XxHash64` with a fixed seed. + +--- + +## 4. Required changes to existing code + +### 4.1 `build` must accept a single sequence + +``` +$ pangraph build one.fa -o one.json +The application panicked: index out of bounds: the len is 1 but the index is 1 +Location: packages/pangraph/src/tree/neighbor_joining.rs:28 +``` + +`build_tree_using_neighbor_joining` (`tree/neighbor_joining.rs:16-33`) skips the +`while nodes.len() > 2` loop for a single graph and then indexes `nodes[1]`. An empty input panics +on `nodes[0]`. + +Fix: return the lone leaf clade when `nodes.len() == 1`, and a clean error for an empty input. The +rest of the pipeline already tolerates a leaf-only tree — `postorder` no-ops on leaves, the root's +`data.take()` returns the graph, and `ProgressBar::new` already guards `n_total <= 1`. Worth a +companion test for `build_tree_from_newick` on a single-leaf Newick string, which shares the shape. + +This is a prerequisite for the whole workflow: appending a single genome requires building a +one-sequence graph. + +### 4.2 Extract `GraphMergeParams` from `PangraphBuildArgs` + +`merge_graphs` takes `&PangraphBuildArgs`, and that type is threaded all the way down into +`MergePromise::solve_promise` (`reweave.rs:40`), `reconsensus_graph` (`reconsensus/reconsensus.rs:35`), +`edit_consensus_and_realign` (`pangraph_block.rs:295`) and `map_variations` +(`align/map_variations.rs:43`). Only five fields are ever used; the rest (`input_fastas`, +`output_json`, `circular`, `verify`, `guide_tree`, `no_progress_bar`) are build-only and meaningless +for `merge`. + +```rust +pub struct GraphMergeParams { + pub aln_args: AlignmentArgs, // #[clap(flatten)] + pub alignment_kernel: AlignmentBackend, + pub max_self_map: usize, // default 100 + pub extra_band_width: usize, // default 5 + pub max_alignment_attempts: usize, // default 4 +} +``` + +`PangraphBuildArgs` and `PangraphMergeArgs` both `#[clap(flatten)]` it, so the two commands expose +an identical set of alignment options. + +Place it next to `AlignmentArgs` in `align/`, and move `AlignmentBackend` there as well. Side +benefit: this removes the current `align → commands::build` dependency +(`map_variations.rs:3`, `reconsensus.rs:1`, `pangraph_block.rs:2`), which is backwards. + +Mechanical follow-ups: the clap `default_value_t = PangraphBuildArgs::default().alignment_kernel` +becomes `GraphMergeParams::default()…`, and tests constructing `PangraphBuildArgs::default()` purely +to reach alignment parameters switch to `GraphMergeParams::default()`. Also generalize +`build_cmd_preliminary_checks` (the "is `mmseqs` on PATH" check) to take `&GraphMergeParams`, so +`merge` inherits it. + +### 4.3 Path names must be unique + +Path names are already treated as the identity of a genome throughout the codebase: +`Pangraph::path_id_by_name` (`pangraph.rs:258`), `simplify` (which unwraps `path.name()` outright, +`simplify_run.rs:26`), and the export commands. Nothing enforces uniqueness, however — +`build_run.rs:72` still carries a `// TODO: check for duplicate fasta names`. The only place that +errors today is `build_tree_from_newick` (`tree/newick.rs:88`), i.e. only when `--guide-tree` is +used. + +Decision: **duplicate path names are a hard error, in both `build` and `merge`.** + +- `build`: after reading the input FASTA records, error if two records share a name. This makes the + existing `--guide-tree` behaviour unconditional and fills the TODO. +- `merge`: error if a name occurs more than once within either input graph or across the two. + This is what catches the common mistakes of merging a graph with itself, or re-adding a genome + that is already present in the base graph — which would otherwise silently produce two + identically named paths and break `export` and `simplify` downstream. + +Error messages should list the offending names, not just the first one. + +Paths with no name at all are left alone by the duplicate check (multiple unnamed paths are not an +error), but `--verify` requires every path to be named, because names are the verification key +(§4.4). + +### 4.4 Reconstruction and verification must be name-keyed + +`reconstruct` emits one FASTA record per path, sorted by `PathId`, with `index = path_id.0` +(`reconstruct_run.rs:56-76`). For a graph produced by `build`, `PathId` is the input FASTA index, so +the output order reproduces the input order exactly and `index` is meaningful. + +**Neither property can hold for a merged graph.** The right graph's paths are renumbered (§3.4), so +while the output stays deterministic (still sorted by path id), its order corresponds to no +original input FASTA ordering, and `index` is merely a position within the merged graph. +Reconstruction itself remains fully supported — every sequence comes back byte-identical — but +*order is not guaranteed*, and neither order nor `index` may be used to pair records up. + +This has three consequences: + +1. **`compare_sequences` must stop comparing whole records.** It currently tests + `left != right` on `FastaRecord` (`reconstruct_run.rs:45-54`), and `FastaRecord` derives + `PartialEq` over all fields *including `index`* (`io/fasta.rs:17-24`) — despite an error message + that only mentions length. It should compare sequence contents and report the path name. + +2. **Verification is keyed by path name, not by index or position.** + + ```rust + fn verify_graph_sequences(graph: &Pangraph, expected: &BTreeMap) -> Result<(), Report> + ``` + + `build` fills the map from the input FASTA records; `merge` fills it by reconstructing each + input graph *before* merging. Both are now sound because names are unique (§4.3). The map form + also keeps working for the intermediate graphs checked inside the build loop, which contain only + a subset of the paths. + +3. **`pangraph reconstruct` documentation must state** that record order matches the original input + FASTA order only for graphs produced directly by `build`, and that consumers should match records + by name. + +### 4.5 The `merge` command + +New module `commands/merge/{mod.rs, merge_args.rs, merge_run.rs}`, dispatched from +`commands/root_args.rs:61` and `commands/main.rs:19`. + +``` +pangraph merge -o +``` + +`PangraphMergeArgs`: two positional graph paths, `-o/--output-json` (default `-`), +`-f/--verify`, and a flattened `GraphMergeParams`. + +`merge_run` flow: + +1. Load both graphs (`Pangraph::from_path`); `sanity_check` each under `debug_assertions`. +2. Preliminary checks: both graphs non-empty; the `mmseqs`-on-PATH check when that kernel is + selected; duplicate path names within and across the inputs (§4.3); a **warning** if the two + graphs disagree on circularity (`circular` is per-path, so mixing is structurally fine, but + `build -c` is global and easy to get wrong for the new-genomes graph). +3. `make_disjoint(&left, &mut right)` (§3.4). +4. If `--verify`: reconstruct the expected sequences from both inputs into a + `BTreeMap` keyed by path name (§4.4). +5. `merge_graphs(&left, &right, &args.merge_params)`. +6. If `--verify`: `verify_graph_sequences(&merged, &expected)`. +7. Write the output JSON. + +`bin/merge_two_graphs.rs` is superseded by this command — it cannot work today anyway, since it +calls `graph_join` on colliding ids — and should be removed in the same phase. + +Exactly two input graphs are supported. See §7 for why N-way is deferred. + +--- + +## 5. Cost model + +`self_merge` runs an all-vs-all alignment over *every* block consensus in the joined graph on each +round. Appending 5 genomes to a 500-genome graph therefore costs roughly what the root merge of a +505-genome build costs: it is **not** proportional to the number of new genomes. + +This is inherent to the existing algorithm and is no worse than what `build` already does at the +root of the guide tree, but users will reasonably assume that appending is cheap. It must be stated +explicitly in the user documentation. + +--- + +## 6. Phased roadmap + +Each phase is its own branch and PR, targeting the integration branch `feat/merge`. `feat/merge` is +merged into `master` last, once all phases have landed. + +| # | Branch | Scope | Depends on | +|---|---|---|---| +| 1 | `feat/merge-single-seq-build` | §4.1 — NJ tree for 1 and 0 input graphs | — | +| 2 | `feat/merge-params` | §4.2 — extract `GraphMergeParams`, no behaviour change | — | +| 3 | `feat/merge-unique-names` | §4.3 — duplicate path names are an error in `build` | — | +| 4 | `feat/merge-relabel` | §3.4 — `relabel_in_place` / `make_disjoint` | — | +| 5 | `feat/merge-verify` | §4.4 — name-keyed verification, `compare_sequences` | 3 | +| 6 | `feat/merge-cmd` | §4.5 — the command, integration tests, docs, CHANGELOG | 2, 4, 5 | + +Phases 1–4 are mutually independent and can proceed in parallel. Phase 1 is not a strict dependency +of phase 6, but the single-genome integration test in phase 6 requires it. + +--- + +## 7. Non-goals (for now) + +- **N-way merge.** `merge` takes exactly two graphs. Merging N graphs along a guide tree is a + natural generalisation — `mash_distance` already works on arbitrary graphs, so the neighbour-joining + machinery would work unchanged — but it would require factoring the tree-traversal loop out of + `build()`, and users can chain pairwise merges in the meantime. +- **FASTA input to `merge`.** Once `build` accepts a single sequence (§4.1), the two-step workflow + composes cleanly. Accepting FASTA directly would duplicate `build`'s reading, `--circular` + handling and name checks inside `merge`. +- **Guide-tree-driven merge order.** Only relevant for N-way. +- **Incremental performance.** See §5; making appends proportional to the number of new genomes + would require a different alignment strategy and is out of scope. +- **Provenance metadata.** The graph JSON has no place to record which inputs a merged graph came + from. Not added here. + +--- + +## 8. Testing strategy + +**Unit** + +- `relabel_in_place` / `make_disjoint`: two hand-built graphs with identical ids `0..2` become + joinable, and `sanity_check` passes on the join. +- Relabeling is an isomorphism: sequences reconstructed from the graph, keyed by path name, are + unchanged by relabeling. +- Path ids after relabeling are contiguous, ordered, and start after the left graph's maximum. +- `build_tree_using_neighbor_joining` for 1 and 2 input graphs; clean error for 0. +- Duplicate path name detection: within one graph, across two graphs, and the unnamed-path case. + +**Integration** (`packages/pangraph/tests/itest_merge.rs`) + +- Split `data/ges-1.fa`, `build` A and B separately, `merge` them, then assert: every original + sequence is reconstructed byte-identically (matched **by name**, not by order); `sanity_check` + passes; the path count is `nA + nB`; all path names are preserved. +- A single-genome right graph (exercises phase 1). +- Two graphs with no shared homology — the merge degenerates to a pure join, and this is also the + case that produces surviving small integer ids, so it is the sharpest test of §3. +- Merging a graph with itself must fail with a duplicate-name error. + +--- + +## 9. Documentation + +- This document, plus `dev/design/merge-implementation-notes.md` for as-built decisions. +- A user-facing tutorial section on extending an existing graph, stating the cost model (§5), the + base-graph-first convention (§3.4), and that `merge(build(A), build(B)) != build(A ∪ B)` (§1). +- `pangraph reconstruct` docs: record order and `index` are only meaningful for graphs produced + directly by `build` (§4.4). +- Regenerate `docs/docs/reference.md` via `docs/generate-reference-docs` after the CLI changes. +- CHANGELOG entry. From 57deceee15f306af7cacbc0d3a7dd0cca21bcdfa Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 13:26:40 +0200 Subject: [PATCH 02/50] fix(build): support building a graph from a single input sequence --- .../pangraph/src/tree/neighbor_joining.rs | 57 ++++++++++++++++++- packages/pangraph/src/tree/newick.rs | 12 ++++ 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/packages/pangraph/src/tree/neighbor_joining.rs b/packages/pangraph/src/tree/neighbor_joining.rs index 8563d54e..76db403a 100644 --- a/packages/pangraph/src/tree/neighbor_joining.rs +++ b/packages/pangraph/src/tree/neighbor_joining.rs @@ -2,6 +2,7 @@ use crate::distance::mash::mash_distance::mash_distance; use crate::distance::mash::minimizer::MinimizersParams; +use crate::make_error; use crate::pangraph::pangraph::Pangraph; // use crate::tree::balance::balance; use crate::tree::clade::Clade; @@ -13,7 +14,17 @@ use ndarray::{Array1, Array2, Axis, s}; use ndarray_stats::QuantileExt; /// Generate guide tree using neighbor joining method. -pub fn build_tree_using_neighbor_joining(graphs: Vec) -> Result>>, Report> { +pub fn build_tree_using_neighbor_joining(mut graphs: Vec) -> Result>>, Report> { + match graphs.len() { + 0 => { + return make_error!("When building the guide tree: expected at least one input genome, but none were provided"); + }, + // A single genome needs no joining: the guide tree is that genome's leaf. Handled here, before + // computing pairwise distances, which would be wasted work. `pop` yields exactly the `Some` we need. + 1 => return Ok(Lock::new(Clade::new(graphs.pop()))), + _ => {}, + } + let mut distances = calculate_distances(&graphs); let mut nodes = graphs @@ -102,12 +113,56 @@ fn join_in_place(D: &mut Array2, nodes: &mut Vec> #[cfg(test)] mod tests { use super::*; + use crate::assert_error; + use crate::io::fasta::FastaRecord; + use crate::pangraph::strand::Strand::Forward; + use crate::representation::seq::Seq; use ndarray::array; use pretty_assertions::assert_eq; use rstest::rstest; const INF: f64 = f64::INFINITY; + /// Builds a singleton graph for one named sequence, as `build` does for each input FASTA record. + fn singleton(name: &str, index: usize) -> Pangraph { + Pangraph::singleton( + FastaRecord { + seq_name: name.to_owned(), + desc: None, + seq: Seq::from_str("ACGTACGTACGTACGT"), + index, + }, + Forward, + false, + ) + } + + #[rstest] + fn test_build_tree_rejects_no_graphs() { + assert_error!( + build_tree_using_neighbor_joining(vec![]), + "When building the guide tree: expected at least one input genome, but none were provided" + ); + } + + #[rstest] + fn test_build_tree_single_graph_is_a_leaf() { + let tree = build_tree_using_neighbor_joining(vec![singleton("A", 0)]).unwrap(); + let root = tree.read(); + assert!(root.is_leaf()); + assert!(root.data.is_some()); + assert_eq!(root.to_newick(), "A;"); + } + + #[rstest] + fn test_build_tree_two_graphs() { + let tree = build_tree_using_neighbor_joining(vec![singleton("A", 0), singleton("B", 1)]).unwrap(); + let root = tree.read(); + assert!(!root.is_leaf()); + assert!(root.data.is_none()); + assert_eq!(root.to_newick(), "(A,B);"); + } + #[rstest] fn test_create_Q_matrix() { // example from wikipedia: https://en.wikipedia.org/wiki/Neighbor_joining#Example diff --git a/packages/pangraph/src/tree/newick.rs b/packages/pangraph/src/tree/newick.rs index 4f41f178..66c16535 100644 --- a/packages/pangraph/src/tree/newick.rs +++ b/packages/pangraph/src/tree/newick.rs @@ -310,6 +310,18 @@ mod tests { assert_error!(parse_newick(input), expected); } + #[rstest] + fn build_tree_from_newick_single_leaf() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("tree.nwk"); + std::fs::write(&path, "A;").unwrap(); + + let tree = build_tree_from_newick(&path, vec![singleton("A", 0)]).unwrap(); + + assert!(tree.read().is_leaf()); + assert_eq!(collect_leaf_names(&tree), vec!["A"]); + } + #[rstest] fn build_tree_from_newick_attaches_graphs() { let dir = tempfile::tempdir().unwrap(); From 59b48b144838a83b0d5330ad9cc89bee9d2e6d5c Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 14:47:57 +0200 Subject: [PATCH 03/50] feat(merge): add command to merge two pangenome graphs --- dev/design/merge.md | 32 ++- docs/docs/reference.md | 68 +++++- packages/pangraph/src/align/alignment_args.rs | 75 ++++++- packages/pangraph/src/align/map_variations.rs | 12 +- .../align/nextclade/align_with_nextclade.rs | 4 +- packages/pangraph/src/bin/merge_two_graphs.rs | 104 --------- .../pangraph/src/commands/build/build_args.rs | 48 +--- .../pangraph/src/commands/build/build_run.rs | 28 +-- packages/pangraph/src/commands/main.rs | 2 + .../pangraph/src/commands/merge/merge_args.rs | 46 ++++ .../pangraph/src/commands/merge/merge_run.rs | 183 ++++++++++++++++ packages/pangraph/src/commands/merge/mod.rs | 2 + packages/pangraph/src/commands/mod.rs | 1 + packages/pangraph/src/commands/root_args.rs | 4 + .../pangraph/src/pangraph/graph_merging.rs | 8 +- packages/pangraph/src/pangraph/pangraph.rs | 207 +++++++++++++++++- .../pangraph/src/pangraph/pangraph_block.rs | 13 +- packages/pangraph/src/pangraph/reweave.rs | 4 +- .../pangraph/src/reconsensus/reconsensus.rs | 12 +- packages/pangraph/tests/itest_merge.rs | 189 ++++++++++++++++ 20 files changed, 831 insertions(+), 211 deletions(-) delete mode 100644 packages/pangraph/src/bin/merge_two_graphs.rs create mode 100644 packages/pangraph/src/commands/merge/merge_args.rs create mode 100644 packages/pangraph/src/commands/merge/merge_run.rs create mode 100644 packages/pangraph/src/commands/merge/mod.rs create mode 100644 packages/pangraph/tests/itest_merge.rs diff --git a/dev/design/merge.md b/dev/design/merge.md index 990c879a..5d7f94a3 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -1,6 +1,6 @@ # Graph merging — design manifesto -Status: **design accepted, implementation not started** +Status: **`pangraph merge` implemented; verification and `build`-side name checks still pending (see §6)** Integration branch: `feat/merge` (merged into `master` last, after all phase branches below) This document describes the design for a new `pangraph merge` command, which combines two @@ -357,17 +357,27 @@ explicitly in the user documentation. Each phase is its own branch and PR, targeting the integration branch `feat/merge`. `feat/merge` is merged into `master` last, once all phases have landed. -| # | Branch | Scope | Depends on | +| # | Branch | Scope | Status | |---|---|---|---| -| 1 | `feat/merge-single-seq-build` | §4.1 — NJ tree for 1 and 0 input graphs | — | -| 2 | `feat/merge-params` | §4.2 — extract `GraphMergeParams`, no behaviour change | — | -| 3 | `feat/merge-unique-names` | §4.3 — duplicate path names are an error in `build` | — | -| 4 | `feat/merge-relabel` | §3.4 — `relabel_in_place` / `make_disjoint` | — | -| 5 | `feat/merge-verify` | §4.4 — name-keyed verification, `compare_sequences` | 3 | -| 6 | `feat/merge-cmd` | §4.5 — the command, integration tests, docs, CHANGELOG | 2, 4, 5 | - -Phases 1–4 are mutually independent and can proceed in parallel. Phase 1 is not a strict dependency -of phase 6, but the single-genome integration test in phase 6 requires it. +| 1 | `feat/merge-single-seq-build` | §4.1 — NJ tree for 1 and 0 input graphs | landed | +| 2 | `feat/merge-cmd` | §4.2 — extract `GraphMergeParams`, no behaviour change | landed | +| 3 | `feat/merge-cmd` | §3.4 — `relabel_in_place` / `make_disjoint_from` | landed | +| 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | +| 5 | `feat/merge-unique-names` | §4.3 — duplicate path names are an error in `build` too | todo | +| 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | todo | +| 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | todo | + +Phases 2–4 were implemented together, since a `merge` command without §3.4 panics on the first +identifier collision and would not be testable. + +Two pieces of §4.3 and §4.4 are therefore only half-done, and phases 5 and 6 finish them: + +- Duplicate path names are rejected by `merge` (across and within its two inputs), but `build` still + accepts duplicate FASTA names except when `--guide-tree` is used. +- Name-keyed verification exists as `verify_merged_sequences` inside `merge_run`, private to the + merge command. `build` still verifies against input FASTA records by index, and + `compare_sequences` still compares whole `FastaRecord`s (including `index`). Phase 6 unifies the + two behind a single `BTreeMap`-based helper. --- diff --git a/docs/docs/reference.md b/docs/docs/reference.md index 24133e6d..3451f1b9 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -24,6 +24,7 @@ If you have Pangraph CLI installed, you can type `pangraph --help` to read the l * [`pangraph`↴](#pangraph) * [`pangraph build`↴](#pangraph-build) +* [`pangraph merge`↴](#pangraph-merge) * [`pangraph export`↴](#pangraph-export) * [`pangraph export gfa`↴](#pangraph-export-gfa) * [`pangraph export block-consensus`↴](#pangraph-export-block-consensus) @@ -55,6 +56,7 @@ Questions, ideas, bug reports: https://github.com/neherlab/pangraph/issues ###### **Subcommands:** * `build` — Align genomes into a multiple sequence alignment graph +* `merge` — Merge two pangenome graphs into a single one * `export` — Export a pangraph to a chosen file format(s) * `simplify` — Generates a simplified graph that only contains a subset of the input genomes * `reconstruct` — Reconstruct all input fasta sequences from graph @@ -115,7 +117,6 @@ Align genomes into a multiple sequence alignment graph Default value: `10` * `-K`, `--kmer-length ` — Sets kmer length for mmseqs2 aligner -* `-c`, `--circular` — Toggle if input genomes are circular * `-x`, `--max-self-map ` — Maximum number of alignment rounds to consider per pairwise graph merger Default value: `100` @@ -127,20 +128,81 @@ Align genomes into a multiple sequence alignment graph Possible values: `minimap2`, `mmseqs` -* `-f`, `--verify` — Sanity check: after construction verifies that the original sequences can be reconstructed exactly from the resulting pangraph. Raises an error otherwise -* `--no-progress-bar` — Toggle to disable progress bar. Notice that the progress bar is only displayed if the output is specified via the `-o` argument * `--extra-band-width ` — For within-block alignment: excess bandwidth for internal stripes. Can be increased to improve block alignment quality, at the cost of computation time and memory usage Default value: `5` * `--max-alignment-attempts ` — For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit Default value: `4` +* `-c`, `--circular` — Toggle if input genomes are circular +* `-f`, `--verify` — Sanity check: after construction verifies that the original sequences can be reconstructed exactly from the resulting pangraph. Raises an error otherwise +* `--no-progress-bar` — Toggle to disable progress bar. Notice that the progress bar is only displayed if the output is specified via the `-o` argument * `--guide-tree ` — Path to a Newick-format guide tree to use instead of the default neighbor-joining tree. When provided, the tree's topology drives the bottom-up graph-merging order. Each input FASTA sequence must appear exactly once as a leaf (matched by sequence name), and every internal node must be strictly bifurcating. Branch lengths and internal labels, if present, are ignored. Accepts plain or compressed files (gz, bz2, xz, zst). +## `pangraph merge` + +Merge two pangenome graphs into a single one + +**Usage:** `pangraph merge [OPTIONS] ` + +###### **Arguments:** + +* `` — Path to the first input graph, in pangraph JSON format. + + This graph is treated as the base: its block, node and path identifiers are preserved in the output, while those of the second graph are renumbered. When extending an existing graph with new genomes, pass the existing graph here. + + Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. The decompressor is chosen based on the file extension. +* `` — Path to the second input graph, in pangraph JSON format. + + Its identifiers are renumbered so that they do not clash with those of the first graph. Its genomes appear after those of the first graph in the output. + +###### **Options:** + +* `-o`, `--output-json ` — Path to output JSON file with the merged pangraph. + + If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. + + Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + + Default value: `-` +* `-l`, `--len ` — Minimum block size for alignment graph (in nucleotides) + + Default value: `100` +* `-a`, `--alpha ` — Energy cost for splitting a block during alignment merger. Controls graph fragmentation, see documentation + + Default value: `100` +* `-b`, `--beta ` — Energy cost for diversity in the alignment. A high value prevents merging of distantly-related sequences in the same block, see documentation + + Default value: `10` +* `-s`, `--sensitivity ` — Used to set pairwise alignment sensitivity for minimap aligner. Corresponds to option -x asm5/asm10/asm20 in minimap2 + + Default value: `10` +* `-K`, `--kmer-length ` — Sets kmer length for mmseqs2 aligner +* `-x`, `--max-self-map ` — Maximum number of alignment rounds to consider per pairwise graph merger + + Default value: `100` +* `-k`, `--alignment-kernel ` — Backend to use for pairwise genome alignment + + Nb: `mmseqs` is more sensitive to highly-diverged sequences, but slower and requires more memory. It is not provided with Pangraph, so you need to install it separately (see: https://github.com/soedinglab/MMseqs2) + + Default value: `minimap2` + + Possible values: `minimap2`, `mmseqs` + +* `--extra-band-width ` — For within-block alignment: excess bandwidth for internal stripes. Can be increased to improve block alignment quality, at the cost of computation time and memory usage + + Default value: `5` +* `--max-alignment-attempts ` — For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit + + Default value: `4` +* `-f`, `--verify` — Sanity check: after merging verifies that every genome of the two input graphs can still be reconstructed exactly from the merged graph. Raises an error otherwise + + + ## `pangraph export` Export a pangraph to a chosen file format(s) diff --git a/packages/pangraph/src/align/alignment_args.rs b/packages/pangraph/src/align/alignment_args.rs index 490a84b8..c070c415 100644 --- a/packages/pangraph/src/align/alignment_args.rs +++ b/packages/pangraph/src/align/alignment_args.rs @@ -1,6 +1,10 @@ -use clap::{Args, ValueHint, value_parser}; +use clap::{Args, ValueEnum, ValueHint, value_parser}; +use color_eyre::owo_colors::{AnsiColors, OwoColorize}; +use color_eyre::{Help, SectionExt}; +use eyre::{Report, WrapErr}; use serde::{Deserialize, Serialize}; use smart_default::SmartDefault; +use strum_macros::Display; #[derive(Clone, Debug, SmartDefault, Args, Serialize, Deserialize)] pub struct AlignmentArgs { @@ -33,3 +37,72 @@ pub struct AlignmentArgs { #[clap(value_hint = ValueHint::Other)] pub kmer_length: Option, } + +#[derive( + Copy, Debug, Clone, PartialEq, Eq, PartialOrd, Ord, ValueEnum, SmartDefault, Display, Serialize, Deserialize, +)] +#[clap(rename_all = "kebab-case")] +#[serde(rename_all = "kebab-case")] +#[strum(serialize_all = "kebab-case")] +pub enum AlignmentBackend { + #[default] + Minimap2, + Mmseqs, +} + +/// Parameters that govern how two graphs are merged into one. +/// +/// Shared by `pangraph build` (which merges at every node of the guide tree) and +/// `pangraph merge` (which performs a single such merger), so that both commands expose an +/// identical set of alignment options. +#[derive(Clone, Debug, SmartDefault, Args, Serialize, Deserialize)] +pub struct GraphMergeParams { + #[clap(flatten, next_help_heading = "Alignment")] + pub aln_args: AlignmentArgs, + + /// Maximum number of alignment rounds to consider per pairwise graph merger + #[default = 100] + #[clap(long, short = 'x', default_value_t = GraphMergeParams::default().max_self_map)] + #[clap(value_hint = ValueHint::Other)] + pub max_self_map: usize, + + /// Backend to use for pairwise genome alignment + /// + /// Nb: `mmseqs` is more sensitive to highly-diverged sequences, but slower and requires more memory. + /// It is not provided with Pangraph, so you need to install it separately (see: https://github.com/soedinglab/MMseqs2) + #[clap(long, short = 'k', default_value_t = GraphMergeParams::default().alignment_kernel)] + #[clap(value_hint = ValueHint::Other)] + pub alignment_kernel: AlignmentBackend, + + /// For within-block alignment: excess bandwidth for internal stripes. + /// Can be increased to improve block alignment quality, at the cost of computation time and memory usage. + #[default = 5] + #[clap(long, default_value_t = GraphMergeParams::default().extra_band_width)] + #[clap(value_hint = ValueHint::Other)] + pub extra_band_width: usize, + + /// For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit. + #[default = 4] + #[clap(long, default_value_t = GraphMergeParams::default().max_alignment_attempts)] + #[clap(value_hint = ValueHint::Other)] + pub max_alignment_attempts: usize, +} + +/// Checks that the selected alignment backend is usable before any real work is started. +/// Only the external backends need checking: the minimap2 one is built into pangraph. +pub fn check_alignment_backend_available(params: &GraphMergeParams) -> Result<(), Report> { + if params.alignment_kernel == AlignmentBackend::Mmseqs { + // check that mmseqs is available in PATH + std::process::Command::new("mmseqs") + .arg("--help") + .output() + .wrap_err("When executing `mmseqs --help`") + .section( + "Please make sure that `mmseqs` is installed, available in PATH and is functional outside of pangraph. For more details, refer to mmseqs documentation at https://github.com/soedinglab/MMseqs2" + .color(AnsiColors::Cyan) + .header("Suggestion:"), + )?; + } + + Ok(()) +} diff --git a/packages/pangraph/src/align/map_variations.rs b/packages/pangraph/src/align/map_variations.rs index 7478f1b8..2330d625 100644 --- a/packages/pangraph/src/align/map_variations.rs +++ b/packages/pangraph/src/align/map_variations.rs @@ -1,6 +1,6 @@ +use crate::align::alignment_args::GraphMergeParams; use crate::align::nextclade::align_with_nextclade::{AlignWithNextcladeOutput, NextalignParams, align_with_nextclade}; use crate::align::nextclade::alphabet::nuc::{from_nuc, from_nuc_seq}; -use crate::commands::build::build_args::PangraphBuildArgs; use crate::make_internal_report; use crate::pangraph::edits::{Del, Edit, Ins, Sub}; use crate::representation::seq::Seq; @@ -40,7 +40,7 @@ pub fn map_variations( ref_seq: &Seq, qry_seq: &Seq, mut band_params: BandParameters, - args: &PangraphBuildArgs, + args: &GraphMergeParams, ) -> Result { let params = NextalignParams { min_length: 1, @@ -204,7 +204,7 @@ mod tests { mean_shift, band_width: bandwidth, }, - &PangraphBuildArgs::default(), + &GraphMergeParams::default(), ) .unwrap(); @@ -249,7 +249,7 @@ mod tests { mean_shift, band_width: bandwidth, }, - &PangraphBuildArgs::default(), + &GraphMergeParams::default(), ) .unwrap(); @@ -294,7 +294,7 @@ mod tests { mean_shift, band_width: bandwidth, }, - &PangraphBuildArgs::default(), + &GraphMergeParams::default(), ) .unwrap(); @@ -338,7 +338,7 @@ mod tests { mean_shift, band_width: bandwidth, }, - &PangraphBuildArgs::default(), + &GraphMergeParams::default(), ) .unwrap(); diff --git a/packages/pangraph/src/align/nextclade/align_with_nextclade.rs b/packages/pangraph/src/align/nextclade/align_with_nextclade.rs index f3832de5..787c35ee 100644 --- a/packages/pangraph/src/align/nextclade/align_with_nextclade.rs +++ b/packages/pangraph/src/align/nextclade/align_with_nextclade.rs @@ -77,15 +77,15 @@ pub fn align_with_nextclade( #[cfg(test)] mod tests { use super::*; + use crate::align::alignment_args::GraphMergeParams; use crate::align::nextclade::coord::position::NucRefGlobalPosition; - use crate::commands::build::build_args::PangraphBuildArgs; use crate::o; use eyre::Report; use pretty_assertions::assert_eq; use rstest::rstest; use std::sync::LazyLock; - static EXTRA_BANDWIDTH: LazyLock = LazyLock::new(|| PangraphBuildArgs::default().extra_band_width); + static EXTRA_BANDWIDTH: LazyLock = LazyLock::new(|| GraphMergeParams::default().extra_band_width); #[rstest] fn test_align_with_nextclade_general_case() -> Result<(), Report> { diff --git a/packages/pangraph/src/bin/merge_two_graphs.rs b/packages/pangraph/src/bin/merge_two_graphs.rs deleted file mode 100644 index 3684bec2..00000000 --- a/packages/pangraph/src/bin/merge_two_graphs.rs +++ /dev/null @@ -1,104 +0,0 @@ -use clap::{Parser, ValueHint}; -use ctor::ctor; -use eyre::{Report, WrapErr}; -use log::info; -use pangraph::commands::build::build_args::PangraphBuildArgs; -use pangraph::commands::reconstruct::reconstruct_run::{compare_sequences, reconstruct}; -use pangraph::io::json::{JsonPretty, json_write_file}; -use pangraph::pangraph::graph_merging::merge_graphs; -use pangraph::pangraph::pangraph::Pangraph; -use pangraph::utils::global_init::global_init; -use pangraph::utils::global_init::setup_logger; -use std::collections::HashMap; -use std::path::PathBuf; - -#[ctor] -fn init() { - global_init(); - setup_logger(log::LevelFilter::Debug); -} - -#[derive(Parser, Debug)] -struct Args { - #[clap(flatten)] - pub build_args: PangraphBuildArgs, - - #[clap(long, short = 'L')] - #[clap(value_hint = ValueHint::FilePath)] - pub left_graph: Option, - - #[clap(long, short = 'R')] - #[clap(value_hint = ValueHint::FilePath)] - pub right_graph: Option, - - // binary flag: if set, verify the sequences in the merged graph did not change - #[clap(long, short = 'V')] - pub merge_verify: bool, -} - -fn main() -> Result<(), Report> { - let mut args = Args::parse(); - - rayon::ThreadPoolBuilder::new() - .num_threads(num_cpus::get()) - .build_global()?; - - // Manually empty the input_fastas field in PangraphBuildArgs - args.build_args.input_fastas = vec![]; - - info!("Reading left graph from {:?}", args.left_graph); - info!("Reading right graph from {:?}", args.right_graph); - - let left = Pangraph::from_path(&args.left_graph).wrap_err("When reading left graph")?; - let right = Pangraph::from_path(&args.right_graph).wrap_err("When reading right graph")?; - - info!("left graph sanity check"); - #[cfg(debug_assertions)] - left.sanity_check()?; - - info!("right graph sanity check"); - #[cfg(debug_assertions)] - right.sanity_check()?; - - info!("Merging graphs"); - - let merged = merge_graphs(&left, &right, &args.build_args)?; - - #[cfg(debug_assertions)] - merged.sanity_check()?; - - info!("Writing merged graph to {}", args.build_args.output_json.display()); - - json_write_file(&args.build_args.output_json, &merged, JsonPretty(true))?; - - if args.merge_verify { - let initial_left_seqs = reconstruct(&left); - let initial_right_seqs = reconstruct(&right); - - // create a map of {seq.idx -> seq} - let mut initial_seqs = HashMap::new(); - initial_left_seqs - .chain(initial_right_seqs) - .try_for_each(|seq| -> Result<(), Report> { - let seq = seq?; - initial_seqs.insert(seq.index, seq); - Ok(()) - })?; - - // verify that the merged graph contains the same sequences - let mut merged_seqs = reconstruct(&merged); - merged_seqs.try_for_each(|seq| -> Result<(), Report> { - let seq = seq?; - let idx = seq.index; - if let Some(expected) = initial_seqs.get(&idx) { - compare_sequences(expected, &seq)?; - } else { - return Err(eyre::eyre!("Sequence with index {idx} not found in initial sequences")); - } - Ok(()) - })?; - info!("Merged graph contains the same sequences as the original graphs"); - } - - Ok(()) -} diff --git a/packages/pangraph/src/commands/build/build_args.rs b/packages/pangraph/src/commands/build/build_args.rs index 5ce74f65..aa490a51 100644 --- a/packages/pangraph/src/commands/build/build_args.rs +++ b/packages/pangraph/src/commands/build/build_args.rs @@ -1,22 +1,8 @@ -use crate::align::alignment_args::AlignmentArgs; -use clap::{Parser, ValueEnum, ValueHint}; -use serde::{Deserialize, Serialize}; +use crate::align::alignment_args::GraphMergeParams; +use clap::{Parser, ValueHint}; use smart_default::SmartDefault; use std::fmt::Debug; use std::path::PathBuf; -use strum_macros::Display; - -#[derive( - Copy, Debug, Clone, PartialEq, Eq, PartialOrd, Ord, ValueEnum, SmartDefault, Display, Serialize, Deserialize, -)] -#[clap(rename_all = "kebab-case")] -#[serde(rename_all = "kebab-case")] -#[strum(serialize_all = "kebab-case")] -pub enum AlignmentBackend { - #[default] - Minimap2, - Mmseqs, -} /// Align genomes into a pangenome graph #[derive(Parser, Debug, SmartDefault)] @@ -43,26 +29,13 @@ pub struct PangraphBuildArgs { #[clap(value_hint = ValueHint::AnyPath)] pub output_json: PathBuf, - #[clap(flatten, next_help_heading = "Alignment")] - pub aln_args: AlignmentArgs, + #[clap(flatten)] + pub merge_params: GraphMergeParams, /// Toggle if input genomes are circular #[clap(long, short = 'c')] pub circular: bool, - /// Maximum number of alignment rounds to consider per pairwise graph merger - #[clap(long, short = 'x', default_value_t = 100)] - #[clap(value_hint = ValueHint::Other)] - pub max_self_map: usize, - - /// Backend to use for pairwise genome alignment - /// - /// Nb: `mmseqs` is more sensitive to highly-diverged sequences, but slower and requires more memory. - /// It is not provided with Pangraph, so you need to install it separately (see: https://github.com/soedinglab/MMseqs2) - #[clap(long, short = 'k', default_value_t = PangraphBuildArgs::default().alignment_kernel)] - #[clap(value_hint = ValueHint::Other)] - pub alignment_kernel: AlignmentBackend, - /// Sanity check: after construction verifies that the original sequences can be reconstructed exactly from the resulting pangraph. Raises an error otherwise. #[clap(long, short = 'f')] pub verify: bool, @@ -71,19 +44,6 @@ pub struct PangraphBuildArgs { #[clap(long)] pub no_progress_bar: bool, - /// For within-block alignment: excess bandwidth for internal stripes. - /// Can be increased to improve block alignment quality, at the cost of computation time and memory usage. - #[default = 5] - #[clap(long, default_value_t = PangraphBuildArgs::default().extra_band_width)] - #[clap(value_hint = ValueHint::Other)] - pub extra_band_width: usize, - - /// For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit. - #[default = 4] - #[clap(long, default_value_t = PangraphBuildArgs::default().max_alignment_attempts)] - #[clap(value_hint = ValueHint::Other)] - pub max_alignment_attempts: usize, - /// Path to a Newick-format guide tree to use instead of the default neighbor-joining tree. /// /// When provided, the tree's topology drives the bottom-up graph-merging order. Each input diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index f821c532..0d59289c 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -1,4 +1,5 @@ -use crate::commands::build::build_args::{AlignmentBackend, PangraphBuildArgs}; +use crate::align::alignment_args::check_alignment_backend_available; +use crate::commands::build::build_args::PangraphBuildArgs; use crate::commands::reconstruct::reconstruct_run::{compare_sequences, reconstruct}; use crate::io::fasta::{FastaReader, FastaRecord}; use crate::io::json::{JsonPretty, json_write_file}; @@ -10,30 +11,10 @@ use crate::tree::neighbor_joining::build_tree_using_neighbor_joining; use crate::tree::newick::build_tree_from_newick; use crate::utils::progress_bar::ProgressBar; use crate::{make_internal_error, make_internal_report}; -use color_eyre::owo_colors::{AnsiColors, OwoColorize}; -use color_eyre::{Help, SectionExt}; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::info; -pub fn build_cmd_preliminary_checks(args: &PangraphBuildArgs) -> Result<(), Report> { - // alignment kernel checks - if args.alignment_kernel == AlignmentBackend::Mmseqs { - // check that mmseqs is available in PATH - std::process::Command::new("mmseqs") - .arg("--help") - .output() - .wrap_err("When executing `mmseqs --help`") - .section( - "Please make sure that `mmseqs` is installed, available in PATH and is functional outside of pangraph. For more details, refer to mmseqs documentation at https://github.com/soedinglab/MMseqs2" - .color(AnsiColors::Cyan) - .header("Suggestion:"), - )?; - } - - Ok(()) -} - pub fn reconstruct_and_compare_graph_seqs(graph: &Pangraph, fastas: &[FastaRecord]) -> Result<(), Report> { // Reconstruct sequences from the given graph. let mut results = reconstruct(graph); @@ -71,7 +52,8 @@ pub fn build_run(args: &PangraphBuildArgs) -> Result<(), Report> { // TODO: adjust fasta letter case if `upper_case` is set // TODO: check for duplicate fasta names - build_cmd_preliminary_checks(args).wrap_err("When performing preliminary checks before building the pangraph.")?; + check_alignment_backend_available(&args.merge_params) + .wrap_err("When performing preliminary checks before building the pangraph.")?; let pangraph = build(fastas, args, args.verify)?; @@ -125,7 +107,7 @@ pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) - right.paths.len() ); - clade.data = Some(merge_graphs(left, right, args).wrap_err("When merging graphs")?); + clade.data = Some(merge_graphs(left, right, &args.merge_params).wrap_err("When merging graphs")?); // increase progress bar pb.inc(1); diff --git a/packages/pangraph/src/commands/main.rs b/packages/pangraph/src/commands/main.rs index 4d94f27c..53c22787 100644 --- a/packages/pangraph/src/commands/main.rs +++ b/packages/pangraph/src/commands/main.rs @@ -1,6 +1,7 @@ use crate::commands::build::build_run::build_run; use crate::commands::export::export_run::export_run; use crate::commands::md_help::print_help_markdown::print_help_markdown; +use crate::commands::merge::merge_run::merge_run; use crate::commands::reconstruct::reconstruct_run::reconstruct_run; use crate::commands::root_args::{PangraphCommands, generate_shell_completions, parse_cli_args}; use crate::commands::schema::generate_schema::generate_schema; @@ -17,6 +18,7 @@ pub fn pangraph_main() -> Result<(), Report> { match args.command { PangraphCommands::Build(args) => build_run(&args), + PangraphCommands::Merge(args) => merge_run(&args), PangraphCommands::Export { args } => export_run(args), PangraphCommands::Simplify(args) => simplify_run(args), PangraphCommands::Reconstruct(args) => reconstruct_run(&args), diff --git a/packages/pangraph/src/commands/merge/merge_args.rs b/packages/pangraph/src/commands/merge/merge_args.rs new file mode 100644 index 00000000..5eefe25f --- /dev/null +++ b/packages/pangraph/src/commands/merge/merge_args.rs @@ -0,0 +1,46 @@ +use crate::align::alignment_args::GraphMergeParams; +use clap::{Parser, ValueHint}; +use smart_default::SmartDefault; +use std::fmt::Debug; +use std::path::PathBuf; + +/// Merge two pangenome graphs into a single one +#[derive(Parser, Debug, SmartDefault)] +pub struct PangraphMergeArgs { + /// Path to the first input graph, in pangraph JSON format. + /// + /// This graph is treated as the base: its block, node and path identifiers are preserved in the + /// output, while those of the second graph are renumbered. When extending an existing graph with + /// new genomes, pass the existing graph here. + /// + /// Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. + /// The decompressor is chosen based on the file extension. + #[clap(value_hint = ValueHint::FilePath)] + #[clap(display_order = 1)] + pub left_graph: PathBuf, + + /// Path to the second input graph, in pangraph JSON format. + /// + /// Its identifiers are renumbered so that they do not clash with those of the first graph. Its + /// genomes appear after those of the first graph in the output. + #[clap(value_hint = ValueHint::FilePath)] + #[clap(display_order = 2)] + pub right_graph: PathBuf, + + /// Path to output JSON file with the merged pangraph. + /// + /// If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. + /// + /// Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + #[clap(long, short = 'o', default_value = "-")] + #[clap(value_hint = ValueHint::AnyPath)] + pub output_json: PathBuf, + + #[clap(flatten)] + pub merge_params: GraphMergeParams, + + /// Sanity check: after merging verifies that every genome of the two input graphs can still be + /// reconstructed exactly from the merged graph. Raises an error otherwise. + #[clap(long, short = 'f')] + pub verify: bool, +} diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs new file mode 100644 index 00000000..2c71d8a0 --- /dev/null +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -0,0 +1,183 @@ +use crate::align::alignment_args::check_alignment_backend_available; +use crate::commands::merge::merge_args::PangraphMergeArgs; +use crate::commands::reconstruct::reconstruct_run::reconstruct; +use crate::io::json::{JsonPretty, json_write_file}; +use crate::make_error; +use crate::pangraph::graph_merging::merge_graphs; +use crate::pangraph::pangraph::Pangraph; +use crate::pangraph::pangraph_path::PangraphPath; +use crate::representation::seq::Seq; +use eyre::{Report, WrapErr}; +use itertools::Itertools; +use log::{info, warn}; +use std::collections::{BTreeMap, BTreeSet}; +use std::path::Path; + +pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { + let left = read_graph(&args.left_graph).wrap_err("When reading the first input graph")?; + let mut right = read_graph(&args.right_graph).wrap_err("When reading the second input graph")?; + + merge_cmd_preliminary_checks(args, &left, &right).wrap_err("When performing preliminary checks before merging")?; + + // The two graphs were built independently, so their identifiers almost certainly collide. + // Namespace the second graph before joining them. + right + .make_disjoint_from(&left) + .wrap_err("When making the identifiers of the two input graphs disjoint")?; + + // Reconstruct the expected genomes from the (relabeled) inputs, before they are consumed by the + // merger. Keyed by path name: neither path ids nor record order survive a merge. + let expected = args + .verify + .then(|| expected_sequences(&left, &right)) + .transpose() + .wrap_err("When reconstructing the sequences of the input graphs")?; + + info!( + "=== Graph merging start: graph sizes {} + {}", + left.paths.len(), + right.paths.len() + ); + + let merged = merge_graphs(&left, &right, &args.merge_params).wrap_err("When merging graphs")?; + + info!( + "=== Graph merging completed: graph sizes {} + {} -> {} paths, {} blocks", + left.paths.len(), + right.paths.len(), + merged.paths.len(), + merged.blocks.len() + ); + + if let Some(expected) = expected { + verify_merged_sequences(&merged, &expected).wrap_err("When verifying the sequences of the merged graph")?; + info!("Merged graph reconstructs all {} input genomes exactly", expected.len()); + } + + json_write_file(&args.output_json, &merged, JsonPretty(true))?; + + Ok(()) +} + +/// Reads a pangraph from a JSON file, and checks its internal consistency in debug builds. +fn read_graph(filepath: &Path) -> Result { + let graph = Pangraph::from_path(&Some(filepath))?; + + #[cfg(debug_assertions)] + graph + .sanity_check() + .wrap_err_with(|| format!("When performing sanity check on graph '{}'", filepath.display()))?; + + Ok(graph) +} + +fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right: &Pangraph) -> Result<(), Report> { + check_alignment_backend_available(&args.merge_params)?; + + for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { + if graph.paths.is_empty() { + return make_error!("Input graph '{}' contains no genomes", filepath.display()); + } + } + + // Path names are the identity of a genome throughout pangraph: they are what `export` and + // `simplify` resolve genomes by, and the only handle that survives a merge unchanged. + let duplicates = duplicate_path_names(&[left, right]); + if !duplicates.is_empty() { + return make_error!( + "Duplicate genome names found in the input graphs: [{}]. Genome names must be unique across the two graphs: merging a graph with itself, or re-adding a genome that is already present, is not supported.", + duplicates.join(", ") + ); + } + + if args.verify { + for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { + if graph.path_names().any(|name| name.is_none()) { + return make_error!( + "Graph '{}' contains genomes without a name, which cannot be verified: verification matches genomes by name. Re-run without `--verify`.", + filepath.display() + ); + } + } + } + + // Circularity is a per-path property, so mixing is structurally fine. It is however most often a + // mistake, since `build --circular` applies to all genomes of a graph at once. + if circularity(left) != circularity(right) { + warn!( + "The two input graphs disagree on whether their genomes are circular. This is allowed, but check that both graphs were built with a consistent `--circular` setting." + ); + } + + Ok(()) +} + +/// Returns the sorted path names that occur more than once across the given graphs. +fn duplicate_path_names(graphs: &[&Pangraph]) -> Vec { + graphs + .iter() + .flat_map(|graph| graph.path_names().flatten()) + .counts() + .into_iter() + .filter(|(_, count)| *count > 1) + .map(|(name, _)| name.to_owned()) + .sorted() + .collect() +} + +/// Returns the set of circularity flags used by the paths of a graph. +fn circularity(graph: &Pangraph) -> BTreeSet { + graph.paths().map(PangraphPath::circular).collect() +} + +/// Reconstructs the genomes of both input graphs, keyed by genome name. +fn expected_sequences(left: &Pangraph, right: &Pangraph) -> Result, Report> { + let mut expected = BTreeMap::new(); + for graph in [left, right] { + for record in reconstruct(graph) { + let record = record?; + expected.insert(record.seq_name, record.seq); + } + } + Ok(expected) +} + +/// Checks that the merged graph reconstructs exactly the genomes of the input graphs. +/// Genomes are matched by name: path ids are renumbered by the merger, and the order in which +/// genomes are reconstructed is therefore not the order of either input graph. +fn verify_merged_sequences(merged: &Pangraph, expected: &BTreeMap) -> Result<(), Report> { + #[cfg(debug_assertions)] + merged.sanity_check().wrap_err("When checking the merged graph")?; + + let mut remaining: BTreeSet<&String> = expected.keys().collect(); + + for record in reconstruct(merged) { + let record = record?; + let Some(expected_seq) = expected.get(&record.seq_name) else { + return make_error!( + "Merged graph contains genome '{}', which is not present in either input graph", + record.seq_name + ); + }; + + if record.seq != *expected_seq { + return make_error!( + "Sequence mismatch for genome '{}': expected length {} but got {}", + record.seq_name, + expected_seq.len(), + record.seq.len() + ); + } + + remaining.remove(&record.seq_name); + } + + if !remaining.is_empty() { + return make_error!( + "Merged graph is missing genomes from the input graphs: [{}]", + remaining.into_iter().sorted().join(", ") + ); + } + + Ok(()) +} diff --git a/packages/pangraph/src/commands/merge/mod.rs b/packages/pangraph/src/commands/merge/mod.rs new file mode 100644 index 00000000..a4481831 --- /dev/null +++ b/packages/pangraph/src/commands/merge/mod.rs @@ -0,0 +1,2 @@ +pub mod merge_args; +pub mod merge_run; diff --git a/packages/pangraph/src/commands/mod.rs b/packages/pangraph/src/commands/mod.rs index 0c11d129..36ff2e18 100644 --- a/packages/pangraph/src/commands/mod.rs +++ b/packages/pangraph/src/commands/mod.rs @@ -2,6 +2,7 @@ pub mod build; pub mod export; pub mod main; pub mod md_help; +pub mod merge; pub mod reconstruct; pub mod root_args; pub mod schema; diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index 5020a3cf..e0d1ce3e 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -2,6 +2,7 @@ use crate::commands::build::build_args::PangraphBuildArgs; use crate::commands::export::export_args::PangraphExportArgs; +use crate::commands::merge::merge_args::PangraphMergeArgs; use crate::commands::reconstruct::reconstruct_args::PangraphReconstructArgs; use crate::commands::schema::generate_schema::PangraphGenerateSchemaArgs; use crate::commands::simplify::simplify_args::PangraphSimplifyArgs; @@ -62,6 +63,9 @@ pub enum PangraphCommands { /// Align genomes into a multiple sequence alignment graph Build(PangraphBuildArgs), + /// Merge two pangenome graphs into a single one + Merge(PangraphMergeArgs), + /// Export a pangraph to a chosen file format(s) Export { #[clap(subcommand)] diff --git a/packages/pangraph/src/pangraph/graph_merging.rs b/packages/pangraph/src/pangraph/graph_merging.rs index 5c2ad4a8..85b2520c 100644 --- a/packages/pangraph/src/pangraph/graph_merging.rs +++ b/packages/pangraph/src/pangraph/graph_merging.rs @@ -1,10 +1,10 @@ use crate::align::alignment::Alignment; use crate::align::alignment_args::AlignmentArgs; +use crate::align::alignment_args::{AlignmentBackend, GraphMergeParams}; use crate::align::energy::alignment_energy2; use crate::align::minimap2_lib::align_with_minimap2_lib::align_with_minimap2_lib; use crate::align::mmseqs::align_with_mmseqs::align_with_mmseqs; use crate::circularize::circularize::remove_transitive_edges; -use crate::commands::build::build_args::{AlignmentBackend, PangraphBuildArgs}; use crate::pangraph::detach_unaligned::detach_unaligned_nodes; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_block::{BlockId, PangraphBlock}; @@ -26,7 +26,7 @@ use std::collections::BTreeMap; pub fn merge_graphs( left_graph: &Pangraph, right_graph: &Pangraph, - args: &PangraphBuildArgs, + args: &GraphMergeParams, ) -> Result { // put the two graphs in a single one, by simply joining // the two sets of blocks and paths. No merging is performed @@ -92,7 +92,7 @@ pub fn graph_join(left_graph: &Pangraph, right_graph: &Pangraph) -> Pangraph { } } -pub fn self_merge(graph: Pangraph, args: &PangraphBuildArgs) -> Result<(Pangraph, bool), Report> { +pub fn self_merge(graph: Pangraph, args: &GraphMergeParams) -> Result<(Pangraph, bool), Report> { // use minimap2 or other aligners to find matches between the consensus // sequences of the blocks let matches = find_matches(&graph.blocks, args)?; @@ -175,7 +175,7 @@ pub fn self_merge(graph: Pangraph, args: &PangraphBuildArgs) -> Result<(Pangraph // Returns a list of alignment objects. pub fn find_matches( blocks: &BTreeMap, - args: &PangraphBuildArgs, + args: &GraphMergeParams, ) -> Result, Report> { match args.alignment_kernel { AlignmentBackend::Minimap2 => align_with_minimap2_lib(blocks, &args.aln_args), diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 16516162..b209f60a 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -2,19 +2,22 @@ use crate::io::fasta::FastaRecord; use crate::io::file::open_file_or_stdin; use crate::io::fs::read_reader_to_string; use crate::io::json::json_read_str; -use crate::make_internal_report; use crate::pangraph::pangraph_block::{BlockId, PangraphBlock}; use crate::pangraph::pangraph_node::{NodeId, PangraphNode}; use crate::pangraph::pangraph_path::{PangraphPath, PathId}; use crate::pangraph::strand::Strand; use crate::representation::seq::Seq; use crate::tree::clade::WithNewickName; +use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; +use crate::{make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; +use log::warn; use maplit::btreemap; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; use std::collections::{BTreeMap, BTreeSet}; +use std::mem::take; use std::path::Path; use std::str::FromStr; @@ -65,6 +68,115 @@ impl Pangraph { self.blocks.values().map(|block| block.consensus()) } + /// Re-derives every block, node and path id of this graph, in place. + /// + /// Block and node ids become `id((salt, old_id))`; path ids are renumbered contiguously from + /// `path_id_offset`, preserving their relative order (path ids double as the ordering index of + /// the genomes, so they are kept small and sequential rather than hashed). + /// + /// Consensuses, edits, names, descriptions, strands and positions are left untouched: the + /// relabeled graph describes exactly the same sequences as the original one. + pub fn relabel_in_place(&mut self, salt: usize, path_id_offset: usize) -> Result<(), Report> { + let (n_blocks, n_nodes, n_paths) = (self.blocks.len(), self.nodes.len(), self.paths.len()); + + let block_map: BTreeMap = + self.blocks.keys().map(|&bid| (bid, BlockId(id((salt, bid))))).collect(); + + let node_map: BTreeMap = self.nodes.keys().map(|&nid| (nid, NodeId(id((salt, nid))))).collect(); + + let path_map: BTreeMap = self + .paths + .keys() + .enumerate() + .map(|(rank, &pid)| (pid, PathId(path_id_offset + rank))) + .collect(); + + self.blocks = take(&mut self.blocks) + .into_values() + .map(|block| { + let new_block_id = block_map[&block.id()]; + let mut block = block.with_id(new_block_id); + let alignments = take(block.alignments_mut()) + .into_iter() + .map(|(nid, edit)| (node_map[&nid], edit)) + .collect(); + *block.alignments_mut() = alignments; + (block.id(), block) + }) + .collect(); + + self.nodes = take(&mut self.nodes) + .into_iter() + .map(|(nid, node)| { + let node = PangraphNode::new( + Some(node_map[&nid]), + block_map[&node.block_id()], + path_map[&node.path_id()], + node.strand(), + node.position(), + ); + (node.id(), node) + }) + .collect(); + + self.paths = take(&mut self.paths) + .into_values() + .map(|mut path| { + path.id = path_map[&path.id]; + for nid in &mut path.nodes { + *nid = node_map[nid]; + } + (path.id, path) + }) + .collect(); + + // An injective relabeling cannot change the number of entities. If it did, two distinct ids + // were mapped onto the same one and entities were silently dropped. + if (self.blocks.len(), self.nodes.len(), self.paths.len()) != (n_blocks, n_nodes, n_paths) { + return make_internal_error!( + "When relabeling graph ids: expected {n_blocks} blocks, {n_nodes} nodes and {n_paths} paths, but got {} blocks, {} nodes and {} paths", + self.blocks.len(), + self.nodes.len(), + self.paths.len(), + ); + } + + Ok(()) + } + + /// Returns true if this graph shares no block, node or path id with `other`. + pub fn is_id_disjoint_from(&self, other: &Self) -> bool { + self.blocks.keys().all(|bid| !other.blocks.contains_key(bid)) + && self.nodes.keys().all(|nid| !other.nodes.contains_key(nid)) + && self.paths.keys().all(|pid| !other.paths.contains_key(pid)) + } + + /// Relabels this graph so that it shares no identifier with `other`, which is left untouched. + /// + /// Two graphs built independently always collide: `Pangraph::singleton` labels the first genome + /// of every build with path, block and node id `0`, and blocks that never merge keep that id all + /// the way to the final graph. Merging therefore requires namespacing one of the two graphs + /// first. + pub fn make_disjoint_from(&mut self, other: &Self) -> Result<(), Report> { + const MAX_ATTEMPTS: usize = 16; + + let path_id_offset = other.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0); + + for salt in 1..=MAX_ATTEMPTS { + self.relabel_in_place(salt, path_id_offset)?; + if self.is_id_disjoint_from(other) { + return Ok(()); + } + // Relabeling is a composition of injective maps, so retrying on top of the previous attempt + // is safe. Reaching this point at all is astronomically unlikely. + warn!("Hash collision when relabeling graph ids with salt {salt}; retrying"); + } + + make_internal_error!( + "When making graphs id-disjoint: no collision-free relabeling found after {MAX_ATTEMPTS} attempts" + ) + } + pub fn update(&mut self, u: &GraphUpdate) { // Consistency check: node ids let old_nodes_set_from_graph: BTreeSet = self.blocks[&u.b_old_id].alignment_keys(); @@ -303,10 +415,13 @@ mod tests { #![allow(non_snake_case, clippy::redundant_clone)] use super::*; + use crate::commands::reconstruct::reconstruct_run::reconstruct; + use crate::o; use crate::pangraph::edits::Edit; use crate::pangraph::pangraph_node::PangraphNode; use crate::pangraph::pangraph_path::PangraphPath; use crate::pangraph::strand::Strand::{Forward, Reverse}; + use itertools::Itertools; use maplit::btreemap; use rstest::rstest; @@ -448,4 +563,94 @@ mod tests { let g = pangraph_with_named_paths(names); assert_eq!(g.newick_name(), expected); } + + /// Builds a two-genome graph whose ids are the small sequential integers that `build` assigns to + /// blocks that never merge. Two such graphs collide on every single id. + fn colliding_graph(names: [&str; 2]) -> Pangraph { + let blocks = btreemap! { + BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap!{ NodeId(0) => Edit::empty() }), + BlockId(1) => PangraphBlock::new(BlockId(1), "TTTTGGGG", btreemap!{ NodeId(1) => Edit::empty() }), + }; + let nodes = btreemap! { + NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Reverse, (0, 8)), + }; + let paths = btreemap! { + PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, Some(names[0].to_owned()), None), + PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 8, false, Some(names[1].to_owned()), None), + }; + Pangraph { paths, blocks, nodes } + } + + #[rstest] + fn test_relabel_keeps_graph_consistent() { + let mut graph = colliding_graph(["a", "b"]); + graph.relabel_in_place(1, 7).unwrap(); + + graph.sanity_check().unwrap(); + assert_eq!(graph.blocks.len(), 2); + assert_eq!(graph.nodes.len(), 2); + assert_eq!(graph.paths.len(), 2); + + // path ids are renumbered contiguously from the offset, in their original order + assert_eq!(graph.path_ids().collect_vec(), vec![PathId(7), PathId(8)]); + assert_eq!( + graph.paths.values().map(|p| p.name.clone()).collect_vec(), + vec![Some(o!("a")), Some(o!("b"))] + ); + + // block and node ids are re-derived, and no longer the original small integers + assert!(graph.block_ids().all(|bid| bid.0 > 1)); + assert!(graph.node_ids().all(|nid| nid.0 > 1)); + } + + #[rstest] + fn test_relabel_preserves_sequences() { + let original = colliding_graph(["a", "b"]); + let mut relabeled = original.clone(); + relabeled.relabel_in_place(3, 0).unwrap(); + + let seqs = |g: &Pangraph| { + reconstruct(g) + .map(|r| r.map(|r| (r.seq_name, r.seq))) + .collect::, Report>>() + .unwrap() + }; + + assert_eq!(seqs(&original), seqs(&relabeled)); + } + + #[rstest] + fn test_relabel_is_deterministic() { + let mut first = colliding_graph(["a", "b"]); + let mut second = colliding_graph(["a", "b"]); + first.relabel_in_place(2, 5).unwrap(); + second.relabel_in_place(2, 5).unwrap(); + assert_eq!(first, second); + } + + #[rstest] + fn test_make_disjoint_from() { + let left = colliding_graph(["a", "b"]); + let mut right = colliding_graph(["c", "d"]); + + // the two graphs share every single id before relabeling + assert!(!right.is_id_disjoint_from(&left)); + + right.make_disjoint_from(&left).unwrap(); + + assert!(right.is_id_disjoint_from(&left)); + right.sanity_check().unwrap(); + + // the left graph is untouched, and the right graph's genomes follow it + assert_eq!(left.path_ids().collect_vec(), vec![PathId(0), PathId(1)]); + assert_eq!(right.path_ids().collect_vec(), vec![PathId(2), PathId(3)]); + + // joining the two no longer conflicts + let joined = crate::pangraph::graph_merging::graph_join(&left, &right); + joined.sanity_check().unwrap(); + assert_eq!(joined.paths.len(), 4); + assert_eq!(joined.blocks.len(), 4); + assert_eq!(joined.nodes.len(), 4); + } } diff --git a/packages/pangraph/src/pangraph/pangraph_block.rs b/packages/pangraph/src/pangraph/pangraph_block.rs index 11a755d1..a5b6e6c0 100644 --- a/packages/pangraph/src/pangraph/pangraph_block.rs +++ b/packages/pangraph/src/pangraph/pangraph_block.rs @@ -1,5 +1,5 @@ +use crate::align::alignment_args::GraphMergeParams; use crate::align::map_variations::{BandParameters, map_variations}; -use crate::commands::build::build_args::PangraphBuildArgs; use crate::io::fasta::FastaRecord; use crate::io::json::{JsonPretty, json_write_str}; use crate::io::seq::reverse_complement; @@ -60,6 +60,11 @@ impl PangraphBlock { } } + /// Returns this block with a different id, leaving consensus and alignments untouched. + pub fn with_id(self, id: BlockId) -> Self { + Self { id, ..self } + } + pub fn reverse_complement(&self) -> Result { let rev_cons = reverse_complement(&self.consensus)?; @@ -292,7 +297,7 @@ impl PangraphBlock { /// Applies a set of edits to the block's consensus sequence and re-aligns the sequences /// to the new consensus. Returns a new `PangraphBlock` object with the same BlockId. - pub fn edit_consensus_and_realign(self, edits: &Edit, args: &PangraphBuildArgs) -> Result { + pub fn edit_consensus_and_realign(self, edits: &Edit, args: &GraphMergeParams) -> Result { // apply the edits to the consensus let new_consensus = edits.apply(&self.consensus)?; debug_assert!(!new_consensus.is_empty(), "Consensus cannot be empty"); @@ -342,7 +347,7 @@ pub enum RecordNaming { #[cfg(test)] mod tests { use super::*; - use crate::commands::build::build_args::PangraphBuildArgs; + use crate::align::alignment_args::GraphMergeParams; use crate::pangraph::edits::{Del, Edit, Ins, Sub}; use crate::pangraph::pangraph_node::NodeId; use maplit::btreemap; @@ -820,7 +825,7 @@ mod tests { ); // Create build args with default values for testing - let args = PangraphBuildArgs::default(); + let args = GraphMergeParams::default(); // Apply the edits and realign let result_block = block.edit_consensus_and_realign(&edits, &args).unwrap(); diff --git a/packages/pangraph/src/pangraph/reweave.rs b/packages/pangraph/src/pangraph/reweave.rs index 8364e183..8bc07213 100644 --- a/packages/pangraph/src/pangraph/reweave.rs +++ b/packages/pangraph/src/pangraph/reweave.rs @@ -1,7 +1,7 @@ use crate::align::alignment::{Alignment, AnchorBlock, ExtractedHit}; +use crate::align::alignment_args::GraphMergeParams; use crate::align::bam::cigar::{Side, add_flanking_indel, cigar_switch_ref_qry, invert_cigar}; use crate::align::map_variations::{BandParameters, map_variations}; -use crate::commands::build::build_args::PangraphBuildArgs; use crate::io::seq::reverse_complement; use crate::make_internal_error; use crate::pangraph::edits::Edit; @@ -37,7 +37,7 @@ impl MergePromise { } } - pub fn solve_promise(&mut self, args: &PangraphBuildArgs) -> Result { + pub fn solve_promise(&mut self, args: &GraphMergeParams) -> Result { // TODO: avoid re-aligning if cigar is only a single match (no indels) // calculate the mean shift and bandwidth of the alignment due to the displacement diff --git a/packages/pangraph/src/reconsensus/reconsensus.rs b/packages/pangraph/src/reconsensus/reconsensus.rs index 8b410b45..1a53bee2 100644 --- a/packages/pangraph/src/reconsensus/reconsensus.rs +++ b/packages/pangraph/src/reconsensus/reconsensus.rs @@ -1,4 +1,4 @@ -use crate::commands::build::build_args::PangraphBuildArgs; +use crate::align::alignment_args::GraphMergeParams; use crate::make_report; use crate::pangraph::detach_unaligned::detach_unaligned_nodes; use crate::pangraph::edits::Edit; @@ -32,7 +32,7 @@ struct BlockAnalysis { pub fn reconsensus_graph( graph: &mut Pangraph, ids_updated_blocks: &[BlockId], - args: &PangraphBuildArgs, + args: &GraphMergeParams, ) -> Result<(), Report> { // there should be no empty nodes in the graph debug_assert!( @@ -406,7 +406,7 @@ mod tests { let majority_edits = block.find_majority_edits(); assert!(majority_edits.has_indels()); // This block has indels requiring re-alignment let block = block - .edit_consensus_and_realign(&majority_edits, &PangraphBuildArgs::default()) + .edit_consensus_and_realign(&majority_edits, &GraphMergeParams::default()) .unwrap(); // Check that the re-alignment produced the expected result @@ -422,7 +422,7 @@ mod tests { let majority_edits = block.find_majority_edits(); assert!(majority_edits.has_indels()); // This block has indels requiring re-alignment let block = block - .edit_consensus_and_realign(&majority_edits, &PangraphBuildArgs::default()) + .edit_consensus_and_realign(&majority_edits, &GraphMergeParams::default()) .unwrap(); // Check that the re-alignment produced the expected result @@ -457,7 +457,7 @@ mod tests { paths, }; - let result = reconsensus_graph(&mut graph, &[block_id], &PangraphBuildArgs::default()); + let result = reconsensus_graph(&mut graph, &[block_id], &GraphMergeParams::default()); result.unwrap(); assert_eq!(graph.blocks[&block_id], expected_block); @@ -535,7 +535,7 @@ mod tests { let mut graph = Pangraph { paths, blocks, nodes }; // Apply reconsensus_graph - let result = reconsensus_graph(&mut graph, &[initial_block.id()], &PangraphBuildArgs::default()); + let result = reconsensus_graph(&mut graph, &[initial_block.id()], &GraphMergeParams::default()); // Check that the operation succeeded result.unwrap(); diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs new file mode 100644 index 00000000..88b8c7eb --- /dev/null +++ b/packages/pangraph/tests/itest_merge.rs @@ -0,0 +1,189 @@ +mod common; + +#[cfg(test)] +mod tests { + use eyre::Report; + use itertools::Itertools; + use pangraph::align::alignment_args::GraphMergeParams; + use pangraph::commands::build::build_args::PangraphBuildArgs; + use pangraph::commands::build::build_run::build; + use pangraph::commands::merge::merge_args::PangraphMergeArgs; + use pangraph::commands::merge::merge_run::merge_run; + use pangraph::commands::reconstruct::reconstruct_run::reconstruct; + use pangraph::io::fasta::{FastaReader, FastaRecord}; + use pangraph::io::json::{JsonPretty, json_write_file}; + use pangraph::pangraph::pangraph::Pangraph; + use pangraph::representation::seq::Seq; + use pangraph::utils::error::report_to_string; + use pretty_assertions::assert_eq; + use rstest::rstest; + use std::collections::{BTreeMap, BTreeSet}; + use std::path::{Path, PathBuf}; + use tempfile::{TempDir, tempdir}; + + /// Reads the first `n_total` records of a FASTA file and splits them in two groups, so that the + /// two resulting graphs are built from disjoint but homologous sets of genomes. + fn read_and_split(path: &str, n_left: usize, n_total: usize) -> Result<(Vec, Vec), Report> { + let mut fastas = FastaReader::from_paths(&[PathBuf::from(path)])?.read_many()?; + fastas.truncate(n_total); + let right = fastas.split_off(n_left); + Ok((fastas, right)) + } + + /// Builds a graph out of the given records and writes it to a JSON file in `dir`. + fn build_graph_file(dir: &TempDir, name: &str, fastas: Vec) -> Result { + let args = PangraphBuildArgs { + circular: false, + ..PangraphBuildArgs::default() + }; + let graph = build(fastas, &args, false)?; + let path = dir.path().join(name); + json_write_file(&path, &graph, JsonPretty(false))?; + Ok(path) + } + + /// Reconstructs the genomes of a graph, keyed by genome name. + fn sequences(graph: &Pangraph) -> Result, Report> { + reconstruct(graph).map(|r| r.map(|r| (r.seq_name, r.seq))).collect() + } + + fn read_graph(path: &Path) -> Result { + Pangraph::from_path(&Some(path)) + } + + fn merge_args(left: PathBuf, right: PathBuf, output: PathBuf) -> PangraphMergeArgs { + PangraphMergeArgs { + left_graph: left, + right_graph: right, + output_json: output, + merge_params: GraphMergeParams::default(), + verify: true, + } + } + + /// Two graphs built independently from homologous genomes merge into a graph that reconstructs + /// every input genome exactly. This is the main end-to-end guarantee of the `merge` command. + #[rstest] + fn itest_merge_homologous_graphs() -> Result<(), Report> { + let dir = tempdir()?; + let (left_fastas, right_fastas) = read_and_split("../../data/ges-1.fa", 3, 6)?; + let left_names: BTreeSet = left_fastas.iter().map(|f| f.seq_name.clone()).collect(); + let right_names: BTreeSet = right_fastas.iter().map(|f| f.seq_name.clone()).collect(); + let expected_names = left_names.union(&right_names).cloned().collect_vec(); + + let left = build_graph_file(&dir, "left.json", left_fastas)?; + let right = build_graph_file(&dir, "right.json", right_fastas)?; + let output = dir.path().join("merged.json"); + + // `--verify` is on, so this already checks that every genome round-trips + merge_run(&merge_args(left.clone(), right.clone(), output.clone()))?; + + let merged = read_graph(&output)?; + merged.sanity_check()?; + + // all genomes of both inputs are present, exactly once + assert_eq!(merged.paths.len(), 6); + assert_eq!( + sequences(&merged)?.keys().cloned().sorted().collect_vec(), + expected_names + ); + + // and their sequences are unchanged + let mut expected = sequences(&read_graph(&left)?)?; + expected.extend(sequences(&read_graph(&right)?)?); + assert_eq!(sequences(&merged)?, expected); + + // merging actually found homology across the two graphs: some blocks are now shared between + // genomes that came from different inputs. (Block counts alone say little: reweaving splits + // blocks as it merges them, so the total can go either way.) + let cross_graph_blocks = merged + .blocks + .values() + .filter(|block| { + let names = block + .alignment_keys() + .iter() + .filter_map(|nid| merged.paths[&merged.nodes[nid].path_id()].name.clone()) + .collect_vec(); + names.iter().any(|n| left_names.contains(n)) && names.iter().any(|n| right_names.contains(n)) + }) + .count(); + assert!( + cross_graph_blocks > 0, + "no block is shared between the two input graphs" + ); + + Ok(()) + } + + /// Appending a single genome to an existing graph: the case the command is meant to serve. + #[rstest] + fn itest_merge_single_genome() -> Result<(), Report> { + let dir = tempdir()?; + let (left_fastas, right_fastas) = read_and_split("../../data/ges-1.fa", 3, 4)?; + assert_eq!(right_fastas.len(), 1); + + let left = build_graph_file(&dir, "left.json", left_fastas)?; + let right = build_graph_file(&dir, "right.json", right_fastas)?; + let output = dir.path().join("merged.json"); + + merge_run(&merge_args(left, right, output.clone()))?; + + let merged = read_graph(&output)?; + merged.sanity_check()?; + assert_eq!(merged.paths.len(), 4); + + Ok(()) + } + + /// Genomes with no homology at all: the merger degenerates to a plain join. This is the case in + /// which the small sequential ids assigned by `Pangraph::singleton` survive into the final + /// graphs and collide, so it exercises the relabeling most sharply. + #[rstest] + fn itest_merge_unrelated_graphs() -> Result<(), Report> { + let dir = tempdir()?; + let (flu, _) = read_and_split("../../data/flu-h1.fa", 1, 1)?; + let (mpox, _) = read_and_split("../../data/mpox.fa", 1, 1)?; + + let left = build_graph_file(&dir, "left.json", flu)?; + let right = build_graph_file(&dir, "right.json", mpox)?; + + // both graphs are made of untouched singleton blocks, with identical ids + let left_graph = read_graph(&left)?; + let right_graph = read_graph(&right)?; + assert!(!right_graph.is_id_disjoint_from(&left_graph)); + + let output = dir.path().join("merged.json"); + merge_run(&merge_args(left, right, output.clone()))?; + + let merged = read_graph(&output)?; + merged.sanity_check()?; + assert_eq!(merged.paths.len(), 2); + assert_eq!(merged.blocks.len(), 2); + + let mut expected = sequences(&left_graph)?; + expected.extend(sequences(&right_graph)?); + assert_eq!(sequences(&merged)?, expected); + + Ok(()) + } + + /// Merging a graph with itself duplicates every genome name, and must be rejected. + #[rstest] + fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { + let dir = tempdir()?; + let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; + let graph = build_graph_file(&dir, "graph.json", fastas)?; + let output = dir.path().join("merged.json"); + + let result = merge_run(&merge_args(graph.clone(), graph, output)); + + let error = report_to_string(&result.unwrap_err()); + assert!( + error.contains("Duplicate genome names"), + "unexpected error message: {error}" + ); + + Ok(()) + } +} From 82d126e09957224ee44659373a98a62b82c8683f Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 15:15:53 +0200 Subject: [PATCH 04/50] feat(build): reject duplicate genome names, shared with merge --- dev/design/merge.md | 41 +++++++++------- .../pangraph/src/commands/build/build_run.rs | 19 +++++++- .../pangraph/src/commands/merge/merge_run.rs | 20 ++------ packages/pangraph/src/utils/collections.rs | 16 ++++++- packages/pangraph/tests/itest_merge.rs | 48 +++++++------------ 5 files changed, 78 insertions(+), 66 deletions(-) diff --git a/dev/design/merge.md b/dev/design/merge.md index 5d7f94a3..320113d7 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -260,14 +260,21 @@ used. Decision: **duplicate path names are a hard error, in both `build` and `merge`.** -- `build`: after reading the input FASTA records, error if two records share a name. This makes the - existing `--guide-tree` behaviour unconditional and fills the TODO. +- `build`: error if two input FASTA records share a name. The check lives in `build()` rather than + `build_run()`, so that it applies to every caller of the library entry point and not only to the + CLI. This makes the existing `--guide-tree` behaviour unconditional and fills the TODO. - `merge`: error if a name occurs more than once within either input graph or across the two. This is what catches the common mistakes of merging a graph with itself, or re-adding a genome that is already present in the base graph — which would otherwise silently produce two identically named paths and break `export` and `simplify` downstream. -Error messages should list the offending names, not just the first one. +Error messages list all offending names, not just the first one. + +The two commands share the primitive, `utils::collections::find_duplicates`, but not the check +itself: the surrounding messages differ (input FASTA records versus genomes of two graphs, with the +merge one hinting at the self-merge mistake), and each is two lines around the shared call. Sharing +a single `check_unique_names(names, context)` would only trade those two lines for a context +parameter threaded through to the message. Paths with no name at all are left alone by the duplicate check (multiple unnamed paths are not an error), but `--verify` requires every path to be named, because names are the verification key @@ -363,21 +370,17 @@ merged into `master` last, once all phases have landed. | 2 | `feat/merge-cmd` | §4.2 — extract `GraphMergeParams`, no behaviour change | landed | | 3 | `feat/merge-cmd` | §3.4 — `relabel_in_place` / `make_disjoint_from` | landed | | 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | -| 5 | `feat/merge-unique-names` | §4.3 — duplicate path names are an error in `build` too | todo | +| 5 | `feat/merge-cmd` | §4.3 — duplicate genome names are an error in `build` and `merge` | landed | | 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | todo | | 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | todo | -Phases 2–4 were implemented together, since a `merge` command without §3.4 panics on the first +Phases 2–5 were implemented together, since a `merge` command without §3.4 panics on the first identifier collision and would not be testable. -Two pieces of §4.3 and §4.4 are therefore only half-done, and phases 5 and 6 finish them: - -- Duplicate path names are rejected by `merge` (across and within its two inputs), but `build` still - accepts duplicate FASTA names except when `--guide-tree` is used. -- Name-keyed verification exists as `verify_merged_sequences` inside `merge_run`, private to the - merge command. `build` still verifies against input FASTA records by index, and - `compare_sequences` still compares whole `FastaRecord`s (including `index`). Phase 6 unifies the - two behind a single `BTreeMap`-based helper. +§4.4 is therefore the only piece left half-done: name-keyed verification exists as +`verify_merged_sequences` inside `merge_run`, private to the merge command. `build` still verifies +against input FASTA records by index, and `compare_sequences` still compares whole `FastaRecord`s +(including `index`). Phase 6 unifies the two behind a single `BTreeMap`-based helper. --- @@ -416,9 +419,15 @@ Two pieces of §4.3 and §4.4 are therefore only half-done, and phases 5 and 6 f sequence is reconstructed byte-identically (matched **by name**, not by order); `sanity_check` passes; the path count is `nA + nB`; all path names are preserved. - A single-genome right graph (exercises phase 1). -- Two graphs with no shared homology — the merge degenerates to a pure join, and this is also the - case that produces surviving small integer ids, so it is the sharpest test of §3. -- Merging a graph with itself must fail with a duplicate-name error. +- Merging a graph with itself must fail with a duplicate-name error, and `build` must reject + duplicate names in its input FASTA records. + +Deliberately **not** tested at the integration level: two graphs with no shared homology, as a way +to exercise the surviving small integer ids of §3.2. Whether two genomes have *no* detectable +homology is a property of the aligner and its parameters, not something a test can pin down — even +distant genomes may share a short segment. The colliding-id case is covered directly and +deterministically by the `make_disjoint_from` unit tests, which construct graphs with ids `0..2` by +hand. --- diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index 0d59289c..731e3bee 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -9,8 +9,9 @@ use crate::pangraph::strand::Strand::Forward; use crate::tree::clade::postorder; use crate::tree::neighbor_joining::build_tree_using_neighbor_joining; use crate::tree::newick::build_tree_from_newick; +use crate::utils::collections::find_duplicates; use crate::utils::progress_bar::ProgressBar; -use crate::{make_internal_error, make_internal_report}; +use crate::{make_error, make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::info; @@ -50,7 +51,6 @@ pub fn build_run(args: &PangraphBuildArgs) -> Result<(), Report> { let fastas = FastaReader::from_paths(input_fastas)?.read_many()?; // TODO: adjust fasta letter case if `upper_case` is set - // TODO: check for duplicate fasta names check_alignment_backend_available(&args.merge_params) .wrap_err("When performing preliminary checks before building the pangraph.")?; @@ -62,7 +62,22 @@ pub fn build_run(args: &PangraphBuildArgs) -> Result<(), Report> { Ok(()) } +/// Sequence names identify genomes throughout pangraph: they are what `export` and `simplify` +/// resolve genomes by, and the only handle that survives a graph merger. They must be unique. +pub fn check_unique_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { + let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); + if !duplicates.is_empty() { + return make_error!( + "Duplicate sequence names found in the input: [{}]. Sequence names must be unique, because they identify genomes in the resulting pangraph.", + duplicates.join(", ") + ); + } + Ok(()) +} + pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) -> Result { + check_unique_sequence_names(&fastas).wrap_err("When checking the names of the input sequences")?; + // If verification is requested, we need to keep a copy of the original FASTA records // to compare them with the sequences reconstructed from the graph. let fasta_copy = verify.then(|| fastas.clone()); diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 2c71d8a0..c65fd849 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -7,6 +7,7 @@ use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; use crate::representation::seq::Seq; +use crate::utils::collections::find_duplicates; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::{info, warn}; @@ -80,9 +81,9 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right } } - // Path names are the identity of a genome throughout pangraph: they are what `export` and - // `simplify` resolve genomes by, and the only handle that survives a merge unchanged. - let duplicates = duplicate_path_names(&[left, right]); + // Genome names are the identity of a genome throughout pangraph, and the only handle that + // survives a merge unchanged. `build` enforces the same invariant on its input FASTA records. + let duplicates = find_duplicates([left, right].into_iter().flat_map(|graph| graph.path_names().flatten())); if !duplicates.is_empty() { return make_error!( "Duplicate genome names found in the input graphs: [{}]. Genome names must be unique across the two graphs: merging a graph with itself, or re-adding a genome that is already present, is not supported.", @@ -112,19 +113,6 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right Ok(()) } -/// Returns the sorted path names that occur more than once across the given graphs. -fn duplicate_path_names(graphs: &[&Pangraph]) -> Vec { - graphs - .iter() - .flat_map(|graph| graph.path_names().flatten()) - .counts() - .into_iter() - .filter(|(_, count)| *count > 1) - .map(|(name, _)| name.to_owned()) - .sorted() - .collect() -} - /// Returns the set of circularity flags used by the paths of a graph. fn circularity(graph: &Pangraph) -> BTreeSet { graph.paths().map(PangraphPath::circular).collect() diff --git a/packages/pangraph/src/utils/collections.rs b/packages/pangraph/src/utils/collections.rs index add05a54..fe1f49f4 100644 --- a/packages/pangraph/src/utils/collections.rs +++ b/packages/pangraph/src/utils/collections.rs @@ -1,6 +1,6 @@ use crate::{make_error, make_internal_report}; use eyre::Report; -use std::collections::HashSet; +use std::collections::{BTreeMap, HashSet}; use std::hash::Hash; pub fn concat_to_vec(x: &[T], y: &[T]) -> Vec { @@ -32,3 +32,17 @@ pub fn has_duplicates>(iter: I) -> bool let mut seen = HashSet::new(); iter.into_iter().any(|item| !seen.insert(item)) } + +/// Returns the values that occur more than once in the given iterator, in sorted order. +/// Each duplicated value is reported once, no matter how many times it occurs. +pub fn find_duplicates>(iter: I) -> Vec { + let mut counts: BTreeMap = BTreeMap::new(); + for item in iter { + *counts.entry(item).or_insert(0) += 1; + } + counts + .into_iter() + .filter(|(_, count)| *count > 1) + .map(|(item, _)| item) + .collect() +} diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 88b8c7eb..c9b50344 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -136,51 +136,37 @@ mod tests { Ok(()) } - /// Genomes with no homology at all: the merger degenerates to a plain join. This is the case in - /// which the small sequential ids assigned by `Pangraph::singleton` survive into the final - /// graphs and collide, so it exercises the relabeling most sharply. + /// Merging a graph with itself duplicates every genome name, and must be rejected. #[rstest] - fn itest_merge_unrelated_graphs() -> Result<(), Report> { + fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { let dir = tempdir()?; - let (flu, _) = read_and_split("../../data/flu-h1.fa", 1, 1)?; - let (mpox, _) = read_and_split("../../data/mpox.fa", 1, 1)?; - - let left = build_graph_file(&dir, "left.json", flu)?; - let right = build_graph_file(&dir, "right.json", mpox)?; - - // both graphs are made of untouched singleton blocks, with identical ids - let left_graph = read_graph(&left)?; - let right_graph = read_graph(&right)?; - assert!(!right_graph.is_id_disjoint_from(&left_graph)); - + let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; + let graph = build_graph_file(&dir, "graph.json", fastas)?; let output = dir.path().join("merged.json"); - merge_run(&merge_args(left, right, output.clone()))?; - let merged = read_graph(&output)?; - merged.sanity_check()?; - assert_eq!(merged.paths.len(), 2); - assert_eq!(merged.blocks.len(), 2); + let result = merge_run(&merge_args(graph.clone(), graph, output)); - let mut expected = sequences(&left_graph)?; - expected.extend(sequences(&right_graph)?); - assert_eq!(sequences(&merged)?, expected); + let error = report_to_string(&result.unwrap_err()); + assert!( + error.contains("Duplicate genome names"), + "unexpected error message: {error}" + ); Ok(()) } - /// Merging a graph with itself duplicates every genome name, and must be rejected. + /// `build` enforces the same uniqueness invariant on its input FASTA records, so that a graph + /// can never carry duplicate genome names into a later merge. #[rstest] - fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { - let dir = tempdir()?; - let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; - let graph = build_graph_file(&dir, "graph.json", fastas)?; - let output = dir.path().join("merged.json"); + fn itest_build_rejects_duplicate_sequence_names() -> Result<(), Report> { + let (mut fastas, _) = read_and_split("../../data/ges-1.fa", 2, 2)?; + fastas[1].seq_name = fastas[0].seq_name.clone(); - let result = merge_run(&merge_args(graph.clone(), graph, output)); + let result = build(fastas, &PangraphBuildArgs::default(), false); let error = report_to_string(&result.unwrap_err()); assert!( - error.contains("Duplicate genome names"), + error.contains("Duplicate sequence names"), "unexpected error message: {error}" ); From 2a599b1c3d39ce9d0bad272e2013788012229971 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 15:35:04 +0200 Subject: [PATCH 05/50] refactor(merge): drop unreachable relabeling retry loop --- dev/design/merge.md | 11 +++-- packages/pangraph/src/pangraph/pangraph.rs | 47 ++++++++++++++++------ 2 files changed, 42 insertions(+), 16 deletions(-) diff --git a/dev/design/merge.md b/dev/design/merge.md index 320113d7..262bd8bc 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -185,12 +185,17 @@ preserved either way. `PangraphNode::{block_id, path_id}`, and `PangraphPath::nodes`. - **Assert injectivity.** After relabeling, the three maps must have the same lengths as before (a hash collision inside one graph would silently drop entities) and the resulting id sets must be - disjoint from the left graph's. `make_disjoint` bumps the salt and retries if not; this is - astronomically unlikely with `XxHash64`, but the check is cheap. + disjoint from the left graph's. Both are hard errors: at ~2^-64 per pair neither happens in + practice, but the alternative is one graph silently overwriting a block of the other during the + join. There is deliberately no retry-with-another-salt path — it would be unreachable code + guarding against an event that cannot realistically occur. - Path ids are assigned as `path_id_offset + rank`, where `rank` is the position in the graph's existing (sorted) path key order, and `path_id_offset` is `max(left path ids) + 1`. This keeps ids contiguous and keeps the right graph's genomes in their original relative order, after all of - the left graph's. + the left graph's. It also means **path ids cannot collide at all** — every one of them is + strictly greater than every path id of the left graph — so only block and node ids are at risk. + Since the assignment is by rank rather than derived from the previous id, relabeling the same + graph twice does not shift its path ids any further. - Relabeling is deterministic: `utils::id::id` uses `XxHash64` with a fixed seed. --- diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index b209f60a..02f5b9b8 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -12,7 +12,6 @@ use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; use crate::{make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; -use log::warn; use maplit::btreemap; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; @@ -74,6 +73,9 @@ impl Pangraph { /// `path_id_offset`, preserving their relative order (path ids double as the ordering index of /// the genomes, so they are kept small and sequential rather than hashed). /// + /// Path ids are assigned from the offset by rank rather than derived from the previous id, so + /// relabeling the same graph twice does not shift them any further. + /// /// Consensuses, edits, names, descriptions, strands and positions are left untouched: the /// relabeled graph describes exactly the same sequences as the original one. pub fn relabel_in_place(&mut self, salt: usize, path_id_offset: usize) -> Result<(), Report> { @@ -158,23 +160,24 @@ impl Pangraph { /// the way to the final graph. Merging therefore requires namespacing one of the two graphs /// first. pub fn make_disjoint_from(&mut self, other: &Self) -> Result<(), Report> { - const MAX_ATTEMPTS: usize = 16; + // Namespace under which the ids of this graph are re-derived. + const SALT: usize = 1; + // Path ids are assigned above every path id of `other`, so they cannot collide by construction. let path_id_offset = other.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0); - for salt in 1..=MAX_ATTEMPTS { - self.relabel_in_place(salt, path_id_offset)?; - if self.is_id_disjoint_from(other) { - return Ok(()); - } - // Relabeling is a composition of injective maps, so retrying on top of the previous attempt - // is safe. Reaching this point at all is astronomically unlikely. - warn!("Hash collision when relabeling graph ids with salt {salt}; retrying"); + self.relabel_in_place(SALT, path_id_offset)?; + + // Block and node ids are hashes, so a collision with `other` is possible in principle. At + // ~2^-64 per pair it does not happen in practice, but the check is cheap and the alternative is + // one graph silently overwriting a block of the other during the join. + if !self.is_id_disjoint_from(other) { + return make_internal_error!( + "When making graphs id-disjoint: the relabeled graph still shares block or node ids with the other graph. This requires a 64-bit hash collision and should never happen." + ); } - make_internal_error!( - "When making graphs id-disjoint: no collision-free relabeling found after {MAX_ATTEMPTS} attempts" - ) + Ok(()) } pub fn update(&mut self, u: &GraphUpdate) { @@ -620,6 +623,24 @@ mod tests { assert_eq!(seqs(&original), seqs(&relabeled)); } + /// Path ids are assigned from the offset by rank, not derived from the previous id, so relabeling + /// twice leaves them where they are instead of shifting them by the offset again. + #[rstest] + fn test_repeated_relabel_does_not_shift_path_ids() { + let mut graph = colliding_graph(["a", "b"]); + + graph.relabel_in_place(1, 5).unwrap(); + let path_ids = graph.path_ids().collect_vec(); + let block_ids = graph.block_ids().collect_vec(); + assert_eq!(path_ids, vec![PathId(5), PathId(6)]); + + graph.relabel_in_place(2, 5).unwrap(); + + assert_eq!(graph.path_ids().collect_vec(), path_ids); + assert_ne!(graph.block_ids().collect_vec(), block_ids); + graph.sanity_check().unwrap(); + } + #[rstest] fn test_relabel_is_deterministic() { let mut first = colliding_graph(["a", "b"]); From 5c9645f92c75c5d89c4d7e30383e6fd771bea2d4 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 15:39:53 +0200 Subject: [PATCH 06/50] refactor(pangraph): relabel graphs by value instead of in place --- dev/design/merge.md | 22 ++--- .../pangraph/src/commands/merge/merge_run.rs | 4 +- packages/pangraph/src/pangraph/pangraph.rs | 89 ++++++------------- .../pangraph/src/pangraph/pangraph_block.rs | 15 +++- 4 files changed, 55 insertions(+), 75 deletions(-) diff --git a/dev/design/merge.md b/dev/design/merge.md index 262bd8bc..7b9e7ee2 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -155,10 +155,10 @@ the left-hand one. The **left graph is never modified**. /// `id((salt, old_id))`; path ids are renumbered contiguously from `path_id_offset`, /// preserving their relative order. Consensuses, edits, names, descriptions, strands /// and positions are unchanged. -fn relabel_in_place(&mut self, salt: usize, path_id_offset: usize) +pub fn relabel(self, salt: usize, path_id_offset: usize) -> Result -/// Makes `right` id-disjoint from `left`. `left` is left untouched. -pub fn make_disjoint(left: &Pangraph, right: &mut Pangraph) -> Result<(), Report> +/// Relabels this graph so that it shares no identifier with `other`, which is untouched. +pub fn make_disjoint_from(self, other: &Pangraph) -> Result ``` Rationale for relabeling *all* of the right graph's block and node ids, rather than only the ones @@ -176,11 +176,13 @@ preserved either way. ### 3.5 Implementation notes -- **Relabel in place.** Block consensuses are the bulk of a graph's memory; building a relabeled - copy would double peak usage on a large base graph. Use `std::mem::take` on the three maps and - rebuild them in place. -- No new constructors are needed: `PangraphBlock::new`, `PangraphNode::new(Some(id), …)` and - `PangraphPath::new(Some(id), …)` all accept an explicit id. +- **Move, don't clone.** Block consensuses are the bulk of a graph's memory; relabeling a cloned + copy would double peak usage on a large base graph. `relabel` therefore takes the graph by value + and destructures it (`let Self { paths, blocks, nodes } = self;`), so each map can be drained + into its replacement and the sequences are moved rather than copied. +- `PangraphNode::new(Some(id), …)` and `PangraphPath::new(Some(id), …)` already accept an explicit + id. Blocks own their consensus behind private fields, so `PangraphBlock::relabel(self, id, + node_map)` does the equivalent from inside the type, moving the consensus and the edits across. - References to rewrite: the three map keys, `PangraphBlock::alignments` keys, `PangraphNode::{block_id, path_id}`, and `PangraphPath::nodes`. - **Assert injectivity.** After relabeling, the three maps must have the same lengths as before (a @@ -373,7 +375,7 @@ merged into `master` last, once all phases have landed. |---|---|---|---| | 1 | `feat/merge-single-seq-build` | §4.1 — NJ tree for 1 and 0 input graphs | landed | | 2 | `feat/merge-cmd` | §4.2 — extract `GraphMergeParams`, no behaviour change | landed | -| 3 | `feat/merge-cmd` | §3.4 — `relabel_in_place` / `make_disjoint_from` | landed | +| 3 | `feat/merge-cmd` | §3.4 — `relabel` / `make_disjoint_from` | landed | | 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | | 5 | `feat/merge-cmd` | §4.3 — duplicate genome names are an error in `build` and `merge` | landed | | 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | todo | @@ -410,7 +412,7 @@ against input FASTA records by index, and `compare_sequences` still compares who **Unit** -- `relabel_in_place` / `make_disjoint`: two hand-built graphs with identical ids `0..2` become +- `relabel` / `make_disjoint_from`: two hand-built graphs with identical ids `0..2` become joinable, and `sanity_check` passes on the join. - Relabeling is an isomorphism: sequences reconstructed from the graph, keyed by path name, are unchanged by relabeling. diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index c65fd849..7782b040 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -16,13 +16,13 @@ use std::path::Path; pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { let left = read_graph(&args.left_graph).wrap_err("When reading the first input graph")?; - let mut right = read_graph(&args.right_graph).wrap_err("When reading the second input graph")?; + let right = read_graph(&args.right_graph).wrap_err("When reading the second input graph")?; merge_cmd_preliminary_checks(args, &left, &right).wrap_err("When performing preliminary checks before merging")?; // The two graphs were built independently, so their identifiers almost certainly collide. // Namespace the second graph before joining them. - right + let right = right .make_disjoint_from(&left) .wrap_err("When making the identifiers of the two input graphs disjoint")?; diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 02f5b9b8..77e26c61 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -16,7 +16,6 @@ use maplit::btreemap; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; use std::collections::{BTreeMap, BTreeSet}; -use std::mem::take; use std::path::Path; use std::str::FromStr; @@ -67,47 +66,39 @@ impl Pangraph { self.blocks.values().map(|block| block.consensus()) } - /// Re-derives every block, node and path id of this graph, in place. + /// Returns this graph with every block, node and path id re-derived. /// /// Block and node ids become `id((salt, old_id))`; path ids are renumbered contiguously from /// `path_id_offset`, preserving their relative order (path ids double as the ordering index of /// the genomes, so they are kept small and sequential rather than hashed). /// - /// Path ids are assigned from the offset by rank rather than derived from the previous id, so - /// relabeling the same graph twice does not shift them any further. - /// - /// Consensuses, edits, names, descriptions, strands and positions are left untouched: the - /// relabeled graph describes exactly the same sequences as the original one. - pub fn relabel_in_place(&mut self, salt: usize, path_id_offset: usize) -> Result<(), Report> { - let (n_blocks, n_nodes, n_paths) = (self.blocks.len(), self.nodes.len(), self.paths.len()); + /// Consensuses, edits, names, descriptions, strands and positions are moved over untouched: the + /// relabeled graph describes exactly the same sequences as the original one. The graph is taken + /// by value so that the sequences can be moved rather than copied. + pub fn relabel(self, salt: usize, path_id_offset: usize) -> Result { + let Self { paths, blocks, nodes } = self; + let (n_blocks, n_nodes, n_paths) = (blocks.len(), nodes.len(), paths.len()); - let block_map: BTreeMap = - self.blocks.keys().map(|&bid| (bid, BlockId(id((salt, bid))))).collect(); + let block_map: BTreeMap = blocks.keys().map(|&bid| (bid, BlockId(id((salt, bid))))).collect(); - let node_map: BTreeMap = self.nodes.keys().map(|&nid| (nid, NodeId(id((salt, nid))))).collect(); + let node_map: BTreeMap = nodes.keys().map(|&nid| (nid, NodeId(id((salt, nid))))).collect(); - let path_map: BTreeMap = self - .paths + let path_map: BTreeMap = paths .keys() .enumerate() .map(|(rank, &pid)| (pid, PathId(path_id_offset + rank))) .collect(); - self.blocks = take(&mut self.blocks) + let blocks: BTreeMap = blocks .into_values() .map(|block| { let new_block_id = block_map[&block.id()]; - let mut block = block.with_id(new_block_id); - let alignments = take(block.alignments_mut()) - .into_iter() - .map(|(nid, edit)| (node_map[&nid], edit)) - .collect(); - *block.alignments_mut() = alignments; + let block = block.relabel(new_block_id, &node_map); (block.id(), block) }) .collect(); - self.nodes = take(&mut self.nodes) + let nodes: BTreeMap = nodes .into_iter() .map(|(nid, node)| { let node = PangraphNode::new( @@ -121,7 +112,7 @@ impl Pangraph { }) .collect(); - self.paths = take(&mut self.paths) + let paths: BTreeMap = paths .into_values() .map(|mut path| { path.id = path_map[&path.id]; @@ -134,16 +125,16 @@ impl Pangraph { // An injective relabeling cannot change the number of entities. If it did, two distinct ids // were mapped onto the same one and entities were silently dropped. - if (self.blocks.len(), self.nodes.len(), self.paths.len()) != (n_blocks, n_nodes, n_paths) { + if (blocks.len(), nodes.len(), paths.len()) != (n_blocks, n_nodes, n_paths) { return make_internal_error!( "When relabeling graph ids: expected {n_blocks} blocks, {n_nodes} nodes and {n_paths} paths, but got {} blocks, {} nodes and {} paths", - self.blocks.len(), - self.nodes.len(), - self.paths.len(), + blocks.len(), + nodes.len(), + paths.len(), ); } - Ok(()) + Ok(Self { paths, blocks, nodes }) } /// Returns true if this graph shares no block, node or path id with `other`. @@ -159,25 +150,25 @@ impl Pangraph { /// of every build with path, block and node id `0`, and blocks that never merge keep that id all /// the way to the final graph. Merging therefore requires namespacing one of the two graphs /// first. - pub fn make_disjoint_from(&mut self, other: &Self) -> Result<(), Report> { + pub fn make_disjoint_from(self, other: &Self) -> Result { // Namespace under which the ids of this graph are re-derived. const SALT: usize = 1; // Path ids are assigned above every path id of `other`, so they cannot collide by construction. let path_id_offset = other.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0); - self.relabel_in_place(SALT, path_id_offset)?; + let relabeled = self.relabel(SALT, path_id_offset)?; // Block and node ids are hashes, so a collision with `other` is possible in principle. At // ~2^-64 per pair it does not happen in practice, but the check is cheap and the alternative is // one graph silently overwriting a block of the other during the join. - if !self.is_id_disjoint_from(other) { + if !relabeled.is_id_disjoint_from(other) { return make_internal_error!( "When making graphs id-disjoint: the relabeled graph still shares block or node ids with the other graph. This requires a 64-bit hash collision and should never happen." ); } - Ok(()) + Ok(relabeled) } pub fn update(&mut self, u: &GraphUpdate) { @@ -587,8 +578,7 @@ mod tests { #[rstest] fn test_relabel_keeps_graph_consistent() { - let mut graph = colliding_graph(["a", "b"]); - graph.relabel_in_place(1, 7).unwrap(); + let graph = colliding_graph(["a", "b"]).relabel(1, 7).unwrap(); graph.sanity_check().unwrap(); assert_eq!(graph.blocks.len(), 2); @@ -610,8 +600,7 @@ mod tests { #[rstest] fn test_relabel_preserves_sequences() { let original = colliding_graph(["a", "b"]); - let mut relabeled = original.clone(); - relabeled.relabel_in_place(3, 0).unwrap(); + let relabeled = original.clone().relabel(3, 0).unwrap(); let seqs = |g: &Pangraph| { reconstruct(g) @@ -623,42 +612,22 @@ mod tests { assert_eq!(seqs(&original), seqs(&relabeled)); } - /// Path ids are assigned from the offset by rank, not derived from the previous id, so relabeling - /// twice leaves them where they are instead of shifting them by the offset again. - #[rstest] - fn test_repeated_relabel_does_not_shift_path_ids() { - let mut graph = colliding_graph(["a", "b"]); - - graph.relabel_in_place(1, 5).unwrap(); - let path_ids = graph.path_ids().collect_vec(); - let block_ids = graph.block_ids().collect_vec(); - assert_eq!(path_ids, vec![PathId(5), PathId(6)]); - - graph.relabel_in_place(2, 5).unwrap(); - - assert_eq!(graph.path_ids().collect_vec(), path_ids); - assert_ne!(graph.block_ids().collect_vec(), block_ids); - graph.sanity_check().unwrap(); - } - #[rstest] fn test_relabel_is_deterministic() { - let mut first = colliding_graph(["a", "b"]); - let mut second = colliding_graph(["a", "b"]); - first.relabel_in_place(2, 5).unwrap(); - second.relabel_in_place(2, 5).unwrap(); + let first = colliding_graph(["a", "b"]).relabel(2, 5).unwrap(); + let second = colliding_graph(["a", "b"]).relabel(2, 5).unwrap(); assert_eq!(first, second); } #[rstest] fn test_make_disjoint_from() { let left = colliding_graph(["a", "b"]); - let mut right = colliding_graph(["c", "d"]); + let right = colliding_graph(["c", "d"]); // the two graphs share every single id before relabeling assert!(!right.is_id_disjoint_from(&left)); - right.make_disjoint_from(&left).unwrap(); + let right = right.make_disjoint_from(&left).unwrap(); assert!(right.is_id_disjoint_from(&left)); right.sanity_check().unwrap(); diff --git a/packages/pangraph/src/pangraph/pangraph_block.rs b/packages/pangraph/src/pangraph/pangraph_block.rs index a5b6e6c0..4c706127 100644 --- a/packages/pangraph/src/pangraph/pangraph_block.rs +++ b/packages/pangraph/src/pangraph/pangraph_block.rs @@ -60,9 +60,18 @@ impl PangraphBlock { } } - /// Returns this block with a different id, leaving consensus and alignments untouched. - pub fn with_id(self, id: BlockId) -> Self { - Self { id, ..self } + /// Returns this block with a new id and its alignment keys rewritten through `node_map`. + /// The consensus and the edits are moved over unchanged. + pub fn relabel(self, id: BlockId, node_map: &BTreeMap) -> Self { + Self { + id, + consensus: self.consensus, + alignments: self + .alignments + .into_iter() + .map(|(nid, edit)| (node_map[&nid], edit)) + .collect(), + } } pub fn reverse_complement(&self) -> Result { From 25b7ca13c567049c3c26cf565e228a99f00fc2e8 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 17:27:10 +0200 Subject: [PATCH 07/50] refactor(pangraph): move sequence reconstruction into the graph module --- .../pangraph/src/commands/build/build_run.rs | 3 +- .../pangraph/src/commands/merge/merge_run.rs | 2 +- .../commands/reconstruct/reconstruct_run.rs | 81 +----------------- packages/pangraph/src/pangraph/mod.rs | 1 + packages/pangraph/src/pangraph/pangraph.rs | 2 +- packages/pangraph/src/pangraph/reconstruct.rs | 82 +++++++++++++++++++ packages/pangraph/tests/itest_merge.rs | 2 +- 7 files changed, 90 insertions(+), 83 deletions(-) create mode 100644 packages/pangraph/src/pangraph/reconstruct.rs diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index 731e3bee..7e673806 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -1,10 +1,11 @@ use crate::align::alignment_args::check_alignment_backend_available; use crate::commands::build::build_args::PangraphBuildArgs; -use crate::commands::reconstruct::reconstruct_run::{compare_sequences, reconstruct}; +use crate::commands::reconstruct::reconstruct_run::compare_sequences; use crate::io::fasta::{FastaReader, FastaRecord}; use crate::io::json::{JsonPretty, json_write_file}; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; +use crate::pangraph::reconstruct::reconstruct; use crate::pangraph::strand::Strand::Forward; use crate::tree::clade::postorder; use crate::tree::neighbor_joining::build_tree_using_neighbor_joining; diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 7782b040..442180bc 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -1,11 +1,11 @@ use crate::align::alignment_args::check_alignment_backend_available; use crate::commands::merge::merge_args::PangraphMergeArgs; -use crate::commands::reconstruct::reconstruct_run::reconstruct; use crate::io::json::{JsonPretty, json_write_file}; use crate::make_error; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; +use crate::pangraph::reconstruct::reconstruct; use crate::representation::seq::Seq; use crate::utils::collections::find_duplicates; use eyre::{Report, WrapErr}; diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs index cf4c57d3..13b4f2ba 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs @@ -1,14 +1,10 @@ use crate::commands::reconstruct::reconstruct_args::PangraphReconstructArgs; use crate::io::fasta::{FastaReader, FastaRecord, FastaWriter}; use crate::io::json::json_read_file; -use crate::io::seq::reverse_complement; +use crate::make_error; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::pangraph_node::NodeId; -use crate::pangraph::pangraph_path::PangraphPath; -use crate::representation::seq::Seq; -use crate::{make_error, make_internal_report}; +use crate::pangraph::reconstruct::reconstruct; use eyre::Report; -use itertools::Itertools; use log::info; pub fn reconstruct_run(args: &PangraphReconstructArgs) -> Result<(), Report> { @@ -52,76 +48,3 @@ pub fn compare_sequences(left: &FastaRecord, right: &FastaRecord) -> Result impl Iterator> + use<'_> { - graph - .paths - .iter() - .sorted_by_key(|(path_id, _)| **path_id) - .map(|(path_id, path)| { - let index = path_id.0; - let seq = reconstruct_path_sequence(graph, path)?; - let seq_name = path - .name() - .clone() - .unwrap_or_else(|| format!("Unknown sequence #{path_id}")); - let desc = path.desc().clone(); - Ok(FastaRecord { - seq_name, - desc, - seq, - index, - }) - }) -} - -fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result { - if let Some(first_node_id) = path.nodes.first() { - let first_node_pos = graph.nodes[first_node_id].position().0; - - let mut genome: Seq = path - .nodes - .iter() - .map(|node_id| reconstruct_block_sequence(graph, *node_id)) - .collect::>()?; - - let genome_len = path.tot_len(); - if genome.len() != genome_len { - return make_error!( - "When reconstructing sequences, genome length mismatch: computed length {} expected {}", - genome.len(), - genome_len - ); - } - - genome.rotate_right(first_node_pos); - - Ok(genome) - } else { - Ok(Seq::new()) - } -} - -fn reconstruct_block_sequence(graph: &Pangraph, node_id: NodeId) -> Result { - let node = graph - .nodes - .get(&node_id) - .ok_or_else(|| make_internal_report!("Node {node_id} not found in graph"))?; - - let block_id = node.block_id(); - let block = graph - .blocks - .get(&block_id) - .ok_or_else(|| make_internal_report!("Block {block_id} not found in graph"))?; - - // Get edits and apply them to the consensus sequence - let edits = block.alignment(node_id); - - let mut s = edits.apply(block.consensus())?; - - // Reverse-complement if on opposite strand - if node.strand().is_reverse() { - s = reverse_complement(&s)?; - } - Ok(s) -} diff --git a/packages/pangraph/src/pangraph/mod.rs b/packages/pangraph/src/pangraph/mod.rs index 201c062e..aebbc33f 100644 --- a/packages/pangraph/src/pangraph/mod.rs +++ b/packages/pangraph/src/pangraph/mod.rs @@ -6,6 +6,7 @@ pub mod pangraph_block; pub mod pangraph_interval; pub mod pangraph_node; pub mod pangraph_path; +pub mod reconstruct; pub mod reweave; pub mod slice; pub mod split_matches; diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 77e26c61..6336a629 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -409,11 +409,11 @@ mod tests { #![allow(non_snake_case, clippy::redundant_clone)] use super::*; - use crate::commands::reconstruct::reconstruct_run::reconstruct; use crate::o; use crate::pangraph::edits::Edit; use crate::pangraph::pangraph_node::PangraphNode; use crate::pangraph::pangraph_path::PangraphPath; + use crate::pangraph::reconstruct::reconstruct; use crate::pangraph::strand::Strand::{Forward, Reverse}; use itertools::Itertools; use maplit::btreemap; diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs new file mode 100644 index 00000000..8795c8c5 --- /dev/null +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -0,0 +1,82 @@ +use crate::io::fasta::FastaRecord; +use crate::io::seq::reverse_complement; +use crate::make_internal_report; +use crate::pangraph::pangraph::Pangraph; +use crate::pangraph::pangraph_node::NodeId; +use crate::pangraph::pangraph_path::PangraphPath; +use crate::representation::seq::Seq; +use eyre::Report; +use itertools::Itertools; + +pub fn reconstruct(graph: &Pangraph) -> impl Iterator> + use<'_> { + graph + .paths + .iter() + .sorted_by_key(|(path_id, _)| **path_id) + .map(|(path_id, path)| { + let index = path_id.0; + let seq = reconstruct_path_sequence(graph, path)?; + let seq_name = path + .name() + .clone() + .unwrap_or_else(|| format!("Unknown sequence #{path_id}")); + let desc = path.desc().clone(); + Ok(FastaRecord { + seq_name, + desc, + seq, + index, + }) + }) +} + +fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result { + if let Some(first_node_id) = path.nodes.first() { + let first_node_pos = graph.nodes[first_node_id].position().0; + + let mut genome: Seq = path + .nodes + .iter() + .map(|node_id| reconstruct_block_sequence(graph, *node_id)) + .collect::>()?; + + let genome_len = path.tot_len(); + if genome.len() != genome_len { + return crate::make_error!( + "When reconstructing sequences, genome length mismatch: computed length {} expected {}", + genome.len(), + genome_len + ); + } + + genome.rotate_right(first_node_pos); + + Ok(genome) + } else { + Ok(Seq::new()) + } +} + +fn reconstruct_block_sequence(graph: &Pangraph, node_id: NodeId) -> Result { + let node = graph + .nodes + .get(&node_id) + .ok_or_else(|| make_internal_report!("Node {node_id} not found in graph"))?; + + let block_id = node.block_id(); + let block = graph + .blocks + .get(&block_id) + .ok_or_else(|| make_internal_report!("Block {block_id} not found in graph"))?; + + // Get edits and apply them to the consensus sequence + let edits = block.alignment(node_id); + + let mut s = edits.apply(block.consensus())?; + + // Reverse-complement if on opposite strand + if node.strand().is_reverse() { + s = reverse_complement(&s)?; + } + Ok(s) +} diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index c9b50344..46efdb0f 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -9,10 +9,10 @@ mod tests { use pangraph::commands::build::build_run::build; use pangraph::commands::merge::merge_args::PangraphMergeArgs; use pangraph::commands::merge::merge_run::merge_run; - use pangraph::commands::reconstruct::reconstruct_run::reconstruct; use pangraph::io::fasta::{FastaReader, FastaRecord}; use pangraph::io::json::{JsonPretty, json_write_file}; use pangraph::pangraph::pangraph::Pangraph; + use pangraph::pangraph::reconstruct::reconstruct; use pangraph::representation::seq::Seq; use pangraph::utils::error::report_to_string; use pretty_assertions::assert_eq; From 844db45bde5072ea754c863707ca6348cf1a622d Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 17:30:10 +0200 Subject: [PATCH 08/50] feat(pangraph): add name-keyed sequence verification --- packages/pangraph/src/pangraph/reconstruct.rs | 368 +++++++++++++++++- 1 file changed, 365 insertions(+), 3 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 8795c8c5..02bd8059 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -1,13 +1,40 @@ use crate::io::fasta::FastaRecord; use crate::io::seq::reverse_complement; -use crate::make_internal_report; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_node::NodeId; -use crate::pangraph::pangraph_path::PangraphPath; +use crate::pangraph::pangraph_path::{PangraphPath, PathId}; use crate::representation::seq::Seq; -use eyre::Report; +use crate::utils::collections::find_duplicates; +use crate::utils::string::str_slice_safe; +use crate::{make_error, make_internal_report}; +use eyre::{Report, WrapErr}; use itertools::Itertools; +use std::collections::BTreeMap; +/// Number of genome names listed in full in an error message before the rest are elided. +const MAX_NAMES_IN_ERROR: usize = 10; + +/// Number of nucleotides shown on each side of the first difference between two genomes. +const MISMATCH_CONTEXT: usize = 10; + +/// How much of the expected genome set the graph is required to reconstruct. +/// +/// In both cases a genome that the graph contains but `expected` does not is an error: the +/// variants differ only in whether the graph is allowed to be missing expected genomes. +#[derive(Copy, Clone, Debug, Eq, PartialEq)] +pub enum GenomeCoverage { + /// The graph must reconstruct every expected genome, and nothing else. + Complete, + /// The graph may reconstruct only some of the expected genomes. Used for the intermediate graphs + /// of a build, which hold the genomes of one clade of the guide tree. + Partial, +} + +/// Reconstructs every genome of the graph as a FASTA record, ordered by path id. +/// +/// Record order and `index` reproduce the order of the original input FASTA only for graphs +/// produced directly by `build`. A merged graph renumbers its path ids, so consumers must match +/// records by genome name. pub fn reconstruct(graph: &Pangraph) -> impl Iterator> + use<'_> { graph .paths @@ -30,6 +57,171 @@ pub fn reconstruct(graph: &Pangraph) -> impl Iterator Result, Report> { + let unnamed = graph + .paths + .iter() + .filter(|(_, path)| path.name.is_none()) + .map(|(path_id, _)| path_id.to_string()) + .collect_vec(); + + if !unnamed.is_empty() { + return make_error!( + "Graph contains {} genome(s) without a name (path ids: {}). Genomes are identified by name, so every path must be named.", + unnamed.len(), + format_names(&unnamed) + ); + } + + let duplicates = find_duplicates(graph.path_names().flatten()); + if !duplicates.is_empty() { + return make_error!( + "Graph contains duplicate genome names: {}. Genome names must be unique, because they identify genomes.", + format_names(&duplicates) + ); + } + + Ok( + graph + .paths + .iter() + .filter_map(|(path_id, path)| path.name.as_deref().map(|name| (name, *path_id))) + .collect(), + ) +} + +/// Reconstructs the genome of a single path. +pub fn reconstruct_genome(graph: &Pangraph, path_id: PathId) -> Result { + let path = graph + .paths + .get(&path_id) + .ok_or_else(|| make_internal_report!("Path {path_id} not found in graph"))?; + reconstruct_path_sequence(graph, path) +} + +/// Reconstructs every genome of the graph, keyed by genome name. +/// +/// Errors if any path is unnamed or if two paths share a name. +pub fn reconstruct_by_name(graph: &Pangraph) -> Result, Report> { + path_ids_by_name(graph)? + .into_iter() + .map(|(name, path_id)| { + let seq = + reconstruct_genome(graph, path_id).wrap_err_with(|| format!("When reconstructing the genome of '{name}'"))?; + Ok((name.to_owned(), seq)) + }) + .collect() +} + +/// Collects FASTA records into genome sequences keyed by name. +/// +/// Errors on duplicate names: the name is the key that verification matches on. +pub fn sequences_by_name(fastas: &[FastaRecord]) -> Result, Report> { + let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); + if !duplicates.is_empty() { + return make_error!( + "Duplicate sequence names found: {}. Sequence names must be unique, because they identify genomes.", + format_names(&duplicates) + ); + } + + Ok( + fastas + .iter() + .map(|fasta| (fasta.seq_name.clone(), fasta.seq.clone())) + .collect(), + ) +} + +/// Compares one reconstructed genome against the sequence it is expected to have, reporting the +/// genome name and the first position at which the two differ. +pub fn verify_genome(name: &str, expected: &Seq, actual: &Seq) -> Result<(), Report> { + if expected == actual { + return Ok(()); + } + + if expected.len() != actual.len() { + return make_error!( + "Sequence mismatch for genome '{name}': expected length {} but got {}", + expected.len(), + actual.len() + ); + } + + let pos = expected + .iter() + .zip(actual.iter()) + .position(|(e, a)| e != a) + .ok_or_else(|| make_internal_report!("Genomes of '{name}' compare as different but share every character"))?; + + let (start, end) = (pos.saturating_sub(MISMATCH_CONTEXT), pos + MISMATCH_CONTEXT + 1); + make_error!( + "Sequence mismatch for genome '{name}' at position {pos} (length {}):\n expected: {}\n actual: {}", + expected.len(), + str_slice_safe(expected.as_str(), start, end), + str_slice_safe(actual.as_str(), start, end) + ) +} + +/// Checks that the graph reconstructs the expected genomes, matched by name. +/// +/// Genomes are never matched by position or by path id: a merge renumbers path ids, so neither the +/// order in which genomes are reconstructed nor [`FastaRecord::index`] can pair them up. Only the +/// sequence content is compared; descriptions are ignored. +/// +/// Genomes are reconstructed and dropped one at a time, so this holds no more than a single genome +/// in memory beyond `expected`. +pub fn verify_graph_sequences( + graph: &Pangraph, + expected: &BTreeMap, + coverage: GenomeCoverage, +) -> Result<(), Report> { + let path_ids = path_ids_by_name(graph)?; + + for (name, path_id) in &path_ids { + let Some(expected_seq) = expected.get(*name) else { + return make_error!("Graph contains genome '{name}', which is not among the expected genomes"); + }; + let actual = + reconstruct_genome(graph, *path_id).wrap_err_with(|| format!("When reconstructing genome '{name}'"))?; + verify_genome(name, expected_seq, &actual)?; + } + + if coverage == GenomeCoverage::Complete { + let missing = expected + .keys() + .filter(|name| !path_ids.contains_key(name.as_str())) + .cloned() + .collect_vec(); + + if !missing.is_empty() { + return make_error!( + "Graph is missing {} expected genome(s): {}", + missing.len(), + format_names(&missing) + ); + } + } + + Ok(()) +} + +/// Formats a list of genome names for an error message, eliding all but the first few. +fn format_names>(names: &[S]) -> String { + let shown = names.iter().take(MAX_NAMES_IN_ERROR).map(AsRef::as_ref).join(", "); + if names.len() > MAX_NAMES_IN_ERROR { + format!("[{shown}, ... and {} more]", names.len() - MAX_NAMES_IN_ERROR) + } else { + format!("[{shown}]") + } +} + fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result { if let Some(first_node_id) = path.nodes.first() { let first_node_pos = graph.nodes[first_node_id].position().0; @@ -80,3 +272,173 @@ fn reconstruct_block_sequence(graph: &Pangraph, node_id: NodeId) -> Result; 2]) -> Pangraph { + let blocks = btreemap! { + BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap!{ NodeId(0) => Edit::empty() }), + BlockId(1) => PangraphBlock::new(BlockId(1), "TTTTGGGG", btreemap!{ NodeId(1) => Edit::empty() }), + }; + let nodes = btreemap! { + NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Reverse, (0, 8)), + }; + let paths = btreemap! { + PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, names[0].map(String::from), None), + PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 8, false, names[1].map(String::from), None), + }; + Pangraph { paths, blocks, nodes } + } + + fn graph() -> Pangraph { + two_genome_graph([Some("a"), Some("b")]) + } + + fn expected_genomes() -> BTreeMap { + btreemap! { o!("a") => Seq::from_str("ACGTACGT"), o!("b") => Seq::from_str("CCCCAAAA") } + } + + #[rstest] + fn test_reconstruct_by_name() { + assert_eq!(reconstruct_by_name(&graph()).unwrap(), expected_genomes()); + } + + #[rstest] + fn test_path_ids_by_name_rejects_unnamed_path() { + let graph = two_genome_graph([Some("a"), None]); + assert!(report_to_string(&path_ids_by_name(&graph).unwrap_err()).contains("without a name")); + assert!(report_to_string(&reconstruct_by_name(&graph).unwrap_err()).contains("without a name")); + } + + #[rstest] + fn test_path_ids_by_name_rejects_duplicate_names() { + let graph = two_genome_graph([Some("a"), Some("a")]); + assert!(report_to_string(&path_ids_by_name(&graph).unwrap_err()).contains("duplicate genome names")); + } + + #[rstest] + fn test_sequences_by_name_rejects_duplicate_names() { + let record = |name: &str| FastaRecord { + seq_name: name.to_owned(), + desc: None, + seq: Seq::from_str("ACGT"), + index: 0, + }; + let fastas = [record("a"), record("a")]; + assert!(report_to_string(&sequences_by_name(&fastas).unwrap_err()).contains("Duplicate sequence names")); + } + + #[rstest] + fn test_verify_graph_sequences_accepts_exact_match() { + verify_graph_sequences(&graph(), &expected_genomes(), GenomeCoverage::Complete).unwrap(); + } + + /// The whole point of keying by name: `index` and `desc` must not take part in the comparison. + /// The previous whole-`FastaRecord` comparison failed this, which is why verifying a merged + /// graph reported a mismatch between two sequences of identical length. + #[rstest] + fn test_verify_graph_sequences_ignores_index_and_desc() { + let fastas = [ + FastaRecord { + seq_name: o!("a"), + desc: Some(o!("some description")), + seq: Seq::from_str("ACGTACGT"), + index: 41, + }, + FastaRecord { + seq_name: o!("b"), + desc: None, + seq: Seq::from_str("CCCCAAAA"), + index: 42, + }, + ]; + let expected = sequences_by_name(&fastas).unwrap(); + verify_graph_sequences(&graph(), &expected, GenomeCoverage::Complete).unwrap(); + } + + #[rstest] + fn test_verify_graph_sequences_detects_missing_genome() { + let mut expected = expected_genomes(); + expected.insert(o!("c"), Seq::from_str("GGGG")); + + let err = verify_graph_sequences(&graph(), &expected, GenomeCoverage::Complete).unwrap_err(); + let err = report_to_string(&err); + assert!(err.contains("missing"), "unexpected error: {err}"); + assert!(err.contains('c'), "unexpected error: {err}"); + } + + #[rstest] + fn test_verify_graph_sequences_detects_extra_genome() { + let mut expected = expected_genomes(); + expected.remove("b"); + + let err = report_to_string(&verify_graph_sequences(&graph(), &expected, GenomeCoverage::Complete).unwrap_err()); + assert!( + err.contains("not among the expected genomes"), + "unexpected error: {err}" + ); + } + + /// `Partial` relaxes only one direction: the graph may hold a subset of the expected genomes, + /// but a genome the expected set does not know about is still an error. + #[rstest] + fn test_verify_graph_sequences_partial_accepts_superset() { + let mut expected = expected_genomes(); + expected.insert(o!("c"), Seq::from_str("GGGG")); + + verify_graph_sequences(&graph(), &expected, GenomeCoverage::Partial).unwrap(); + } + + #[rstest] + fn test_verify_graph_sequences_partial_still_rejects_extra_genome() { + let mut expected = expected_genomes(); + expected.remove("b"); + + let err = report_to_string(&verify_graph_sequences(&graph(), &expected, GenomeCoverage::Partial).unwrap_err()); + assert!( + err.contains("not among the expected genomes"), + "unexpected error: {err}" + ); + } + + #[rstest] + fn test_verify_genome_reports_first_difference() { + let expected = Seq::from_str("ACGTACGT"); + let actual = Seq::from_str("ACGTTCGT"); + + let err = report_to_string(&verify_genome("a", &expected, &actual).unwrap_err()); + assert!(err.contains("at position 4"), "unexpected error: {err}"); + assert!(err.contains("ACGTACGT"), "unexpected error: {err}"); + assert!(err.contains("ACGTTCGT"), "unexpected error: {err}"); + } + + #[rstest] + fn test_verify_genome_reports_length_mismatch() { + let err = report_to_string(&verify_genome("a", &Seq::from_str("ACGT"), &Seq::from_str("ACG")).unwrap_err()); + assert!(err.contains("expected length 4 but got 3"), "unexpected error: {err}"); + } + + #[rstest] + fn test_format_names_elides_long_lists() { + let names = (0..12).map(|i| format!("g{i}")).collect_vec(); + assert_eq!( + format_names(&names), + "[g0, g1, g2, g3, g4, g5, g6, g7, g8, g9, ... and 2 more]" + ); + assert_eq!(format_names(&names[..2]), "[g0, g1]"); + } +} From 8627366cf790468ffe1cda429a8f57701ee38b17 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 17:32:09 +0200 Subject: [PATCH 09/50] fix(build): verify reconstructed sequences by genome name, not by index --- .../pangraph/src/commands/build/build_run.rs | 62 ++++++------- .../pangraph/src/commands/merge/merge_run.rs | 90 ++++++------------- 2 files changed, 55 insertions(+), 97 deletions(-) diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index 7e673806..0f906099 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -1,12 +1,12 @@ use crate::align::alignment_args::check_alignment_backend_available; use crate::commands::build::build_args::PangraphBuildArgs; -use crate::commands::reconstruct::reconstruct_run::compare_sequences; use crate::io::fasta::{FastaReader, FastaRecord}; use crate::io::json::{JsonPretty, json_write_file}; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::reconstruct::reconstruct; +use crate::pangraph::reconstruct::{GenomeCoverage, sequences_by_name, verify_graph_sequences}; use crate::pangraph::strand::Strand::Forward; +use crate::representation::seq::Seq; use crate::tree::clade::postorder; use crate::tree::neighbor_joining::build_tree_using_neighbor_joining; use crate::tree::newick::build_tree_from_newick; @@ -16,31 +16,23 @@ use crate::{make_error, make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::info; - -pub fn reconstruct_and_compare_graph_seqs(graph: &Pangraph, fastas: &[FastaRecord]) -> Result<(), Report> { - // Reconstruct sequences from the given graph. - let mut results = reconstruct(graph); - - // Check that the reconstructed sequences match the original FASTA records. - results.try_for_each(|actual| -> Result<(), Report> { - let actual = actual?; - let expected = &fastas[actual.index]; - compare_sequences(expected, &actual)?; - Ok(()) - })?; - - Ok(()) -} - -pub fn graph_sanity_checks(graph: &Pangraph, fastas: &[FastaRecord]) -> Result<(), Report> { +use std::collections::BTreeMap; + +/// Checks that the graph is internally consistent and reconstructs the genomes it should. +fn graph_sanity_checks( + graph: &Pangraph, + expected: &BTreeMap, + coverage: GenomeCoverage, +) -> Result<(), Report> { // check that graph internal structure (blocks, paths, nodes, edits...) is valid #[cfg(debug_assertions)] graph .sanity_check() .wrap_err("When performing sanity check on the pangraph")?; - // Reconstruct sequences from the graph and compare them with the original FASTA records. - reconstruct_and_compare_graph_seqs(graph, fastas) + // Reconstruct the genomes from the graph and compare them with the input sequences. Genomes are + // matched by name: path ids do not survive a merge, so they cannot pair sequences up. + verify_graph_sequences(graph, expected, coverage) .wrap_err("When comparing reconstructed sequences with original FASTA records")?; Ok(()) @@ -79,9 +71,12 @@ pub fn check_unique_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) -> Result { check_unique_sequence_names(&fastas).wrap_err("When checking the names of the input sequences")?; - // If verification is requested, we need to keep a copy of the original FASTA records - // to compare them with the sequences reconstructed from the graph. - let fasta_copy = verify.then(|| fastas.clone()); + // If verification is requested, keep the input sequences, keyed by genome name, to compare them + // with the sequences reconstructed from the graph. + let expected = verify + .then(|| sequences_by_name(&fastas)) + .transpose() + .wrap_err("When collecting the input sequences for verification")?; // Build singleton graphs from input sequences // TODO: initial graphs can potentially be constructed when initializing tree clades. This could avoid a lot of boilerplate code. @@ -135,14 +130,12 @@ pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) - clade.data.as_ref().unwrap().paths.len() ); - // perform checks only in debug mode and if requested + // perform checks only in debug mode and if requested. An intermediate graph holds + // only the genomes of its own clade, hence `Partial`. #[cfg(debug_assertions)] - { - if verify { - // verify the graph if requested - graph_sanity_checks(clade.data.as_ref().unwrap(), fasta_copy.as_ref().unwrap()) - .wrap_err("When performing sanity checks on the merged graph")?; - } + if let Some(expected) = expected.as_ref() { + graph_sanity_checks(clade.data.as_ref().unwrap(), expected, GenomeCoverage::Partial) + .wrap_err("When performing sanity checks on the merged graph")?; } Ok(()) @@ -169,10 +162,11 @@ pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) - .take() .ok_or_else(|| make_internal_report!("Root clade of the guide tree contains no graph after graph alignment"))?; - // verify the final graph if requested - if verify { - graph_sanity_checks(&graph, fasta_copy.as_ref().unwrap()) + // verify the final graph if requested. It must hold every input genome, hence `Complete`. + if let Some(expected) = &expected { + graph_sanity_checks(&graph, expected, GenomeCoverage::Complete) .wrap_err("When performing sanity checks on the final pangraph")?; + info!("Pangraph reconstructs all {} input genomes exactly", expected.len()); } Ok(graph) diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 442180bc..1e4bd04e 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -5,11 +5,12 @@ use crate::make_error; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; -use crate::pangraph::reconstruct::reconstruct; +use crate::pangraph::reconstruct::{GenomeCoverage, reconstruct_by_name, verify_graph_sequences}; use crate::representation::seq::Seq; use crate::utils::collections::find_duplicates; +use color_eyre::owo_colors::{AnsiColors, OwoColorize}; +use color_eyre::{Help, SectionExt}; use eyre::{Report, WrapErr}; -use itertools::Itertools; use log::{info, warn}; use std::collections::{BTreeMap, BTreeSet}; use std::path::Path; @@ -30,9 +31,8 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { // merger. Keyed by path name: neither path ids nor record order survive a merge. let expected = args .verify - .then(|| expected_sequences(&left, &right)) - .transpose() - .wrap_err("When reconstructing the sequences of the input graphs")?; + .then(|| expected_sequences(args, &left, &right)) + .transpose()?; info!( "=== Graph merging start: graph sizes {} + {}", @@ -51,7 +51,11 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { ); if let Some(expected) = expected { - verify_merged_sequences(&merged, &expected).wrap_err("When verifying the sequences of the merged graph")?; + #[cfg(debug_assertions)] + merged.sanity_check().wrap_err("When checking the merged graph")?; + + verify_graph_sequences(&merged, &expected, GenomeCoverage::Complete) + .wrap_err("When verifying the sequences of the merged graph")?; info!("Merged graph reconstructs all {} input genomes exactly", expected.len()); } @@ -91,17 +95,6 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right ); } - if args.verify { - for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { - if graph.path_names().any(|name| name.is_none()) { - return make_error!( - "Graph '{}' contains genomes without a name, which cannot be verified: verification matches genomes by name. Re-run without `--verify`.", - filepath.display() - ); - } - } - } - // Circularity is a per-path property, so mixing is structurally fine. It is however most often a // mistake, since `build --circular` applies to all genomes of a graph at once. if circularity(left) != circularity(right) { @@ -119,53 +112,24 @@ fn circularity(graph: &Pangraph) -> BTreeSet { } /// Reconstructs the genomes of both input graphs, keyed by genome name. -fn expected_sequences(left: &Pangraph, right: &Pangraph) -> Result, Report> { +/// +/// Cross-graph name collisions are already rejected by `merge_cmd_preliminary_checks`, so the two +/// sets cannot overwrite each other here. +fn expected_sequences( + args: &PangraphMergeArgs, + left: &Pangraph, + right: &Pangraph, +) -> Result, Report> { let mut expected = BTreeMap::new(); - for graph in [left, right] { - for record in reconstruct(graph) { - let record = record?; - expected.insert(record.seq_name, record.seq); - } + for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { + let genomes = reconstruct_by_name(graph) + .wrap_err_with(|| format!("When reconstructing the genomes of graph '{}'", filepath.display())) + .with_section(|| { + "Verification matches genomes by name. Re-run without `--verify` to skip it." + .color(AnsiColors::Cyan) + .header("Suggestion:") + })?; + expected.extend(genomes); } Ok(expected) } - -/// Checks that the merged graph reconstructs exactly the genomes of the input graphs. -/// Genomes are matched by name: path ids are renumbered by the merger, and the order in which -/// genomes are reconstructed is therefore not the order of either input graph. -fn verify_merged_sequences(merged: &Pangraph, expected: &BTreeMap) -> Result<(), Report> { - #[cfg(debug_assertions)] - merged.sanity_check().wrap_err("When checking the merged graph")?; - - let mut remaining: BTreeSet<&String> = expected.keys().collect(); - - for record in reconstruct(merged) { - let record = record?; - let Some(expected_seq) = expected.get(&record.seq_name) else { - return make_error!( - "Merged graph contains genome '{}', which is not present in either input graph", - record.seq_name - ); - }; - - if record.seq != *expected_seq { - return make_error!( - "Sequence mismatch for genome '{}': expected length {} but got {}", - record.seq_name, - expected_seq.len(), - record.seq.len() - ); - } - - remaining.remove(&record.seq_name); - } - - if !remaining.is_empty() { - return make_error!( - "Merged graph is missing genomes from the input graphs: [{}]", - remaining.into_iter().sorted().join(", ") - ); - } - - Ok(()) -} From db167cec518208431edd431a8f1bc29e406e87ad Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 17:33:46 +0200 Subject: [PATCH 10/50] fix(reconstruct): match verification sequences by name --- docs/docs/reference.md | 14 +++- .../commands/reconstruct/reconstruct_args.rs | 21 ++++-- .../commands/reconstruct/reconstruct_run.rs | 75 ++++++++++++++----- 3 files changed, 83 insertions(+), 27 deletions(-) diff --git a/docs/docs/reference.md b/docs/docs/reference.md index 3451f1b9..ffa6bc09 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -370,9 +370,9 @@ Reconstruct all input fasta sequences from graph * `` — Path to a pangenome graph file in JSON format. - Accepts plain or compressed FASTA files. If a compressed fasta file is provided, it will be transparently decompressed. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. Decompressor is chosen based on file extension. If there's multiple input files, then different files can have different compression formats. + Accepts plain or compressed files. If a compressed file is provided, it will be transparently decompressed. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. Decompressor is chosen based on file extension. - If no input files provided, the plain fasta input is read from standard input (stdin). + If no input file is provided, the plain JSON input is read from standard input (stdin). ###### **Options:** @@ -380,10 +380,16 @@ Reconstruct all input fasta sequences from graph If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. - Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. See: https://en.wikipedia.org/wiki/FASTA_format + Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + + Records are written in order of path id, which reproduces the order of the original input FASTA only for graphs produced directly by `pangraph build`. A graph produced by `pangraph merge` renumbers its path ids, so consumers should match records by genome name rather than by position. + + See: https://en.wikipedia.org/wiki/FASTA_format Default value: `-` -* `-f`, `--verify ` — Path to the FASTA file with sequences to check the reconstructed sequences against. If this argument is provided, then the sequences are not being printed to standard output (stdout) as usual. Instead, if any differences are detected, a diff will be printed between the expected (original) sequence and reconstructed sequence. +* `-f`, `--verify ` — Path to the FASTA file with sequences to check the reconstructed sequences against. If this argument is provided, then the sequences are not written out as usual: nothing is produced on success, and the first difference found is reported as an error. + + Genomes are matched by name, so the order of the records is irrelevant. The file must contain exactly the genomes of the graph: a record the graph does not contain, a genome missing from the file, or a repeated name are all errors. Every path of the graph must be named. Accepts plain or compressed FASTA files. If a compressed fasta file is provided, it will be transparently decompressed. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. Decompressor is chosen based on file extension. If there's multiple input files, then different files can have different compression formats. diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs index 50dc8261..4d5741e6 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs @@ -1,17 +1,18 @@ use clap::{Parser, ValueHint}; +use smart_default::SmartDefault; use std::fmt::Debug; use std::path::PathBuf; /// Reconstruct sequences from a pangenome graph -#[derive(Parser, Debug)] +#[derive(Parser, Debug, SmartDefault)] pub struct PangraphReconstructArgs { /// Path to a pangenome graph file in JSON format. /// - /// Accepts plain or compressed FASTA files. If a compressed fasta file is provided, it will be transparently + /// Accepts plain or compressed files. If a compressed file is provided, it will be transparently /// decompressed. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. Decompressor is chosen based on file - /// extension. If there's multiple input files, then different files can have different compression formats. + /// extension. /// - /// If no input files provided, the plain fasta input is read from standard input (stdin). + /// If no input file is provided, the plain JSON input is read from standard input (stdin). #[clap(value_hint = ValueHint::FilePath)] #[clap(display_order = 1)] pub input_graph: Option, @@ -21,12 +22,22 @@ pub struct PangraphReconstructArgs { /// If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. /// /// Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + /// + /// Records are written in order of path id, which reproduces the order of the original input FASTA + /// only for graphs produced directly by `pangraph build`. A graph produced by `pangraph merge` + /// renumbers its path ids, so consumers should match records by genome name rather than by + /// position. + /// /// See: https://en.wikipedia.org/wiki/FASTA_format #[clap(long, short = 'o', default_value = "-")] #[clap(value_hint = ValueHint::AnyPath)] pub output_fasta: PathBuf, - /// Path to the FASTA file with sequences to check the reconstructed sequences against. If this argument is provided, then the sequences are not being printed to standard output (stdout) as usual. Instead, if any differences are detected, a diff will be printed between the expected (original) sequence and reconstructed sequence. + /// Path to the FASTA file with sequences to check the reconstructed sequences against. If this argument is provided, then the sequences are not written out as usual: nothing is produced on success, and the first difference found is reported as an error. + /// + /// Genomes are matched by name, so the order of the records is irrelevant. The file must contain + /// exactly the genomes of the graph: a record the graph does not contain, a genome missing from + /// the file, or a repeated name are all errors. Every path of the graph must be named. /// /// Accepts plain or compressed FASTA files. If a compressed fasta file is provided, it will be transparently /// decompressed. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. Decompressor is chosen based on file diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs index 13b4f2ba..d61e4252 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs @@ -3,9 +3,12 @@ use crate::io::fasta::{FastaReader, FastaRecord, FastaWriter}; use crate::io::json::json_read_file; use crate::make_error; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::reconstruct::reconstruct; -use eyre::Report; +use crate::pangraph::reconstruct::{path_ids_by_name, reconstruct, reconstruct_genome, verify_genome}; +use eyre::{Report, WrapErr}; +use itertools::Itertools; use log::info; +use std::collections::BTreeSet; +use std::path::Path; pub fn reconstruct_run(args: &PangraphReconstructArgs) -> Result<(), Report> { let PangraphReconstructArgs { @@ -15,21 +18,18 @@ pub fn reconstruct_run(args: &PangraphReconstructArgs) -> Result<(), Report> { } = &args; let graph: Pangraph = json_read_file(input_graph)?; - let mut results = reconstruct(&graph); if let Some(verify) = verify { info!("Verifying sequences reconstructed from pangenome graph"); - let mut reader = FastaReader::from_path(verify)?; - results.try_for_each(|actual| -> Result<(), Report> { - let actual = actual?; - let mut expected = FastaRecord::new(); - reader.read(&mut expected)?; - compare_sequences(&expected, &actual)?; - Ok(()) - })?; + let n_verified = verify_against_fasta(&graph, verify) + .wrap_err_with(|| format!("When verifying reconstructed sequences against '{}'", verify.display()))?; + info!( + "Graph reconstructs all {n_verified} genomes of '{}' exactly", + verify.display() + ); } else { let mut writer = FastaWriter::from_path(output_fasta)?; - results.try_for_each(|fasta| { + reconstruct(&graph).try_for_each(|fasta| { let fasta = fasta?; writer.write(fasta.seq_name, &fasta.desc, &fasta.seq) })?; @@ -38,13 +38,52 @@ pub fn reconstruct_run(args: &PangraphReconstructArgs) -> Result<(), Report> { Ok(()) } -pub fn compare_sequences(left: &FastaRecord, right: &FastaRecord) -> Result { - if left != right { +/// Checks that the graph reconstructs exactly the genomes of the given FASTA file, and returns how +/// many were verified. +/// +/// Genomes are matched by name, so the order of the FASTA records is irrelevant: a merged graph +/// renumbers its path ids and reconstructs its genomes in an order that matches no input file. +/// The file is streamed and genomes are reconstructed one at a time, so only a single genome is +/// held in memory at once. +fn verify_against_fasta(graph: &Pangraph, verify: &Path) -> Result { + let path_ids = path_ids_by_name(graph)?; + let mut remaining: BTreeSet<&str> = path_ids.keys().copied().collect(); + + let mut reader = FastaReader::from_path(verify)?; + let mut record = FastaRecord::new(); + loop { + record.clear(); + reader.read(&mut record)?; + if record.is_empty() { + break; + } + + let Some(path_id) = path_ids.get(record.seq_name.as_str()) else { + return make_error!( + "Verification file contains genome '{}', which the graph does not contain", + record.seq_name + ); + }; + + if !remaining.remove(record.seq_name.as_str()) { + return make_error!( + "Verification file contains genome '{}' more than once. Genome names must be unique, because they identify genomes.", + record.seq_name + ); + } + + let actual = reconstruct_genome(graph, *path_id) + .wrap_err_with(|| format!("When reconstructing genome '{}'", record.seq_name))?; + verify_genome(&record.seq_name, &record.seq, &actual)?; + } + + if !remaining.is_empty() { return make_error!( - "Sequence mismatch detected: expected length {} but got {}", - left.seq.len(), - right.seq.len() + "Verification file is missing {} genome(s) that the graph contains: [{}]", + remaining.len(), + remaining.iter().join(", ") ); } - Ok(true) + + Ok(path_ids.len()) } From eea2d66fa0f1b39b2c8fcf9251078e65fa984e34 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 10 Aug 2026 17:38:54 +0200 Subject: [PATCH 11/50] test: cover build --verify and name-keyed reconstruct verification --- dev/design/merge.md | 43 ++-- packages/pangraph/tests/itest_merge.rs | 21 +- packages/pangraph/tests/itest_reconstruct.rs | 238 +++++++++++++++++++ 3 files changed, 287 insertions(+), 15 deletions(-) create mode 100644 packages/pangraph/tests/itest_reconstruct.rs diff --git a/dev/design/merge.md b/dev/design/merge.md index 7b9e7ee2..465e865f 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -1,6 +1,6 @@ # Graph merging — design manifesto -Status: **`pangraph merge` implemented; verification and `build`-side name checks still pending (see §6)** +Status: **`pangraph merge` implemented and verified; only user docs remain (see §6)** Integration branch: `feat/merge` (merged into `master` last, after all phase branches below) This document describes the design for a new `pangraph merge` command, which combines two @@ -301,21 +301,39 @@ Reconstruction itself remains fully supported — every sequence comes back byte This has three consequences: -1. **`compare_sequences` must stop comparing whole records.** It currently tests - `left != right` on `FastaRecord` (`reconstruct_run.rs:45-54`), and `FastaRecord` derives - `PartialEq` over all fields *including `index`* (`io/fasta.rs:17-24`) — despite an error message - that only mentions length. It should compare sequence contents and report the path name. +1. **`compare_sequences` must stop comparing whole records.** It tested `left != right` on + `FastaRecord`, which derives `PartialEq` over all fields *including `index`* (`io/fasta.rs:17-24`) + — despite an error message that only mentioned length. It is now deleted; `verify_genome` compares + sequence contents and reports the genome name and the first differing position. + + This was not only cosmetic. Two bugs followed from index/position pairing: + - `build --verify` indexed `&fastas[actual.index]`, which **panicked out of bounds** whenever the + input records' indices were not exactly `0..n-1` — as they are not for any programmatically + assembled record set, including the repo's own test helper. + - `reconstruct --verify` paired records positionally, so surplus records in the verification file + were silently ignored (exit 0), too few gave a misleading `expected length 0 but got N`, and a + merged graph failed spuriously with `expected length N but got N` at identical lengths. 2. **Verification is keyed by path name, not by index or position.** ```rust - fn verify_graph_sequences(graph: &Pangraph, expected: &BTreeMap) -> Result<(), Report> + pub fn verify_graph_sequences(graph: &Pangraph, expected: &BTreeMap, + coverage: GenomeCoverage) -> Result<(), Report> ``` + The third parameter was not in the original sketch: `build`'s intermediate clade graphs hold only + the genomes of their own clade, so they need `GenomeCoverage::Partial`, while the final graph of a + build, a merged graph and `reconstruct --verify` all require `Complete`. Making the *final* build + check `Complete` is a small gain — nothing previously noticed a genome going missing. + `build` fills the map from the input FASTA records; `merge` fills it by reconstructing each - input graph *before* merging. Both are now sound because names are unique (§4.3). The map form - also keeps working for the intermediate graphs checked inside the build loop, which contain only - a subset of the paths. + input graph *before* merging. Both are sound because names are unique (§4.3). `reconstruct + --verify` does **not** build the map: it is the only one of the three that does not otherwise + need every genome resident, so it streams the verification file against a name → path id index + and reconstructs one genome at a time, sharing `verify_genome` and its error formatting. + + Reconstruction itself moved out of the command module into `pangraph/reconstruct.rs`, since it is + a graph operation — `pangraph/pangraph.rs` was reaching up into `commands::`. 3. **`pangraph reconstruct` documentation must state** that record order matches the original input FASTA order only for graphs produced directly by `build`, and that consumers should match records @@ -378,16 +396,13 @@ merged into `master` last, once all phases have landed. | 3 | `feat/merge-cmd` | §3.4 — `relabel` / `make_disjoint_from` | landed | | 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | | 5 | `feat/merge-cmd` | §4.3 — duplicate genome names are an error in `build` and `merge` | landed | -| 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | todo | +| 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | landed | | 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | todo | Phases 2–5 were implemented together, since a `merge` command without §3.4 panics on the first identifier collision and would not be testable. -§4.4 is therefore the only piece left half-done: name-keyed verification exists as -`verify_merged_sequences` inside `merge_run`, private to the merge command. `build` still verifies -against input FASTA records by index, and `compare_sequences` still compares whole `FastaRecord`s -(including `index`). Phase 6 unifies the two behind a single `BTreeMap`-based helper. +Phase 6 turned out to fix two latent bugs rather than merely unify style, both recorded in §4.4. --- diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 46efdb0f..82de39a7 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -36,7 +36,7 @@ mod tests { circular: false, ..PangraphBuildArgs::default() }; - let graph = build(fastas, &args, false)?; + let graph = build(fastas, &args, true)?; let path = dir.path().join(name); json_write_file(&path, &graph, JsonPretty(false))?; Ok(path) @@ -79,6 +79,7 @@ mod tests { merge_run(&merge_args(left.clone(), right.clone(), output.clone()))?; let merged = read_graph(&output)?; + #[cfg(debug_assertions)] merged.sanity_check()?; // all genomes of both inputs are present, exactly once @@ -130,6 +131,7 @@ mod tests { merge_run(&merge_args(left, right, output.clone()))?; let merged = read_graph(&output)?; + #[cfg(debug_assertions)] merged.sanity_check()?; assert_eq!(merged.paths.len(), 4); @@ -155,6 +157,23 @@ mod tests { Ok(()) } + /// `build --verify` used to pair reconstructed genomes with input records by `FastaRecord::index`, + /// which is only valid when the records' indices happen to be exactly `0..n-1`. Here they are + /// `[3, 4, 5]` for a 3-record slice, which panicked with an out-of-bounds index before genomes + /// were matched by name. + /// + /// Note that the intermediate-clade half of this check only runs in debug builds. + #[rstest] + fn itest_build_verify_with_nonzero_record_indices() -> Result<(), Report> { + let (_, right_fastas) = read_and_split("../../data/ges-1.fa", 3, 6)?; + assert_eq!(right_fastas.iter().map(|f| f.index).collect_vec(), vec![3, 4, 5]); + + let graph = build(right_fastas, &PangraphBuildArgs::default(), true)?; + assert_eq!(graph.paths.len(), 3); + + Ok(()) + } + /// `build` enforces the same uniqueness invariant on its input FASTA records, so that a graph /// can never carry duplicate genome names into a later merge. #[rstest] diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs new file mode 100644 index 00000000..4ea3b8ff --- /dev/null +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -0,0 +1,238 @@ +mod common; + +#[cfg(test)] +mod tests { + use eyre::Report; + use itertools::Itertools; + use maplit::btreemap; + use pangraph::align::alignment_args::GraphMergeParams; + use pangraph::commands::build::build_args::PangraphBuildArgs; + use pangraph::commands::build::build_run::build; + use pangraph::commands::merge::merge_args::PangraphMergeArgs; + use pangraph::commands::merge::merge_run::merge_run; + use pangraph::commands::reconstruct::reconstruct_args::PangraphReconstructArgs; + use pangraph::commands::reconstruct::reconstruct_run::reconstruct_run; + use pangraph::io::fasta::{FastaReader, FastaRecord, FastaWriter}; + use pangraph::io::json::{JsonPretty, json_write_file}; + use pangraph::pangraph::edits::Edit; + use pangraph::pangraph::pangraph::Pangraph; + use pangraph::pangraph::pangraph_block::{BlockId, PangraphBlock}; + use pangraph::pangraph::pangraph_node::{NodeId, PangraphNode}; + use pangraph::pangraph::pangraph_path::{PangraphPath, PathId}; + use pangraph::pangraph::strand::Strand::Forward; + use pangraph::representation::seq::Seq; + use pangraph::utils::error::report_to_string; + use pretty_assertions::assert_eq; + use rstest::rstest; + use std::collections::BTreeMap; + use std::path::{Path, PathBuf}; + use tempfile::{TempDir, tempdir}; + + const GES: &str = "../../data/ges-1.fa"; + + /// Builds a merged graph from two disjoint halves of a FASTA file, and returns its path together + /// with all the input records. The merger renumbers path ids, which is what makes this graph the + /// interesting case: neither record order nor `FastaRecord::index` can pair genomes up any more. + fn merged_graph(dir: &TempDir) -> Result<(PathBuf, Vec), Report> { + let mut fastas = FastaReader::from_paths(&[PathBuf::from(GES)])?.read_many()?; + fastas.truncate(6); + let right_fastas = fastas.split_off(3); + let all = fastas.iter().chain(right_fastas.iter()).cloned().collect_vec(); + + let args = PangraphBuildArgs::default(); + let left = write_graph(dir, "left.json", &build(fastas, &args, true)?)?; + let right = write_graph(dir, "right.json", &build(right_fastas, &args, true)?)?; + + let output = dir.path().join("merged.json"); + merge_run(&PangraphMergeArgs { + left_graph: left, + right_graph: right, + output_json: output.clone(), + merge_params: GraphMergeParams::default(), + verify: true, + })?; + + Ok((output, all)) + } + + fn write_graph(dir: &TempDir, name: &str, graph: &Pangraph) -> Result { + let path = dir.path().join(name); + json_write_file(&path, graph, JsonPretty(false))?; + Ok(path) + } + + fn write_fasta(dir: &TempDir, name: &str, records: &[FastaRecord]) -> Result { + let path = dir.path().join(name); + let mut writer = FastaWriter::from_path(&path)?; + for record in records { + writer.write(record.seq_name.clone(), &record.desc, &record.seq)?; + } + drop(writer); + Ok(path) + } + + fn verify_args(graph: &Path, verify: &Path) -> PangraphReconstructArgs { + PangraphReconstructArgs { + input_graph: Some(graph.to_owned()), + output_fasta: PathBuf::from("-"), + verify: Some(verify.to_owned()), + } + } + + /// The headline case: a merged graph verified against its genomes in a different order. Before + /// verification was keyed by name this failed spuriously, reporting a mismatch between two + /// sequences of identical length, because whole `FastaRecord`s were compared and `index` differs + /// once a merge has renumbered the path ids. + #[rstest] + fn itest_reconstruct_verify_merged_graph_ignores_order() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, mut records) = merged_graph(&dir)?; + records.reverse(); + + let verify = write_fasta(&dir, "verify.fa", &records)?; + reconstruct_run(&verify_args(&graph, &verify))?; + + Ok(()) + } + + /// A record the graph does not contain used to be read past and silently ignored, so the command + /// exited 0 while the verification file and the graph disagreed. + #[rstest] + fn itest_reconstruct_verify_rejects_surplus_record() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, mut records) = merged_graph(&dir)?; + records.push(FastaRecord { + seq_name: "not_in_the_graph".to_owned(), + desc: None, + seq: Seq::from_str("ACGTACGT"), + index: 0, + }); + + let verify = write_fasta(&dir, "verify.fa", &records)?; + let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); + + assert!(err.contains("not_in_the_graph"), "unexpected error: {err}"); + assert!( + err.contains("which the graph does not contain"), + "unexpected error: {err}" + ); + + Ok(()) + } + + #[rstest] + fn itest_reconstruct_verify_rejects_missing_record() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, mut records) = merged_graph(&dir)?; + let dropped = records.remove(0).seq_name; + + let verify = write_fasta(&dir, "verify.fa", &records)?; + let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); + + assert!(err.contains(&dropped), "unexpected error: {err}"); + assert!(err.contains("missing"), "unexpected error: {err}"); + + Ok(()) + } + + #[rstest] + fn itest_reconstruct_verify_rejects_duplicate_record() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, mut records) = merged_graph(&dir)?; + records.push(records[0].clone()); + + let verify = write_fasta(&dir, "verify.fa", &records)?; + let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); + + assert!(err.contains("more than once"), "unexpected error: {err}"); + + Ok(()) + } + + /// A real sequence difference must be reported with the genome name and the position, rather than + /// the old "expected length N but got N". + #[rstest] + fn itest_reconstruct_verify_reports_mutated_base() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, mut records) = merged_graph(&dir)?; + + let mutated = records[1].seq_name.clone(); + let seq = records[1].seq.as_str().to_owned(); + let original = seq.as_bytes()[100] as char; + let replacement = if original == 'A' { 'T' } else { 'A' }; + let mut bases = seq.into_bytes(); + bases[100] = replacement as u8; + records[1].seq = Seq::from_str(std::str::from_utf8(&bases)?); + + let verify = write_fasta(&dir, "verify.fa", &records)?; + let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); + + assert!(err.contains(&mutated), "unexpected error: {err}"); + assert!(err.contains("at position 100"), "unexpected error: {err}"); + + Ok(()) + } + + /// Verification matches by name, so a graph with unnamed paths cannot be verified at all. + #[rstest] + fn itest_reconstruct_verify_rejects_unnamed_path() -> Result<(), Report> { + let dir = tempdir()?; + let graph = Pangraph { + blocks: btreemap! { + BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap!{ NodeId(0) => Edit::empty() }), + }, + nodes: btreemap! { + NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + }, + paths: btreemap! { + PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, None, None), + }, + }; + + let graph = write_graph(&dir, "unnamed.json", &graph)?; + let verify = write_fasta( + &dir, + "verify.fa", + &[FastaRecord { + seq_name: "a".to_owned(), + desc: None, + seq: Seq::from_str("ACGTACGT"), + index: 0, + }], + )?; + + let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); + assert!(err.contains("without a name"), "unexpected error: {err}"); + + Ok(()) + } + + /// The non-verify path is untouched: every genome is written out, and matching them by name + /// recovers the inputs exactly. Their order is deliberately not asserted. + #[rstest] + fn itest_reconstruct_writes_every_genome() -> Result<(), Report> { + let dir = tempdir()?; + let (graph, records) = merged_graph(&dir)?; + let output = dir.path().join("out.fa"); + + reconstruct_run(&PangraphReconstructArgs { + input_graph: Some(graph), + output_fasta: output.clone(), + verify: None, + })?; + + let written: BTreeMap = FastaReader::from_path(&output)? + .read_many()? + .into_iter() + .map(|record| (record.seq_name, record.seq)) + .collect(); + let expected: BTreeMap = records + .into_iter() + .map(|record| (record.seq_name, record.seq)) + .collect(); + + assert_eq!(written, expected); + + Ok(()) + } +} From 47e17def61407e0378b213ace6bbe47a6b25d63a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 15:01:21 +0200 Subject: [PATCH 12/50] docs(tutorial): add page on merging two graphs --- docs/docs/assets/svgs/merge.svg | 234 ++++++++++++++++++ docs/docs/assets/t5_merge.png | Bin 0 -> 80963 bytes docs/docs/tutorial/t04b-merging-two-graphs.md | 41 +++ .../t05-example-plasmid-rearrangements.md | 2 +- ...6-example-resistance-gene-neighbourhood.md | 2 +- 5 files changed, 277 insertions(+), 2 deletions(-) create mode 100644 docs/docs/assets/svgs/merge.svg create mode 100644 docs/docs/assets/t5_merge.png create mode 100644 docs/docs/tutorial/t04b-merging-two-graphs.md diff --git a/docs/docs/assets/svgs/merge.svg b/docs/docs/assets/svgs/merge.svg new file mode 100644 index 00000000..2dfa536f --- /dev/null +++ b/docs/docs/assets/svgs/merge.svg @@ -0,0 +1,234 @@ + + + + + + + + + graph A + + + + + + + + + + + + + + graph B + + + + + + + + + + + merged graph + + + + + + + + + + + + + + + + + + diff --git a/docs/docs/assets/t5_merge.png b/docs/docs/assets/t5_merge.png new file mode 100644 index 0000000000000000000000000000000000000000..63d8c1910bcfc3b8f79102d38539bfc53b8a4b08 GIT binary patch literal 80963 zcmdRWWmp`+wki^!NFVS%z`(#DiHiv-z`(qZg@J*61ib;SJQOA118*dTVhYkQ zFdkoEVEh7LU><=>eg`lx&R=0*j`Uz)xRPODuD<#~V);0z>1g`-K}~ zHqfI)L`+}q{bo(&55y;7^A<&xEFdv5+8z6!JvLmC+cq~cqDcaN)J*NazW^5K-snW7l+*y`yN~bPrEPL%EFPL?1ly&)KApA9$elSJ+$9QsuzkvW&hT3QNp6| zpC5fHLiLe{h56IqpyOV!xiKl3MbS&zPAY~)jL$?V$*LbgBGh>!^{?|LqW|QPfQ0a) z=UHnPmWYT;I!qjB=b7KY`0L7Ak^k$)`?vp|`Dm_t z!ykR8`k(H7{=&g5jf%eVnC!w~HOkdezqpoT`7-{Kc%Ge*pLu_rSUfcOIOX3s_;u!= z{2s*7>zp6=nGTfw{o?W}A!s_UxeL$GBO>D19Tu=-%*%u}V@J_W46Y?|PXa zq9G;K@Y8vCXY^|S|5pA^ph6MV&Sz;o(dc5mgjtiglvnQL;S_ktbutT|%TBng@k)L1 z4Sz}f-wh9ue>FeACK>PA$AMF}q?Pt)Q4$jg1nnAR!gJd_qA}d5A`~jf_cZonLa?YYk@dEcTW%KG%F2(^oVpP2=_%;n3EWRCuObP-q9FRg& ziYE}*h;b&AQR?JGDz1hX;DiD|1TQUnsYFAuUB*3-{sW=Rt?Kz=66w% z(yDy5l+M?FFBQ{nG8utdqBP$Fk?UMgR zLsf3(RZ7gor0;~iPnR(``(Rx}rXXWXAL`9w{*Ztjz4XUh$=^=h~tsP$7Kx=5o*t|h2IF7KxLel5{ z^Zu+WBZ01y7d#j>zU--OW7_t*H}3?Wa!Qferi0Ob@P~~IA_6Qi*r>R4%=>A3>BKYr z%hme$M~;U(qw(}P$yvdde5+0o1d+~t-rZZLX>Q7#%OzQ zb{3lPpyIgPUnfS5!%Jg%xH?VF1cnS))mlG$L949YgA(gelnMH$f+=WQOf?`PNfbFQ z;fBX3qIb~P=_w;KUnZMxJ+-@JU+4)O4tH%fi-u5$pWcJ7ujko?dxDa$=hpd`8_pW0 z}DXdvWb74Hwt=_ z*-k_RjcvxK-+$Evn~U;y1C@1218h9?9~tFP)heCzqo+;Fe%>Xc&?xBz=Uut#Bs zJrr90_N&#*&}`>)avSEf_HVxtN9^B%6B=ptm}zwP=kb|_>7n)NHFDIx+PhHvOoz=c zJ~`6qHJbfq6k~V_qpR1W1_LZ4FBNR9kySCPabvQNWShm|J}GlgPJ}y!CA#9$e6e>Y z%aoMPI)xagncZ<#lg@11j9oE&bgo$9Kb4lTPjll@<2cJrA@j*KeX}ZOuSAFc_77%J zG(GOb?BsiKw=8Qq&&_?v<2r+FZL<%jTFs48_mZe6inxN?d3)KaZXjyc*BLxJbsn~( z$aVU!h#D;&c>nfB6`WMQ%ZX`Bd^Tsd_6M!kY)_(+)i&P^-i8GhjR4dWJEw)dZ1xG6 zf~Nc^dwEeJpXRNGQEaFZ*18JLf|^Q``yTqzVe9p5!hYJO+ANW}F=*`Ea!1^Hg_bmXdzJz0_$MXH zu6BPl89vjd?yok?D9leheoI9SM~g9ng*kX%YSOZ3Ml)PdQZdo)@x* z#|3qJn6|WGs!v{TeG0~^yVyc9OC98wf4wZz<-Z>riaFzqxX)>W04vt?Vd!s^d+bmS}?f4zGe)P7Hx$zHEk@8x*EaNGK{riGS< zXMgyI%s1od@_~Q3`$L(O$8kMzbjF(M3?9e%pKNK<`PRnJe5muo#ERL%eeZV+?vF&! zoyY1{uA0fcN|$Gqx@7{B+_G$ma8#~J#{IM;cHP#v%7gmmNksq5j?M69htWI+Yc{nz z8xC)ev%rG()++C(Ti>lY->9RXUvkH)_j;w{k9Fb8=Tq5yy$@zie*~|0sNmwx9A@it zm@qG|Hy%Ibr}}DZGO{SDFP!(*;4`F`YrI^)8DC?EVf?7ubiB=62Kx~fMVFUMf9TKG z(3n!&suQl|+E=?#nvPG%AQh~07Y}4gix&dq-&3rWp0>j6aV4t! zoSF|fyOpQmgA0Tug}rh#3}1HjM*Fs%5&h@>mL~HD)mw<+hyHdx@>MK zQx9;Ok`kW!;H_alB1&pili0k{>L*X-PqGp6DqnUp028Awo#wy?C9y9*eSW>{-kQ0q za$s<~OB88e{BzMKtg+BA`bC!2=hL$BmsJ|=mh2mW zt^}Q^AOPveDr2*@-<^+t$uQoD7>S)`{z0|_SJp({;IThWk&djYC_U_2@uIA~vXH~} z1tlWWt~X}j-RIpL%doV1sFT$4FIUb&Rt6gn=k8TP6{7{(C}_(aymcE9IG&Bx=kXG4 za!@}$oSs*=WjC$+XeqG;bY?n37UOg@)L@fh`Q(Ma|S>A)uQ$@k^bB*-KOT`*W${3CJ_e(I>X;4;11S41SdECto_PzgjrUe66baMdMh(X0{8 zaLMxX4Y+O2$c)Vw+BXNke#isfJ>OgDJ2D~Yr(nQOWaXj~qpDIT2rQs>ef3%~v1d_Y z?YP=lg8FzeN3RcJvIuOQ@wS#1<$re9yIyc#ln>Q^1C~{+;I8Flzmr3N9Q^4^FP#+6 zH8JtHnr~kBOAeH^;#;gdw!(mj(USPMdyVws!e5CNmZuPSj;8g8A4#pA?S!A~;`x&? ztGxDlcXMl7fH7iqSGak&b%%c(t z8Nrecp`lu}!DMnFwlDqteFam%#Fgbotf&aYqte#iMF;PYh>p#{sv>M*oxXgU3IYt) zeP4J@si>hZ!>>FywE~-^YYooumKv&!rO7zcuRGNjlcFE$Sh`EO)vh)YihS<(-@;tO zK5k9T4SfW4_q9C@0WFxEra)11z8S-0*KQ@qRYZc1;&6hKcUuUMT|>lYtev(NX}bmb zTL~Rw_BDc*iqBxBFh)Q<_{{SSSgA_iQI#<~@%dw?`XTe2cW>Owl{y}?rNz$gDp4XY zJVvR>27U^<4kjbHxje(0-aHO*nTN;j0jllgEJQ*+a{G%PC*c+Im{^Kzo{JUu%xA5G zG-n0UWliTdve)4@nFff1mt+0Yer@Xo`uJUPd=U#Uv2H*aK@AW!A zU;t%fa9^Q8;>rn{kNB^alXi_G#Beytwo?+aQjk=nGs+L7lpoLU!(E?_gIhG;VqsL^ zIBfgwH88vl6-OZE`+V|%-!V0F7-c=4R%vH9cko-%wC=T=Hwepdsq&s%SBDfOR;#(Q zq2hHGuM&re@#b_sv0-2LH%fF?y;n~vHap>xQ+i{LFV<=)+SaWMh}&_vT+6rlxcF)eknDMotF9f zzpHksCr`4|BMlWc=16K0x?k_}3$SZDRB_4Bfw_CR4K3xbsVIEPP>ZM^&wL>4u1ts? zNp1HsDkz~a)nc_c*r}cAtit9`VfpH@5MNuM;$_vqHBL;$TCc{X{Yg=^94+KQ*2_WA ztGW1Pa6~?Id(XJT>1rNu(dwnzPJ#_Tszi49MNi0GP!DW9*)6lN)M_>mJV}9BaYFssh z0a>lK#Y174_~=1GP}Q*a%~qZATL!xQhn>soIkAAny~i6nYt?RI&7X-L;>BVS3Nx9h zllrrR#*>IwSKm|TLbY2Fjb1~<(6YXPUc7##n&`c-!2F^6ybhIU(E?=$2w%}_lY&D- z@ot8E8CozrYmmtXi`IRol>fW|f8w&5=!Fe;xlkPe2fPW8lhgqpJ=R#x&_LTb^d_n8 z2fG@Zm{B1i`e7Z=yppQHdD?&1!{S3~PMhj(s`iYc;B8l%8u*sHN3F}C%g#4+X{NL~ z{BBc0o70zT&~(&u_|j&NJ_g-b(rkE7I2ah@&qxvNKoJ%TJGF&9^$6}u=a@@~c$I~eg$%r^{;24~0qie#AGX$K59|qAHba}V>68JJTs*wa!2=(JAtxQ+1gY?` zl(6b<4Cw9y8_iSnrEGjdxnt~rFgp)tg`f^XP<9D%y0f`>$nX%i#*$o-Fep%Hpu)Fx z+f(>5B<bJ` zAYSNa7S_G97=|nA>=#+PhgL0awP7zUzM|oSle27!@PV#)XC!u3v$gf8mursF8Bx1& z_ghu+8LyNy{iabr;n9Pt%p3Yw^ptDutKUA|0yMx+j>c9@Vt4MX1e3gbFifMro` zb2dHpFLxjyrHDLXnVS$;lAfoW;vGEiveQDxh(yiNF28+~LO6iF9WMyy@|X)-Qeo~b z*Kwz($`IRXAYU*JZ<<%fSIJV$o9r(68W`j+stKKUyKihSoLS2e1c@8wing9-bw;oE z$5r!b0yN@SWM;zb;g!6&ei=%Tj08rA+p5(s+t>ME_p#TUx#4+vW@d3dz#Ro+z}M<@ zX7Vac&GsI#qKuz9mBB&|uGhNe;0^Z>|6Y4!_EYqap0uu1>sV6r>V~5#^wPw^yC@{+ z;*m?>e)31HPCW^vpjpbY)gJIe#bak^nGT}zg5`7&fm>#WUax&DL;0f38NaTMAR8*h zVeq)?h-#qFRlq`1pRnJ)ZKJ#0^4t*mSV_-Re44I^PY`njmI8@ybRVrn3tfep3jgs;GN&i ztxfE;$9O4maP{6SStSQN(qwcRF7|4h2J*g1Al5~#kjSiC)doH_K5%|_G(ZiWRw4n^ zYg+7Uh)geOIbv+*iUDL_BF8(!B%H8h;JBKb4OrsYMLn;Xi5C?S4iVYLvFUDkw{ z7?eK;$tK+J^08CdH;vpHs6RtaTKuu_aZCzu|E6C%&Kb#mz>#nBWEM5wF62U=yk|xn zZGA@?DN$|_VPFU>#tROR6%SI@!U`pyT9@awESl5le8Pk)Cn>JpI( zgE*~Z+Unz!HzkzimMYz50xzHI+i_BI(OZ`CSx@bXeqdmhUBt`0#VRA+TG9|GI(9md zt!Ljp*c0ZKHagI8|IXcy?!EXDk&$*{U!Xj=Zl8txnQd7da|?Nwb2*mB;Lo};Gc|)# zW`69Sd88>mIW=m@ZVl(UwZEJoW%q^v3p$p9jpGX|+AtW73( z8ne7CK~lDm3?DSF7`0z&Yev5f$L5t1*T0DFn+JyN~?+OG_n#0`Pzjn z^m|O)U0>ZLNSg_gX6NkA&4wS|2&xAP7PuZDB1&^#!yWHV5%f%I*9&%mv{F&$mvUy8 z3bBG*N~Or7A`Xfj;brQ5UEQiP>7TM&3{0d?&dH4%RBhMnAM+ZlEJm9e_~)>br0>t2 zcP>%@so-8)l?HF=EK<^+OE+qDRy0Fq8Y^hw{326Y*Etizlyz~59T_>v^}dK&XK{4j z%u`@!z|B~7IiNSd>CwycF0OTZcYOWacz@Y!seaP(HMF3X1$TLW?j{_LEV0%%Sh#uF z`g+$DKgI5$YYK5RKC^<3S5;9npRHbP1g>OQV6ij`na+@YmMHz`Id40G69f)$IvT6k zh>r~`@2iJKF;?Qxf~ySIp7Ce@>2tkzkyybfEBC=f zX>EcO_|zn%ccy+!gZAX|tuFcLvq$Io-HMZiq3Zm?N&1-Yo(XfyMNUQ;?jc+^$uWP+ zi9_i`jKt);5AGD@U{Wy#K;mtieFfHOI0?o=TAmYd5!X7I!%CYT)L^(tLvwit=jsz(mwgr?xHvNZsn62f8Z z>e+AaS;6BH9TywzHZ#|+rF1tH$Nr?N3)bvsAVKCVo)=9^C^lmMR8U_5Vi>>y2MY}t zcN+Zob7woU2mmP?7aL~R2D|PXS0nhdEmGv*6~V0Uz~(3pmHG8mqnYMFugbF>Cpjmj zg^zO@ud>lbV#RUZ=))ofUm=qA+(0cCWr;row23n(%@x$J6My)@G6Ix6LwQ)5jizl~ z45ofNJ%aa{KN>K~+ndXLp>a=(za+cbi?_3wV{IL z9VZ?rTjry3Z4LXe25~Kg*1I1u->rpNMwIJ#moWgf!6rB3*_O~R?Ts5~4e|n5B$8p9=HEEWhusw^x!$U{{zruM7w%W=; z1AI z;0g)jcW0~k5^pwNx8C)IQZim$ zMoxWq1z^Y1kGy9_bBTbMQGqgUwdq4PO&QkOq@(L^PnT?h*xU~;uDiyr4y}cB>bY-g zMrZlF*6dKy%9?GbLm%fF3^ohJs(!UNQ_6#L2Hy&zK}1R{uR=Z8?@(Y4vJCdd%)I5E zSI0AKelbwbW{w_r($sn*b0cP~V^cp|%n!4@s{>|994nVW@{jR$YwnoE>6g25?WpYE zrCZS*8pNV9c>OnkXy7MFW@=PBH^9-eu?bMEnOWS42O51Qk0U}f$OtKrWA?o)%N4N-OC_0+H;D#K- zTP5+f!=|hc9@S8CWzE%X!G~02ODP;U?@QIiieQG05n#!A;^$umJ8d~ly88aiW#mAG z048!L^CGG-)5USP`90j=(e9z38TzXZ-h#e`5#|@aWSb4A%ui7ERF-eaQgv>(?8#^| z846#Z$Ap7aBI>+`*od0WWco7Wt$dB)*&;iiohd&B^eL=qZ{5`y=L?|KuogI=LIS?U zc&s{=uyl-7rowsiDwP)91Ov zn(#~a#E7h|wQaF}+j4l()>7g)!Rj32?WEG`^W=7i%OkQ=B{Vvn`6-NxC&jhYYC5%E zr=D3g>U5C&yXmjYK-m%E>{L#215ugW$)o1W-tKv+sEim~YRsR~9#4}Wmu`YBwLirX zky+C59DbKC>YS92FPkf7<-%%nyPsPO#jSD(Gx<=Abc`~;r*bfDXyk?7!J_3*1*out zKNH1ePqq2@zR<|fqs4r_$v?YONni)We2$>DDz3CM0d+k#i6iO%==(+TQ)pC4XM;SJ zVs3@$6^tBFIunWPsTgGBNxnz`VEPoFnr^>;)OyEnO8E%L^*s#fgv^e`-*Eqa#i=tL zODKC)e`+c49QK%}Sd@qf*OmREJ(aKZDx`@kIPD))z+jIGo z>jp>t(V)BS%FA2nNl*6la$2iox89A)1HGiUVmd|Qhhr`4qUJP1`~CxU^)j?{K%FZn zpG8VZrdAGdw%C5iH5P5ibmS_szk(xN`?)+yM2Zm~IZH>(Q*Sfy8+Or9@kh7umyEm( zqutoDeT{8Gz$$&C_%pfdI2&ldIxr*(1YkV0#XViIAY;GWln+(7zSOAslUL5$jPR+i zY}t?e5ZyXkVY+{8kX;~9%3Hd;zJoURCSg)QU$fP*&&a zdsTJwQ?hU$VM$$qV|a(I>ho$dP|VA*DGu9t`|^}+TPO4^?Qti?#s`njq^B?`E3tt5 z!Pfu=2G#5x=^7ca&J#dWJKkb5B69iGUO!JxWuDb*TUqpYLmRfb`+e)w3F`8Wh@rO2 zz5J0f$Sp6upJmJL8%s8aPgCe)!+oGuS;OM936@7gtHnGS>4c8+C6fG}kLh7Z*j2OG~H`$Un<>k4Uf!C)Z zpAJ9PvymL?#wk4suF(P|_))W~Xu!`n$qPFp~*c`>^@Ufpx@fJ=p(KL#Z zYJI9Ro*0kE0MG|!mEZkE>|_T)@q@nY?4bhT=vNRpG$x&qZ4R>JUt<{cmE1GU#RQ|;%Z!p%Fw;`N#|jX zuZPnNif4(>l5U1(wykt5^3QVX6w{SjASwU=5t zvLq{r7>C}Ml=lT0nNG;4-t{sVBiK7OygIzTj2E-n)S0JgW_=P4=w(Qcb3QBkp%?g7 z3HN74C~DWhSojS`WM3=;KOv_xWHOZ)KRVN_%vS*PJ~pn;c&&-c={`m>c!RvcIjj2j zeR(v2}R1gb&zVl3;Sgaf>I>FBjTvlrXJFIu%KqQH{k^+wIA3+eB(Eed}=jP?y zhggG@U(-lJ3i`)z5(&>!Sz*RqfS4%nlz%j_x3bdx8Jqo5uJ{Kad;kcH&FRw5_e74= zY&Dykkg{5YqV0M(Bdwjw%|ce+lX+|)V8Z^Y+Ekl)$)M7i8LNAuRe4)vbcCmt^X@*X zH9s}+_Wc+1l&^ebOrBPk$%}6p2sx}AmL!i)S8De*fZhk<2(Tl?5wdE{4r=qd?cWgd zfD`&Dl z-pvq2vU1y5G)jSw3en6;iy=I7_e2gesfB|VW|rVw&&VT)_DWdRfNZq{i^$;5sNAdy zGg`E@btA&U{2B)WhVo}g0|K%o5N)3dd7!5SP0%|q#B2oG7E}8?l=*i(-#q^>n}8UI zs@ZG#_M`5L-O=JaDnu}|k^ibZZP`R@nk^Em`|Ee6-`;=@2jhXAtm)Jz}#91sQ~DykXg0a1E! z@8?iSnPCx@DO!eN;e>iJg8=ab%dHjZ2da(H#nEm&ZwXT^%5VM;* z1q`^FYk@l$5()e4WGxhl2;5o$4TZ~>QJ)2JhLQX2Xer~JudG6)DDAh0vTkg4NkgA? z9bfm=c9QrLPZ4^azcN12l=`j@-A#Ufi}ELfz=j2%aG&}b1WpNw_@M^`(cKdYP9woD zFLk`ngE#~9HcXqW=R?H7Zyn+rKHuM^;{+ei>Hqnf#@H*ZkH_zWfp`_^0mRRCg}VSw z;+r@zwZCZKPopkDgG}A4nYU^h`~8c|^F4Y^g=ki+-&}>x^#hbzQ>ImPR9?N3x7(F^ z0BTV7$fO+^?DNM`qzi<)&i2RExOthVG&V{p%;yT04{eJNmlc8he@h@BeVk@P`e$h% zqEFUJx0>2zto8s`ac2+#n_rfjAW+xK^>!3R1IoO8tE@Ht33%>74{_ERGmECk*nk)z z!E$!G&}2$6L*}X(+Y%~daFMad|1IzR>onB58)Vfp6S4Ex?0Nl0&JiWS>!vFUx#c66 z#b^qqK95%*3qYp2gH+SI@rS4oeFuS`5~LV$5TeP#nq8HHld*aUbk}i7qFD5if5Cje zY?+jmGTaZ#m>Jr7lZK535#9)+QekmBd05+IPA{m;^I9zZj<-y-=T!vi_zMF$&G9Tq z5pWbp-G$Vg?Cct)Xfc`z%9}q}D>`d-P=`vn4cDmJ@lFrdB>xR=*g3sF;fMYaO#Gpwmdc2KEP>zyIBR>cCm_ z_(9q`Fuxn_ZBWnZ^~jI&1)Z2=)RbT3?g7K%O$18P;?sHfM$wWwFEejYiUWS|p%`HC zfw8ZUTe&`WEZOB0=GAs8&08@qY5^lHCPE=Bh=RrhF%rKbH@>MSYbKhMBv!)ttq9qq7=duj*1?NB;=YU-+O)sIKF)OKBhlnJS+?h?*C2q zgycQ{&HaBJ4*dTbQLOO6TN#0iVRi2d-Rhq0Uj z1alxwVlruICau(#Sv)K6%#}0-Sx!gn{c6JPkIJ-Y1_wqXj{9i8xDJi`SOwyY{nYmn zU4RWx27q*qSrW&oXv|RPiZ#Ri;V8p`Y<4!0BS=X=$m%O6We`8DP7LAbejI8Spd-K= z;shOY`pKweNcm)tHT1>m?iX6P%pOv<40rn$=`m*F3`v4MV+aniVDi(1^^4NkCbDLRCvdU$>Z^nO<~>U?oQ2tCHxZ(JV*pw`>LDC-ZHD)bF6|dXj0m2~X4} zGPiM1e3H){6O4jSFif06f`i5Yj3q%AqW_ermsh-a(=9)p=!g|BMRz47LUntbM@_nmn@Qv6%qywZ?Zr$Vi zy^xDu zyzN^UKW1P}it`q(m%YQ4NKMz04S3HNN=HE3ZWF7qhf+fImH(Ejp%TYq-`2`96~lM=D+%X>)6Oi zadkPE&tP%njb#I)KhNe3fve=)k^Fq%ffeaDn}ROE*VkA>pSwU6!@Wv)zk754Pq2)zc1>dB5M(-P3`d62@ubO4H`Y@^xUv0%P!G>-LmRpT0p zDKhuy5e3As?;!U7Jbm4$b`s^abhK=tLn_a}??3$!86z&@o7T`5$<{Muds+ua> z*%Qj`h=V4!^nb#eyhCt5H~vr4CkhH9XT15%C1+v1i9CDtE`0ak1uq$3Uf%W4wuTBE z+%fMpxE?&IdGzL?$M1X$hfFE)lvDiA#B3F`&9i+`(ayHF=ZzgsOQ7;N9YTE*Rt7tP z@@=Hc^5|yf%D4V{O(V@JmZRRI^?EPV0p#%ZU!AXAu;3lXqf@WyV4KNz6l$1+#~D6+ z$B>746>{tJhvIQ&#j+jqOZhH-)8^oYob-{Su>R(>@&OpbIPr$)!asZ5RzYj9~@ z-V?l^2M$}@V*wS`d!f2YTXv4`*us1lMYsm(OPvs;R z^U1I0YX*E=p;kv{GXu)Qf^FdJ0g!ZVy7CkG$2D53OXBU;3WWqj$oV-V5Mxn4FxuEp z$b4Wu&}OBZdC=%DlJK<8ou89m=O;T@np8MlXgyyvqs?>i;F5nUd?-wd`46wduL#SEDq$krA8B<7S#HngQic4y z9TnZQKs(RXx$~(RE(P+YjQFJ{ZKJ;%7OZ6oR(ES#tgzsPC$Vyov>l4X&OzTpqEUYi z1c1O*T9k<&gwV6qzMzT+8l-#`qlz5>2fS-VFu)?3(mZVOo%qD+QG@m=XaZP=t`R3| z5o0A}?rkpL-$c>vmo8t?(V5uNn{}c#8|VC;6L(l21(YWsxrL7omg?l5iknU^s&74+ z4hZIDW;H6XJl@0~*1Ys(ehQLA*!OleOpbwn&&dtw36Zm-bde&~xjOSkR_079I%X4d z$A8&~>uWwg1n{V#6MTsa6a%6WvSY(027XOY3Y!_lD8aIGz*+)pNx=MDEGDqeF6zZ4 z+SC9EaS%1vFEgjwk@|~L+VfWU zqJ=#ayBRAPm?;?~Qw!iG;9wDz{dqE*)<}ypXOY9h$g7zEfon3dU)!%uaCYn%1q`&n zU6=+uYLIuKc4+Tbn}IOTxd^t$OyautH*4naFw01=Q7{;YsACa0pES#6o92kvp)P%Y z?cDd1Hs{1IH7SM~x&B#qB?l=e>ad3+YIwsF*6)u(k+b%dd)#zx@d-lqPTFw7ns!tt z$@Mnl^i#_<(YmCkWuOD3cmm#5+0Vc%KU+dmeL+rMO;=t{*Eze>Xn9-Q+LmWR95GZpg;+k6;@?F8c6~Y>%3W!qhP~)UD?#ODZ8Du zLxK6|?|5SX#&K<;G&L_G>zGj25nwGY%KxH~>&(hBg-5sjcSjR_`^w%JghnJ2K!pIj zqjgG3@0sRb{ft99>>v_*fJ;36x^UX9X!Zs5N?>M@BYK8Ev)oX}>@rzcM@uO->1F zlDDS~<;}e}b1pC0=nPK#K1THhy&!~n_;-MFEs4`-&$sTg6;IajCsuW%Rc?f%)vi3^ z;-U2#_Ibp~$0SH^gwjU@uj|d}Fq&Vx>H7QS+zty~91d#v^GBvU*|V3#;tas9v#C5q zxE*>i_Z=UfmOh{fhBw_y-2_f21QyT$Q5+H2*v{}gdDLcqCK`dTYg{sncVTq@?7$V{iU==Ym>eH(&6Cs7rKRDI#Bjitsw~yb;=Fl#t)@wLG00b|LZeNs!Jjl> zH&{Q{jN;n$hqj=T!rx!lOy`4#gngPXRI;5{A0stvjWZmwFnggE&T0&!^HW8UO~H~5 zCFle64{b7qEg`aI#{AJdCWQY(1_FziTEm$2Hk9>*84+Dn}(y@GykkcEn|F zz_{8lX0hgWnz=ryrc()O&g>rzu=d*i3db6=Bq0?}&%L0=Q? z5>_e>51Lq`fJJ>_5j)iPHf+Ovv5jiBAzQ)IOZ4^78@o6hZ(}P06A0z1q&g%sF(Ush z>S5mi+)~#s{S;=EiCJ{R&lf5!jX_69fB05hBlat02dd0U zRJ!3o4|8phpfMjNi>ID3Zvbj*f((jRpWk{NPj|4^5Mt6(u&rQrQoP|EDk`_;Y$t>E zj6D~VW>vHvUv=g9i^_~2=%4>EnE;&QrR^^$Dl+dwMkrtFa`UFAguM^9J{n*p=~qwB zd(F*kOj>XS#6TiCnpuq0(si-5+zI_B;9G9i2`^gPu7Er4#{EWgU|>K(@%4-6he>^` z>x^@mb5@scrAQNX>SzzvxI-)D%J<)VtGMok-8OwfDh(>dq4GAuGNo{^wV5vCHaFou z3T+De-2LV{F2nK)p-l5Up<`TTGd&3)Q>ny*0pDxgBVkT;IF9*lNm-491RjWl{H~{{2Yf zp+WPu+(f`zy2fau(XeL(>R8rHe^9<`_x@cEtloa=c$ymB7)k7&TC`wZhs&+(TodC^^3_%TeB4^P z@ykO#!HlTzE$Q}87*cCY@@%HRa8Rz}bBFzsg|7rCK~fbfXc!!e*>lgT`+k8PJVeI_*&0uus^&vz<8 zkF}1hs$=t9rw{Vm>I(+1Bp1*}t$r(jEpjg9c46ngYAwl6JPvh($Hm6D9Jt^OnLD-m5 z*Jtc~^eM6eT0%CDxo_V?f>AA)22Ozzna^AY}($ zG?+I_fSc}$S;^7yGR`CH_vxpr!pP8gYO&fJ-o1yjn#m`4>rb>I%kAsqe>Gflnqsxn z-`}T-Q77B=%2^D=FvxO^+`c-4X?OJJqhrf6Nyy%U5eMOVXu0;)BY{I4E-cU&%SVSl zIkQzvPu<70$hU32N)84mCurWUFTn9(KSBAR@%e)_kLarFMNbL0O49ojio9WLK}jW; z=IZ?BM`Nm0v%_{hQ2hBNPMmtgBwIgWGW+aN4XulUY)ngIWgQXHA;0E&?Z#PwacS zJ8C2`k;uM;VuD%xHBjfUkF2ZX%eP)!_3b1$qK`u>VHm?@IJ8DCW$})0N=qt*s6r}K zgK2tRa*Vp$Ki2opTbkc?@X?5f5v7!EcmSv0C!*!>%JwpYt>94{dC`1PrPI(K*V5$D zF#IGB%dSfsj#l|y(=x#x#^Rv)n%5-m_UZKU>XweSkYMX}-QBD=j-Q)YHQRZ;ZuJ*5GpnvCI$Nce+}6x*2p1$CEeKW$-w zW_o!q+)onCxoa8l-S!q&u3=Umfl;uqNCQ9wWo>k%I*X!9MF3d#t0CqtYCrZVt_nR4 zy+^^t#C<$``jPGbZ?u2kbxL@NI%7Gtg?0pXUctW?LK=3Vp#JL${j-G{qf2d%l~J?= z9H4r}*wt!hg?{;TrED*u+{(nnsDxRe)Eij1l~C4%{<`nf$l~lFtu3hyD8LX+-RvTo z`D7#?>BsHR&yEG-qgXCPNmKvt#>kERh&(&=)^<}Zm&ZdU)323 z?SNKAY0zFzMdL}kxFE>jKo}ShQZgqyjuM$cFTuOT11dE&Jq)xz4AOxe(h$+``Di&2 zTgO=Ih5iY1`^rDdL(1if8bHh6U>0L-y~9SA_MIC<$phe4r1Ld96no&OU+&?gqj+ zTZf;OOCYqL7rU!|xukcr*Xh$crm7B({m4ix`CX6jyXOd7Ud}a)q*xd@ zLqRB(FT_N}Zl!r~`YA=|0y!XJ0#pVmO_We3_+&gm_m0=j#VW$O_Vy65|3~7|+-jhuB}_ z!F>pDz2`NC_f=q{ao<^jPm1L(n5F7qs`hNr0S|UBDZQdO<%E zy&3cdpn+Ulm!~e}TdHzP zwzXBKlDs1e3d@&5m(Ca&A38U&MNV}|A8$BawOy@Q_u+zsVL+ZEWUg=nbtivhd|{WAWkgU-MO*8vE5&9vO*YEJM8Nw-faooj ziZR{4{mcO!c>3`%Jf$zVj~yTKhZN6PYVAhq$QbK9U)&uW(BE!$hBpF^X@NlC&yjh+ zxCS*`?zl?O7G;!@Ys%&ypeL%(pHcg}VajRklDP4bk_VX}1+mr^*K5ZX8Y^HS^SYya-nu%H#z3Fj+kG_ef$> z!%}d6F3NY}NSu?pGL>J`3@?Lh#0qd$$PJOO>h6)se3~@Gx8ZXjP-abc?M^Z;qOijri_6Erqk@t6b6!5I9-adpst1G#E@O{t9i5JUP^?KK(^_W*un75 zH)ZPQm!FfrPtKKR;*Lf$8@w1#zI~OHc)ote!nwFU04vgI=vtYH)358T8_>^$CAQ$R zrZ(NXWxtqnS4t%yw^7XCwZKBh?dRAj?8}V&eu5;0$J%_pddP~(Y8~=8<&2SewX!+= zq2+RdjHzCi<;teop=(++Kf67N15-6t9X{@>p#(_pp~=!5Xov$1f^IK2_t+L}TIw}} zUtdts%4>tL>MXY%z-YpjmPV=@@B@ z1E#9Uj;p+fJK?tXO`YQ3sOSAgCcf<5=7Khc7s6u5o7&PenY-PEa2gx#I(Q79ZH1#k zhO%7+D(FkzPXBZaUmf}Q@({!pnGQs!WrMxkz6)q=9i<+zlejH3iGKwA`#U7`r=Jka z`&SsH<=d{lUw-o(9->T~4R}K0Q!{>g4ghzI^M}X>N!)ugrVu8;{0=5J#s#3dqoZV_n+sBj+S4XW9}HrSiG*Ee-g52egS-O)LR82b7*+0Y8;UZbSih z?)#&Zv!1xkS}-}Xc*@owvbJ#+m4Mpg6<^`PokJLbtX!oI9`aOV6tDcq&6$tkT4U2* z0vZ(+w7Qn*^sx64#@c1NHsQIP!$k z=p(Df#gj&MNE`4`d|<9C;;pLY<*s^wv;!WVuR4m0$AGl(0*i>K7n6`TJcmRHafmQ7 zpJQzP`TbG)Rdmpp=JMd?ux9plQL&@1$Ped#_}1jU!;V}H#n6QSoH?kC;Jz!Y2-LlN z#qTVR=&s@IzIk%<0;X9Y(0p>V;V_2F9h&kk4&TS;hpyQy0V3%Gqx%UzzXG3I$b5W- z`U;Br0xq-Vy2sz z=)(nTW)hhH^(HnX<#ziW*70@ z`2M3I!77U`(|~jHVuZ(ddAw#0omdKlJf;OQ_h(E=z;4yOnvOq#sfq1+mAiGZ1~(u; zddhOk#`RS5%S7Z>l{RT3_xE-_%V$&asOz=QcV%KldTaJTxCAgD{{IQfHh0`cDp*N~ zIhf6M=L&!{;qk5Qu;Fl+liy#+(bi&W`rv9?mK@SzBm<_RHx)z1^N?S~#9P*1S(zUF z3E3_(4jN*yN8fIR`P2)bibs$I%FpvO@pf>NZYZ~zvh0RIP-WXG~f2(2Q+x-$=ZeYySy8P-h-a4 z=QIz$dEp_r`?axCG1G`ZnNApM1s|j4UrvM#fJV{JtK(7tosc?VcdcBZke4Rxb=#@U zC@oEUoiXV^ZZfT1|CTrcMIG9B`j?uHKZ=%#`zi8+qi-(hAFcgV?uKJVCvMVTs{geU z^@u5qLVTBWd@cpto}J^NiPvntA(-j*%epF={-4=O=i|6MwEk0rle^I*v3_ zQfrikPXat1E%mgzPhHTWXJwOi0rnM&dUf@}4=y;67r&mU@mV0Nv}j)-g7ON{)(AbxPC1Lk_(e$pZa^jNJU6YKFSy`FW-} zt^ft9kf)cpGh9og?I|*93UjmTv>gi*5%{zC&wxw2tW7je!XA_N1N0a;oF;f*woage zOH#yFUy_GDMt(q03qixi%uMV&#N|5Qimk(i5(g6l!*kow(hST z-h%oIUChewt-q-GZv7TNdm*;*W4Am+7Hw=v@0YNY@h%Oq-2HuCCDdh>^~1*ln>k4q z&tP*<@T|1_(vZgvn4^@@pzS&Ol7Q}I!*SkTONL%_zhmH2V;hPa!-3~udnWry>V{5F zyJ*5L`e|OAjXgc&Prc*HXJ!vu`5A%E{*+x3)bDl}uy_Zvpwb7Bb;on4l#t|Ygk6$a zBiY@U|NB-y^HbnM4~23$GJ;AjiDTyqhD*2b58|UjbQv%_RA; zu`)!Vx962zI1RlpE2q=coekVoD6uh5#~EuRmdD!hyZwbQsLwUOap$Q`Ju`M8(@xE) zP>m^|j|T*r;8obYG~mwmIPA;PT(=W&>RwyvzKL$7Fqvj$8(eLK!=`c8m}}t}OEin> z5%OqZ`7_$-A#j0vmDTrgd`ks0ls`&-PzG0+Kk8^3)Jc9%^IdbV+}>Py;B&{CW1nFJ zJ%#FLH+(JPcc*K7vF|903nQ?^xS~wPqB6Ps90Q!Z>Q>Ah(vC}?EXw`NDJ2-aV9+nb zA4J82TFILX<13cT%l<@c_3;P?_`Rcn^>O>&C~;D5e6{C41-ML$Zs${IUvqr?Yx@gi00PKmTq|EQb)04L&u+f&mS>pyjgxip?b_X$nar z2$KXHi(DBKJ2~D@+Xa}ysxJ$QHgGNV^DxY=9*8VyeffxN5LtARLZ2N^3HV9vWKBbO z%5?6`(h4Ax@L*kW+Yk5Xf>3F7-8h0aFbP6$acWCow+rLP#gr7$8XvzQ=fQofm#b*O zH@uK@@(L}ft=@b)eaJgkE5a|P#JVsyVBa0r{QeU?vg>ID$<`p0(vRLiNMamqYmxZ) zcaII{2Z&edug+df-1Bd6^ny9khV1NpF^HulQwLoRVNtxA+ghw?K_fN5cktI^ezX(| z_^$p%<&|LL58$>Qvn?$qtuaC?-^> z2Ld&3v+2tDJ>Tq8zm0jnUF|wQ{GLgw+5_Q#AZLhGZt4)EOqKo4+-K@kNq0)j!<;P` zR9fNsuLe_>N3SyS6EI zfQ3=(b;R6K&r%L^fJzQ7%_A%iWu>Lg76>?sctQ`2_^ITYHFT3(A5xaD5-%8^nl*R> z25mUg1kN4&>PB?aWCA*ASBQ~)fyqa)SbSQ>&Tp|b1MUa!g@TvFQT^ema&+=WS*+zwj=Q)aT4 z8ZKi@p2o|XguFl^xs%y~{`+eK+=^+bZx=*ifqEZ~)=K_~g{ZEMS|b;zv%Ahglvcu6 z6VYF;ef{uSzABdOS{N9U=<_QfP{aeGm*LxXXjHnNk8fR%H~zDJ*P%wf=-wZp`Au*a zX1nu^Ae2e7FT~-63PdmzQ`}@?R53PFmIj=)1VmAv8Cd(|);`y5tiNq14 zzcjn$@NiR*Yp|!CRp$Df&_l&->sRmL;Y0CVVOPVkz*#qY&hSY2fT$&E(eF17rCY~K ziuDL!ylt5%72o-0nu7C54>SqoBo*|$PI$}qeGz;Kc^`V~cURll%Nky@OpW-nmbhqc zYX}ajEJM5v6_}=3C^XnUMsa0my(_%&`a+KrOEz?a4ZJ{6s#dj=PYc!Zv349^-+5U( zWzah)HoEQV%4e*!!7y+f;&7S?`bf41KpbX*mkl5!=~%-zQ*Z{9W`#bv(fATehbPf9 zX!v{4NnqSG83u-MP?;S>_pCeA_Zh)T0RWBuQ ze81W?U0Ci##VUi2qjngSeD>(@! zFr)N{g$x@4tjt{GEyi#6sJy~5ipcPVTESRbvNQw_Rq*n~UA?Rv#spzq`;&wuUO_#E zE=187ZCy)$P1S7(4s(=7^4UWeRHdVOv{&{`z92q|k!ACSA944Gg@>P4PY7E!UViHD zJB`wRdmWfIi;9rJQx^m67iq*KoiJC|wh=8mjpt)+V~!dszbq8n+S=c4B^RTmy?y6p zyJ#NRE!kSNV|(WX24)SzG{4@Tw43*6v;3B{p#N!Awxj{Y{E>$qe~HHvn%%*6KYsSB zm2aLmbe#-5K8jNm{Pb&_y8oIcvm@2pzu~oo`gARr(&6{fCcUG`t>>0RLEw_=-s|9J~?mXX)qKPq%uH;umCU{Ea&^5h%`+^x5?A9|V%AVISOqXOq5^lXS=b9B#FU9Wzt(${Z zRF-jvuVRx)BQtbnqT^&BUsZYi&GghYYaWl}9$rIuvgAY@HhM-=!%zWE+3Z9b4aaNJ zF4oG=9qdwqWuhJsSg<`R_lH~J410)Y!O0}HMi6Tt#p6C16;t%lBHV_vHT9q8B&jHA zIu}5W_uu&B+5SR3hpRb55ttoD{q(xnz*DxxP}OM*luv=52K+b{h3pqac3I+T&f zXyoi-yfnJJ^o%@=ZLW^e6uBevB_>FVu~c*uy;KEF*)34zcQLMYc++FYO9Sw$zAalr z(v8{tpx5&oqitz#fl==qU}ASVY=~Wab5Pu>yyrLivGx4ReF%xhQ!0{trN8khU=<@F zqedpF=gage>(9Fp`(vIJS5_H~^$~tJqBqClGmEY9s`Y0v#b+n!VDb#L>!Y;3-`qzE zZtfFanlVYMqy*XykUUG^O7g4!DLp32{F9T%Z&7!$pcI{4P{8)zWOahLIHGxOC&Z(F^8YW=jl4JwB0Aob zkBPr+Yo^m6`ZT#+9=gG1bzS4z*lOT^5aib_3+DE%FrSk?GnO@K)l3bgp>JY{FK(D- zUoUm{e@dG{!kPPc3dKCY-B3aA;~>9O>_(6>+`Q^8JP52+yz21(ZEMz{b^L#!&&31f z^?F*!xY&eOX5YYy>J_dgMuLN&%{T{9iLa!_k3N2GF2Z0%yjeSb4aJ8Y#a~(?D`4(} zH3SBq+PItyxJhpCmxoXV3w^JVpHq)}2w)P-ak;o`Ft1ntzs?xdXJS2NIa0GWMdAq( z33Q_Vn-$+^Bce)(k?@Zm>v-V3eLHRcar6r~f;&VD?+myW1y3OKyC!kt8l&XBT_G>O zgj8oX9#MWtH^3)J$+# z*`o(o5q;a@XXgBCJ!O%+Vlzgaj-nx^~)dn9;t z@u<&U&X(?ZV=u5-$a&{zyunwO+Ap$j1>`RqOS6hS8idptR3uUP%Yn-&%qDv7|X0Q4G9IL4ydvch8S`6~l& z0XUXCv1U@2&-1Q+tFo%UmR5PUaY&UNV2`+5B8)8i)v2NWj^J`6nADIf2kp>gtgZP^ zx|jF$5?atlmDix~&x-#*h}NoO54;(vc8y|{MBF`SGP zyO3Ir&2|}QBH2NkznNHCW*KT_<(3IcWd<#I)*tM_%>k4b`lYar@1pP5{Hw+rt;LuY z$k9Ln%yYKu&Rf6mD`Kq4+Yv)0qX`olk0bx`&Un@eC0dG*3jY)R3#%meQ7UH7>$F{x z^OwzlqS-N%{ZQoIIO@~0cqJW1K#tarQM+|7{lEEowyd6eY7f6%63Aik(bk}rPETLd z3ukD~p6W8+nRadIzPd}^Hsa(=5MO`Bh$*<3pbh(39K!pNVJkcYJB_94dDW<%ol(e6 zjd)QH?ygDeHVn!NBn6mM zM5P8%n&NW_`Y+BPb^Np=yl0{(HW>KMrPDRVlhdY!tpvkzsM}+t^ce43p7!W)6AKW7 zS2(DRAfeNyt|^=uzSvr2Q#uQ$Rd{IOd9q#c4lp7CC5!DA6F2T$r2Gd-Z6;d(k}rC) zy*&PnC;?Dp8nxTL+c}zu1RB)|FV1%}vm!LK)QJN(F!xwckEZNZ*-gqKCMrlIZLM~3PSERc2# z5%R6c1EtrqxyW^{C*!@~uknqhqDx~72t~I|y^-?ZXq)An=*sjH9wv^gEUsc`gV5Z{ zO@JqL)H=9p9~(fYI||x>o)3L0w%2kv%v%-|ksZ zhn&_LIj}&F(qNTfx8MP|{`$@=Mp>X!l)-FNp4*8bdaG9MDkZ;%w0}Cj`ZYuzF+2h7 zS-|JAXH9C}l6qtiIuc=JMb(swnyPa=3!D?7hq7{&v1aJG8G`&n)oPlczr^}89U{fK zmEs;*`<-JFrnMbv3Z*O~UeT&c1%0>XdI3!M6i1Vr{e>Y#cA{c5LC?*#?_h#=rQh0n zJUCXS$9;XaW|sBepbZt?x!5QFX{|Od@-lS@GOvI>D^qq3_D~#(d5uHwsc|IH4}_a> zK$^b2!xa^@x`G_W>3cNsx1-7n$Edc&K=-=<>yqXf^?qvB=kd^WBGM(YRcqXSWa=h; zn#;dcW8N&)^286fyXV*0t7WHhK!mUzsS~*JmY3s!%4O(s|E2yXkrA1LBlq~5wp3V~ zWb{Et+OMr(k1&Ar_xEKIBD(WOYoZLJ`YhY~j`hgML$7ATW0jG9$qttu5r*G5Fr9G( zk<9nUM=RVyxq8_PWoOB(YUzGQH&$g;AA?`OtEqgBY0RH54Jp7*B>&aq2TxyDqY=it z(N3XNK3BL0b0dw}oS;{?_W6r*KD(&Cuy?EQG~WrNq!eN*tCB-pu zW|Jabwah)dQL~FGD_xYlF=UQBoKBm0V)3H|*xPMYESRV#)Cg&!B6*}PGvGwK7=)iy;>2OaFzP@bN1!| zm~CBw3^X~evhH!;P6t@#E>G?IluM(RHHsj0<^EEPOO239J7PlWCrnuqy5iC9?kC%| zbf8wz)%RO{Ir$ic7wUJ8w@`-|05>;Q`Q$e}f7nX$bys3^$TxDS_V$g0&$IJ{g=rV< z$q9kH6O3qu$#E2W;8lz1_)X@IR-X}F!oULe8G~s8z&+ja7tGyy$|aEYkJg?`F^&jC?UWEj!rG1p~h{a_1JU(n^{{Z7bL06;%e09-vg@TLWALcR`eyG z+Sflf_`^i4i~16sJqE4z8=OfdFAm6{YYPC`L+X1^=p8j4^#mkA?^@q(|CQlw3-El9 zjFviC`h(zk{+R_Qp$^Y*t6f_-$J(}(kP}7u)DQE$Q69J$le%OM{jtcL{biqJ$@z~E z1O2WrI)Lh@>z>l6cJ7Z%&=lrCJNsOGmdnka&0gAIlSX!J{Ey2PFqSnszH6zx0=gE8 zWE(G^0$(bj;q&{ACqXm9r)B*P#=F&)dBqQ0%tXIp)Etx8tH$#H2S86DEEB8nD?zZe zE`*NpB5e1`KRyl&qO~YG-l3sD^cr<2PpXUl5<`9Fi0za@z^$mwrM25`V@qt<=jf_`XePnC#%S9Il96?bvgr)6wP~3TNd#p( zls8_78B;1%EaLmp;3xykxs}&VD*zN{-x{FwWia+)>y;3Z(9WeI_S1icMqZ+r;3x=gz?fT?>XL)~+%_)pK~ZD@aTG$m|#=oVhsGxUT@Px=$nj%{(F3RKuoW#y|qPaA(+Wu83=Oc zEpWy_3Hb*HH4yahK?OF$ykN70)5+rL$}#=!=~4N1E5)y9+)Z#+)znz-@$q4qux{*x zS&W&;O!{g&S^v~(lEoMgJxG7uVG{rkl|-?Iv*J<)qmz@%=k4Wk}Ap&2R zd(Ffkw7o^JUj_#x1R#Sw5(4NwSULNRgR%Jut9?9>=+!>0AJIcnx+KvD0g~dT+Hdggh|S>eRcpbrZ8;9ai8}41@Oa z6;==ARe%V0^UC}(?qgHc7!0tjwjVTidyDTD^<{GL^G4NHt8QXxG0S=k;qjW(K1SZ% zr79b>UH#XoS{M2vW!pY6;_v7yU_Mn?SLJtSs@POHlY=n=K#yrfmc`?Fb}wCj220bx z+S=B#GW^C@^DI%(C3|~Zc@>D!yybG1=>;%0xM+`tl6)Ya$h`d-mR3jLiCSwL02;N) znvZ@$_VA6-4Ht8<%^sfA!q%dtfDLQHYB645=Dgjc&~@X}vb6n6kT7e~kv1Hb$mJmk zpkv#chj+14Y-)BJ6GW+D*n}v-2XG~)ls+}B<=viRBJFJ}GZw&azE-aN*-Gk1{T$_n z1c>Lc&*pY0eLd}D0B-k#%|PUlf}Y)EF1V0RQZ6LW1D7b^0`>g>;sEMTi*T``!24Le zPcq%QTtc(}5Vi5Ejq5eXW`22>4}}uDMI&lA9;h*{b{TX1EvGie|LCF8A#t?vrQh+a z;}a_8_G7gdWmYGZ$^I~#gy*zzgfcf(7-`(KIX3-xdi&eZOAIhX`|jt}hpz(@{(CO}4FL$Bh%rNh3Z7D>0zqU=az>Gr3S?u! z*8NYy_%Cy83*dLnB)r(Gr!7qIkghlx*9F#Wk6*{kh7Vn}WwCu?z1D7l=Ib=oLtSkz zid^EdP9_-v)@R_!eip8&t?i#q?^JT^+wwbQ0K}O1C~wgW6i`;3RmL5}-MiuGfOS*9CFh`NZ-L2^lpUIYOy%3T>Rsl-xk_Tl zhkhp@Nry!X%>%qQiAISoxE&hOzhYq2cErkOTKz(N{_TUtiVzJjO4Y88XnH@(5gxLv<ZipY;-c&U3I$@wm2J}3Sk4=^(rjW>Zqp#|9BYjr$Au?4l4E%RfJ=$<#5{@rx zph@1WIQf~v?`(`_H#F1I+VmE6su`yprTRu}TMMd8&0%RLqr+2cS^DJB+RvV?ud#88 z{CcPD#at~{Uz98Qdk1N#n!=AiaL2cb<4Sy)z`~5hMfUXFR2Yg*njp^OQ_r`VzAGsT zUzD;F&==oWzK+cs!9zP^59L^!MPFco2oc#0Jt_O`n%g+dlLwAWqgOoY4y&6c{ZC-wFTl*BGyao{j#W~)Eg52K_!^_L$JzQ zWy<9rEW)?a2(Si{Z}V)Am`la6;^V~j4YjJcu^6Q-jEHpT^rar7J=?C(q-6)yfuGc1p%3-zU|xi<=mj|6qq$?UAVDvSoBqfMf1 z_&^e=YLY$vg61^LfMK4esGm;Fn(Ydd_d$JW{z>cfY>&6}kQ6SbKZWZd+VT`VoHLoAD(5 zW%QemE9iB~r`>SEwP88aRd5l75(oWl-GpJG+s-b>kq6>k@0Dc-=vE5ZjJG(L=+lm< zFS0h1SakG##>QeU;y3=TY(ta?1N4U^**#syXHsrw)Nk{pEaWlK+}NT!3-J$r=i~iR z(_>^MT=MVUPQvW)ANx=0OAZDx@wkpJksk^$4n?3{q!Al?h`u(4d@XgWe2$hJm|3P9 zfo4-&&6nN%Nb4K$PigyInHwg}Ze!WI+<>kjv$Um7%da>6uXS=`);sz4Ow>6tlR_Uc zNl)r^MQ^12tYt-mze97GP^10(%LvZ9xHNbpG!T{;veiWPy7S2iMoaPcg32~O&{s9@duFo*?po*;(R+1?Z|TGu8~;P_RY6rmzfVVej`<|e-DkpZr{m?9f9|HLTK&6E z4UTf>!t1EXE1~nIa5z4|J!lwLJp7$4SrGWi-s$_VAE&_9wa@uD{{oQpjp-M?b<~*Ta7;PWoV9yElK!*V9}#hECm>(rp%mDc#>I7?mU*6&&Cb zM5Qk^h16V^6u-|-P(UY|C(ew}YB;lQKESwZh#TXNKtlMh8S+@$(av9cyZwUPgG?7y z8Ce0q3bK@*ig9LI6qN*{C=lI$}sEX%K>s2`l?g&)XeWiP%YDBzW6&wC@dgk z-Ov~X>`D9nx&-aLo+od#*5AaSIKq)KH$B%ng^CLgOr9L!$D6m0%-VxD zdDr|}@!zsIX8u?ZJ9+A_hgDV@qI(%1(|noA6oLas1kc?@*WLA*f%ZS6f8LGD);iO! z7zaBeGPl_fEl7hzln~yfv_vznf`1^HaC-mlwxkq}mi*EMha!iU2UnJ41)iaNf3|XP zw)z0e??goYyk)BtX^)|x@qBX(?+%Mc8axuN9PzEF=OcG?%Ge;5nZzYrQ(x$paDY%? zVG#i(UwPlB0;uE3UkQ9%2VV}`>d|*y_kN{g7$W{XtotJt-1Q}b zze1AP=y!o-7q{JWRv;HKUIq!S$)x6#=qRg|*uO0i&gFR;D!eAE>~X2;#o`{KeB&Hp6|p78P8HwQco@OlQwg$GJUes)NwB2Uk z7T;-C7=xw2`pWgxo_Wfitl`m$v=bkg)Zs;rss#`Gg-}j0Q~qRiTswY-FxQ8j#)8KM~Y zz`BRGHUgBM>UkgT#LjyfS_-gGAc`je%oIuvCCc$wCgM=SA#_k66>QvcT(q-1!Tj2b-1q(^Sp%83b&w zN;12T)y-*n5oVU5!7>~m{o7?a?~$t`0fI)uo8YOs+Qw`lP&-xh`%lH6IQg|Tx!uzt zhbKgHDcC|8TGj9khbu7>9yMxWEK%Bx^t-olikiO!Ma+SAhiyo znG+@s2n1;oR8Eg13++Fm=dYSe?QAs;_hFiWMMq$~q`FGi6B{+bOg<*~z`IZ?QqVP( zq($lL<7+q)9J@5FzUQ)OWie&|6o`?_#3%jE(rZDAS@uk1l;{Mh`arB zCqS0%baDxXwt--x0{ZdeRn3i**B+WpGl(F|T+yDeYyN6-|7R2K8 zSI@j%aP4wz5k= znw1>`tU|GH$IKLFOFRk!wGD|!spRM+oL-#2y`-Hv@+yH~liC}v2jSj~1lsONsvk3y zS%hOY+YZW#bG9x9BW5n5ZO@Kl@=I1`Dmz|tZAu%3s-tN8ey+bGMA*XwdCZ*UoY(*Y zOny|l^A>$=qW^aAJ^~AIrHo4@vB_|Xb4h=j7l$=Gw9rv#2ct2Zt(NX{(#3|M{eF7_ zPt?`;ynnmxN~jBcv^K)r?>Ws#eg5Z!*@K5OwsNPf7&AY7BMRC8))E@MjotApm`{Y# zRcLW+Ee>GMdLDdcjV2JtBH$*(UelHuCb*gHTCF6_OrxF1-1bU{8UyHj2*p?+4+VAQ z?bw`HPgaKmJyj(18+Qdh5Rb(0P&IEJ89GO5+L$C+cC{-vITwSX{&B`4zcJn%(Ge;6 z54W1nKO%uGKb)d#`Ci%$W1zHY6} zP2bDjk7&{zuT1bvUZeud30YvwB#)Cy+Jd=8H>m5$;`qLo&TgH>l_8*JJ%uVtiF?C> zT7#UX41^jIoNC(PG6lIj$6ZaDfP?c&E1NKgW+Uf)l1#AR;psuPmwI>#PSgK~=jL3)e*gy=+n|K~Kp z-8cP!kCbgVcqC%uS#cqu-i|q6On%rbOC9!ZX5XscB$w4j+D`%PO&A+-(0StLb+Srk zdsBY7QmpmsQL%9EpjTPm&&l}R&FfrdKQl0^^#kVw@iS*pSlpVA5iH`Be0(dbLtXo3 zHI%9bwRjo61Al>U7rSN=3-OteWHV5Ob$s*6%H=LwYcAWkf(jB7bJ|Te-suP8jkRTd zl~BwKD)!P}FaCfoo{HDRWtSU!eCIzN^1{^em5uPj$sQr>Bpv76Z`;S=r1QJOVmuK- zv`Ofu0F$vOk5o0r(t$I2y-p^+#g=Ij|G0(lCi6NB3t12gumJgGSy}0ki#qsJH_d4P zf_WALHwhDK^yWT85m%PqT3x>ZDar;;M(2m`&7nYxJd8jIkb0pHXPj>y@hpx!S_dg^ z@PK{LdCl)CSH)PPd5o8wF_1RE2ln2UZtM@)StO*gh4H%6bsSd5bT6c21+OaLM(eC7mGoCw$cDc z!r-R!OXVI0=<+YtcH`xy@;>!H9VS@_5YQuQe0kWV{;!Il2CdQJ zBqUk9rylSu%Ngt=aHV!NOtsl&nO#(k?t4O*(49gI+IES3T1E?fcRTe3v@e{6v9Q$8 zP)8cF`SCie_ zlSqIdjp(EASv@6urW6yXQHf#O-}fz?0_8}I2RSch;1F1{+VkJdTlwf4h9>l~6ciZz zmDO_$>GB^u@Snd&1qOXtZ6Gv^i@tHfvJhsTy~iufh3pd-*N}x7=5qc!R%Sd4sE-x_ ztVrC;%}K&fqWKgX^Kyc8w!t22W1y#An5Cv@{4zGun6?Y2UqUR&Ueec1H9@5&=W)~t z$6;Qxm-QG$xiV8Km-c^!6Z6DcK0mPJ z75zlEpmeHAkeJUF3&_}g(2^Jj39l~7y9T96jR-=m$_p?~TS19A3FcQUKp%FomX!ha)kpU)n_j0Nc)SP0CcyOImJD9EgjV037wL}T^zD{^ zU2HJHvkrR6V;T2y&<&xfo;W2u)m{?VCcHYiB zwoe$%twhU&;5QbT%M4f-Q8D)i~ox!dy;PFlCSn&d>G zHaypbaDc4`l9dou>r5=i@jgW3;Gr&V-|#&KXeWGJi5G z&;D1wcuStI!1T!sIrLo@uBeb8Gw8AYaI4>h|MG7vQHSAqCwK_31qX=qV(u_X)xH|n zYAlZy2@c=bPu3l{EHzv)0d?T&kL7<|oxrBkHP+ABM+ZT#J3GXP7_w>}tmi2PERd3u zsR}WW&j0|TdF{O-Lr?CP0v4-qMk6;u@ive=j8UPl?|noNyM6SPBBwZ%i{qMJfh_uW zHlJ^_Rk^r5i!qzu6Ls0b`-$w$qaK)CsP5ewbA1I~k!XI|X_&Aes2N9qISq)?joEHh z^>n^UEF63*>jdI5i<&kLT*RakulusjGsM7eCZms_l@E6VXgDQGe{SA_v5a89y&In$XI;j*&)S;nV##;Y^f0k3e>)z#X0 zw$i+rKSJxW*Ev4hyEJz=R>SXkt36ez2IAIv*xLop4^sKf$T9D>xi4c)pi;q3iUaPV zmWFp18}(c1DytUDh9StvJSqOqz}mJ{B)Bl)JBdlq;5Gs>nr1>B{`>7>LzVP5{khHD zvQL0W^GCquav$*j>1B!qPBte7C&Z^|~@>vVj%wN`|#8#>&Y6 zU-%Ay>A?`m{m>U_s$Je;p^MQl>83PXMjebwtkuh;LM?Dm^EU-_SmjtE((+nOXGdY6 z$H~ftV_Xx!u3Fb~nwE5ZVNedGeqM)40vkWKx_cqyFd(}u+L;6VwQEDem2Znfnk<`M zj}!qLH7Fh~Bzh2;ojt0Sv8pP&Khtx#CCz|86JAV!g9Ht2;9H6e6S(3fWiGq7j)I>Q zG(ozYpsD@0(;7L{1lSIVG_a{i14=En!&&WBM*xmDDI2X!q@qJZcDM}s78J) zmh`9#u+QyfD#tjS;%p+jMs2dUUP&qN)v6=`xeL@Vy?8i|{Ce+J5m$k6`0H%k|89~V zNPtB$>#Eu;>{RMVKgx#_zdXJHdRAlsp4eiut)X8oj+7~>Q4prn36GdnsRfO^rPBE~ z)D~z;nf|u53coc`*4z#ApZX^;0xEiQn1Eu-jKzC|mLe_WJQOrUIzTrAgrsYX;NY*( z8XV+?4cVIha#aM_!Dx#W4UL_&L$Eh<0#OUxIO<$A%%*%4a*yTGFSIb{%z#z1-Fd(K zjchV~@kQ`#TpcR7%4(4o3EVdzz@Cch-w}I4pEW`LHbpG(D;YR}B9fQq(Y8#K>C8OR zskqemM}A@Y5`%@oUp^7UPcGs7s9z+-AO3=*#4}vRvYN(Cq0VZG6;_9ktI6LkSe2Kk z2bD~as6IZmf+&#$tVw{W1|8-%*sAaCi%5UJ0*fq309lu6%JGud&$3p+I^=LKye3>P zOdVlVkNf`yP#10U;s^oR;Tn$cmMG-LoRLwtrV+cP~+i^Y-T`h1lV zjvc6(kX_PbzMvD2={BNI5Jlg^Ru<17;-dx!v5TEG>(>|yNx2C8h_7g0!xa1q+D`(O zPh2t1%i=jh3Fy#55$ZtI0SReHXo4=q*X0_?6u}s&wOYFjTShgMYojs9;c0z!!n@v+ zBAe-DY;49m{{=IDCbAElP@bdch-UL6eqwpx4ey`HwZBPv?~(wQM-QUeN`2%}Q_z|? z1x;<(Wu=xBI%!12N65r3lz^6ulg4yTCHT!*C0PDrq9j^VHX*2O?t;|w2KP9T3DEka zYxY=cgS3*&hT%WCal=^>+wqt<_hFM6#LMD5B~OahCMkFFCg$a3R0a+ zAV`)3DEJhJMa@fHriB;e6L;ht2+KWSJ3pY0*}gskH(`{0CJmM+dpb$LEA0PRy2_}i zy0(4jQECVQ>COQJ=?>|V5C)_}N?N+RQ%X{#M3C<8kW{)mq(i#C!}ES?@dwthXUBDS zN&aFgc^|c>TrOL~r}n`-$si8r?=WIU*jZ|Dq_MufR=*2=b~i}VP+T5y;CcLsK@&&% z0b^mL&-FAqcy0d-zX>sF+`xFsw6BpC?^qg2Fu89$9Sk3v+Q~ z>;0F5@5jHp&=%?wzz1su&?6~(mDltdGC*_vvqQrM9*CnaMG&ghs|qkmQo2k8cohE> zB{dT?{ky2{&_9Rt6-|Pk1!qw3rg2@M*ZKPH8Cnl1ySb{Lx|lHy?#{BoqqBz}V)hjT-qj&;{-?$e(uymN^zHoDP#!**ageT2GwBHeO|5z=){cTDaN zRiTg_F_2Ks4DO@1PjcyqjD7=QB5#e|t3&cz`T@?{b=M~=rgE_Sk@FdhPx1jRBnAVY z3o4q5RK75EH4GN&kTO=jw1GW$!HOk>$g;DCt(-=kWk%Dn(xWx~HJ-69MomD5p(7x$ zDzdp5Gws!(M!hrOP$gM!wWGojJj84m9;Sf#ErTX z)leA&b3Obc7A9U4>B#v!*lcW5JfSD4PBGxcGVo9Lm=9%M1N~l3xN!k!8{joHic)R0hmxP3kPp>3h32~(Y3_gd$KU~V1 zjllH;`TVqE1@YDyGs@x(qp)|dQtIUtpWcdddYab-n{WqStTyFboD{6?MyxycLg z+hDo(I2z9|!l-et9aYtwY(BqSZLsrcKQ}@eW3jniC6cU)3^Z_Hq16UfxI5PIKZfo! z*?8Q{zl@9=rf_afhIt$6ZI_94M-CS16+@9O*i3lWP z+5B>znR)jq%Xi&Pwi0M>3h1lQ&#>j2cmL5-d!|_vZ(LLCKD$p%Z@WhMLAAK%!6y-H zqxz_qi9liXb^2Mt@u5L#r^|$(FTp(Nz6d2?xOaD(T|vgV(wkHn#0O?zw~LljqGXAb zvtAo-c^U7>3ejUd6K)tVDgGhYdX_!sKlgfj_kwie<_hefe(~60Tam-bX>m32sni@( z%z!2eKWt5^B!tK@y1L3l1R-$DRYs}g;xm4)!ZY(l{@-Mz;0-Nn{Hp!6au**DrWjxO z{s8_hfaAdAm|ZsQ(>`5*6TCp8+!@@xq|nSZbafm^8LT{P2;K`v8bcxNOEhxRA>c3~ zw);J~aYxs^bK<7POi=Qv@FBRU-fKFgjX(48IRT;94fnJRiddPd+mCOhx?NWNCB-u9 z$p3~=+iyWD6mFw3Z>cWccier5X#FKkZ)>8nLqx7p4C3A{R7mVJ{9JEg?cS-{VN($57Kn*Yqc~wHv`6o&K{4C%?R+ z6oe}61BQo|iR7%gbwD8P>GxL>H~e_!jK^K^2G&jkR_fVF|_9xUm3s zP%dowhOIo~-@GzfHGI0tW!eZruw)o3y6c|m=C#K)r8bxbi*fU*t>b5^gxG);5H|=X&w#(qJ9z zF$ek7OE3sF;Dlp-X*NxjWnKYOw*P6Lb~p~R5frJ!(1-qzY;nDL& z`U zxF6Hw%hu!81`w4_f6;*>6N2IfEQ*S3ONwlFMZgXQKjxYjbq5;}34!}zaJrn{$6fnX zWD)Oc$baCYCt>Wl-Y@+ZeQh%+3B%etLz1p7tN&P!{Tbtqxa!JLrXk79tYh*uE-QC}~v^Cm7i9rHkORtZeN#KMEK0rA5v*}bzT_tg6 z{daJzOF#J3kWU}qXJBQ;ebe|~Y?3o0cCd2B8JYJ*8c~Ee#JhI6y;iUPE0K?8BDS6F zHT#FvPrpA3@kbS{2HapY%A_Ot5 z40&2WlHSV z8e6teVdG&n^tz>9d;8n$Pi2+pm1u>Nyute0JHo0f7gxICgKJu!`Hvuz^8VF=KI1L- z8so+Vy9xId@G1Ax0cv5qUh#qd#v0C*O+T1|tG6~pDUh;Kc z-F5{rCzvrqI`<7>g-~2_v!s?c8P2??{fWfVdED(w*<;>4eih@7*O`pAAGkNt1}bo# zFQa}0j;p|fN`8I;wmn@m7_NPe0c4t_t{qKR5L3md(Xq7!?71VYD1kvvKQla11xfYE zEwU%ATP)G{TeH-#@d>{88XkdIq&jf-CId!K3roIS<9hwv#C0(qXCOPdgDt%a$tOa) zWLCT52NI&3q3cj^HOCm*qKIJl3o9E(a+ka?@DS|yoBuWvj`>oMHyzA28*A(5ob7>e+6y2%@lG3Rd;@>l zv1L|cQ+qa*b9T z1_&)XV#|p>gXjB=tFILQ&S5^Gq&>m++rr1{hPD=2=6>?*&L_+oATuMd2T2)mlIB#T zzAqv`jv-H7EyR=Z?S{3QXuq*{lec#f%3ur>v7WuM_!6eqdS4bgYsQQ)Cf;Gb{1%Hq zD(Mxe%6o52FfX)SE3e{rMO6Wanu|e->c~Y*BG8Wl&WI|X^O=ZJYz3xB5LfIzs2zWq12~>?#;rMP@}fiS0c=;|Bj^M0Raof=^%m$y_pZ#i>JNjoZ}+arq-?yq zw8wJyTFZ z!fCjO+7yS@e$R^I`ah--py>{+?S^w8W@R^i0os$m>AR_1bj4KVLdb38QlPyj@8y1SD8@mpXMcwO z8k41uP0H&0j+pY7(r>@_A1;VON;45>`@b3(HlbS0|3wAoNrUGp%NyqJv%2lx)G|iy zjuBrb^zG5jO6;*DeVVR451tqS0%<^t)V2}*Bu@;UXI-B6+U;z{M z=&9>px#g^+hYrpG<6rINZ<1{&5q9#oFd`wzr;E!=hk13|T;NrW-R?Y(Boh!Wrjk=2 ze9FbU^T&@|D@*ONlHELd|5P}!Fg~$I{3C5~S)ZI*?jbBl`!dAl!ysQ%ZJy zHJ9ig;eZzFhZ^to)paaku58r%MX)b;NiEp+0W&n13b20W9Wy$1bE3ru3qU}=IzswNCkcKq9!l8o`9!|R~4nLtMi9iFwM6Q--T7SxhIq( zGP-XMZX;@*Di5RNb0-8=x{niAOkYIyyyZcwVIWKn1qeQ&NQL}yysYbc1-oq(d|K5f z$$nEvb6(4CvFAJyVFg#|H%cM%!m>W&Rw1^^Q=aiZ*SohO>|vjLZ!Mypbv-ay;xNk&&hCcfG%PDSbZYEwv}joDz#Rn3QTKk^5xLacqG-C0lhcl!N8H zK?8iZk{9oW*1&1=IXzns7hv7thQE<+3p!wLw#w*|)0kQON3afY<# zD-pp@q~;MVj>wkniIUoEQHdUgoEq{S*947(Y6}ANt|-mvfdqmlOrWLfx!a*5zQ1nCC?VN{2{tjBD;&&xpZ)@(6A84#nwOD`v8qqg8{gJ!O1T z+AhH~=cf;rMxUMv6>M603fCw>k5&unLv{kG004)NA#2JvEiAVtbTBX^xGKhjrvLJP z9yvFh$b#V<$A@|hx@kKUU?Ej7LvJX+1{ZIqZl<8DkJZq17F8^pN+AlJxP48tJik@d zpu2P$46sQ`8MSn6fzL0#uJ)AFyF!sn>G~e}k1+b;)XG9zw-Sy9Zy1hVj^5ug%}v=9x-YVGQC82 zx+Js+rytrj*&LkyVEQ8P!12LD_UGVF+J&}%y{RTmzZ_?NqMYGmcf&3!bcs0mNq*P` z_}KKW*kAH~AgUI{R%2~9tKKjux2y)sCq+E?U>FGZFXvPBx>ilVL_q)xKikX!A<&4? zSr!Je9^j5fgxlEq;qM^N1HZMC#?frRZw;`gQ)yvTT&yuciQ_Z}1~@Tp^Trz-nyd~F zIry4~QeSd&zfTbIogU=FzW{qVuudOZ0{WTG!(UfN_WUpeZh#Q+9`mP)ZE9sE><`n0 zV4GicqL6R7<#sCBJT93Z9eRy#t>@`C9I1kci_?`n4(B~RML(!ce`GD%n%Zi zZ9l&SfRu;qmZlh|-CyK@Pjmq|-JI1CLt zSwG;2x;M{IAyLu?P_NUQm>$B@Tk7&O_P--QzK;};7ymxIzRJj9G7iN6__Pj>Ct`Lh zU43<$#w{ElF3O1c9tIDTwKISzH=K8fAy7UkXbu){Osi9a~~DrbE-gpHrq~k z$q89cXb;v)KflfqZ4Cbm6hIuzfz+hFcV9&utrT{Chgld?o0y!zYXq7f{B%BY=*!!% z7!8vJc_y=oFqp+029Ty3mQuwBe>N}Kj^a$g+Chxn0|Tj@$vBt$FOkA*Q%r_ffXw7H z0tEXVUk{AS!%PuZcLNFf$>Ld38sYZ!RY-(Oh2(|vNPy9`I_8JbzIEGmG)mn~25YY8 zQ7t-XEJm3q^~} z2UhMlBXET&KJHegdD^U@?yq}w6iHV_t_o9#;u2^Beqgo%b@=KRBD>J>=nqYic!E~Y zyv7*PORsWzR&8=KsGunjATg904_|MJ$$a_^F|VRA(Ah-S%}}@>1wdAlrji~(q?*u} zrLt`o8PO*F$zvQ%kz9m#d4&KW&#mNeB!Dp_@)(y`K7XxTIBx6u??g0I@~&3V*dd)2 zKI((ng&Il533O0<{Cib$Qh!?xK=fqll1-vjM*h2hW;zA}YcG?eetvX{OUw@Db~&hp ztWHpUAGX*rQ6^qE_k%{DdwX)A``+-SfS}+lk=Xb@(QJ45{Z~D|y-VJ#a7;BxVFvf2 z0e6Z?sg$uXjxH|le66?LOcj!-lwxVt0u{Z5TM{FPev~iLNtx7Tb+LN8PLz_UQ>u?o z1R`np_F$ar6FAJ_|NFw^oLr`;B}|4rv_k>(F|-A=wy_c%zbdKKW*Ws8lW#-tFraXf zxn~Js%A{Xo-vJOGWI0+>!^B6s(q0Aan+$d*0s^%6)BYz8JY13thaE^zjYLjn7X5?k ztIwG9;Ep=J*WsI7voR&@o;h`cIHU|n>M^l#@hrYVnd~@v7N>B^osN1eNnI%4+Y`-zvmsVX^B zQUrM0;@z+6q1Mes-*oolI@~ja+&wmbO|qpZ?2C2E*s=6}#x0Sl2mIO{yj?JTNZSM;Vc zQ<>vLA^VHI^*YWcG!SvXsxmIAA`%jLzaNJHc!#U>?7f_x8e=uCZXMP( z{D_gmM9r>Bvlfp84SKGA^5c(j%ine}!y>zlmntmGR6(Cm$~lnzmf!aF$kM#}(^nuL zm83-3KJ;PGe6a0mE%F5qiI?7mf;HLFfjs93jV>Uz(v!ck1@m|lHC4!uo*~o9^jrSf z!TR&BKv3g1eAcOl(K(lT2vVC-X=jsnM87=Y6W(KZSh~(S%15ZLk}~+N*U-zz8ATXf zD5*-S8VIeuyoi@Tn}8Cmcd1c(DUJU7!oOjJ`ZZSW6eWQ4CP*F^kLuFP3$vsOXfrD< z?IA83ki~miRQU~-^2j$*ozNg4w5!5#b2nDxrn+vL(FI0TR7jTl?#ml_k~QZi=DLF} zj>|C+pA~!bkt2fW=cpq=Go_*M>2Q=*IGBi!{4DatS%=se=*g_*at`OTBIgkuj`onq#wA%&H!7+2-f^#3ENL~Y|F<}|7l9O^_+Mrh$ zRX+E+~3YJy>D z?#GEkJFJMm4F?$$02wmK9|T5qmTAF#t4Mlx?Ul9E_}oh;2OqtSHT+o>76h{197;}t z^`BOU6%`l^n%_kRwI^I>DtfyF(18&aVa;&v*8DyymwrmPJzGjK;d-SSItm{GABCA$ z$(!gC)1IY2^x)bgeGY=PM_%jabtJ^E)LM`}Q+{2-ijpabuOP~F1pwm?kK*UehWpiF zFMfd6d>|cFFiea>hDU|Bmx=)S@R!e15-MTSLS!clT_PpZgVzAuMFZdyiM^$UI9nxb zciFK1G?u|=o>*Es8JRyS(`}gBbPOj`IIbkJu_f#cz?UT5fB_JM`oBHo84LM#*D&+n zAT^8daibo|aRNv(-^Bg7$sGw+2(gAF{C-`*pO}(t5$>801;!psB8?J)7J=RRZ>_jc zB_)DPA&%2^-3I8>vBv_S=(Z!Be*){uIzzqcMmmEW)`KR8ar_XZSPLf|qihFkT8=Fs zcSB24G2Hq%B&<*HV4_nV>7066H*5Bz_QPL71FwqC4}oh_+m&g@E;+m3xSzmR&ljyP zrA9OhM5+*|xPi}~vQ zK7pYA3JD#9a_d!RF}U#vGFkfyq7(8`tAFKL#C}jQs4SU(1vm|z9S4H&n1x_v;IHDi0asK5%$ z1hi7jlTYi8`6U@)gl0$)Dvwahv%J{zLxA|e`6J0$dT|-RXFDJ|hZ%yq_l|pctVzZ1 z=>f|3D>Zlb9tHN+T9aGltnHo{DK9ubx^qwfWp@k_u<|k*EBB(Ufw%iIB=sN)qzlE+ z7O_`|HJA`1wHmZqU>SNT*2_m!^B*2mrwhUJa$TN(t5`(sr7FW@ob~&_A zx{?+uVzQ4I2$ZHErpv!byEyyJ{%{H}k9o!w|6J})S5$nL(;V?BMP#+ITi9dCPuR}_Z7evmMAY*l<4Z|;oTeLh^d_W zyV`Nnsv;NSVq{h;%todJDgd)9Z|M@?P>Q z^XmIkH{c(Z0riw687Kh3Wz(1*gGxCnPFDMd;|09vpQ<4`il8pOs z-`Xr{ij-^M>IKBh<+iGjGf() z426$?h5heuQs^2;G}n7B#zMsLOqAvbV*PO}4pZ6~-&?3?#bv=ivQ`5x^N=ntVm$&q;ev>;I%>F?jHRUihVCdLd_8gRPaL*s8Xwo zsp!poykxC%6!jo^arT2-VR*LH7n+Xw#hIYp7tXLmFDZ|}&4;iP*>2)h?U-sO2IcuR z$gJ=4_nd6r^|{2158)YKthB{Y%{KtRiBIJv%aFyE_pTX1G_ix=SAhej>(dJ}Y%N@$ zL;V{xv0a4#r099m3$G{Ba^KD0V!`(E=A2~Nds~!@EYrAejSxOdbmfRZnP}K*#6QHG z+`5+AN(G)CSf=7=JG7qrsxK@CIh&@BqIib5f|o&->t0^mBm=J0zq?1G?|D9azQ^lG z+WT1u93&fGG#J30h4WXu$nFmsO!=7@vD35X;PKxn9I4rC7m4T&Z>wAhdVqJ4P}z9q?dAqW*s=zn!c zXkXWd1ia~P1Wka%AY2xXz4Q!}DONm96Ab{B618dz-PTS6nrwd=QjQe`&hEhw%OJ@u zp{D~iRCWkgv~-pIZ!;+=6l1hCSGA0c;Q0ZN^#bkcu)uiJZdRXFN$Pz}9OUkDp#?Jp zd#ZYFgYt-#2o?rjnVe(sldNcxIylUm+x#1&)$ebEH;;kF#wLg^jyH1O)Nt{c+%k@h zVVY0E_WKEFN#QyYAUzX@c7u1uxMs`#ipPmkV)K0OhB0J`zd`x&E@M_F-*%Vw6wj z$SZeIJW-*Z5UNbI4BS@A8CfnOesnm*sw!!OZ6G?%5Rt}P(IM)S;C z1PlSk_a#DsC#taVJ&;FvdbB*uS*L`P1;su^e^$PzJxvpN()@Th9XfoVO-I7S(BA z+g}4zUB=ZQ_H%G;hW+gv2~<-f=>-(y0#H^06#33!)S!63qA1c_zc&01-VX&RJ>lz{ zA+q56`ow2tx%4DRsj8|Pqfyg4BTE;lroVIqX&A#PcKEC#O*RSca6@mNqdNASY4f=L z(1H+zb$z)$i?lF0>jtd-@#(DRE5g#KhGGsbtD7(hk4zLk_FxI00Xiq?XmU9y)p*9B z$^?%5DB6>1DiPK|$Y7q+b;Wqhnmq}P_#;o!By=CQ%Owe(9o_1{!9xU&<|*0!2tGg- zc5Pktla1an7+%F2L^%*wYL4K;E-HF?I!I{cAfnNg`rJA*c!25{Q5@^v8*G58<4eN2R}*u#rNql|0+94ppQIwZo!Wu|^FQP~TLV~!on0{<{UrA4DK zxK3?m%whtaP+ybl``dl70Lau5G>edwCq)g3kO2q+N<0XNy~B<$#UVX!3=^xLxYNI! zpM)lvQbJFcz=+53eC`akx3jO-+Y?5{4RrMXp544Q)TYI+f}kObW03HeKA4>)%16Eb z=MS)$`$IV|XP#u?vB^ToU`z#-FzH2z;W=BdGXbD0TQ+&%Zb=mMR)SWSz%RQ9_c-l% z_Wj(Gq;%KbBN3*6@rrRo#+ko*qu=Ugq^VgjHxj^!bwGL-P}@xnbTn`gDLi^a$*WZ(3h~w zDJ1b=+5w04-k~po3d|t0QHc8^W5Sg$0XeTu2X#Zh-eHD6@)g`1r?|^aRg2# zM1i`Zi<#+0LGgAJ@lFB4^YU7Fp}?jh zFOM%6_qAZ6e^b6_e4Trvh3tvgkw4XaDetwGY?)?~VB|(+KT;e25C{sMHm${o-%b!J zeC#KHK=F&BqAR#PJ<7~%tZ<$z$oQZ6N#;bOeZ}gZLEnJkXIWbH0>v~Q@8?pgpyq@4 zk7I61KqaynwA=s*2^l=)P#B_TdMfsH$&mAhW@2lv7hBOUK@-)Pr1w3I4vV_%sxkVV z{L5I?Kb^X&pilvjX{_~-rreaQB0wjg(BHq7*g|ecL)zn1Myomf@Tl6dr_IxT;yQN| z;iHzPNxJ^nfW(%4zY-+dAR+IrFIe6<%sj<`1ReUJ!?8rH{miXF0(zk#caj+zgKDAI)KRl|ukQS;ONw$>HG;Wi-pbF%~cU}HTRXbMPAnONK*2<@}yJ;h|ZRX7R z(Gw{D7{AMp&Z5_JAZ$l^B{s2AQq>=YXJhaZqC7YBX%tORut`@~T02kq$xq2f09vfL zHS**rEy|LTK0b%v1_`C#7%un(z6KR1GNl&MLG|iUO(!JaE+-dJz_si}t_mjzdwy!is°@#D!8^L5ayBQxlx!$=9IBp@D&RCl zQAA2N=bAJ({bMh$>hGSCUSJVGyZgrANvGw|{A-|dDO{jAOY(~Yc#gIlWu<}*qRhhR zC!V`p$x~AwO+?XCADJ#jDJ3nv|}+OIL~RgrEy|O{U|j8K3B!b_2)4L&rk)fNW7)hltWk7&IgeK)%8$GHjbVq z+zKb-RLd*p!C_bSRa~w_tzrzJ^a?j01s9E%t z*6@u)bZqcX`Vk-!@IBabwQBrlW}_Vq)Q+#%#+p_?rzi)C)sXYnI_E|-qf{H=V zdEH@&1DaI>fS8-0zB#Ua>oQtC=t0Z~6%j?V_L`1gfASDO z{|X0J@FEjv`}xy|VA7bcRyNJ2znWu21Q5d=s(w1_WyPxeiooGDG`Ik{F6QWjcT3Yi zS4V(f%w$>tb_~HKJ^-b~(pWiQAxRUlcOt+R_R1bJ4=|?`HF}Lf76n0c&j9uWyo4^d ztb1aJ{k|F&S=(2elvu`+YEV8|`?8KMu&Jxw2%ZI78qk$z@ne4lzI1lN#P4S3n|*dI zfZj~cC}T|t!gBN?#pFq8!{t*^kmW12CW&1gIg@a=vY{QGlAjPvE6q#s8LvZ z@Fwv7{e#XgXl1Fj_gkny@A4Qv2c(eA(7`>H#9Z$=u(~T-^Sj5enNa$}>Q75VBzb)0 zCttMlRq7Ki@dYkSN~+@iI`2-*0xAb4)Gulq$#I8sqwK-n|KLYR(j-Cup5*fG@HMre&-gR6q*L1yhhOgJ9h#{E+V_9lZ zYc0tBy&Ga9h`MoX4C=%==U!fW7s6T7`^c2KtF{>7WP;UjgYd;`E550#WSgd^4RM^n<{M4Qwx9GYJ>`NKRV3(J zkP~`y^-B3{Iy|YVV;#S=p~x=TCcejmB11)i-$20*v0#AJ07&yw>qP0lpb{odm{0Vm zx3J*=Uw>Z*sz?|XU;t*3zG@rXXtV@;P#yt-{w0uOcchm0)B9*JG6KfwB22F3GnC#} z48$}4l-U0sNC~PNb3VB8I)45N(6K4ps0^M6S8mv87Vh5FzumbmSmy(l; zb?GNhJ9R&KLKOGICx>Vdk>Gc@)MHtD9Uf~`NJ>Mf<|x>iKgXFk(bTywK)WqWT(WjE zMEIols}TGkQ)HbxaZ%7Tgd_ZwFIPo)R$8SeUS3^|N&_iF_)iBJ1Le`w#M-snJGB5F z7D-V0LB-{5Gc?#71YctJWn;u-pe&CMIy*bqkm;H;OwZEs=`}CLAnV&VJZm7OdNaml z7DkI1pV%QLMItLnn;-jztCt22NFdrGj4T*|LB6QCY)4RpF@X;^3TjoFgmoZ~!%(!w zsx^+z=0ILzYNr8XXL$Hb_F2iUHcR!0PmaO0uNIuSuR6}= zvko=Y-%G0K%CA_3qAM3Rm2`TaaygwQHkAZeTW&TsqVV<_Um0LeLFE`2qXUBc@tg}9 zUDCJLbF2S|Tk3%E!W?meqDWHR8Sm=uu9K3k*-B^gUw8b*XkdlS>J|*wTP1ovS<@nea-&KT=OlO7y4Pu?t}FKR;<|v#T8e0nu}PjJ>VQmi zVk`wKEAt!tg$BR3YVOUyH;+HX<4Z8Qx`EJn95UdK+(&j!5dS@KaBSZ^Ibh5BxA{}h z)uI6cB#$D3Lz9Lk=mSPLU5=XYF}!mkJZl@Ai*;`86Oe&IWTBWe*;4pRh=P4msym9D z>OVwTDE9tvt@9Ll*my|$tKenuC{-?)(R3+LYwP-Pet>6PPsu&n`pEv{!rDwd3e_7c zuG13}{MUsnACz;yYhJ&;O8(hIRQ&FnCK#Uu{MBy{u;J@?g(~!hDS|}pYG>>3bb^QX z{{{|>_i(eJ(WuLMosKt*u-&FPUnsrT?Fnt3KUrmI?lFF}thz584x=@bL}OuZ(x@>c z|45F|u|HD^T1xVILp(oa-ggGH=G<#L^GN$BGSvRFtc3A_@{3FwWUd`*c!Ox9yA@H5K4>k z0f@Od-&%@$cCN3ksu^wa-?Q=0k zRKnH?PzI@STzX9W>}~QzqRgu~n6tC%slTAi|5HD(-W@{>#Df!3`-pu%F)T&=tEyoA z*}H9EOD^27w-ieDzK&*!w}$yHU>$r@7vZbEf_Oa7$7`YYq0?UxfVR#>A~nBjo=XI;SPO^5E^-S4=@NsjFG7L6&mP+O-hd@cjG z2C4+@+NPcV)z7Sg5n`Va5jgC{m+pwZ06fwZ31ZB?yB z*5~%>uYgLzr(-z4&- zvtDyafv>W;oFQct!NY)1mOP&#Xjp_o1bt|#La@HeSaskG_gyZh)XmrB0RwiEzFMa+ z4MN1#vWL)Y_*2jyH<~yC^yT!HGx5XMvF*;j3u3FQPP$Jc*`%;LeVq44-p_y8G0^ZR zcAGR^W8j^8dlmlBMhxilHRdAj-E)~5^GKon_6}W^{1V$Bys@{aC>xwBvJilKkatU$m6)cIB?K!P;)`j!y4VWT>V#oD7}FUPzJn!Fb2CIWMp6TG+h8 z;v@BxTUDmE9>)73@zh%jC$g!lRLJaqi4M(q%1SY$8h+gQyH#c2BLI(9%bx9U9e-{* z?*YW8R0??BCc}s3;SZNqhRI*5TLLW#+5`sLjum- zjX5e6I(uAXkk>KS``~ly-Ho}z=@MXgZ!ER9e_cw`*Nue|^z+wxw9Pru&wyu$m|0Mf%a^uY;jdgNI|~jB22LoKqKP$w|qg|N9)M zUtIg#KDU8yEH|BPCz12>5?7=ZUW;ekj?yAiiNd1fG69A@kGZwM3SZK6R6D^;B?10; zWrZXxHi0uyIc0)Os7il#QTZSbR}HKF2ao#$gv?e1pDI^Y{kpW0j)5DC=o!`QNUWjQ z221H(v~(-sqw>y^PQg8^2k$e=iV<|+ffBOQa?TA`r`AYNkm(KwbAixqI-C1Df`u8B z-9_^6*4fXF>f<$;dSb``gC9B-e{57@@_&oT-u_np%?11Yf$nHLRBdO~RYP@@0sBquv z9k?0w=+o0+?mIDGrWVpKmO=XJ)hpYJm1MONruA)10>Rxg$lDQWR!wdyQLD?pY@>x< zHBBe}DB5zU8JJ*DzHYd0USU@Wu3nm+MxCMt>KL7GE^4$0);C}a((04GZG%k*9u-Jv z?dE(b#s3y#D+(M zW;!yBoi=r3)I^&3!2^QA?R7de5B#mQqIO94Qrv6X)EKSs&znJ)yeq=-sFeLJH%R(P zOlj1%-Cpa!Sm=aQiagCLopEqh>(XPQInjUnW){7j`@Iwn&POjOe#U7(tpLVB0>pC7 z-6s?Tr)TZ0ryFHQu%Dp^#9=WREj5 zFB|*FlcPxwV}&~TY5!Wz3ED;9Z=K~yHc>x5XwAN*V9z7W7%9D_;AlCCX~RYiYDu0e z9Y;IX1dz9Tq4@n7C`vffN=hARYsOcPAMp!?umO?F8{&OU{kmagLNK|?_SCj5EAPYw zCClRs32ueZDY2VXs@ovzxVm@*_-&M=ZtTyX-?RD$X>Un7DWO3&9JHGvsZ2S~ z|FEo0|BP6!miC;s(a`<)XfTeh@u_+%nPko9j?HJCc=%lFv(<8q8y@od8M~|KgI2f1 zJLQ#@VFR*%&1tj}vLr>!q-1$@h6yUlCeI)Z`SMf}#WqtYTf$E&X#T+LJB`o#lnScmnX^MzG}Q z?RH)}`~KqV7+aUv%>JQNx+qd6xrh|n9(7bMREH8G#l6PUHCpr${ zM~8ceGF^j_$L{>z58-TM=Sudn(yQt7j?*G!KSDQgx2-+S1#Q=%0jQuoG%Mn;>F2|( zGvcVH$mYi0k(q7-G>a!NZmWknGRX3C^%u)0YoMIO_wS0`(!#FG!`5LMe$bu_?(Di< zL_$4VeZK_MEmXl?fInTYd~COE z#ffXKXggQEc*aypZt~YN8^hxp!Vl@aFVOU);&BQs7^s=hwGEV)0= zVWXp>Xl0^#zy9X?Xch{xIP<0~Qz5l8B**D+5G7{4F1ixdn|FBal0_HNl74pVl)LF8Oa10R)z&X~!@3zu zX2@Ieq4hdou|7nwKBep@KFm!e5Hb8MGxcE;i_tdyc~T_K@8v=M?GfQj*CqDyl9^Bx ziQbCL7kaGHB~0pU*t5hy|)eNvCjKKlP|8`BJ8wX`u?yybTL$ti&kYL z{Y>eyLYeV0grFyiESx*osZqK0B5TGUX$8~C+2)0hQZn`31UZ1Dc@ekK9hI%XKa`!G zkY9bf*>{{VHa@JSW#8jDQ^WG*NXx#d?sT&drAl=l&z@XEv3fOWd%g8mEn#-KF(ig{ zzKP<7rgL`U8ET~x3+N8hvfMD)Mm_3<(#nrei>Ucb2GK5PKf+ainYg!AkNrsmXEeYs!faZ&#>HH6Z`uFJ* z7Mp7h;ZLH_;4bmxH_eId0CKY6RiQ7nEJa}4pW4EDvpu#2?ecy^uy}!;gjx%n^tQPN zT{>0EhVJvUHU`Xt+U@y^Ca2d1@52|^DpUqWAzu_93;T@Qbd(voFSngx*f5?;#9sir`1MAzyU(rwH_hUT$sh=j4a8 zEOnLKL@c((mlaEQs6LYrarp$^RfP8V?HP5;qt{Y}W${cXVt|&Ow)Maga|sho!4~lv zbvM=>^&9Js8(*yr<~%oryt(KR^R{5)3`(x@jN|Q5-0e!_2;Iq1ciS8L^@|w_YeT=QhZeGWSBlpuF`!BliN|V= zQ<$-Y8KPv1!C;reJB zo68EIn*r+28%eJA=5z)EudTYFAp4>G*W5U3>WnU(OduGyZ1l3VHesBKe~T@%h>FR(DC7@`F`oaRTX8mFH(wtslj1Z^2Ea-hb<9DOX9TZZ^HR0%xn14b40*zd z!W|FyXUar#G2)k$%zs!nQFu#qUy0kF8prqS7fTw(qu}zI=Om2#!V{wfiyMb8wse}2 zaRG#j83qP>>oO?v&3KYKh;-C%;@HcQEE{ljwFu#$C~ZMrg=;Wnar}{uRuE@*Cr_<7Or}`-!4r30rYx*8cs}Xro}ug;q@tDv#Sjw z;RgsSjkVUN@G|SG2x8=jvWoBoU0y-KOmru^7A#lIqoPW6J6}eZeIO&ds%o!a)Sxqy zj`t10l55jU1rpSQC#y20_F>NGUwZ8roa+}aHz+1e>+JB0Cr~nziEj^5KI!~$9rw(> z=2d@3Bfmij@qWxEt5;`h@F9)XHc*36Z^IZ9uFl@gj3&QK_T9+EBgMYI5F4{Ra1gm) zR$I`1&UEgp0NyNAsbjIOKR4IjJk#rZF)Lx*xDTt{oz&x$zYfQ9^>RgYiQ)HmQ2Y0T zcG-8%&-CCY1&Dw~DS8fF;D~8=3DhLPoY)%Y@Mv~PRz5%`cL12*Oi$<9DPp|5^s|S^ur!Q?FzugF$jnSL` z>pxhTklPIFK^A+IncfnH!eOB!u(5}#Cv+XA1?6(sFord4)N%= zc$d-b+bL&`n~ygYW|lq9Y)RSG|Bs`qfQhSH!cd@Ci#x@=IK|zqI24!S?k>yX?(P&Q z?(R_Bt+;D(m&M`T|0Op|0?Cqd?va`KWMl_o)>U`@s|hOV>1|0=2OV_N3#nUU@wq|y z-u*+dcMCS@FFkLg6gyq&5l(G7{M!<4>}Z}|ZWSYNUk1FoOL`sWu^yVY?ScMn#kcrh z0Xz8!erUbdz1N_FWx&k2?|g4oePEooa4q}oCo&pwhE`%Ub(M^x3@hrMxm%sb#t5*nIO1>-VP;-Fw5L7A|cm+rEM5L z*qqI})25i$PzRbKrVf){LV&9*`2V?l7VCPNoxvp3=N~w*6mbnZl!q=I6BOVB`QTJWo*%<5U~3ZuFPMpv$k2Yc-4>>yo*t2be9m;Bsf(6RXuz_BR+A(QQ}1kl-Sh2G zsaTPoCjuLY&ydx-BEu<829Sq$P@esFDblF3>!=#5!HsCw#zK9vQ1o@Qnq}>f&_W<< z%zC@cQ?}Big$^(aHS6&r=hc+6o3=1KhGE3#JTEP-NOHv=%B7@)o)-_} z#RgDF{HE)>2OlLI4HVspqT=uwd%WZe0%N0-q{J=5rD5<>g^SGMT7%Wxi1BlU#SalP zFxKq);mVGG_f0OUk;RY|7^CBdbs7WdS&x}BkzfShjO%I8ep@*r;>PmFysGknhaOd- z%P+AU)-yz^PNAW4$!p&NsDR?OY=lY!p&rd~4gKaR@4V~L@PoALFcvzrq1o6~O!3=^ z{CyBuSA$z{lho}20F-{w%1c_UkCHgjXj>iA1@n8&vHUj$N_O?8tFG>C@@8w{wYyTN z?|L5kJ_$9Ft@dASc=sUk>{Pv1P$pCdJS|7izan^BI zBX+kmz~Fu6qh)Q-K7eQoo1QKvpZ2x&#mZ}ezs`a-FY@-J2MGtW-BiH+lyv9Sz&fxf zf5SI09yexD3QJ*Fnxs6n$&OH! zDB?hH7b{eXQF{W5_@(<`W9YQ<4EUOlS(wZn%N?enxR~%l5I_6C(a@zfJn$rl_0;ZpBAInHDO^cW0PhgQDUQD`b zHc{KbU_*Ed_q2_Gu3{K~XvhGz6F9GUXegPs!o80Cc|_>cw7SXNaqf_bi}~KhPU0o` zLGki~Q;TQ$cZU|Z^2p|Pxb{rogNCA}Hdn;^?apsqcCX3Sw*|RU%lhTLVpzbkZ6d;&EC)MyGyt z(?RK@p|!nKHsKUVMWT@J?d_n;M)Z3tC1S`-F8k^Qm4x^fGMYl0b{`1q_1fk$&EZ+%gwSgWxr|RiaC%X6@ zgD-0>snx$O9A+=cbHQ~1ZbotnCbY~6;gT&l6PL?M$)=)6f%VZZwk{;4?%0cMvKj0D zY~@^OW|zjA(p&8a?cLt8nz$2d{)}so2BAq{Xj5fp{~}6Co}RO;Uz+STa}w>^CRxPk z1%k=(IXW4;g3@OT#B*^eozPVig)~8#Qwk`UfZ7J5A;$Zg2LU*XLZg*y_q~suOMqp( zG_Tsh)A6r~z8+wad=JB8+<5a$3|yp_=|u8g6d+4XWOX!A5VQ=#!Q-ma_VKw4AZ_5J zC=ZTOytXz}cwCwRD9@Ou+yY133P14%Zt314kzfNuA|SEvj|G07)odQ33jwPDZL2*s zfYm%N4xr?dcdf2QJTMBwQUZ3XT?h>4fxr#%Y%xiNtXtqN!D*?)d54~^ZxT)nnGUN zxtbm$5@-Zxy8_EG4tMVMw$3!Zr^n(QYSCI=sE!7o>c)!C6dJUJBQG9^k*zM(1O5L7 zWh_U_L-}DvEE-I#KBlLHBdFA*`RG)tvOth3<;8u-$?h~Tp+uRAjf_uExQr%@M(H^g z87Z)kge2g^hn!!mV}0gV{ZpeT=&m;oQZ4TCo-Bm$05~UakG3#!8&&P60e!ocua3+C zkg_3Zh5ZnN)8B=gGu*b~UEYnZ5}2RdjqWz1&5E+{&b3GwmSg+!)t1Nnc|OzqGhsZU zB19O0l=iDte_Td&_q|c5^y(z&>E(;9dfO5_bT1G&@!5I)x;H7sbyme4C%Qfa0i%b? zi@=@!cH))&X%KtiEU_^#@prUoogZbluT7noqP??FD&~ zy|BFfe-;iv1 zY#NP1%oTj0;e}XMBcWlOvfz~UNq5l zPC@nWme{Ww0U|0vWEr=f`vJQTPXwaWG@q2aJbO0dad0%dUkJ)gP*wFyTOYjwie9lW z(3HK6r@X;h1CAbC{9mJAHV&#k4Y-kYBW(OtWtZr1U1W*w&AeZoRMkTTJ7l4DUFZMG ztzdd{iS*VB2R-VKg(N}>K3Az@0==suoIi~|-0?9m%iJ`ZJ{n2ELgn;6+ePN`Z$yS? z%{J~XJTtm}{%@&@N#%OB?q|;H%TZcb?dZIxNLma4p0T0;jN1VpF?)^m6<-9+LauGC zon**Cu*?sH*z|6hugErY=vzHeK(t6)m0zjX0Ss1%F}zKYn8x|6Or z%5+uIbTC}HMv(NTZrT-g;o8Po7i0QWWS9`Y@K3khy0?2mu=t1mMo6Ln_D~H89F^&S zoB^B4>C!WO$$u;Fv0x`vK^9_nUaWI$@YbaQ9CjNg4G!LASq3;d8pO3kRqn1@uLk_H ziwHrV;=O{VzX4=A%8~&Tcqsef0&<$J*L?T0?-M^b`0PP`17E#S{shlgYI`4!n0ICp zav92PcQyla0~OSyfUGeaURz79B7vg#>Po+`_a!)KaHCysW>U@dp$cUHUj?7Tg%UI( zN-j7rUfgr_RwF5 zMReNyf{Wep-&IjaYjrSE#aY0GzW_2_C*TnvN0ucH!Jwf?YGQ5wN+dE%YeYOtNjO%PWQMoSlU1fiY%+Q|kFc;DZ@3wg()l*KB zUiLfaD7RaxJjsXz7AntHD%jZ@kO)0e1tuldf!8s}^AHdvws3aE5jgIg@Mkbu`-d8j z5z?dN#wc&&OWHA6Ulq8PY(;gxrDsL>yn<*(rW$>(d3x!>b$#*I1zh9Xb#T(t@L?LyEF=juV6}WMcw)yGt$_972yTWi5aHvalLP@LI`w<6^=d zr?xx!*kg8Q>$k#ZH%KtW$=>$8hR+DNs}F{DpTXDrxz|NP6Nz*nHEVU7ym+7_9zFiI z+u?@vV!tlS;W;xD%)j&%coEL&L@!mc2Vhjq7|xc{bC%H`K)hkZt@gK{Gzjujed!}9 zx3bf6f3yB&+kxn;qRzeGWsu+UFJ!+M7^$Xp2QK5g>sHel828`fxfBo)C^QrXg|0k8 ziaUOCppTQmT?7Ig5C1@>Z{HB1o_we8I0%!~@s=ubIP1;L6SD%xZp>@5i1BwI{0QLH zoMEI>KY37$0IMmbdXdWPISXgj+tCwj%+`;xX)a0)m|p7m ztr^pQoTZ<|M%V(8yhFqM zrXfzDfma6{*b>TpLrR#%WTjkM$p&JLt1V}Q@qMXOCG(5A!5ka4*zG112w&}RpU-pE zd2N=)A{)Hs|Av}{6Lm2Ms{m*4?OxprzL-%*CcrYLAjy7kP|aR~XPxG_ue1JsN*}vq z#ryh{O+g)l3sK{xf7jSu89?f)cK_@rZ8Q{3aDOe-xpAx0Y;`udO0tvXtFwvRE4_OD zx@zCDdJ-fd?zZ3GY(Uzf8HfVKO|ll5PBwxo=a2}SkwH&$)h%;sXShq(-hGdrC{ucy zH5N8p^41adleK=8i#&A-XSEDxoW=nO@f&usu2k{+B*tYkv;LBj#{Pt)zcq1oL5z|I zE2~BD5-yJnn*9b3`!|Ju>n*C4LgpQj{`y12X&TeNH={URna_w%)vsP$o>howz zTItwkN=e=bJCtBxa($eNxSg(neBEa&D~ALwYy6dowVbcr7j0B(R$hZtCX(GxU5G{c zZbN2!1=z)~6#t-i@pr_;Jx zN|Ts6wLXWKH71k(D%MDH(^>ez1wsQSD(Q6ro9= zsSqeriX=+QVp94~TGH6LT*iXdLDXZ4XhA`uA4C?(TeHT=j|R(#xJ{9)9wGz2ZI zS$fv%@!f1l7N}w@C`ZX9Hl{71e1nAa0K8QZd7S>|)-F;Boya5pxa)mJ_p&iDHBJHc-P-~XU0kd(acM>Fcn%?(-Dd+ewHXLRTE z*#9LS*g$D%4twhSqS5Q|^r^COo+$+Eq$T)oAfMf@ZvttEV4Z5B>L;ysUJ`~EcA-Xs zHcx3vJQ7`A0=g{S(SW^3QbxXI{^TyRJwR3MoeJxI&jkQZ*W%PE2QP&bJ6)IcTjmYb zV(Lg>$SqmE2L;AcUB`DWvecQ^SD8-%y0x(|eh#xn{li)&1JX>qA(Yd%`~#zXoLs)k z)tx^SvIFF%A7AxD#>dIXT~4bi^)If`v@=@+pG#&ipGcy7jwwY8vG_Mw50+YoHV@z* zbT!Ja*1Icqt}e0>5?Wr5g1_2oo+0s2^g}{k08a1CBSpcS(}hs6;F6|yA3Z!{OwIal zQeJ)GV(m406zy1!h{PHoF%lQhOQOwAVl-e;bKUira5d(*`A9cpJ{pNpXsPT{k)krz zA)}ea%8gFZzG=4KR;lEU7KEKpMAP+b1U~;%`B^@*x~=furS(4qBlyo1e9nUu7y}1Z zA_jc6oHo-pAczIk8=Z+M=N5I+T7ef^^|#T<$th+n(!&GVmdQ0O+rOosGT-PB-x)Fa z-h1L9L-uXXkwlz4d2}cW;={kVFsY^vzl8;D;JCbAEp**De<)8@4A-nMN*MnYg8m>; zvJX@|#b#Ye1h`eGuins_QeBw5%uJ&dgHcw94jK@L64Cc1!y3)CG*u4-*;ZavgE1_dNx+NEU!LC6Lx4p zA@+WEp z>EH?D7HoplrKU2Z4nOO6f6#DBvadJ+Q(nvL)(-OBpWfWGLJLwL92S7zmel6A2i8u> zMpD<=lus!xQ_=pbH&gEsZnS!HgK8(7K(y9%y4Q2(FxUb{B(e`%a4sd2jn`p~vlzd0 zZfQ9AwOxOtwJZ!3fkwt&Mvq^RpN| z_qRL%iHSq2sto!wuncsnH}G`+{?2>z*>`UvE3*pczmX{tr7gO$5=5yNVb<-TBiUE2 zwb`ygoXq{Y`1g>!26f37o<(^85J}YD9bAycjbh#LtY!%{`S~*ZVq?Y2;l`4g`YXpZ z-&nv*4q(G<|5_adf#=+Q_x!scR)h85 zX_$MEd*m>liwrFvG#j4%)S!zoeN0*UQ9GjEtpIL8%a4UBHX`e7!239f^1?tAO;+F2 zd;yDpbxnY(dT^ z$@a~_5!hn^d1d+HrP|ah7c4TrxETlU9U~Ee0o4>n#b#g4hn7*=Gw%Y;9_Ook+|S%yYt_0 zVngF}SkDMA^k4p?aIcyd3%T!-){%GuWOrt;K`Zd8h$KBmLo0^{tND!tX7(VnQsg4P zlBB)V{db-mGb}wLq7Eb7r;t8XSj|@g+K(WPA4V_0iKx;4lmF$hb3@fsh_R#Tzba$w zN7xSsE@$2kQ}AyW7y9??0>;yQ?qa24|c#{{Voswl$H@!UR~oDTUC*({QI}2_hT}H ziCVZgkn#v|HY|<%!tACv2Kj8`0R49>z%EU{9)-%r@vE*a+npHdGF{(Tqqd;9MyJHNnw{#OsP_;EAic=L0oo5MV4G3v5eb2hScFpAM2+f z)-Stex+saWj_YIa7hROQSM*M$Kiefx*uKO<)cL-@#?4bBCSg;IM5Y?qb|LTsjv51S zTt7))EbH0Ks-J958r?Om2s?by{dd3XdlrK>dl2pUH>W-!ASeJeePWLn%jV_3Q;Ya7 z!<|47Kn9tCVj=xm7;n0ff|sb&pC$UQtbmi5U<&cWiuQ3FHhTDz-`bWEmVoEScH4sK z==T~`w58W}N@pDMR<2b;O3&s0MlU-MRqD@81Ep&2d@*(pwruuL-3iJc@y_WKB`Lsu zCjCI>vjFV8udC_mp-#PNd5GzOJAaYLcU2lgxC=Q|iqvK?jy!SINdtaal&a;al(<@f zq6C>Zs5zLKK+_sSBjJj~%+~)gMSI4McNV`iKk^%r<%Q*oAI%rct2$$gr@bEXagrg`SuH6m;3ct2$vzC27kMS zr)@(N{__(XRPa=5phH2^Sv53*Y@u3Ou!>WXTxl-rWYy7T>B7DAs_!P7pIEUs3){@Sgy-H$-Y zA$@kO{vP*RE^h~GX7RQ7#(r}y0u0q4 z&%4it+$DanACWwh3 zW0JrBeu*l?(Up?&oBVj0R=WDM12S-7hj?&-o)Y3!nMjx*BCrY;Yxp@D<>B7$p`J6D zw(g*X#^aI+l3aO8Mm+;2t=nG2NElwhmt&Lq=i)>++>?4jhsLO^G^%c|1~biwPxQ#7 zy!ewVV*0|L;I)uhl%yLo@ZVn*uh;NjL1ZGdp!eCvdYtA9K3WJX^u-l}4uGubRJZnR ztZYoB=KmO_Bk79wyfdneMfCFz9-u>$hCHJb&?*ClYX6u(xbzi_h`K8QN~K3Ytod&u z1nd04u*V2ulut=BcGKfi&B{{z!K0+P@8y(eOvkFdBQr+;SiF7p{eoeBNF!MTBWCxQ z^xjP1^0R{V6!pYhp;(0c<w3?Hz{_vJw)Wf@v2=pT|9+at7gPS62D#P&{` zwH-GpioDH9yd75XyPpGouVM7LGLt;?h0=Ic`XA?G`FKE|RXBX9x}QL@O%Q&(`fFM+ z2MjJRt&&?TB)#4*0Z{6ZS$e%#PoaQz$mU`@C(IG}4In+{CuaI6*-Byv#ER@W8wLg5UfxPZqSURX^^#FGgZgUqk=N7=@;u`zKj`U>QqD*WRLjrN=hqdW)2WIJ z`Ku7O*=IYzed+56B^*Wy1Q@2hCsWE!TJHwqD@z@B8GI;jkGXaQGN&*wxHBD|Lo4a2 zso}L~ zk@T>wKUzWMbyE+!H=|Gzmm?QJ6TrDA5#+DkYXxD$lop2dF;?){*`HpCD5C<~N#}PK zm-LbA&Dk%bo)_5#ujdLVKc&>R%Y@}}K~-vx<yOy@;T$=EXJkev7RR8s$tm89j+anUnVm_nu09K|-svuy zRL!QyS*a6~clB$eQTgFwAx5ca-a_`{`}3H$*{kb8qCpnhickGHX&Mjy_7G+hQ#q>O zNxdY_fp0hWsgv zNbn!9t=<32+ZtzaIjc5L6LEpE^&uu|H26$Qd)pKf8p=;moR;OBKn}$~zh%gxgeMrO z+B?352iRGaMc+tq&s69*49jpiq=O;qT~$ErsOc?+mb8tfqi1>2ZTI}UeE#r`p26D# z5_a@@Mo!WeaDP5CfK+YA336Yeb|dZnD?YfwIQR|uMN0vwnXJ2eXOd{Afo`JMS5{B$ zC+aws<6LZpZC~n0#N3+LDEy_Tan~ zlWp~#RCl82DBd4~zOss(?h;PJ=2Yu6LYC^Qd3k-*>$J_L6@5;Em#zv11i7K0AiT)X zWF-3cl!u3OJQ230C>Kr3PW&q>`wlFkh5>U^2Z-qL%5z)jYpkdG8f!CwCO+f|o=Le> zjRY0AxVqD9y6nXf)*D_gI>oPCcc00rfzcq1axCD=!v+JFsS5zTxLrmSB4XibWM5(9Lwj zx{_Ah#oy^;{rb*0((WRzoA&zl|Jm<+r)P-3_+2n-YM8!IyHioGG!_8T!i0hr9x>T9 z@RZEWb$)qV^jzSw{hir-YO(1~dN5dibI~Rf@_2~&AP!WQq~5ObYa4n)tMPeCe*S%~ zF2UIS|BQ&HoiN}j)U?AUkEbsBYO!K)(VtdbJC(PP{sQvK#6XhNqx(>46MeGHnXe7( z+w#sltqBEl!;ZwCsadMO!vEyx+Nv=>3s=l}j5qeA8rd(YEJgCJ>&+o=F1N_~{%ap* zy!f47m?<%l&hkHmE<dKc)e@EOt2 zh|}^6nhabKXo3F&SAs^zz51l+PV(TfZqr#RY%HU?BAuJG>(28y%GOnPIu1s!y;9p& zs&hd2!Hp&J+7%_u5QHYD#RZ+G2+`bFw?oNK^DVBg{d?B26_bpv^)!TYAjk0MF#SuA zCNV(GvuA~zpqGfqPmF2=-A6w~lenfdd94ha`4qj(n|}OHza7S+Zhd8}$o2A&Xm6li z7CGGcn9;FXomz5*oCh82ibW#$`1EYs4ta!_QW4yhfM4iruY77_QK2pgIkLy=?)msh z-cNUi@gI@E;&tVJj~?e2J*ke5wBV3Rb5o&iGj(n7B?%}xTW!4kgMO9O_HEQkH-m)!-k#1j2sxRa?%pbpP`cC952~ZJl{d& zLxU5q?j@4bS8jw4JhuDDueR2)H>&~@tFpXzR+eTR5ePkJ7fg;1NxNQZ1Dslx4NXrRa>um3tz5NkWL;QFM&(U2EK}H zQ8=-I{u*OvRo-mP>`3MEU;WkT(x$$NEZGh$2EW;jwmexzW z%s|jr^>afXqYK3Nrfy(_3*s;m-!Ps#FauCP*fL=g(d{!PWF-j@SXmY*Z z$}%#mijDeASUoPkxf3XEJWg}hX-mL4cp{G>$=?BmgSDp50Ry6ggP5|(OXj`i0srv1 z$B-4P&L6ka^iGAD6!I1l_avM&q{^e4I~_fqa@~C!%i|3P>^uXGepZ%Otv(0AN#vKd zMvU-mqyi1hvdtz#R0F?pL=hB0ZGD4{cfR4M?|zp{Oxan*b#uG_;tiR<7@8D(X+8k? zS_`ls@^9YGF61(`o6TZ#HJ#Bx&LR`p1@>a^r=ILuc{#LBVc?*$LS-fUEm(;+7F?c+ zjAlIL$5U{rAlCRk&Cxopa|?6bQZBd>(Rh)2OH<`7#-V87_*Hml=S4(+l8=`PI$~2J z&&2cz?%~*eKO1WZyWHP41|H zB`VZV{z{h@pZGykoo32GG&Q;U$i$c5Kb8K-*1jRLUEs@?%49tbx}$x3%d2>@OGD+^ z5gR5~4*@+{OlPFzEA;Iq;-iiTYrT7L_?71^q*0k-cFU6)>R^aBi^c%deA}rQ6scF@ z=}$TWC>osXJ2~xkG;2}WDOqu>C%DSTgkIKGmQG_AHE3vJyw40~C_=+Qzo3A!PeZ=9 zePeOziRH(qlXjz5f~gGv)Z6#9#UMjq|0>VPz8srs?xV-i48}cYu$uE=`&Jwa4Tv$% zH1rQ1k*UwuC5tH^FIal;sh-Q+-)~+sjpzqZ_B2cs{P(WQN7ewP%@OVu$qN#~>+X>L zCN%>&*51;m#}>KeI1|hKbMhftQnmS>q(;MATIG@kKmpA=QPsSl-u7;k1HFVPfg0fu!6l@)kf0yj4vp4e5$prp>r`^NYA-|GTE_%F z&pRKcxa^l_I!vYJ!pO81_0ye>7_^U4^%k~^i}zdxis_Fvst;ZkXqZnj=tG1a@m?aK z&|cUIjyeIH+V0eJy@K@aU_TN_voa{3R1;Lurv0@w+S4;r4yvzlOth&Dbf_AdjV5*! zi795;o=ar^xO;lhTsz}EAyiyJrIP--8@sRaa$m*7?`dpEtmZ!RdaE|+`8k5U7=Hw? z1$U{g7PUIM25wtptu%+J`sZ{9Iz&d5kJu`V+E&MiVnM6Bj~*ghI@Sta_be=YBA|ak zzssqqB&-8Z;@j65D|oF2R!$u_I}z}{k*WS$pIWS#7`^7pa zXaK1MfO3cbd9*gp7Ozyu5MTt2m^xw(e)>awkOK&E2k&&sx*m#7h7sRivE;Q!OTe#i zINeZaUl(#}ggT2BZ>}zMasL%3;F$Ai9`Hsy;vL<)$U3?Cs}$dkxmExu459k2+xf4M zSTg5p=G*2JMZt^5v036SEVB6Db6EiweD97+x+Jg06Wf)hViaEXidEQrS;(XaR%<3B zKe*6WroaCTf7x>Xhn}g3fipvJTozw87FybIqQc4}g zbJE>K_k2Hkoi1hsK4JKZ+T4QG`-KzOw9W}?wD6BtR!0lb_z3QMdrGVEW4w@*R+R6foP&1hJXxH z>G>>U@N=eC`~5R4bHwkX)00#b&lhM-eQ05Xk+8vbM{dmnB;iRQ8IQsZ+9ei}-z- zSd4wi#z>;cV8jp2?(pCj7SMp}ZIV{r9X*-ZW?bULxlr}-zo+I&3Q>^_L9Lr9iFhFU zVza+MLj0hW^)mA@3Y9O%SkEnLXXHQuSLsJN;Pfyim3cy+$;dA8Tbzy#+#JzW1Dc=o zvOlpKv?~1$PxsmeVMZV-b+7KB3uG%7X62-SMalqNW|9yc)5mZq^bZAn|AsP3Ol}`@ z0A3abknMx)0V&OPF?rYzG!o^qmh)!cxg*wzA}`IT$PjsvhqEe2*^J@Wn{67 zUp4*+8wigxMGQ|069m_zu7bM_M+F!&NM*jy>$*4E$0Cph!9v?UK5eywwJbS!fd1+P zH{-NXzJGRpuDx7|`MB~V3xz4s4#W(q=AV5&YP?zmEzMY7y~!q#&-_(B=A*Vw?y@LZ zo*QHGC8S^->PYt z6giA$EShpiLFn`-G;~4%eBu6>Roe4I)~tbI*~cw&bjniZ@$XqYRJJqx@@=utX|QC# z#mw*0;5i#JnA^0Zv-05*69`1Mq|PT=kE{Z)!2V=Ve~=eniVMp6W?hT0!Z`)MRL8>~ zzUt2o*w8!Qf_3r~WE7@U#UI;*2@O<7h?WAmq^ja{oAx?>Flry@Z3~_s^?v@hwmd`9 z@o%*cdS8E43T)tPq&UwEc2cx*%^Re2)6F4Qvl?-2RyC#$Ckiv4iAb*ZxXnDz#@4baM>% zF%@S7oG)aSExY`61fuBx0lyNms zzl1P~74DO=1YaScu<}PY&b!nx(V6U=W~b5#C?g+* zoAwtbVWfCbiO;wlmejO%V}ra&c8a4&AO})_&cG=5Ba|>WS~k(}9|KVCg-NfMo}NJt zDdEg&(9wIJ$LmFgCI>@UK>{p?<)DpfX$KEH38v>qIixS!GNIjms!^h8q7bwO)KXkd&U;ji+mro-LqdB>_q((>x zL&&sa=>P_5ix5RJ?M|h9b=S!S_^)qy+(M1$ooMBe1_nkKjnBUxlXOMp#uLoNpHJ!7 z+g$^v{hM)P3AobbCb*v!PYF3)H{4?7fmgz?DPTL4Dx%CgL$NCp#GDxJi{|yOyuK35;QhT-wgxZ1&W_zN#hkLA|w8ObtRB3Pd3>< z#AOH@o+>oSKO=W^yTkiF=B=1|pQ4WUoqjx-gI~Rt%RseX9M$)BBZ?jueGcli4vJKF z{ZR{7-G1hvDl97p_Yg(IPW|C1iOBkg4T%c*QMi^eHbIHmpMG&YHyF5jh*N~6P2!tTfgVlVp-}I8qo2d)XK7P+kBEK#uZ{y z|K90pnaQ?{(@g|&Pq6tCmsM68_9R6WE`MVqPZeldT>UgVBM z_2N#Anxo0(-bzel`I&rL{&xC!r^h&m{JDj_$W{N-pqHdbd7@m7NRdfgDZcsp7=&ZM zWRK7)=!u}KD1n~uC>84?WBkS=8qk5|!k~5YLw0@3+3X~y{L*V;0mmHJ4lwM0?93e8 z-Mq7*zq#ci1&~d?zN>|{<9$1!gU%-rYBFPTlUm+Zb(wN(jLLu*B^h?4jFeHj)`dvi zFNtI%347IrOg)OihuT>Y%LsXJV#(n2Vq8EI-m5+}@)wi2Yrf94_}w>1g-Iq#m1@As zB_ood;E;n4HTSq?Y4SBAQPSeyjPiAfv}ck?**qFMm2#h&Nm~>V9=)RuhdY8abu8jo6ts_(ayr9S{K%y9L zaRzf`R5=BZtS(BOyt<;=>_2jF@aPa%9xXD4PtOk6y#OvP=?Rk2xlcDPA(jkyWZe9( zOmT`|NZNN)H^039F+Q&ay*}-Cp2$8CBMEX>Ktb_&$%u=nK|IW~B0d-s`U4jU@UFZ_ zaE7kt|E+~ntog5trYg3VC;YyJgqE<1G$+#jQXE5FXzh^JEHT|H|Na?T8n_%1Zt;W$ z_V(Q>b-fL&Y6Rhxg^ zt>?;O*7EtUvXyfFL~N>LS&4k{*+z1fkW_QB4rp9Gj+7{wI?e7QaHAbV+UE zGxisWcpkX_62=JY#|$L9DNeUdg}rMD4#1lUe3_xfR2&j@+f)9CayO@{2zMGP&^)GwC;G zt_xFBc1BL#`?IlgdFo}1JbB^J5aMOhRP{cF0Q4YH9ULke&rR&|5X6Wpt;;ZfYQ#$; zRkG?ZGf~!c%h0b(@<^-U4x|jz3{)vjZgyYsh;IvrR{_{-HQoGd`;Wz+#3B&BqAB7+ zLG6$u%o+cyAr|AjbJB=Wi*)7rof5Cb`WNM!x$EQBH!b0kzTO}1b+UBmrPtSDu}WVb z+pIcBzR#H0@QV&#KZ5n(V?6}F?d0%L;vc;VT|yV`X!d1-U(6O!hw$zP?q^VE2|_KR zbGo@XoW8tv|M>PQfhKN(#bH7z0Ak7uM_b8=E7+80g!AXC%N?Dh$v5)B{LKt&m#*$f zsHz*e4*#}ciC*ycLvz1c`5$gF1<8lP_irM<0D*(XyXi|tw?Yfh=9K3-D^S)a%2$mE zQcWt6f8==$tuJ$?S@5hxv~h7c?lS;2bq$3keVp_gT~Vm3#A9}vhFE3L^HGlww=j#9 z)6~PTLj4TX#?K_nW9pKijo%cBf{-EGWwIZIO)|dP!Dc-2At1yC4hm+tqsPNqn$UYm z;D?dmysB*YqnMglBFIhA(D7PFmZ&^trGx%Prk@%uURhHWAQSc9Adn5E5OE8I#{nHE zzO368-M`fxKk}t=sWo_WQ|X1z$f1#mM?+pGkwr`M4;bQvo>TVPRuMtn?I~jxWepxqju z{H{GR^`1xk0dXeTJ#Xq4Vh`xfj+rpNlZ5{|N(ujxY0L;{w;ySL4GAIPRP%wjE^ZqW zE6kM@id`G!U#_xbr%F;~^YXSncvzd;U#kDQ3|K=;_ng39Uew1Quha*O2b0kmM z-~8L(%3ajadn1!@DmOZP!MEh!W94M?V6$0-4bQKo@RMMgj1t$*DvWrGWp0#lL6a7^2Jw7D{5>z)u~4?FJT!FW#>@9 zR*D42RLDVVA%w!#;LK{xu`crF$5rmzh;(@5tg#S}T=;Kf%#SYk$U)PwG^d)I69{>E zBY!&{YCxk%!f_OqytY>_4Ffk5^R9pr+Ti94b#k%_`E!vp8=rh?W)%&n%}q-*aW|Ce z3$3N~S^wk;d>)WhI@xKau-T`euKohr*v+kxn`&i_+*|?FKKK(|e_a8Peyd{w9%EzU z+>GswJfT{Ul02^J;>6ydj8x&s9)@DK`0#Hm{X&2)GX;)Q(y_Uca>6;Z56K7#6q?wP#P+h=G9ihW(Y<} zkY;N$zl~A#?`8RKR8OAcY^=w;+Pn90GaqMtfbx;cOy|4}5B>Ol)dkq?S#@d8ci5)> zdD~a{K|)5Tg?CU4`a&raMf}D6(8SxBZ!p+WNx7K1SWl1{1SFl3fZ7)dDqw(0Vw9^n zLo3VN>EetfR<^GT?T)82L6=?bhwd)_asVjMi>$*>R%ackf=`=hi3Ic)TxY?GnEi7a zLN$I#N--NMV$}2>DOR{%4o8+^=sSD7dbq#1V0ZO?B+902R)rmG_hUeqRwp9oReze! zUCz!-!_M{4%Soi?>8;X{nCd2HVZA34=84q9_K0M_aGCq7OwJ&FHER`ymwno_d;~b( z2&v6Ki#d+P!mqAG@7NqiE*P4VP@9Y-|Mk~T*nZ94XQI~^6cmWaAw4FLhWfKCo$+7A z$`ZmZC1t)|)(GU=ZX`m%uw?WS8fmtdXV($VV0MwHuMwC^*5AEpE1Tl*Z%b1rPy5-B%M|DxBLR=O&iMFSqT(x^XL-!S6bItFrA%i+ljv9<}I3eEeoC< zBR|qe8D*134d z|H&frBTgdBg)#EInXgF-@Vu2;BLS`pRXo#mfZ#Zk?iU=h!z!C=S}U~vARy@nXn{LW zZr67!wHf|7N6@6SbVm6%01EUznR(jSIN5BV;?fMe6^a2%poQ1IxW0mY+sJ6*rPtda zQ;NLbFHfxa7uq%q1qy-jyHr{2fApzE&dfp6QSsK{_fATIPLm)nso;Vx!fIcU&8#Sg zmJQeZk!(U5{1gV&TgQ6RqrF5espw*oi;`>l<+rQuY z5W3}IXdYa@I@sMYhlaJuXjH@fSiE?Jj!rKIp<88 zz~r0#r<=WC;?ViV=b_ybp;hzKhKFO0&ni(DF65j&7eG_b@8t0tUP74IeQ%eIxb?vc z)cZsBl#$igWY_PA;*%fB{lAG?*u|?H|6uy{zT)UakDhkt4~``dS6+Sso_C#u)xWz- zP94T4;gf*m1}Btxw$t4DyuobH%ZnLQm(I%jAn;@IpEn~{49TxT{2T-O{-;Xr-4MrP zQqK@5s0V?ruEE#iZvmwSVF@(yP#~mT*KMGT|gzB@HMVPf$M!_UuSrVTW-muBoi#SsaEj zpL$)d&E0$nV!vGfMl-o;a_3#~RmQ-^Tt<8)qP)Ic|ZxJyZh282$CWxCEY0@-F4}Z?gl|Zy1QGX zySux)q`%wG`!~Fd^8;gWfphjgd+oL7nsY8wrP||6sv9Nlq-V>xHRsF36E}e-4q62c zZ`{{prLaf0|HQD$bot)1f4?SYgohyVwN?&Z?X&CCFbxNVD{SZP^_%@y*DoB8PsW{n zDxD^jo{pj^0D-g$#dd~oaEz3)>krK*sjecr{5$=0d=c5S;1pD&uAX|V2C6RiISIBh@s#raPT2`< zFkf^i8tN|{a}#9LqHJI>sGr2=uJ*C(7FjR7C(dB=YvIpjSv>-8vgsO_D4|liQAGnI(_34+D0Vp*z-Iv; zf?mr)lnH9H^A??&cn^PGPJkU+?g*V*n4Ttgivi?(KqaVxCzi@PGPm@nLTo0OR#mK; z8mbbL9~&Jk;cn2fQ}zpAV-z<0JtQIqtv8(FyLVy4hhuHsN-yEykvgB0FAQ0%4SEmF zKWX2>tk2zXGmu04XD<&2{Fbe*^T=$VYyzH)C>tsMV-aToBO%ewA7|_O=kZZ@vgrGs zxkK$u;$tNINEQ@K7Ba32G#- z6sOujB@t2G4);&5dM}E-bkT>aie}v!LyUL72?ap^kDCW)_nG7k;1A~YTugygRp#L! zqx2oMAtkH_#Smasq)g_a)T&E#)tt7_h_9dsn7&%X*gKf=`_XHZxzW(*mGoSktv+$zvL-kSs#2>?G!1B!2}Hw zX7g|%kWrw))FrD-wQ6T<^a3>2W7Lb5iSL(sXkReF(PU5la@j0z#N0k@=ot69@io1l z2!nGYi#J%y>sd(&5hxEFISo34xM)6x9P^RK#a#6bmyZm4U`+X;F^b z0tk4^LsS%K$u+a^6b+9QPu5=Ou8rl}6-2Fx%lB%l8&___f7UzikM_LSQOk~sy47-} zqUl&!#7|^^^EX||d)9(78L|>ut2wu$dhHqvAaDr3qB9ET;?#Rr&iV`hUWRs zAGo3`95mu>uow8FYSWK*RYHNEQ`hc3(dLbXCddnaPVHj?(n1M|(pLYV5Iq(5Ls6XL zFnslPUTX}bKK*pic54?8E+f!LY_Xyl3$}jVk{}p7-p4IyPj$mPMrgJsr~CzK4}2(` z?!FZv3I$Q+zjSd=$ZEf!KhIo5*l`SHc6h0A2;AODC+mPcx%nIN)y56^hsW zjb3?G5^TtDP-##m!jF}al>SBBkD#BQ;dE0pup6>%>Y{(JkN!Bw8GJeW*R5^$4+FtH z3X)iBO}h=DaKwcx_PF&hTQwpHcN6;p-!64Ir8sg8do0}s z*Xp~kCFZQd`!Tu}tgyu!nwxRum*ISLbrkF7``^f(3W!k#4!!%Qxn*e zM{e!sPBYFc07p-=)sTa`KXNjWq*OpzYeHp(q?hbVROgR5|HKt;8XkGCrx)1Utju^? z+$etN9=9BeKKyDwjT`tEI^G&n#G=CsrL~nkHtH3l9Y`IjZG6rPbeDCD`6SCase2zkGsc_{Z8A?SYk`$A)pZ`P(KR^er)18 zkJ!>yGne+1U+7vs)$579xWcOPg;A}GdbFNWjf$(e z`6Ky7NH&aL$b^b7ow8w_C!>J02^pr%0XxU%F!>P#MTS7!^~c^$0S^U(aOW;A3?ZDO zK`LEXo2R3axJ6aCTvQcGTq^FdxbMILO=H^(d1OzoIyv=-gaFi-fn_dw)sG z?9Y2({YAAXW@}t%Bk=e+qlmlV~@KZmRNO6 z5reD2&}#?}(JMJE)Ms5C&$EzOSXn1@*};n~NhR`-WZ%ihsgSX5J3)J@n$)8JdvL`3TTc|%;vY!^)SK8HC7idZ6Z%4e*{ z29yyy!cx0)C_lcSa{Mvq zdIQlDD^E_sA#?kXmrmbu}e&jYI89A&}pJn2wiZ zMte^U(?56mUYTj5mF#-D6bRD42?qny`zzB3m(KO55?F8S%j639QxJT&iWj44;2UyP zR6vSP`=dOQdOKpK{!r_RJcalCQ1_l7Hd|IaGD3=@Gmk(&mMUxA5kuY#O?NSnE8)b7 z-LVFrS2GOMf=Y1>q%2jXERDb3c{tN~8O|>rbmN8({CPZwQd0UwsGz+2#eB@-AZ%W%AwqJ#~)KNn)gd-7SlOez*4H@wmX}fZ#Sg)#yXu*$ZJz6B$9Z zEha?2G>u;;wRSqiTlZZBxHpwIJh*78OYC2Y5Rc&-y87B0rkeNjOVZ2im(;E@^%#Im z3SvWgh^C9e*$f&Pj@si%mhP%*y0PVXr||?X1t|IxQ8*pE+$#YZO^BZ_C>gmGrY`h1 zyWj)KpyNuSHmmNpQF>ok`0taw_)Vaz4N{sLd#S4YYU8RlfYv2t=U+f%H)m0QWDyeo z)M<3oTlEZj!b~?FPdBb$m&A#i3cLGJun6ImjD0Sui8j-Y$Ts?&|HY;ywan9n+sPm#slu@5nnpND$W-InSno4HPO zd`9>A$e5}*gFQO&xRB$M3E5(Jn@1*)tngOkq>MJA}^jH|X6E)T;rAl!8P+U~l@lz(<_-)GRl;FX!$D zo+p=)mvvsl%=F6OaUAij$HY6%#{VF2VkXw#>K-?{L|nJ|`jkJKU_jOFNd9MC>H5*1 zw;D~`jVLBiS%?_&_Uq#jn};QILp#0^aEN{i9J?Qx2}>IG7BBMyCXvjLGLUAcTIu!A zFX}jmT1Iz$4NuIWO>3|bC>^wj7143v+b~+#Z@S-Y6_lb&(#n(@kO+}0+@k){AKzvg z-&?QN-9_n_;)FMF${E_Bhoh(K^$buv36#KL9nNm_m9hgDO5~07Fl%dJzQEqE&C!6R z(b21((%0svKAcG>pBaalW^GZ$b&CemBu(DCArjL0G-CaBo%U%PO=bNHA_j=acT)ii z0!LUP%*Lo8p??PfCQ17yBtFSo&4R6XtW#*%X_uc z{k{S3s55`ffO^2{`U^ZVO^eCDtR;+|1q!o4<1jVy9>xGjNUVY+pS{Z_bGk#Lnl)+p z2co%!MA%0!e?1esaxDp08D^@_c|h~jU&xyT29SlZ`zAVgdX_FHtf0R&rEfdq~Swqo#(*`E~Y$$v&`mw&M zw-bAh1dU?ccqRC9Gf4y}d$ECD6p}&)fgpw*s`X(d?l-3`n{S*vW*#(kJTCPy;)aND zSHyI6D2E0VR_ary3GA9B^te7Ln)m>})ai)o*!KG0-1+F#@^5(_ftcgz&;C7Yi)}Mt z?jcf8qC$bug|0H~byEFr9}}t|=P(1L(~o%n%oLG-N9ws$XTeb!K*<4?$&n{laH{zx ztx{#1b>(KArLx|s*ww+6%Zttg2XI5@rtJdp4(2?|=6WBU|9F-Uiz?qg+Cpjt;rv$$Tslom>|iR;9iTuEvF0i;6F%!CN^Kxi=k?E7Y>MR#2$ym%u`00veP|3G zYy7z19EDbO5UKxPhtE}ktjC{X`j7HHh=hsJDUHZN@YgAgbosXvpsYU|92^u>UQ66w zvgxoG&y=0Jb7d$gQgw%TNDfD`^3qp3CkK-QX?6dhcNgw24}JD?99HZ1>e~1>;U%=k zXdq><*Nzh)HMyDn+tYSnWFu)!>6iW9w(In?tld|)Ilgq85?t>_$gK0=Qdb{*6aYgXaDLNFV7>erX*tEg31gs7cI z14ytqtdJ|r{J&n-uW^rDKqC}3^8VgZzkqyNUF1++80~OfN27YH;WyO-C}LDO7nz3SoAel7 z)ArHt@_T+OP7YWM9C|_5%@l~b8sFjDo0biq0%|@^UZ9qJ#P|ieGlrOO*yfZtUy<-K zFIK#qspj*KyWVOm^2H$47}K5z_7L`*62H29C||<2s|9W>MW0;qm#4m6spC-w2ya;q zCU0DG^c-<$Gj$-zn@Mgr-9PIPwG~XBtyv9J9fS#_mCra-wld4}QAj;nh43%`Y@+OH2ez6LRue(6~7?%Fk@N+)|JdA?xh7`=Be&5J~QQ}a+@K#EFB zmz>u0_G<+~Dd|Z(mGTN%&DydUsky_-qsxqxDxv#SVMjY6TG?(lvw;t~f@JW|ZgV)h z!tX-aA?`PgyU+Kf#$Mr(1Eie0c#09R-D)1!zy0H*HfwYZli>{xP(c;$43to}DyTKE4WNlsf zCdy81CS+|8h&Rme{n@!f$5J{hzRqPOC1T4T#PHg9-e-pbtMYW!!@Ga%i#f+`_L4h8 zOiKQc;fUx?6R)L5J2lN8hh|xRp@uHW1Y7eS;pL2*ufJ2XA!7x~4w1a$_(043i{J$t zZ{$K7=*$C7V}`COzhtAJ*!bbJaLvP%Hr%0uCVh*PG*$-kesV5E_K$%i&uZA*JS}Za z?rxS3St|~StN%D{s*x%O%KsIG_$Wv=H@`nms_CwQmOoGN0a_eTohCkDNc@5)A<3b2 zllh2sh z2**dH@Y^N02r>yXGp-NXRbyj}EMvN|Y5w{Sm~V^l>72=cSIEwc;P$REpI6}Evh;qB z)r=|}tceZFiuZ}R0WXV{@;WVn}ok?9fAJu6;0ix{2xw|)(RoH%I?xjt7OdD$e> zE0(1o`j%Hn4(*KEC^2m}%2JQ&(gWI%o$>M33`e0ZHPwYd^q= zfQQW!&QnAPA!k2mKP(hf#*hP~^bkZ|SDf4F8u=T~IpvwDvTA?t(MHVsYr7uc`T(j3 zMZ(!kjZ4h#6)WeW6;sE!EN{J8Ew`jJ36h17FdRZCrVB{l&3P*Hr5m@{qZGN{6eV1v zw(%*xMp9_VHnHeFEpQR0{Z`~Y`J&U@Pysn(3dtxdaKpdWsI@RR%`h&fU-NFe^ATTBg_tr*y+p9aamN0V_)~Iu(L`6DmLM1T>C(4aOH12M* zci$>%W{%~fxe|om@LLF@HND!a!s)P9j^8rV#>fs8YsQk`Nr@pDs!Tm#G*SJey8Y@u z4WS%7+RbqB0kWiIZaN9dH@lV=x`^M2*}gB-bX^b}9y5=x4yZfpNMKW7do$)&A>5JO zm(tIJQjn`PUO-K9J=j|M!ebZT>#)D++^xff~;2i5LyQ!^A~SNZE|ROeJ0pWUW8JJ z^#`A}PpmMMo|%4-8g zy*LFywOLBdjz428gygUFDf1irtPWmdibQl-=FydFQ|L^jCpOp}fKdCA!A`};&Z5ym zE#xrevT%tx#mx33-^Pl^@}%af=t2}s*SdnX_lSZ`%y=hN6;%jk4%nDoe=wkm`Xi0E z4bOZNk?KT+hj{x4*y5Y>@f%TQppHfefC$A~J<`!GRFqdT_5wnVy%)FHG$Z5 zlxkAQJ*VYSIX6yZ#Vaz&^Ucgk%S=aWcr-3WkKXPZ;^9~>@=K}$><0eQq+O>f&1nf- zW-Q**s<*FKwRygYb+H8iK~9c#Pe@c&MWOqOcKZLjHWcjiuimxS-2J9v;|4q@m`&LZdwmHJ zbt02c@xahA53y)_C$2p5s&ONz!S zKMnFTZ0^|G1Fab$CtPl z6KbB?#Hig>Bjc6LfG$MsbDT1y1^2|vqK^B`$#=WsKjiVBqm6KalQ(9Eg+LTjU zRMXm&H`MuR$Uv+hEJ9}TGtF)$mczTg=)SpkeRCC(6`>|~@Eo=;RfdHCCh?|-1|<iLVE84!tT=xpeYIqf`R9{C?I-EFce=7Fl5?4Cw_J z>utNo+1Iis`%@Sp;Fad?@N7xnRBO#!np`s4H+$z;L)f%1n4qI+9oV&KzS$ypm7X*A>Ukn`=?LB`AmVWI=qr!^X0gHMaYE65J) zz=2K7Bt|Q&j%P*Hba%sj^Lw|>mL?ELyO>E~2chg65}e*s#FiHr@R)q9ESL3Pp#ZID z9NVjuwGO_WC?tJmUgEDZ-$4s_4TM@08t`RoQHCp8gDBqTYxk+!(O3yx-c_b8#9SgqO72zj5Yhv#} zQOQ~|a+)uZX`0ikt50!CT-smfUe^-5hqxRWO_7q)NJjsfB+C`T|IGW62p>xGYYoWj^T9qG_(9C8df{QsN(#dH z?i8-@#HhteA0HoH2smU2ycIr}A$6jb0B$(*z22POhz0<4J`6a)2t6z6I-2W_OqIq( z#dv4YZlOO7y3_d+v$6da4AIh7pn|S6(=z0niDeV)!b-#0@_mam6;5LnI&r9!R+g~t zH^1Ep1`BGes!C#1q2QdXptr0?{E&Jj_y^AWtvQKx5b&5|W#!OP$j0VH_d{n8k7|Vpzo%wu7K`nCXzxd}4td%xon3&M0s_UB zw3vB5_;z}c>;@>hgkyc*=fFWuBSxn{@dTpU_(Tw1Q7FGE^tFZ%+}7f*!<;Itn~5Zc zv{yDB_*}I%^BW1NaK+^id+#=&VEEh|FgnyW33#X$dxN4kkT3s_q@RQ_dsVo8{N@%o zbEfI!lf_cFIi-$ug{aIrX{4?1A-FaeLcFB!WD0#+Ju0uKCFUPFhOcNM9yw&ZlvEq@ ztLa=765JIc^lf4s$i1Flpbx_fmb*3{!BL2ORq4D$`kwo)bpapv=0Wg(O|qRw&tb&c zZ;q6(b6&r^x_*-`H9ZkI9E_icG#7#{Mw=@qEN0h(3iOiKC!6P?`4T<{e>?+57lz!= zIggoK`z;omrJoT*hu6b^KyBjnWxDDWC((b8NTQg#-DkX2rP1epnBZ~&uE0ioSd6Gh^iIU_AvYV8E}fI#Z?yxs@%)AHuzOI4N zswV+3o-vi9`@a~u*(GI#pI6c3Pjyua4$zgUz?X{9_3mEwOEeuQkdZ+_Vg_U)l?L}y ze~=;Q`iJ24$DC&uEb9q9vz2ed{5P^0JO^PwSRj#P^; zcoTYSVNR_x-8b{MRz$|t7c$#BtaL}uPd?f2LlhLp2pKelMGN}y-{2JG0Qe{>;pNxW z^>;U^eIja3p*3!zae!n^j@V^VkcYQ#o%{>vAzaK0s0K0dK+X6@HnKMy#aZ2C)i_2} z)xN7KdcmJVs$4^h^&^ejp(!Qpa@$#h_HmugDsY35rABl;7JBY>dLfut#YOc&=6sL& zT@zy}U_Gi<-_zL3$I(4@6bEH~J+gUcBW}7ri1`}04nzEC7tLqAt^Z(Ks1(Pn`YDjC z{=1GL%YELfh#CIp)Z-m!@W#fvt$>vmOTXudTx>1)uZ_;W-YGXTn>xt85=G_r2u znHh!7j)FZCA#W_wQPueN^N4mQ&dK)AeOoAU%TlQqKSj$<x_$#aGO}Kf-HqF<*6IRLlDp#c-3th0O(=MV zL9_g8!q5E=t*tN3o9UE=z`#(0xY{;2Bf!P3Z(p!cv&F)j(sq)u9v`PyiSn)A6%)@!TaWtYl_)yu(r< z*eHR8m21I!zlC#+C+zVm4^&*(lfEUfeq5D`khE&8d*)z8s0s#{q16iNNhA%2Z(38Y z*Uo+>V*AEH^Vf}SzXYAqj6lo%Z*CUN-ta|=z(Fi7TSsDFSxJ&Q9)~JgE-D`YA~xI2 z&oA4zO{1k;r``Le>fqluLyvLhHTj99J8Al#uf)hZ{saCyJ4XK6@xw=3!yX(mWvYhv za>WUR>GWjjG-QbszC36D5%x5s-J<}`&=m?gz^qpp0K_liGNm4{F{`aPd-Cz-fo+e^ z(C&6c=e3-0EDBV6$K(A~7)+42r zoVkU*1s2B1uIekC#1u$zm=Zgkl6D)}+_o`);Da?@)-e2TKopMKynY27Lk~ptLUj?f zj|O+1^;A72BJicclRtgi(Jp6hoM=~$w}0hiA9zC**z$a)xRECpYZT;g?uoz+)*U1x zTv~dKk8J4TVeRq7J>2&7x+NJ+)j1p%J{X`6@nJurXr6QAf5l<|zy?I>+*nHvTQOeH z+M%hW8S4KwPJvQPcu!d=EPyS)kD4DaWcxy*>8Q(3{?j7naP~8t8yZX`yS{J)54}+1 zzfgk^jh&Pt+1ZWd+A-oLrlzucgfXxAH}~#|`tz_L?(2kQhq8R=iSfVk7X*DCanNe* zaQKT)pZaMIjl2DS-SGY~gS|+fzCs{xM|){R1ix343%U-cH*vQc?zkN_clD*WB#O{d z7-9o6+c82()CQzcfy+S`q0dB3E+y7F5j&uPfZtsFYrLd51`sELP(vI9<>0}K0&x!S zr~mXmTy47J%-s>e>kb~M?pA!>9jL66i0ecriq`)73`qR`MfCsDTYk5JN(!#D*dj@m zK1HT}P9z^ed3ZG*!U8CELj1UrQU(XRlH1)M>DS+J#yR`&+WHAN5}PA)JJ;(9!-0=@ zbBQas?sQDn)9d%x+KRZx96mn@itPS#iTHodex9G1p|)`wpy9;4Oo_qaNheKW6*>YLi(phF%wZLj;WUvrKphGgP%pL^rv3)#uCa3!V=TFo1X(Nv9WM ze*~7pZo?<45qLUWVbKt8Vo~{TSK$a{%$Z54Q<`mrGD*p>sOK;7FbD*{`0w`{1U-Wl zq8ZQIjm5z-Dfx)=LYSW6zswQ-#r$i`g$?8CkWpmGFIouWD>Kg}WzYpqG`%px`|0_BlMg%$ zd`FkgPk+`HNg==Jo)posH^T#L4(t1`>f*}O6{9qTB4XbTR5r6ie*dD8SV^?*MI|%S zH*s$tKUXe#>9(kpFhw?}cHb57>x^vrJZ~m$xy?P>2^rF>JMC6{{hPm!EWPiYu*0@` zyty@MUV7S%Q8P&@0)&}zuGMkX$NB@e^Wg0HD*-~*rs4)KSHm)5Vl^+_#`^}^?f@)k zB4M#J%?a`nEqG$XX)B(6Y*wxNiqI?^W_$qx`$g3GMs6 zJHwvXFU^H>DPSbd^FHD^Ne{T~K}C&Ud^!++zLb7mf}ys{dOAu^y2wnrSiLt;;c|4H zeaCEJ?sXyi=%H>cdsp|k*IIgsCY$%_`pq`?*x>L!35>&a4+%1=KhEz}%75v5nZw_4 z7iV&vG9JQ6)I$AHmj?x+(f`|bMWWV{AfX_OE@=Ii4$&W^qB;-eeLh(B^-Bm{^`-yP z)fWNTG!NI_m4+arbk)Ygjk6})t(MM!Jtl|={oKLxe%nTFaUSmp`LpicIb*-7JKG61 zI-p%=i+r$uuxtx)ZC-JHg*H@yNGCsQ&mNa%$SEjE*c=#P-{i<5`T<`S1(K zTb^AoNVa|UO=dSMu`SnT!`uw<{f>2=7t66Xo)AbF;lDZ1owj#aU2dAdbK1cnPvL2? zI~p1*o^v^Yhx|6ltb(jgM$umNkd^R1$enf{P(5rqx38ZU9(34lTr9&uyyeA=U;L)0 zJ~%3PH1w$42#)4nqfFr{Xg)W4r3Qh#|NQT#Q2x`II|DQ*Zymzboeh7M5`%VQX0!gc zw?-Dr0|*fO5(ivJN@k_O(p5>Dq!V8f2|u6TM8FZ&dT3%|f(Kp7dzup(I7pZkxLX-Z z^@QUCNzXc}+sEO<(Se~6#e#18oh}sIo*mXV{aekd2w<^;zjvBje4}=^t_$W7%)?wp zV%2h5oxnAvMLK%gJND#vj6B9xd@xVBg3T826bax#y|MCVDyagi43_51pI=up?YdD( z#oB0T&Q6ntS))L|phBhlc_|^E7$?+0oQ8;E|C5LI(f tv^H{p>DhgQz+f;IQ!8_O13jB>EY^0$Kacsq$r%EX5S0-r75e)9{{V Date: Fri, 14 Aug 2026 15:01:21 +0200 Subject: [PATCH 13/50] docs: add changelog entries for the merge command --- CHANGELOG.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e6538773..9116aef9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +## Unreleased + +- added `pangraph merge` command, to combine two existing pangenome graphs into a single one. +- `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. +- `pangraph reconstruct --verify` now matches sequences by genome name rather than by position. + ## 1.3.0 - added `--guide-tree` option to `pangraph build` command to allow users to provide a custom guide tree in Newick format, see #180. From dc44757c2bc0ced0afccf5f1a079c23652d4ea80 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 15:01:21 +0200 Subject: [PATCH 14/50] docs(design): mark merge roadmap as complete --- dev/design/merge.md | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/dev/design/merge.md b/dev/design/merge.md index 465e865f..cb2312a9 100644 --- a/dev/design/merge.md +++ b/dev/design/merge.md @@ -1,14 +1,12 @@ # Graph merging — design manifesto -Status: **`pangraph merge` implemented and verified; only user docs remain (see §6)** +Status: **complete — `pangraph merge` implemented, verified and documented (see §6)** Integration branch: `feat/merge` (merged into `master` last, after all phase branches below) This document describes the design for a new `pangraph merge` command, which combines two pre-existing pangenome graphs into one. It covers the motivation, the identifier model, the -required changes to existing code, and the phased roadmap. - -An accompanying `dev/design/merge-implementation-notes.md` will record as-built decisions and -gotchas as the phases land; this file describes the *intent*. +required changes to existing code, and the phased roadmap. As-built decisions and gotchas are +recorded inline in the sections they belong to, rather than in a separate notes file. --- @@ -397,7 +395,7 @@ merged into `master` last, once all phases have landed. | 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | | 5 | `feat/merge-cmd` | §4.3 — duplicate genome names are an error in `build` and `merge` | landed | | 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | landed | -| 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | todo | +| 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | landed | Phases 2–5 were implemented together, since a `merge` command without §3.4 panics on the first identifier collision and would not be testable. @@ -455,10 +453,20 @@ hand. ## 9. Documentation -- This document, plus `dev/design/merge-implementation-notes.md` for as-built decisions. -- A user-facing tutorial section on extending an existing graph, stating the cost model (§5), the - base-graph-first convention (§3.4), and that `merge(build(A), build(B)) != build(A ∪ B)` (§1). +All of the following have landed: + +- This document, which also records the as-built decisions inline. +- A user-facing tutorial page on extending an existing graph: + `docs/docs/tutorial/t04b-merging-two-graphs.md`, at `sidebar_position: 5`. It states the cost + model (§5), the base-graph-first convention (§3.4), the duplicate-name and `--circular` + constraints (§4.3), and that `merge(build(A), build(B)) != build(A ∪ B)` (§1). Its scheme is + `docs/docs/assets/svgs/merge.svg`, exported to `docs/docs/assets/t5_merge.png` at 300 dpi. + The two pages that follow it had their `sidebar_position` bumped to 6 and 7; their filenames + were deliberately *not* renamed, since the docs site configures no redirects and renaming would + break the published URLs. - `pangraph reconstruct` docs: record order and `index` are only meaningful for graphs produced - directly by `build` (§4.4). -- Regenerate `docs/docs/reference.md` via `docs/generate-reference-docs` after the CLI changes. -- CHANGELOG entry. + directly by `build` (§4.4). This lives in the `reconstruct_args.rs` doc comments, and hence in + the generated reference; no tutorial page documents `reconstruct`. +- `docs/docs/reference.md`, regenerated via `docs/generate-reference-docs`. +- CHANGELOG entry, under `## Unreleased` (the version bump to 1.4.0 happens in the release commit, + per repo convention). From 5b40fdd29705a358d7a968627f15d0525d09ed13 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 15:04:22 +0200 Subject: [PATCH 15/50] docs: add PR references to changelog --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9116aef9..14078002 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,8 @@ ## Unreleased -- added `pangraph merge` command, to combine two existing pangenome graphs into a single one. +- added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #192. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. -- `pangraph reconstruct --verify` now matches sequences by genome name rather than by position. +- `pangraph reconstruct --verify` now matches sequences by genome name rather than by position, see #193. ## 1.3.0 From 1f9e8de0fac5f9c1438956803e74976081d8665a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 15:05:02 +0200 Subject: [PATCH 16/50] docs: remove merge design document --- dev/design/merge.md | 472 -------------------------------------------- 1 file changed, 472 deletions(-) delete mode 100644 dev/design/merge.md diff --git a/dev/design/merge.md b/dev/design/merge.md deleted file mode 100644 index cb2312a9..00000000 --- a/dev/design/merge.md +++ /dev/null @@ -1,472 +0,0 @@ -# Graph merging — design manifesto - -Status: **complete — `pangraph merge` implemented, verified and documented (see §6)** -Integration branch: `feat/merge` (merged into `master` last, after all phase branches below) - -This document describes the design for a new `pangraph merge` command, which combines two -pre-existing pangenome graphs into one. It covers the motivation, the identifier model, the -required changes to existing code, and the phased roadmap. As-built decisions and gotchas are -recorded inline in the sections they belong to, rather than in a separate notes file. - ---- - -## 1. Motivation - -Today `pangraph build` can only construct a graph from scratch, out of a set of FASTA sequences. -There is no way to add new genomes to an existing graph: the only option is to rebuild everything -from the full sequence set. - -The proposed workflow for extending a graph is a two-step composition of existing pieces: - -```bash -# 1. turn the additional genomes into their own (possibly singleton) graph -pangraph build new_genomes.fa -o new.json - -# 2. merge it into the existing graph -pangraph merge base.json new.json -o extended.json -``` - -Step 2 is exactly the operation that already happens at every internal node of the guide tree -during `build`. The command is a thin CLI wrapper around it. - -### What this buys us - -- Adding genomes to a large graph without redoing the whole build. -- Merging graphs built from different sources or with different parameters. -- A directly testable, user-visible entry point into the merge machinery, which is currently only - reachable through the full build pipeline (or the undocumented `bin/merge_two_graphs.rs`). - -### What it does not buy us - -`merge(build(A), build(B))` is **not** expected to be identical to `build(A ∪ B)`. The guide tree -and therefore the order in which homologous regions are discovered differ, so the resulting block -decompositions will differ in detail. Both are valid pangenome graphs over the same sequences. - ---- - -## 2. Current state - -### 2.1 What already exists - -The core operation is complete and generic. `merge_graphs(left, right, args)` -(`packages/pangraph/src/pangraph/graph_merging.rs:26`) joins two graphs, iteratively self-merges -homologous blocks until convergence, and removes transitive edges. It makes **no assumption** that -its inputs are singletons or that they come from the same build. `bin/merge_two_graphs.rs` is a -development binary that already tries to call it directly. - -Distance computation also already works on arbitrary graphs: `mash_distance` takes `&[Pangraph]` -and sketches all block consensuses (`distance/mash/mash_distance.rs:9`). - -So this is primarily a plumbing and identifier problem, not an algorithmic one. - -### 2.2 What is missing - -1. **Identifier collisions** between independently built graphs (§3). This is the hard blocker. -2. `build` **panics on a single input sequence**, so the "wrap my new genome in a singleton graph" - half of the workflow does not work (§4.1). -3. The merge parameters are entangled with `PangraphBuildArgs`, which carries build-only fields - (§4.2). -4. **Path names are not guaranteed unique**, although the rest of the codebase treats them as the - de-facto identity of a genome (§4.3). -5. Sequence verification is keyed on input order / FASTA index, which is meaningless for a merged - graph (§4.4). -6. There is no `merge` subcommand (§4.5). - ---- - -## 3. The identifier model - -### 3.1 The problem - -`Pangraph::singleton` (`pangraph/pangraph.rs:29`) assigns - -```rust -let node_id = NodeId(fasta.index); -let block_id = BlockId(fasta.index); -let path_id = PathId(fasta.index); -``` - -i.e. small sequential integers taken from the position of the record in the input FASTA. Every -later identifier is content-derived, **but only for entities that are actually touched by a -merge**. Blocks that never find a homologous partner keep their original small integer id all the -way to the final graph. - -Measured on two graphs built independently from disjoint subsets of `data/ges-1.fa` (6 + 6 -sequences): - -``` -paths A: 6 B: 6 overlap: 6 → {0,1,2,3,4,5} -blocks A: 43 B: 33 overlap: 0 -nodes A: 94 B: 85 overlap: 0 -``` - -and on a graph built from two unrelated sequences (influenza + mpox, no homology found at all): - -``` -blocks: {0, 1} nodes: {0, 1} paths: {0, 1} -``` - -So: **path ids always collide**, and **block and node ids collide whenever a block never merged** — -which covers every singleton graph, every accessory block, and every divergent genome. -`graph_join` (`graph_merging.rs:74`) panics on any duplicate key, so `merge` fails immediately -today. - -### 3.2 Two properties of the current model - -**Identifiers are labels, not content hashes.** `circularize::merge_blocks` deliberately *reuses* -an existing `BlockId` for a newly concatenated block (`circularize/merge_blocks.rs:110-142`). -Nothing in the codebase relies on a block id being a hash of its content, so we are free to -relabel entities as long as all references are rewritten consistently. - -**Identifiers are re-hashed during the merge**, and those hashes consume the ids we relabel: - -| Site | New id derived from | -|---|---| -| `pangraph_node.rs:45` | `(block_id, path_id, strand, position)` | -| `reweave.rs:132` | `(qry.name, qry.interval, reff.name, reff.interval)` | -| `pangraph_interval.rs:129` | `(block_id, interval)` | -| `detach_unaligned.rs:99` | `(node_id, seq)` | -| `pangraph_path.rs:44` | `(nodes, tot_len, circular, desc, name)` | - -This is safe **provided the inputs are made disjoint first**. It also means a half-measure — for -instance deduplicating keys only inside `graph_join` — would not work: colliding inputs would -produce colliding *derived* ids later in the same merge. - -### 3.3 `PathId` is semantically load-bearing - -Unlike the other two, `PathId` currently carries meaning beyond identity: `reconstruct` sorts paths -by id and uses `path_id.0` as the output `FastaRecord.index` -(`commands/reconstruct/reconstruct_run.rs:56-76`), which `build --verify` then uses to index back -into the input FASTA records (`commands/build/build_run.rs:44`). For a graph produced by `build`, -`PathId` == input FASTA index, so reconstruction order == input order. - -That correspondence cannot survive a merge (§4.4), but path ids should still be small, contiguous -and ordered rather than becoming arbitrary hashes. - -### 3.4 Decision: relabel everything after the first graph - -Before joining, the right-hand graph is relabeled so that its identifier space is disjoint from -the left-hand one. The **left graph is never modified**. - -```rust -/// Re-derives every block, node and path id of the graph. Block and node ids become -/// `id((salt, old_id))`; path ids are renumbered contiguously from `path_id_offset`, -/// preserving their relative order. Consensuses, edits, names, descriptions, strands -/// and positions are unchanged. -pub fn relabel(self, salt: usize, path_id_offset: usize) -> Result - -/// Relabels this graph so that it shares no identifier with `other`, which is untouched. -pub fn make_disjoint_from(self, other: &Pangraph) -> Result -``` - -Rationale for relabeling *all* of the right graph's block and node ids, rather than only the ones -that actually clash: it is uniform, trivial to reason about, and trivial to test. The alternative -(remap-on-collision) would preserve more identifiers across an append, but block ids of an appended -graph are of limited interest to users compared with those of the base graph — and those are -preserved either way. - -**Consequences, to be documented in the user docs:** - -- Put the **base graph first**. Its block, node and path ids survive the merge unchanged (except - where blocks are genuinely reweaved); the second graph's ids always change. -- `merge` is therefore *not* symmetric in identifiers, even though it is essentially symmetric in - content. - -### 3.5 Implementation notes - -- **Move, don't clone.** Block consensuses are the bulk of a graph's memory; relabeling a cloned - copy would double peak usage on a large base graph. `relabel` therefore takes the graph by value - and destructures it (`let Self { paths, blocks, nodes } = self;`), so each map can be drained - into its replacement and the sequences are moved rather than copied. -- `PangraphNode::new(Some(id), …)` and `PangraphPath::new(Some(id), …)` already accept an explicit - id. Blocks own their consensus behind private fields, so `PangraphBlock::relabel(self, id, - node_map)` does the equivalent from inside the type, moving the consensus and the edits across. -- References to rewrite: the three map keys, `PangraphBlock::alignments` keys, - `PangraphNode::{block_id, path_id}`, and `PangraphPath::nodes`. -- **Assert injectivity.** After relabeling, the three maps must have the same lengths as before (a - hash collision inside one graph would silently drop entities) and the resulting id sets must be - disjoint from the left graph's. Both are hard errors: at ~2^-64 per pair neither happens in - practice, but the alternative is one graph silently overwriting a block of the other during the - join. There is deliberately no retry-with-another-salt path — it would be unreachable code - guarding against an event that cannot realistically occur. -- Path ids are assigned as `path_id_offset + rank`, where `rank` is the position in the graph's - existing (sorted) path key order, and `path_id_offset` is `max(left path ids) + 1`. This keeps - ids contiguous and keeps the right graph's genomes in their original relative order, after all of - the left graph's. It also means **path ids cannot collide at all** — every one of them is - strictly greater than every path id of the left graph — so only block and node ids are at risk. - Since the assignment is by rank rather than derived from the previous id, relabeling the same - graph twice does not shift its path ids any further. -- Relabeling is deterministic: `utils::id::id` uses `XxHash64` with a fixed seed. - ---- - -## 4. Required changes to existing code - -### 4.1 `build` must accept a single sequence - -``` -$ pangraph build one.fa -o one.json -The application panicked: index out of bounds: the len is 1 but the index is 1 -Location: packages/pangraph/src/tree/neighbor_joining.rs:28 -``` - -`build_tree_using_neighbor_joining` (`tree/neighbor_joining.rs:16-33`) skips the -`while nodes.len() > 2` loop for a single graph and then indexes `nodes[1]`. An empty input panics -on `nodes[0]`. - -Fix: return the lone leaf clade when `nodes.len() == 1`, and a clean error for an empty input. The -rest of the pipeline already tolerates a leaf-only tree — `postorder` no-ops on leaves, the root's -`data.take()` returns the graph, and `ProgressBar::new` already guards `n_total <= 1`. Worth a -companion test for `build_tree_from_newick` on a single-leaf Newick string, which shares the shape. - -This is a prerequisite for the whole workflow: appending a single genome requires building a -one-sequence graph. - -### 4.2 Extract `GraphMergeParams` from `PangraphBuildArgs` - -`merge_graphs` takes `&PangraphBuildArgs`, and that type is threaded all the way down into -`MergePromise::solve_promise` (`reweave.rs:40`), `reconsensus_graph` (`reconsensus/reconsensus.rs:35`), -`edit_consensus_and_realign` (`pangraph_block.rs:295`) and `map_variations` -(`align/map_variations.rs:43`). Only five fields are ever used; the rest (`input_fastas`, -`output_json`, `circular`, `verify`, `guide_tree`, `no_progress_bar`) are build-only and meaningless -for `merge`. - -```rust -pub struct GraphMergeParams { - pub aln_args: AlignmentArgs, // #[clap(flatten)] - pub alignment_kernel: AlignmentBackend, - pub max_self_map: usize, // default 100 - pub extra_band_width: usize, // default 5 - pub max_alignment_attempts: usize, // default 4 -} -``` - -`PangraphBuildArgs` and `PangraphMergeArgs` both `#[clap(flatten)]` it, so the two commands expose -an identical set of alignment options. - -Place it next to `AlignmentArgs` in `align/`, and move `AlignmentBackend` there as well. Side -benefit: this removes the current `align → commands::build` dependency -(`map_variations.rs:3`, `reconsensus.rs:1`, `pangraph_block.rs:2`), which is backwards. - -Mechanical follow-ups: the clap `default_value_t = PangraphBuildArgs::default().alignment_kernel` -becomes `GraphMergeParams::default()…`, and tests constructing `PangraphBuildArgs::default()` purely -to reach alignment parameters switch to `GraphMergeParams::default()`. Also generalize -`build_cmd_preliminary_checks` (the "is `mmseqs` on PATH" check) to take `&GraphMergeParams`, so -`merge` inherits it. - -### 4.3 Path names must be unique - -Path names are already treated as the identity of a genome throughout the codebase: -`Pangraph::path_id_by_name` (`pangraph.rs:258`), `simplify` (which unwraps `path.name()` outright, -`simplify_run.rs:26`), and the export commands. Nothing enforces uniqueness, however — -`build_run.rs:72` still carries a `// TODO: check for duplicate fasta names`. The only place that -errors today is `build_tree_from_newick` (`tree/newick.rs:88`), i.e. only when `--guide-tree` is -used. - -Decision: **duplicate path names are a hard error, in both `build` and `merge`.** - -- `build`: error if two input FASTA records share a name. The check lives in `build()` rather than - `build_run()`, so that it applies to every caller of the library entry point and not only to the - CLI. This makes the existing `--guide-tree` behaviour unconditional and fills the TODO. -- `merge`: error if a name occurs more than once within either input graph or across the two. - This is what catches the common mistakes of merging a graph with itself, or re-adding a genome - that is already present in the base graph — which would otherwise silently produce two - identically named paths and break `export` and `simplify` downstream. - -Error messages list all offending names, not just the first one. - -The two commands share the primitive, `utils::collections::find_duplicates`, but not the check -itself: the surrounding messages differ (input FASTA records versus genomes of two graphs, with the -merge one hinting at the self-merge mistake), and each is two lines around the shared call. Sharing -a single `check_unique_names(names, context)` would only trade those two lines for a context -parameter threaded through to the message. - -Paths with no name at all are left alone by the duplicate check (multiple unnamed paths are not an -error), but `--verify` requires every path to be named, because names are the verification key -(§4.4). - -### 4.4 Reconstruction and verification must be name-keyed - -`reconstruct` emits one FASTA record per path, sorted by `PathId`, with `index = path_id.0` -(`reconstruct_run.rs:56-76`). For a graph produced by `build`, `PathId` is the input FASTA index, so -the output order reproduces the input order exactly and `index` is meaningful. - -**Neither property can hold for a merged graph.** The right graph's paths are renumbered (§3.4), so -while the output stays deterministic (still sorted by path id), its order corresponds to no -original input FASTA ordering, and `index` is merely a position within the merged graph. -Reconstruction itself remains fully supported — every sequence comes back byte-identical — but -*order is not guaranteed*, and neither order nor `index` may be used to pair records up. - -This has three consequences: - -1. **`compare_sequences` must stop comparing whole records.** It tested `left != right` on - `FastaRecord`, which derives `PartialEq` over all fields *including `index`* (`io/fasta.rs:17-24`) - — despite an error message that only mentioned length. It is now deleted; `verify_genome` compares - sequence contents and reports the genome name and the first differing position. - - This was not only cosmetic. Two bugs followed from index/position pairing: - - `build --verify` indexed `&fastas[actual.index]`, which **panicked out of bounds** whenever the - input records' indices were not exactly `0..n-1` — as they are not for any programmatically - assembled record set, including the repo's own test helper. - - `reconstruct --verify` paired records positionally, so surplus records in the verification file - were silently ignored (exit 0), too few gave a misleading `expected length 0 but got N`, and a - merged graph failed spuriously with `expected length N but got N` at identical lengths. - -2. **Verification is keyed by path name, not by index or position.** - - ```rust - pub fn verify_graph_sequences(graph: &Pangraph, expected: &BTreeMap, - coverage: GenomeCoverage) -> Result<(), Report> - ``` - - The third parameter was not in the original sketch: `build`'s intermediate clade graphs hold only - the genomes of their own clade, so they need `GenomeCoverage::Partial`, while the final graph of a - build, a merged graph and `reconstruct --verify` all require `Complete`. Making the *final* build - check `Complete` is a small gain — nothing previously noticed a genome going missing. - - `build` fills the map from the input FASTA records; `merge` fills it by reconstructing each - input graph *before* merging. Both are sound because names are unique (§4.3). `reconstruct - --verify` does **not** build the map: it is the only one of the three that does not otherwise - need every genome resident, so it streams the verification file against a name → path id index - and reconstructs one genome at a time, sharing `verify_genome` and its error formatting. - - Reconstruction itself moved out of the command module into `pangraph/reconstruct.rs`, since it is - a graph operation — `pangraph/pangraph.rs` was reaching up into `commands::`. - -3. **`pangraph reconstruct` documentation must state** that record order matches the original input - FASTA order only for graphs produced directly by `build`, and that consumers should match records - by name. - -### 4.5 The `merge` command - -New module `commands/merge/{mod.rs, merge_args.rs, merge_run.rs}`, dispatched from -`commands/root_args.rs:61` and `commands/main.rs:19`. - -``` -pangraph merge -o -``` - -`PangraphMergeArgs`: two positional graph paths, `-o/--output-json` (default `-`), -`-f/--verify`, and a flattened `GraphMergeParams`. - -`merge_run` flow: - -1. Load both graphs (`Pangraph::from_path`); `sanity_check` each under `debug_assertions`. -2. Preliminary checks: both graphs non-empty; the `mmseqs`-on-PATH check when that kernel is - selected; duplicate path names within and across the inputs (§4.3); a **warning** if the two - graphs disagree on circularity (`circular` is per-path, so mixing is structurally fine, but - `build -c` is global and easy to get wrong for the new-genomes graph). -3. `make_disjoint(&left, &mut right)` (§3.4). -4. If `--verify`: reconstruct the expected sequences from both inputs into a - `BTreeMap` keyed by path name (§4.4). -5. `merge_graphs(&left, &right, &args.merge_params)`. -6. If `--verify`: `verify_graph_sequences(&merged, &expected)`. -7. Write the output JSON. - -`bin/merge_two_graphs.rs` is superseded by this command — it cannot work today anyway, since it -calls `graph_join` on colliding ids — and should be removed in the same phase. - -Exactly two input graphs are supported. See §7 for why N-way is deferred. - ---- - -## 5. Cost model - -`self_merge` runs an all-vs-all alignment over *every* block consensus in the joined graph on each -round. Appending 5 genomes to a 500-genome graph therefore costs roughly what the root merge of a -505-genome build costs: it is **not** proportional to the number of new genomes. - -This is inherent to the existing algorithm and is no worse than what `build` already does at the -root of the guide tree, but users will reasonably assume that appending is cheap. It must be stated -explicitly in the user documentation. - ---- - -## 6. Phased roadmap - -Each phase is its own branch and PR, targeting the integration branch `feat/merge`. `feat/merge` is -merged into `master` last, once all phases have landed. - -| # | Branch | Scope | Status | -|---|---|---|---| -| 1 | `feat/merge-single-seq-build` | §4.1 — NJ tree for 1 and 0 input graphs | landed | -| 2 | `feat/merge-cmd` | §4.2 — extract `GraphMergeParams`, no behaviour change | landed | -| 3 | `feat/merge-cmd` | §3.4 — `relabel` / `make_disjoint_from` | landed | -| 4 | `feat/merge-cmd` | §4.5 — the command itself, plus integration tests | landed | -| 5 | `feat/merge-cmd` | §4.3 — duplicate genome names are an error in `build` and `merge` | landed | -| 6 | `feat/merge-verify` | §4.4 — name-keyed verification shared with `build` | landed | -| 7 | `feat/merge-docs` | §9 — tutorial, `reconstruct` docs, CHANGELOG | landed | - -Phases 2–5 were implemented together, since a `merge` command without §3.4 panics on the first -identifier collision and would not be testable. - -Phase 6 turned out to fix two latent bugs rather than merely unify style, both recorded in §4.4. - ---- - -## 7. Non-goals (for now) - -- **N-way merge.** `merge` takes exactly two graphs. Merging N graphs along a guide tree is a - natural generalisation — `mash_distance` already works on arbitrary graphs, so the neighbour-joining - machinery would work unchanged — but it would require factoring the tree-traversal loop out of - `build()`, and users can chain pairwise merges in the meantime. -- **FASTA input to `merge`.** Once `build` accepts a single sequence (§4.1), the two-step workflow - composes cleanly. Accepting FASTA directly would duplicate `build`'s reading, `--circular` - handling and name checks inside `merge`. -- **Guide-tree-driven merge order.** Only relevant for N-way. -- **Incremental performance.** See §5; making appends proportional to the number of new genomes - would require a different alignment strategy and is out of scope. -- **Provenance metadata.** The graph JSON has no place to record which inputs a merged graph came - from. Not added here. - ---- - -## 8. Testing strategy - -**Unit** - -- `relabel` / `make_disjoint_from`: two hand-built graphs with identical ids `0..2` become - joinable, and `sanity_check` passes on the join. -- Relabeling is an isomorphism: sequences reconstructed from the graph, keyed by path name, are - unchanged by relabeling. -- Path ids after relabeling are contiguous, ordered, and start after the left graph's maximum. -- `build_tree_using_neighbor_joining` for 1 and 2 input graphs; clean error for 0. -- Duplicate path name detection: within one graph, across two graphs, and the unnamed-path case. - -**Integration** (`packages/pangraph/tests/itest_merge.rs`) - -- Split `data/ges-1.fa`, `build` A and B separately, `merge` them, then assert: every original - sequence is reconstructed byte-identically (matched **by name**, not by order); `sanity_check` - passes; the path count is `nA + nB`; all path names are preserved. -- A single-genome right graph (exercises phase 1). -- Merging a graph with itself must fail with a duplicate-name error, and `build` must reject - duplicate names in its input FASTA records. - -Deliberately **not** tested at the integration level: two graphs with no shared homology, as a way -to exercise the surviving small integer ids of §3.2. Whether two genomes have *no* detectable -homology is a property of the aligner and its parameters, not something a test can pin down — even -distant genomes may share a short segment. The colliding-id case is covered directly and -deterministically by the `make_disjoint_from` unit tests, which construct graphs with ids `0..2` by -hand. - ---- - -## 9. Documentation - -All of the following have landed: - -- This document, which also records the as-built decisions inline. -- A user-facing tutorial page on extending an existing graph: - `docs/docs/tutorial/t04b-merging-two-graphs.md`, at `sidebar_position: 5`. It states the cost - model (§5), the base-graph-first convention (§3.4), the duplicate-name and `--circular` - constraints (§4.3), and that `merge(build(A), build(B)) != build(A ∪ B)` (§1). Its scheme is - `docs/docs/assets/svgs/merge.svg`, exported to `docs/docs/assets/t5_merge.png` at 300 dpi. - The two pages that follow it had their `sidebar_position` bumped to 6 and 7; their filenames - were deliberately *not* renamed, since the docs site configures no redirects and renaming would - break the published URLs. -- `pangraph reconstruct` docs: record order and `index` are only meaningful for graphs produced - directly by `build` (§4.4). This lives in the `reconstruct_args.rs` doc comments, and hence in - the generated reference; no tutorial page documents `reconstruct`. -- `docs/docs/reference.md`, regenerated via `docs/generate-reference-docs`. -- CHANGELOG entry, under `## Unreleased` (the version bump to 1.4.0 happens in the release commit, - per repo convention). From 755acf711b1722abb582f6e7667b2e3ed221ad64 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 18:04:26 +0200 Subject: [PATCH 17/50] test(merge): drop redundant sanity checks from the integration tests --- packages/pangraph/tests/itest_merge.rs | 4 ---- 1 file changed, 4 deletions(-) diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 82de39a7..717a0a45 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -79,8 +79,6 @@ mod tests { merge_run(&merge_args(left.clone(), right.clone(), output.clone()))?; let merged = read_graph(&output)?; - #[cfg(debug_assertions)] - merged.sanity_check()?; // all genomes of both inputs are present, exactly once assert_eq!(merged.paths.len(), 6); @@ -131,8 +129,6 @@ mod tests { merge_run(&merge_args(left, right, output.clone()))?; let merged = read_graph(&output)?; - #[cfg(debug_assertions)] - merged.sanity_check()?; assert_eq!(merged.paths.len(), 4); Ok(()) From cb83e23b8dc2bb2e4384b650c5d5b1fcaa61e164 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 18:04:26 +0200 Subject: [PATCH 18/50] fix(merge): vary the relabeling salt so repeated merges do not collide --- packages/pangraph/src/pangraph/pangraph.rs | 58 ++++++++++++++++++---- packages/pangraph/tests/itest_merge.rs | 33 ++++++++++++ 2 files changed, 80 insertions(+), 11 deletions(-) diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 6336a629..c9556d1f 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -12,6 +12,7 @@ use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; use crate::{make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; +use log::warn; use maplit::btreemap; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; @@ -151,24 +152,35 @@ impl Pangraph { /// the way to the final graph. Merging therefore requires namespacing one of the two graphs /// first. pub fn make_disjoint_from(self, other: &Self) -> Result { - // Namespace under which the ids of this graph are re-derived. - const SALT: usize = 1; + const MAX_ATTEMPTS: usize = 8; // Path ids are assigned above every path id of `other`, so they cannot collide by construction. let path_id_offset = other.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0); - let relabeled = self.relabel(SALT, path_id_offset)?; + // The salt must differ from the one used by any earlier merge whose relabeled ids survive into + // `other`, otherwise those ids get re-derived a second time and land on themselves. The path id + // offset provides that: a merger salted with `offset(P)` produces a graph with at least one more + // genome than `P`, so a later merge with that graph on the left uses a strictly larger offset. + let mut relabeled = self; + for attempt in 0..MAX_ATTEMPTS { + relabeled = relabeled.relabel(id((path_id_offset, attempt)), path_id_offset)?; - // Block and node ids are hashes, so a collision with `other` is possible in principle. At - // ~2^-64 per pair it does not happen in practice, but the check is cheap and the alternative is - // one graph silently overwriting a block of the other during the join. - if !relabeled.is_id_disjoint_from(other) { - return make_internal_error!( - "When making graphs id-disjoint: the relabeled graph still shares block or node ids with the other graph. This requires a 64-bit hash collision and should never happen." - ); + if relabeled.is_id_disjoint_from(other) { + return Ok(relabeled); + } + + // Relabeling is a composition of injective maps, so retrying on top of the previous attempt is + // safe, and path ids are assigned by rank rather than derived from the previous id, so they do + // not drift. Reaching this point needs either a genuine hash collision, or an operation that + // breaks the monotonicity of the offset: `simplify` drops paths without renumbering, so + // `build -> merge -> merge -> simplify -> merge` can bring the offset back to a value already + // used as a salt. + warn!("Identifier collision when relabeling graph ids (attempt {attempt}); retrying"); } - Ok(relabeled) + make_internal_error!( + "When making graphs id-disjoint: no collision-free relabeling of block and node ids found after {MAX_ATTEMPTS} attempts" + ) } pub fn update(&mut self, u: &GraphUpdate) { @@ -643,4 +655,28 @@ mod tests { assert_eq!(joined.blocks.len(), 4); assert_eq!(joined.nodes.len(), 4); } + + /// Appending to a graph that already absorbed a relabeled graph. With a constant salt the ids of + /// the third graph were re-derived exactly onto those the second one left behind, so the second + /// append always failed. Every graph here carries the same small ids, which is what `build` + /// assigns to blocks and nodes that never merge. + #[rstest] + fn test_make_disjoint_from_after_a_previous_merge() { + let first = colliding_graph(["a", "b"]); + let second = colliding_graph(["c", "d"]).make_disjoint_from(&first).unwrap(); + let joined = crate::pangraph::graph_merging::graph_join(&first, &second); + + let third = colliding_graph(["e", "f"]).make_disjoint_from(&joined).unwrap(); + + assert!(third.is_id_disjoint_from(&joined)); + third.sanity_check().unwrap(); + assert_eq!(third.path_ids().collect_vec(), vec![PathId(4), PathId(5)]); + + // and the three of them can be joined without conflicts + let joined = crate::pangraph::graph_merging::graph_join(&joined, &third); + joined.sanity_check().unwrap(); + assert_eq!(joined.paths.len(), 6); + assert_eq!(joined.blocks.len(), 6); + assert_eq!(joined.nodes.len(), 6); + } } diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 717a0a45..04fbf286 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -30,6 +30,13 @@ mod tests { Ok((fastas, right)) } + /// Reads the first `n` records of a FASTA file. + fn read_records(path: &str, n: usize) -> Result, Report> { + let mut fastas = FastaReader::from_paths(&[PathBuf::from(path)])?.read_many()?; + fastas.truncate(n); + Ok(fastas) + } + /// Builds a graph out of the given records and writes it to a JSON file in `dir`. fn build_graph_file(dir: &TempDir, name: &str, fastas: Vec) -> Result { let args = PangraphBuildArgs { @@ -134,6 +141,32 @@ mod tests { Ok(()) } + /// Appending to a graph that is itself the result of an earlier merge. Identifiers relabeled by + /// the first merge survive into its output whenever a block or node finds no homologue, and a + /// constant relabeling salt then re-derived the third graph's identifiers onto exactly those + /// values, so the second append always failed. The three genomes here are mutually unrelated, so + /// nothing aligns and every identifier survives; homologous appends never hit this. + #[rstest] + fn itest_merge_appends_to_an_already_merged_graph() -> Result<(), Report> { + let dir = tempdir()?; + + let first = build_graph_file(&dir, "first.json", read_records("../../data/flu-h1.fa", 2)?)?; + let second = build_graph_file(&dir, "second.json", read_records("../../data/sc2.fa", 1)?)?; + let third = build_graph_file(&dir, "third.json", read_records("../../data/mpox.fa", 1)?)?; + + let merged_once = dir.path().join("merged-once.json"); + merge_run(&merge_args(first, second, merged_once.clone()))?; + assert_eq!(read_graph(&merged_once)?.paths.len(), 3); + + let merged_twice = dir.path().join("merged-twice.json"); + merge_run(&merge_args(merged_once, third, merged_twice.clone()))?; + + let merged = read_graph(&merged_twice)?; + assert_eq!(merged.paths.len(), 4); + + Ok(()) + } + /// Merging a graph with itself duplicates every genome name, and must be rejected. #[rstest] fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { From 2d77cce3fb247ae8d0a3c10074190e84b9b7f6f2 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 18:04:33 +0200 Subject: [PATCH 19/50] fix(cli): keep non-alignment options out of the Alignment help section --- CHANGELOG.md | 1 + docs/docs/reference.md | 14 +++++++------- packages/pangraph/src/commands/build/build_args.rs | 8 +++++--- packages/pangraph/src/commands/merge/merge_args.rs | 8 +++++--- 4 files changed, 18 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14078002..f4d07e56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #192. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. - `pangraph reconstruct --verify` now matches sequences by genome name rather than by position, see #193. +- in `pangraph build --help`, the `--circular`, `--verify`, `--no-progress-bar` and `--guide-tree` options are no longer listed under the "Alignment" section. ## 1.3.0 diff --git a/docs/docs/reference.md b/docs/docs/reference.md index ffa6bc09..e9572c6b 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -104,6 +104,12 @@ Align genomes into a multiple sequence alignment graph Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. Default value: `-` +* `-c`, `--circular` — Toggle if input genomes are circular +* `-f`, `--verify` — Sanity check: after construction verifies that the original sequences can be reconstructed exactly from the resulting pangraph. Raises an error otherwise +* `--no-progress-bar` — Toggle to disable progress bar. Notice that the progress bar is only displayed if the output is specified via the `-o` argument +* `--guide-tree ` — Path to a Newick-format guide tree to use instead of the default neighbor-joining tree. + + When provided, the tree's topology drives the bottom-up graph-merging order. Each input FASTA sequence must appear exactly once as a leaf (matched by sequence name), and every internal node must be strictly bifurcating. Branch lengths and internal labels, if present, are ignored. Accepts plain or compressed files (gz, bz2, xz, zst). * `-l`, `--len ` — Minimum block size for alignment graph (in nucleotides) Default value: `100` @@ -134,12 +140,6 @@ Align genomes into a multiple sequence alignment graph * `--max-alignment-attempts ` — For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit Default value: `4` -* `-c`, `--circular` — Toggle if input genomes are circular -* `-f`, `--verify` — Sanity check: after construction verifies that the original sequences can be reconstructed exactly from the resulting pangraph. Raises an error otherwise -* `--no-progress-bar` — Toggle to disable progress bar. Notice that the progress bar is only displayed if the output is specified via the `-o` argument -* `--guide-tree ` — Path to a Newick-format guide tree to use instead of the default neighbor-joining tree. - - When provided, the tree's topology drives the bottom-up graph-merging order. Each input FASTA sequence must appear exactly once as a leaf (matched by sequence name), and every internal node must be strictly bifurcating. Branch lengths and internal labels, if present, are ignored. Accepts plain or compressed files (gz, bz2, xz, zst). @@ -169,6 +169,7 @@ Merge two pangenome graphs into a single one Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. Default value: `-` +* `-f`, `--verify` — Sanity check: after merging verifies that every genome of the two input graphs can still be reconstructed exactly from the merged graph. Raises an error otherwise * `-l`, `--len ` — Minimum block size for alignment graph (in nucleotides) Default value: `100` @@ -199,7 +200,6 @@ Merge two pangenome graphs into a single one * `--max-alignment-attempts ` — For within-block alignment: number of times Nextclade will retry alignment with more relaxed results if alignment band boundaries are hit Default value: `4` -* `-f`, `--verify` — Sanity check: after merging verifies that every genome of the two input graphs can still be reconstructed exactly from the merged graph. Raises an error otherwise diff --git a/packages/pangraph/src/commands/build/build_args.rs b/packages/pangraph/src/commands/build/build_args.rs index aa490a51..850fb5ce 100644 --- a/packages/pangraph/src/commands/build/build_args.rs +++ b/packages/pangraph/src/commands/build/build_args.rs @@ -29,9 +29,6 @@ pub struct PangraphBuildArgs { #[clap(value_hint = ValueHint::AnyPath)] pub output_json: PathBuf, - #[clap(flatten)] - pub merge_params: GraphMergeParams, - /// Toggle if input genomes are circular #[clap(long, short = 'c')] pub circular: bool, @@ -52,4 +49,9 @@ pub struct PangraphBuildArgs { /// are ignored. Accepts plain or compressed files (gz, bz2, xz, zst). #[clap(long, value_hint = ValueHint::FilePath)] pub guide_tree: Option, + + // Declared last: it opens the "Alignment" help section, and `next_help_heading` applies to every + // argument declared after it. + #[clap(flatten)] + pub merge_params: GraphMergeParams, } diff --git a/packages/pangraph/src/commands/merge/merge_args.rs b/packages/pangraph/src/commands/merge/merge_args.rs index 5eefe25f..6ed6e35c 100644 --- a/packages/pangraph/src/commands/merge/merge_args.rs +++ b/packages/pangraph/src/commands/merge/merge_args.rs @@ -36,11 +36,13 @@ pub struct PangraphMergeArgs { #[clap(value_hint = ValueHint::AnyPath)] pub output_json: PathBuf, - #[clap(flatten)] - pub merge_params: GraphMergeParams, - /// Sanity check: after merging verifies that every genome of the two input graphs can still be /// reconstructed exactly from the merged graph. Raises an error otherwise. #[clap(long, short = 'f')] pub verify: bool, + + // Declared last: it opens the "Alignment" help section, and `next_help_heading` applies to every + // argument declared after it. + #[clap(flatten)] + pub merge_params: GraphMergeParams, } From e6bb9e9b0a50d31cb7e129298b5cc43b4302186c Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 18:04:33 +0200 Subject: [PATCH 20/50] refactor(merge): verify merged genomes one at a time --- .../pangraph/src/commands/merge/merge_run.rs | 60 +++---- packages/pangraph/src/pangraph/reconstruct.rs | 146 +++++++++++++++--- 2 files changed, 149 insertions(+), 57 deletions(-) diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 1e4bd04e..22343ec9 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -5,14 +5,13 @@ use crate::make_error; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; -use crate::pangraph::reconstruct::{GenomeCoverage, reconstruct_by_name, verify_graph_sequences}; -use crate::representation::seq::Seq; +use crate::pangraph::reconstruct::{path_ids_by_name, verify_graph_against_graphs}; use crate::utils::collections::find_duplicates; use color_eyre::owo_colors::{AnsiColors, OwoColorize}; use color_eyre::{Help, SectionExt}; use eyre::{Report, WrapErr}; use log::{info, warn}; -use std::collections::{BTreeMap, BTreeSet}; +use std::collections::BTreeSet; use std::path::Path; pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { @@ -27,13 +26,6 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { .make_disjoint_from(&left) .wrap_err("When making the identifiers of the two input graphs disjoint")?; - // Reconstruct the expected genomes from the (relabeled) inputs, before they are consumed by the - // merger. Keyed by path name: neither path ids nor record order survive a merge. - let expected = args - .verify - .then(|| expected_sequences(args, &left, &right)) - .transpose()?; - info!( "=== Graph merging start: graph sizes {} + {}", left.paths.len(), @@ -50,13 +42,18 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { merged.blocks.len() ); - if let Some(expected) = expected { + if args.verify { #[cfg(debug_assertions)] merged.sanity_check().wrap_err("When checking the merged graph")?; - verify_graph_sequences(&merged, &expected, GenomeCoverage::Complete) + // Compared against the inputs one genome at a time: reconstructing both graphs up front would + // hold their entire sequence content in memory for the duration of the check. + verify_graph_against_graphs(&merged, &[&left, &right]) .wrap_err("When verifying the sequences of the merged graph")?; - info!("Merged graph reconstructs all {} input genomes exactly", expected.len()); + info!( + "Merged graph reconstructs all {} input genomes exactly", + merged.paths.len() + ); } json_write_file(&args.output_json, &merged, JsonPretty(true))?; @@ -95,6 +92,20 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right ); } + // Verification matches genomes by name, so it needs every path of both inputs to carry one. + // Checked here rather than at verification time so that it fails before the expensive merge. + if args.verify { + for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { + path_ids_by_name(graph) + .wrap_err_with(|| format!("When resolving the genome names of graph '{}'", filepath.display())) + .with_section(|| { + "Verification matches genomes by name. Re-run without `--verify` to skip it." + .color(AnsiColors::Cyan) + .header("Suggestion:") + })?; + } + } + // Circularity is a per-path property, so mixing is structurally fine. It is however most often a // mistake, since `build --circular` applies to all genomes of a graph at once. if circularity(left) != circularity(right) { @@ -110,26 +121,3 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right fn circularity(graph: &Pangraph) -> BTreeSet { graph.paths().map(PangraphPath::circular).collect() } - -/// Reconstructs the genomes of both input graphs, keyed by genome name. -/// -/// Cross-graph name collisions are already rejected by `merge_cmd_preliminary_checks`, so the two -/// sets cannot overwrite each other here. -fn expected_sequences( - args: &PangraphMergeArgs, - left: &Pangraph, - right: &Pangraph, -) -> Result, Report> { - let mut expected = BTreeMap::new(); - for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { - let genomes = reconstruct_by_name(graph) - .wrap_err_with(|| format!("When reconstructing the genomes of graph '{}'", filepath.display())) - .with_section(|| { - "Verification matches genomes by name. Re-run without `--verify` to skip it." - .color(AnsiColors::Cyan) - .header("Suggestion:") - })?; - expected.extend(genomes); - } - Ok(expected) -} diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 02bd8059..b475cecb 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -9,7 +9,7 @@ use crate::utils::string::str_slice_safe; use crate::{make_error, make_internal_report}; use eyre::{Report, WrapErr}; use itertools::Itertools; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; /// Number of genome names listed in full in an error message before the rest are elided. const MAX_NAMES_IN_ERROR: usize = 10; @@ -105,20 +105,6 @@ pub fn reconstruct_genome(graph: &Pangraph, path_id: PathId) -> Result Result, Report> { - path_ids_by_name(graph)? - .into_iter() - .map(|(name, path_id)| { - let seq = - reconstruct_genome(graph, path_id).wrap_err_with(|| format!("When reconstructing the genome of '{name}'"))?; - Ok((name.to_owned(), seq)) - }) - .collect() -} - /// Collects FASTA records into genome sequences keyed by name. /// /// Errors on duplicate names: the name is the key that verification matches on. @@ -212,6 +198,63 @@ pub fn verify_graph_sequences( Ok(()) } +/// Checks that `graph` reconstructs exactly the genomes of `sources`, matched by name. +/// +/// Used to verify a merged graph against the graphs it was built from. Both sides of every +/// comparison are reconstructed on demand and dropped again, so this holds two genomes at a time +/// instead of the whole sequence content of `sources`. The saving is proportional to the total +/// genome length and modest in practice — peak usage during a merge is dominated by the graphs +/// themselves — but it keeps verification consistent with `reconstruct --verify`, which streams for +/// the same reason. +/// +/// Every genome of `sources` must appear in `graph`, and `graph` must contain nothing else. The +/// `sources` are expected to have disjoint genome names; a name appearing in two of them is +/// verified twice rather than reported, since the callers reject that case up front. +pub fn verify_graph_against_graphs(graph: &Pangraph, sources: &[&Pangraph]) -> Result<(), Report> { + let path_ids = path_ids_by_name(graph)?; + let mut n_expected = 0; + + for source in sources { + for (name, source_path_id) in path_ids_by_name(source)? { + let Some(path_id) = path_ids.get(name) else { + return make_error!("Graph is missing genome '{name}', which is present in the input graphs"); + }; + + let expected = reconstruct_genome(source, source_path_id) + .wrap_err_with(|| format!("When reconstructing genome '{name}' from the input graphs"))?; + let actual = + reconstruct_genome(graph, *path_id).wrap_err_with(|| format!("When reconstructing genome '{name}'"))?; + + verify_genome(name, &expected, &actual)?; + n_expected += 1; + } + } + + if path_ids.len() != n_expected { + let expected_names: BTreeSet<&str> = sources + .iter() + .map(|source| path_ids_by_name(source)) + .collect::, Report>>()? + .into_iter() + .flat_map(|ids| ids.into_keys()) + .collect(); + + let extra = path_ids + .keys() + .filter(|name| !expected_names.contains(*name)) + .copied() + .collect_vec(); + + return make_error!( + "Graph contains {} genome(s) that are not present in the input graphs: {}", + extra.len(), + format_names(&extra) + ); + } + + Ok(()) +} + /// Formats a list of genome names for an error message, eliding all but the first few. fn format_names>(names: &[S]) -> String { let shown = names.iter().take(MAX_NAMES_IN_ERROR).map(AsRef::as_ref).join(", "); @@ -280,6 +323,7 @@ mod tests { use crate::pangraph::edits::Edit; use crate::pangraph::pangraph_block::{BlockId, PangraphBlock}; use crate::pangraph::pangraph_node::PangraphNode; + use crate::pangraph::strand::Strand; use crate::pangraph::strand::Strand::{Forward, Reverse}; use crate::utils::error::report_to_string; use maplit::btreemap; @@ -304,6 +348,31 @@ mod tests { Pangraph { paths, blocks, nodes } } + /// A one-genome graph, so that a pair of them stands in for the two inputs of a merge. Every id + /// is `0`, exactly as `Pangraph::singleton` assigns them, which is also what makes two of these + /// indistinguishable by id. + fn one_genome_graph(name: &str, consensus: &str, strand: Strand) -> Pangraph { + let len = consensus.len(); + let blocks = btreemap! { + BlockId(0) => PangraphBlock::new(BlockId(0), consensus, btreemap!{ NodeId(0) => Edit::empty() }), + }; + let nodes = btreemap! { + NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), strand, (0, len)), + }; + let paths = btreemap! { + PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], len, false, Some(name.to_owned()), None), + }; + Pangraph { paths, blocks, nodes } + } + + /// The two single-genome graphs whose merger `graph()` stands for. + fn sources() -> (Pangraph, Pangraph) { + ( + one_genome_graph("a", "ACGTACGT", Forward), + one_genome_graph("b", "TTTTGGGG", Reverse), + ) + } + fn graph() -> Pangraph { two_genome_graph([Some("a"), Some("b")]) } @@ -312,16 +381,10 @@ mod tests { btreemap! { o!("a") => Seq::from_str("ACGTACGT"), o!("b") => Seq::from_str("CCCCAAAA") } } - #[rstest] - fn test_reconstruct_by_name() { - assert_eq!(reconstruct_by_name(&graph()).unwrap(), expected_genomes()); - } - #[rstest] fn test_path_ids_by_name_rejects_unnamed_path() { let graph = two_genome_graph([Some("a"), None]); assert!(report_to_string(&path_ids_by_name(&graph).unwrap_err()).contains("without a name")); - assert!(report_to_string(&reconstruct_by_name(&graph).unwrap_err()).contains("without a name")); } #[rstest] @@ -415,6 +478,47 @@ mod tests { ); } + #[rstest] + fn test_verify_graph_against_graphs_accepts_exact_match() { + let (left, right) = sources(); + verify_graph_against_graphs(&graph(), &[&left, &right]).unwrap(); + } + + #[rstest] + fn test_verify_graph_against_graphs_detects_missing_genome() { + let (left, right) = sources(); + let extra = one_genome_graph("c", "GGGGCCCC", Forward); + + let err = report_to_string(&verify_graph_against_graphs(&graph(), &[&left, &right, &extra]).unwrap_err()); + assert!(err.contains("missing genome 'c'"), "unexpected error: {err}"); + } + + #[rstest] + fn test_verify_graph_against_graphs_detects_extra_genome() { + let (left, _) = sources(); + + let err = report_to_string(&verify_graph_against_graphs(&graph(), &[&left]).unwrap_err()); + assert!( + err.contains("not present in the input graphs"), + "unexpected error: {err}" + ); + assert!(err.contains('b'), "unexpected error: {err}"); + } + + /// The name matches but the sequence does not: the mismatch is reported against the source graph + /// the genome came from, with the position of the first difference. + #[rstest] + fn test_verify_graph_against_graphs_detects_mutated_genome() { + let (_, right) = sources(); + let mutated = one_genome_graph("a", "ACGTTCGT", Forward); + + let err = report_to_string(&verify_graph_against_graphs(&graph(), &[&mutated, &right]).unwrap_err()); + assert!( + err.contains("Sequence mismatch for genome 'a'"), + "unexpected error: {err}" + ); + } + #[rstest] fn test_verify_genome_reports_first_difference() { let expected = Seq::from_str("ACGTACGT"); From bad3c2b8a7be7dab7dbb0963475f4c982e8eeb8e Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 18:19:03 +0200 Subject: [PATCH 21/50] docs: simplify changelog --- CHANGELOG.md | 1 - 1 file changed, 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f4d07e56..14078002 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,6 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #192. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. - `pangraph reconstruct --verify` now matches sequences by genome name rather than by position, see #193. -- in `pangraph build --help`, the `--circular`, `--verify`, `--no-progress-bar` and `--guide-tree` options are no longer listed under the "Alignment" section. ## 1.3.0 From 4d51586256287b0af9a2da18b182417e0634c505 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 19:31:47 +0200 Subject: [PATCH 22/50] fix(pangraph): reject unnamed and duplicate genome names in merge and simplify --- .../pangraph/src/commands/build/build_run.rs | 28 +--- .../pangraph/src/commands/merge/merge_run.rs | 39 ++---- .../src/commands/simplify/simplify_run.rs | 25 +++- packages/pangraph/src/pangraph/reconstruct.rs | 132 ++++++++++++------ packages/pangraph/tests/itest_merge.rs | 29 ++++ 5 files changed, 162 insertions(+), 91 deletions(-) diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index 0f906099..d22d786a 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -4,15 +4,16 @@ use crate::io::fasta::{FastaReader, FastaRecord}; use crate::io::json::{JsonPretty, json_write_file}; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::reconstruct::{GenomeCoverage, sequences_by_name, verify_graph_sequences}; +use crate::pangraph::reconstruct::{ + GenomeCoverage, check_unique_sequence_names, sequences_by_name, verify_graph_sequences, +}; use crate::pangraph::strand::Strand::Forward; use crate::representation::seq::Seq; use crate::tree::clade::postorder; use crate::tree::neighbor_joining::build_tree_using_neighbor_joining; use crate::tree::newick::build_tree_from_newick; -use crate::utils::collections::find_duplicates; use crate::utils::progress_bar::ProgressBar; -use crate::{make_error, make_internal_error, make_internal_report}; +use crate::{make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::info; @@ -55,28 +56,13 @@ pub fn build_run(args: &PangraphBuildArgs) -> Result<(), Report> { Ok(()) } -/// Sequence names identify genomes throughout pangraph: they are what `export` and `simplify` -/// resolve genomes by, and the only handle that survives a graph merger. They must be unique. -pub fn check_unique_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { - let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); - if !duplicates.is_empty() { - return make_error!( - "Duplicate sequence names found in the input: [{}]. Sequence names must be unique, because they identify genomes in the resulting pangraph.", - duplicates.join(", ") - ); - } - Ok(()) -} - pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) -> Result { check_unique_sequence_names(&fastas).wrap_err("When checking the names of the input sequences")?; // If verification is requested, keep the input sequences, keyed by genome name, to compare them - // with the sequences reconstructed from the graph. - let expected = verify - .then(|| sequences_by_name(&fastas)) - .transpose() - .wrap_err("When collecting the input sequences for verification")?; + // with the sequences reconstructed from the graph. Names were just checked to be unique, so no + // record is lost to a key collision here. + let expected = verify.then(|| sequences_by_name(&fastas)); // Build singleton graphs from input sequences // TODO: initial graphs can potentially be constructed when initializing tree clades. This could avoid a lot of boilerplate code. diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 22343ec9..b14f9394 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -5,8 +5,7 @@ use crate::make_error; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; -use crate::pangraph::reconstruct::{path_ids_by_name, verify_graph_against_graphs}; -use crate::utils::collections::find_duplicates; +use crate::pangraph::reconstruct::{check_unique_genome_names, verify_graph_against_graphs}; use color_eyre::owo_colors::{AnsiColors, OwoColorize}; use color_eyre::{Help, SectionExt}; use eyre::{Report, WrapErr}; @@ -83,28 +82,20 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right } // Genome names are the identity of a genome throughout pangraph, and the only handle that - // survives a merge unchanged. `build` enforces the same invariant on its input FASTA records. - let duplicates = find_duplicates([left, right].into_iter().flat_map(|graph| graph.path_names().flatten())); - if !duplicates.is_empty() { - return make_error!( - "Duplicate genome names found in the input graphs: [{}]. Genome names must be unique across the two graphs: merging a graph with itself, or re-adding a genome that is already present, is not supported.", - duplicates.join(", ") - ); - } - - // Verification matches genomes by name, so it needs every path of both inputs to carry one. - // Checked here rather than at verification time so that it fails before the expensive merge. - if args.verify { - for (graph, filepath) in [(left, &args.left_graph), (right, &args.right_graph)] { - path_ids_by_name(graph) - .wrap_err_with(|| format!("When resolving the genome names of graph '{}'", filepath.display())) - .with_section(|| { - "Verification matches genomes by name. Re-run without `--verify` to skip it." - .color(AnsiColors::Cyan) - .header("Suggestion:") - })?; - } - } + // survives a merge unchanged. Checked unconditionally rather than only under `--verify`: a merge + // whose genomes cannot be told apart afterwards is not useful either way. Checked here rather + // than at verification time so that it fails before the expensive merge. `build` enforces the + // same invariant on its input FASTA records. + check_unique_genome_names(&[left, right]) + .wrap_err("When checking the genome names of the input graphs") + .with_section(|| { + format!("{}\n{}", args.left_graph.display(), args.right_graph.display()).header("Input graphs:") + }) + .with_section(|| { + "Genome names must be unique across the two graphs: merging a graph with itself, or re-adding a genome that is already present, is not supported." + .color(AnsiColors::Cyan) + .header("Suggestion:") + })?; // Circularity is a per-path property, so mixing is structurally fine. It is however most often a // mistake, since `build --circular` applies to all genomes of a graph at once. diff --git a/packages/pangraph/src/commands/simplify/simplify_run.rs b/packages/pangraph/src/commands/simplify/simplify_run.rs index 58de97b6..114464ab 100644 --- a/packages/pangraph/src/commands/simplify/simplify_run.rs +++ b/packages/pangraph/src/commands/simplify/simplify_run.rs @@ -4,27 +4,31 @@ use crate::io::file::create_file_or_stdout; use crate::io::json::{JsonPretty, json_write}; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PathId; -use eyre::Report; +use crate::pangraph::reconstruct::check_unique_genome_names; +use eyre::{Report, WrapErr}; use std::collections::BTreeSet; -#[allow(unused_must_use)] pub fn simplify_run(args: PangraphSimplifyArgs) -> Result<(), Report> { let PangraphSimplifyArgs { input, output, strains } = args; let mut graph = Pangraph::from_path(&input)?; let strain_set = strains.into_iter().collect(); - simplify(&mut graph, &strain_set); + simplify(&mut graph, &strain_set)?; let f = create_file_or_stdout(&output)?; json_write(f, &graph, JsonPretty(true)) } fn simplify(graph: &mut Pangraph, focal_paths: &BTreeSet) -> Result<(), Report> { + // Genomes are selected by name here, so an unnamed path cannot be resolved either way: without + // this check the filter below would silently drop it rather than report it. + check_unique_genome_names(&[graph]).wrap_err("When checking the genome names of the input graph")?; + let path_ids_to_remove: Vec = graph .paths .iter() - .filter(|(_, path)| !focal_paths.contains(path.name().as_ref().unwrap())) + .filter(|(_, path)| !path.name().as_deref().is_some_and(|name| focal_paths.contains(name))) .map(|(id, _)| *id) .collect(); @@ -46,6 +50,7 @@ mod tests { use crate::pangraph::pangraph_node::{NodeId, PangraphNode}; use crate::pangraph::pangraph_path::PangraphPath; use crate::pangraph::strand::Strand::{Forward, Reverse}; + use crate::utils::error::report_to_string; use maplit::{btreemap, btreeset}; use pretty_assertions::assert_eq; @@ -212,4 +217,16 @@ mod tests { assert_eq!(graph.paths, expected_graph.paths); assert_eq!(graph.blocks, expected_graph.blocks); } + + /// An unnamed path cannot be matched against the requested strain names. This used to panic on an + /// `unwrap`; the graph must be rejected instead, and in particular the path must not be quietly + /// dropped as "not among the focal strains". + #[test] + fn test_simplify_rejects_unnamed_path() { + let mut graph = graph(); + graph.paths.get_mut(&PathId(3)).unwrap().name = None; + + let err = report_to_string(&simplify(&mut graph, &btreeset! {o!("pathA"), o!("pathB")}).unwrap_err()); + assert!(err.contains("without a name"), "unexpected error: {err}"); + } } diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index b475cecb..a2f47dbc 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -57,36 +57,53 @@ pub fn reconstruct(graph: &Pangraph) -> impl Iterator Result, Report> { - let unnamed = graph - .paths +/// The genome name is the only identifier that survives a merge unchanged: it is the key that +/// verification matches on, and what `simplify` resolves genomes by. A graph whose genomes cannot +/// be told apart by name is therefore rejected rather than processed. +/// +/// This is the single implementation of that invariant for graphs; [`check_unique_sequence_names`] +/// is its counterpart for FASTA records. Note that it inspects the paths directly rather than going +/// through [`reconstruct`], which masks unnamed paths behind a placeholder name. +/// +/// Path ids are only unique within one graph, so when several graphs are passed the reported ids +/// are ambiguous on their own. Callers that pass more than one graph are expected to name the +/// graphs in an error section. +pub fn check_unique_genome_names(graphs: &[&Pangraph]) -> Result<(), Report> { + let unnamed = graphs .iter() + .flat_map(|graph| graph.paths.iter()) .filter(|(_, path)| path.name.is_none()) .map(|(path_id, _)| path_id.to_string()) .collect_vec(); if !unnamed.is_empty() { return make_error!( - "Graph contains {} genome(s) without a name (path ids: {}). Genomes are identified by name, so every path must be named.", + "Found {} genome(s) without a name (path ids: {}). Genomes are identified by name, so every path must be named.", unnamed.len(), format_names(&unnamed) ); } - let duplicates = find_duplicates(graph.path_names().flatten()); + let duplicates = find_duplicates(graphs.iter().flat_map(|graph| graph.path_names().flatten())); if !duplicates.is_empty() { return make_error!( - "Graph contains duplicate genome names: {}. Genome names must be unique, because they identify genomes.", + "Duplicate genome names found: {}. Genome names must be unique, because they identify genomes.", format_names(&duplicates) ); } + Ok(()) +} + +/// Maps each genome name of the graph to the id of the path holding it. +/// +/// Errors if any path is unnamed or if two paths share a name, via [`check_unique_genome_names`]. +pub fn path_ids_by_name(graph: &Pangraph) -> Result, Report> { + check_unique_genome_names(&[graph])?; + Ok( graph .paths @@ -105,10 +122,12 @@ pub fn reconstruct_genome(graph: &Pangraph, path_id: PathId) -> Result Result, Report> { +/// The FASTA counterpart of [`check_unique_genome_names`]: sequence names become genome names when +/// the records are built into a graph, so `build` enforces the invariant on its input to guarantee +/// that a graph can never carry duplicate genome names into a later merge. +pub fn check_unique_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); if !duplicates.is_empty() { return make_error!( @@ -117,12 +136,18 @@ pub fn sequences_by_name(fastas: &[FastaRecord]) -> Result ); } - Ok( - fastas - .iter() - .map(|fasta| (fasta.seq_name.clone(), fasta.seq.clone())) - .collect(), - ) + Ok(()) +} + +/// Collects FASTA records into genome sequences keyed by name. +/// +/// Names are assumed to be unique already: pass the records through [`check_unique_sequence_names`] +/// first, or records sharing a name will silently collapse into one entry. +pub(crate) fn sequences_by_name(fastas: &[FastaRecord]) -> BTreeMap { + fastas + .iter() + .map(|fasta| (fasta.seq_name.clone(), fasta.seq.clone())) + .collect() } /// Compares one reconstructed genome against the sequence it is expected to have, reporting the @@ -208,11 +233,13 @@ pub fn verify_graph_sequences( /// the same reason. /// /// Every genome of `sources` must appear in `graph`, and `graph` must contain nothing else. The -/// `sources` are expected to have disjoint genome names; a name appearing in two of them is -/// verified twice rather than reported, since the callers reject that case up front. +/// `sources` must have disjoint genome names: a name appearing in two of them means the two graphs +/// describe overlapping genome sets, and is reported rather than verified twice. pub fn verify_graph_against_graphs(graph: &Pangraph, sources: &[&Pangraph]) -> Result<(), Report> { + check_unique_genome_names(sources).wrap_err("When checking the genome names of the input graphs")?; + let path_ids = path_ids_by_name(graph)?; - let mut n_expected = 0; + let mut verified: BTreeSet<&str> = BTreeSet::new(); for source in sources { for (name, source_path_id) in path_ids_by_name(source)? { @@ -226,25 +253,19 @@ pub fn verify_graph_against_graphs(graph: &Pangraph, sources: &[&Pangraph]) -> R reconstruct_genome(graph, *path_id).wrap_err_with(|| format!("When reconstructing genome '{name}'"))?; verify_genome(name, &expected, &actual)?; - n_expected += 1; + verified.insert(name); } } - if path_ids.len() != n_expected { - let expected_names: BTreeSet<&str> = sources - .iter() - .map(|source| path_ids_by_name(source)) - .collect::, Report>>()? - .into_iter() - .flat_map(|ids| ids.into_keys()) - .collect(); - - let extra = path_ids - .keys() - .filter(|name| !expected_names.contains(*name)) - .copied() - .collect_vec(); + // Every genome of `sources` has now been found in `graph` and checked, so anything left over is a + // genome the merged graph invented. + let extra = path_ids + .keys() + .filter(|name| !verified.contains(*name)) + .copied() + .collect_vec(); + if !extra.is_empty() { return make_error!( "Graph contains {} genome(s) that are not present in the input graphs: {}", extra.len(), @@ -390,11 +411,25 @@ mod tests { #[rstest] fn test_path_ids_by_name_rejects_duplicate_names() { let graph = two_genome_graph([Some("a"), Some("a")]); - assert!(report_to_string(&path_ids_by_name(&graph).unwrap_err()).contains("duplicate genome names")); + assert!(report_to_string(&path_ids_by_name(&graph).unwrap_err()).contains("Duplicate genome names")); } + /// A name may be unique within each graph and still collide across them. This is what makes + /// `merge` reject merging a graph with itself, and what stops two overlapping source graphs from + /// being "verified" against a merged graph that holds their genomes only once. #[rstest] - fn test_sequences_by_name_rejects_duplicate_names() { + fn test_check_unique_genome_names_rejects_name_shared_across_graphs() { + let (left, _) = sources(); + let other = one_genome_graph("a", "GGGGCCCC", Forward); + + check_unique_genome_names(&[&left]).unwrap(); + let err = report_to_string(&check_unique_genome_names(&[&left, &other]).unwrap_err()); + assert!(err.contains("Duplicate genome names"), "unexpected error: {err}"); + assert!(err.contains('a'), "unexpected error: {err}"); + } + + #[rstest] + fn test_check_unique_sequence_names_rejects_duplicate_names() { let record = |name: &str| FastaRecord { seq_name: name.to_owned(), desc: None, @@ -402,7 +437,7 @@ mod tests { index: 0, }; let fastas = [record("a"), record("a")]; - assert!(report_to_string(&sequences_by_name(&fastas).unwrap_err()).contains("Duplicate sequence names")); + assert!(report_to_string(&check_unique_sequence_names(&fastas).unwrap_err()).contains("Duplicate sequence names")); } #[rstest] @@ -429,7 +464,7 @@ mod tests { index: 42, }, ]; - let expected = sequences_by_name(&fastas).unwrap(); + let expected = sequences_by_name(&fastas); verify_graph_sequences(&graph(), &expected, GenomeCoverage::Complete).unwrap(); } @@ -505,6 +540,19 @@ mod tests { assert!(err.contains('b'), "unexpected error: {err}"); } + /// Two sources holding the same genome name describe overlapping genome sets. Counting one + /// expectation per (source, name) pair used to make the totals disagree while no genome was + /// actually extra, reporting the nonsensical "contains 0 genome(s) that are not present ...: []". + #[rstest] + fn test_verify_graph_against_graphs_detects_name_shared_across_sources() { + let (left, right) = sources(); + let duplicate = one_genome_graph("a", "ACGTACGT", Forward); + + let err = report_to_string(&verify_graph_against_graphs(&graph(), &[&left, &right, &duplicate]).unwrap_err()); + assert!(err.contains("Duplicate genome names"), "unexpected error: {err}"); + assert!(!err.contains("0 genome(s)"), "unexpected error: {err}"); + } + /// The name matches but the sequence does not: the mismatch is reported against the source graph /// the genome came from, with the position of the first difference. #[rstest] diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 04fbf286..d0ddd6aa 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -186,6 +186,35 @@ mod tests { Ok(()) } + /// Unnamed paths used to slip past the duplicate-name guard entirely: it scanned + /// `path_names().flatten()`, which drops `None`, and the check that would have caught them ran + /// only under `--verify`. Merging such a graph with itself therefore succeeded and silently + /// emitted every genome twice. `verify: false` is the whole point of this test. + #[rstest] + fn itest_merge_rejects_unnamed_genomes_without_verify() -> Result<(), Report> { + let dir = tempdir()?; + let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; + + let mut graph = read_graph(&build_graph_file(&dir, "named.json", fastas)?)?; + for path in graph.paths.values_mut() { + path.name = None; + } + let anonymous = dir.path().join("anonymous.json"); + json_write_file(&anonymous, &graph, JsonPretty(false))?; + + let output = dir.path().join("merged.json"); + let result = merge_run(&PangraphMergeArgs { + verify: false, + ..merge_args(anonymous.clone(), anonymous, output.clone()) + }); + + let error = report_to_string(&result.unwrap_err()); + assert!(error.contains("without a name"), "unexpected error message: {error}"); + assert!(!output.exists(), "a rejected merge must not write an output graph"); + + Ok(()) + } + /// `build --verify` used to pair reconstructed genomes with input records by `FastaRecord::index`, /// which is only valid when the records' indices happen to be exactly `0..n-1`. Here they are /// `[3, 4, 5]` for a 3-record slice, which panicked with an out-of-bounds index before genomes From f03429cf53ec0a82e8c155ad273b6f908fe47b71 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 19:41:09 +0200 Subject: [PATCH 23/50] perf(reconstruct): drop redundant re-sort of an already-ordered path map --- packages/pangraph/src/pangraph/reconstruct.rs | 35 +++++++++---------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index a2f47dbc..ed203721 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -32,29 +32,28 @@ pub enum GenomeCoverage { /// Reconstructs every genome of the graph as a FASTA record, ordered by path id. /// +/// The ordering comes for free: `paths` is a `BTreeMap` keyed by [`PathId`], so iterating it +/// already yields ascending ids and the records stay lazy. +/// /// Record order and `index` reproduce the order of the original input FASTA only for graphs /// produced directly by `build`. A merged graph renumbers its path ids, so consumers must match /// records by genome name. pub fn reconstruct(graph: &Pangraph) -> impl Iterator> + use<'_> { - graph - .paths - .iter() - .sorted_by_key(|(path_id, _)| **path_id) - .map(|(path_id, path)| { - let index = path_id.0; - let seq = reconstruct_path_sequence(graph, path)?; - let seq_name = path - .name() - .clone() - .unwrap_or_else(|| format!("Unknown sequence #{path_id}")); - let desc = path.desc().clone(); - Ok(FastaRecord { - seq_name, - desc, - seq, - index, - }) + graph.paths.iter().map(|(path_id, path)| { + let index = path_id.0; + let seq = reconstruct_path_sequence(graph, path)?; + let seq_name = path + .name() + .clone() + .unwrap_or_else(|| format!("Unknown sequence #{path_id}")); + let desc = path.desc().clone(); + Ok(FastaRecord { + seq_name, + desc, + seq, + index, }) + }) } /// Checks that every path of `graphs` carries a name, and that no name occurs more than once From 0c79a24562b6caa5b8500413aca95b67d0c21c71 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 19:41:44 +0200 Subject: [PATCH 24/50] refactor(reconstruct): drop redundant record reset before reading --- packages/pangraph/src/commands/reconstruct/reconstruct_run.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs index d61e4252..fa6f83bf 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs @@ -52,7 +52,7 @@ fn verify_against_fasta(graph: &Pangraph, verify: &Path) -> Result Date: Fri, 14 Aug 2026 19:42:44 +0200 Subject: [PATCH 25/50] fix(reconstruct): elide long genome name lists in the verification error --- .../pangraph/src/commands/reconstruct/reconstruct_run.rs | 6 +++--- packages/pangraph/src/pangraph/reconstruct.rs | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs index fa6f83bf..d980ee95 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs @@ -3,7 +3,7 @@ use crate::io::fasta::{FastaReader, FastaRecord, FastaWriter}; use crate::io::json::json_read_file; use crate::make_error; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::reconstruct::{path_ids_by_name, reconstruct, reconstruct_genome, verify_genome}; +use crate::pangraph::reconstruct::{format_names, path_ids_by_name, reconstruct, reconstruct_genome, verify_genome}; use eyre::{Report, WrapErr}; use itertools::Itertools; use log::info; @@ -79,9 +79,9 @@ fn verify_against_fasta(graph: &Pangraph, verify: &Path) -> Result R } /// Formats a list of genome names for an error message, eliding all but the first few. -fn format_names>(names: &[S]) -> String { +pub(crate) fn format_names>(names: &[S]) -> String { let shown = names.iter().take(MAX_NAMES_IN_ERROR).map(AsRef::as_ref).join(", "); if names.len() > MAX_NAMES_IN_ERROR { format!("[{shown}, ... and {} more]", names.len() - MAX_NAMES_IN_ERROR) From 81f79c8ff8ce70f469714131047334e7cba43c1a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 19:44:34 +0200 Subject: [PATCH 26/50] fix(cli): make the Default impl agree with the clap default for output paths --- .../pangraph/src/commands/build/build_args.rs | 1 + .../pangraph/src/commands/merge/merge_args.rs | 1 + .../commands/reconstruct/reconstruct_args.rs | 1 + packages/pangraph/src/commands/root_args.rs | 38 +++++++++++++++++++ 4 files changed, 41 insertions(+) diff --git a/packages/pangraph/src/commands/build/build_args.rs b/packages/pangraph/src/commands/build/build_args.rs index 850fb5ce..51801cd4 100644 --- a/packages/pangraph/src/commands/build/build_args.rs +++ b/packages/pangraph/src/commands/build/build_args.rs @@ -25,6 +25,7 @@ pub struct PangraphBuildArgs { /// If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. /// /// Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + #[default(PathBuf::from("-"))] #[clap(long, short = 'o', default_value = "-")] #[clap(value_hint = ValueHint::AnyPath)] pub output_json: PathBuf, diff --git a/packages/pangraph/src/commands/merge/merge_args.rs b/packages/pangraph/src/commands/merge/merge_args.rs index 6ed6e35c..abb2a68b 100644 --- a/packages/pangraph/src/commands/merge/merge_args.rs +++ b/packages/pangraph/src/commands/merge/merge_args.rs @@ -32,6 +32,7 @@ pub struct PangraphMergeArgs { /// If the provided file path ends with one of the supported extensions: "gz", "bz2", "xz", "zst", then the file will be written compressed. If the required directory tree does not exist, it will be created. /// /// Use "-" to write the uncompressed data to standard output (stdout). This is the default, if the argument is not provided. + #[default(PathBuf::from("-"))] #[clap(long, short = 'o', default_value = "-")] #[clap(value_hint = ValueHint::AnyPath)] pub output_json: PathBuf, diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs index 4d5741e6..594b4f0f 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_args.rs @@ -29,6 +29,7 @@ pub struct PangraphReconstructArgs { /// position. /// /// See: https://en.wikipedia.org/wiki/FASTA_format + #[default(PathBuf::from("-"))] #[clap(long, short = 'o', default_value = "-")] #[clap(value_hint = ValueHint::AnyPath)] pub output_fasta: PathBuf, diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index e0d1ce3e..903f0671 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -122,3 +122,41 @@ pub fn parse_cli_args() -> Result { setup_logger(args.verbosity.get_filter_level()); Ok(args) } + +#[cfg(test)] +mod tests { + use super::*; + use pretty_assertions::assert_eq; + use rstest::rstest; + use std::path::PathBuf; + + /// The `Default` impl and the clap `default_value` must agree. They are declared separately, so a + /// field can easily get one and not the other: an output path then defaults to `""` in code that + /// builds args programmatically with `..Default::default()`, and writes to a file literally named + /// `""` instead of to stdout. + #[rstest] + fn test_clap_and_rust_defaults_agree_on_output_paths() { + let stdout = PathBuf::from("-"); + + let PangraphCommands::Build(build) = PangraphArgs::parse_from(["pangraph", "build"]).command else { + panic!("expected the build subcommand"); + }; + assert_eq!(build.output_json, stdout); + assert_eq!(PangraphBuildArgs::default().output_json, stdout); + + let PangraphCommands::Merge(merge) = + PangraphArgs::parse_from(["pangraph", "merge", "left.json", "right.json"]).command + else { + panic!("expected the merge subcommand"); + }; + assert_eq!(merge.output_json, stdout); + assert_eq!(PangraphMergeArgs::default().output_json, stdout); + + let PangraphCommands::Reconstruct(reconstruct) = PangraphArgs::parse_from(["pangraph", "reconstruct"]).command + else { + panic!("expected the reconstruct subcommand"); + }; + assert_eq!(reconstruct.output_fasta, stdout); + assert_eq!(PangraphReconstructArgs::default().output_fasta, stdout); + } +} From 166ab57806133775e95682827bca6c1783fe8130 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 20:16:59 +0200 Subject: [PATCH 27/50] fix(pangraph): report malformed graph ids instead of panicking during relabel --- packages/pangraph/src/pangraph/pangraph.rs | 139 +++++++++++++++--- .../pangraph/src/pangraph/pangraph_block.rs | 28 ++-- 2 files changed, 136 insertions(+), 31 deletions(-) diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index c9556d1f..df0a6fe3 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -10,7 +10,7 @@ use crate::representation::seq::Seq; use crate::tree::clade::WithNewickName; use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; -use crate::{make_internal_error, make_internal_report}; +use crate::{make_internal_error, make_internal_report, make_report}; use eyre::{Report, WrapErr}; use log::warn; use maplit::btreemap; @@ -90,39 +90,52 @@ impl Pangraph { .map(|(rank, &pid)| (pid, PathId(path_id_offset + rank))) .collect(); + // The three maps above are keyed by the *map keys* of this graph, so looking an entity's own new + // id up by its key cannot miss. Ids taken from an entity's *contents* are a different matter: + // they can dangle, or disagree with the key they are stored under, in a graph that pangraph did + // not write, so those lookups are fallible. let blocks: BTreeMap = blocks - .into_values() - .map(|block| { - let new_block_id = block_map[&block.id()]; - let block = block.relabel(new_block_id, &node_map); - (block.id(), block) + .into_iter() + .map(|(bid, block)| { + let block = block.relabel(block_map[&bid], &node_map)?; + Ok((block.id(), block)) }) - .collect(); + .collect::>()?; let nodes: BTreeMap = nodes .into_iter() .map(|(nid, node)| { - let node = PangraphNode::new( - Some(node_map[&nid]), - block_map[&node.block_id()], - path_map[&node.path_id()], - node.strand(), - node.position(), - ); - (node.id(), node) + let block_id = *block_map.get(&node.block_id()).ok_or_else(|| { + make_report!( + "Node {nid} refers to block {}, which the graph does not contain", + node.block_id() + ) + })?; + let path_id = *path_map.get(&node.path_id()).ok_or_else(|| { + make_report!( + "Node {nid} refers to path {}, which the graph does not contain", + node.path_id() + ) + })?; + + let node = PangraphNode::new(Some(node_map[&nid]), block_id, path_id, node.strand(), node.position()); + Ok((node.id(), node)) }) - .collect(); + .collect::>()?; let paths: BTreeMap = paths - .into_values() - .map(|mut path| { - path.id = path_map[&path.id]; + .into_iter() + .map(|(pid, mut path)| { + path.id = path_map[&pid]; for nid in &mut path.nodes { - *nid = node_map[nid]; + let old = *nid; + *nid = *node_map + .get(&old) + .ok_or_else(|| make_report!("Path {pid} refers to node {old}, which the graph does not contain"))?; } - (path.id, path) + Ok((path.id, path)) }) - .collect(); + .collect::>()?; // An injective relabeling cannot change the number of entities. If it did, two distinct ids // were mapped onto the same one and entities were silently dropped. @@ -251,6 +264,37 @@ impl Pangraph { #[cfg(any(test, debug_assertions))] pub fn sanity_check(&self) -> Result<(), Report> { + // Each entity is stored under a map key *and* carries its own id. Serde keys the maps by the + // JSON object key, so the two can disagree in a graph that pangraph did not write. Everything + // that resolves an entity by key while reading its id off the entity itself depends on them + // agreeing, so check it first. One integer comparison per entity. + for (block_id, block) in &self.blocks { + if block.id() != *block_id { + return Err(eyre::eyre!( + "Block is stored under id {block_id} but reports id {}", + block.id() + )); + } + } + + for (node_id, node) in &self.nodes { + if node.id() != *node_id { + return Err(eyre::eyre!( + "Node is stored under id {node_id} but reports id {}", + node.id() + )); + } + } + + for (path_id, path) in &self.paths { + if path.id() != *path_id { + return Err(eyre::eyre!( + "Path is stored under id {path_id} but reports id {}", + path.id() + )); + } + } + for (node_id, node) in &self.nodes { if !self.blocks.contains_key(&node.block_id()) { return Err(eyre::eyre!("Block {} not found in graph", node.block_id())); @@ -427,6 +471,7 @@ mod tests { use crate::pangraph::pangraph_path::PangraphPath; use crate::pangraph::reconstruct::reconstruct; use crate::pangraph::strand::Strand::{Forward, Reverse}; + use crate::utils::error::report_to_string; use itertools::Itertools; use maplit::btreemap; use rstest::rstest; @@ -588,6 +633,56 @@ mod tests { Pangraph { paths, blocks, nodes } } + /// A graph read from JSON is keyed by the object key, while each entity also stores its own id; + /// serde never checks that the two agree. `relabel` resolves entities by key, so a disagreement + /// used to abort the process with a bare "no entry found for key" panic and no indication of + /// which file was at fault. It must be a reportable error instead. + #[rstest] + fn test_relabel_reports_block_stored_under_a_mismatched_id() { + let mut graph = colliding_graph(["a", "b"]); + let block = graph.blocks.remove(&BlockId(0)).unwrap(); + graph.blocks.insert(BlockId(42), block); // key 42, but the block still reports id 0 + + let err = report_to_string(&graph.clone().relabel(1, 7).unwrap_err()); + assert!(err.contains("does not contain"), "unexpected error: {err}"); + + // `sanity_check` never compared key against stored id, so it used to pass this graph through. + let err = report_to_string(&graph.sanity_check().unwrap_err()); + assert!(err.contains("stored under id 42"), "unexpected error: {err}"); + assert!(err.contains("reports id 0"), "unexpected error: {err}"); + } + + #[rstest] + fn test_relabel_reports_path_referring_to_a_missing_node() { + let mut graph = colliding_graph(["a", "b"]); + graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; + + let err = report_to_string(&graph.relabel(1, 7).unwrap_err()); + assert!( + err.contains("Path 0 refers to node 99, which the graph does not contain"), + "unexpected error: {err}" + ); + } + + #[rstest] + fn test_relabel_reports_node_referring_to_a_missing_block() { + let mut graph = colliding_graph(["a", "b"]); + let node = graph.nodes.get_mut(&NodeId(0)).unwrap(); + *node = PangraphNode::new( + Some(NodeId(0)), + BlockId(99), + node.path_id(), + node.strand(), + node.position(), + ); + + let err = report_to_string(&graph.relabel(1, 7).unwrap_err()); + assert!( + err.contains("Node 0 refers to block 99, which the graph does not contain"), + "unexpected error: {err}" + ); + } + #[rstest] fn test_relabel_keeps_graph_consistent() { let graph = colliding_graph(["a", "b"]).relabel(1, 7).unwrap(); diff --git a/packages/pangraph/src/pangraph/pangraph_block.rs b/packages/pangraph/src/pangraph/pangraph_block.rs index 4c706127..81f838ac 100644 --- a/packages/pangraph/src/pangraph/pangraph_block.rs +++ b/packages/pangraph/src/pangraph/pangraph_block.rs @@ -3,7 +3,6 @@ use crate::align::map_variations::{BandParameters, map_variations}; use crate::io::fasta::FastaRecord; use crate::io::json::{JsonPretty, json_write_str}; use crate::io::seq::reverse_complement; -use crate::make_internal_error; use crate::pangraph::edits::{Del, Edit, Ins, Sub}; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_node::NodeId; @@ -11,6 +10,7 @@ use crate::pangraph::pangraph_path::PathId; use crate::representation::seq::Seq; use crate::utils::collections::has_duplicates; use crate::utils::interval::positions_to_intervals; +use crate::{make_internal_error, make_report}; use derive_more::{Display, From}; use eyre::{Report, WrapErr}; use getset::{CopyGetters, Getters}; @@ -62,16 +62,26 @@ impl PangraphBlock { /// Returns this block with a new id and its alignment keys rewritten through `node_map`. /// The consensus and the edits are moved over unchanged. - pub fn relabel(self, id: BlockId, node_map: &BTreeMap) -> Self { - Self { + /// + /// Errors if an alignment is keyed by a node the graph does not contain, which can only happen in + /// a graph that pangraph did not write. + pub fn relabel(self, id: BlockId, node_map: &BTreeMap) -> Result { + let alignments = self + .alignments + .into_iter() + .map(|(nid, edit)| { + let new_nid = *node_map + .get(&nid) + .ok_or_else(|| make_report!("Block {id} aligns node {nid}, which the graph does not contain"))?; + Ok((new_nid, edit)) + }) + .collect::>()?; + + Ok(Self { id, consensus: self.consensus, - alignments: self - .alignments - .into_iter() - .map(|(nid, edit)| (node_map[&nid], edit)) - .collect(), - } + alignments, + }) } pub fn reverse_complement(&self) -> Result { From c65487920a086c44ba725540a7c823b1123b29f9 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 20:31:45 +0200 Subject: [PATCH 28/50] fix(cli): keep --jobs out of the Verbosity help section --- packages/pangraph/src/commands/root_args.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index 903f0671..962fa13c 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -53,7 +53,11 @@ pub struct PangraphArgs { pub verbosity: Verbosity, /// Number of processing jobs. If not specified, all available CPU threads will be used. + // Declared after the `Verbosity` flatten, which opens a help section that would otherwise claim + // every argument registered after it. A heading set on the argument itself wins over that + // section, and keeps this option in the default one. #[clap(global = true, long, short = 'j', default_value_t = num_cpus::get())] + #[clap(help_heading = None)] pub jobs: usize, } From 774a08c51c3b03682581c1ba0646453c3aecfb0a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 20:34:12 +0200 Subject: [PATCH 29/50] test(cli): pin the help section of every argument --- .../pangraph/src/commands/build/build_args.rs | 6 ++- .../pangraph/src/commands/merge/merge_args.rs | 6 ++- packages/pangraph/src/commands/root_args.rs | 48 +++++++++++++++++++ 3 files changed, 56 insertions(+), 4 deletions(-) diff --git a/packages/pangraph/src/commands/build/build_args.rs b/packages/pangraph/src/commands/build/build_args.rs index 51801cd4..7c65839d 100644 --- a/packages/pangraph/src/commands/build/build_args.rs +++ b/packages/pangraph/src/commands/build/build_args.rs @@ -51,8 +51,10 @@ pub struct PangraphBuildArgs { #[clap(long, value_hint = ValueHint::FilePath)] pub guide_tree: Option, - // Declared last: it opens the "Alignment" help section, and `next_help_heading` applies to every - // argument declared after it. + // Declared last: it opens the "Alignment" help section, which then applies to every argument + // registered after it, including later fields of this struct. To add one anyway, give it its own + // `#[clap(help_heading = ...)]`, which wins. Pinned by + // `root_args::tests::test_arguments_are_filed_under_the_expected_help_section`. #[clap(flatten)] pub merge_params: GraphMergeParams, } diff --git a/packages/pangraph/src/commands/merge/merge_args.rs b/packages/pangraph/src/commands/merge/merge_args.rs index abb2a68b..70ed0ebd 100644 --- a/packages/pangraph/src/commands/merge/merge_args.rs +++ b/packages/pangraph/src/commands/merge/merge_args.rs @@ -42,8 +42,10 @@ pub struct PangraphMergeArgs { #[clap(long, short = 'f')] pub verify: bool, - // Declared last: it opens the "Alignment" help section, and `next_help_heading` applies to every - // argument declared after it. + // Declared last: it opens the "Alignment" help section, which then applies to every argument + // registered after it, including later fields of this struct. To add one anyway, give it its own + // `#[clap(help_heading = ...)]`, which wins. Pinned by + // `root_args::tests::test_arguments_are_filed_under_the_expected_help_section`. #[clap(flatten)] pub merge_params: GraphMergeParams, } diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index 962fa13c..284367ac 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -132,6 +132,7 @@ mod tests { use super::*; use pretty_assertions::assert_eq; use rstest::rstest; + use std::collections::BTreeSet; use std::path::PathBuf; /// The `Default` impl and the clap `default_value` must agree. They are declared separately, so a @@ -163,4 +164,51 @@ mod tests { assert_eq!(reconstruct.output_fasta, stdout); assert_eq!(PangraphReconstructArgs::default().output_fasta, stdout); } + + /// Returns the ids of the arguments of `cmd` filed under the given help section. + fn args_under(cmd: &clap::Command, heading: &str) -> BTreeSet { + cmd + .get_arguments() + .filter(|arg| arg.get_help_heading() == Some(heading)) + .map(|arg| arg.get_id().to_string()) + .collect() + } + + fn ids(names: &[&str]) -> BTreeSet { + names.iter().map(|name| (*name).to_owned()).collect() + } + + /// A help section is not a property of a group of arguments: clap keeps a single cursor on the + /// `Command` and stamps it onto each argument as it is registered, and `#[clap(flatten)]` shares + /// that `Command` with the flattened struct. A flattened group that opens a section therefore + /// opens it for every argument registered afterwards, including later fields of the *parent* + /// struct. Declaration order is load-bearing, and getting it wrong is invisible outside `--help`. + /// + /// This pins the assignment so that mistake is a test failure. An argument appended after a + /// flattened group shows up in one of these sets; an argument added before it does not, so there + /// are no false alarms. The escape hatch, if a trailing argument really is needed, is to set + /// `#[clap(help_heading = ...)]` on the argument itself — that wins over the cursor. + #[rstest] + fn test_arguments_are_filed_under_the_expected_help_section() { + // `jobs` is declared after the `Verbosity` flatten and used to be swept into it. + assert_eq!( + args_under(&PangraphArgs::command(), "Verbosity"), + ids(&["verbosity", "silent", "verbose", "quiet"]) + ); + + // Exactly the options of `GraphMergeParams`, which `build` and `merge` both flatten last. + let alignment = ids(&[ + "indel_len_threshold", + "alpha", + "beta", + "sensitivity", + "kmer_length", + "max_self_map", + "alignment_kernel", + "extra_band_width", + "max_alignment_attempts", + ]); + assert_eq!(args_under(&PangraphBuildArgs::command(), "Alignment"), alignment); + assert_eq!(args_under(&PangraphMergeArgs::command(), "Alignment"), alignment); + } } From e1f6dbc98740cf2714358af971c643336ee105c6 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 20:37:05 +0200 Subject: [PATCH 30/50] docs: update changelog for merge and stricter genome id checks --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 14078002..1e82ca2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,8 @@ ## Unreleased -- added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #192. +- added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. -- `pangraph reconstruct --verify` now matches sequences by genome name rather than by position, see #193. +- more strict checks on input sequences ids: duplicated or absent ids are now rejected. The `--verify` options compare the input sequence by id and not by order. ## 1.3.0 From 8f5d2f8e48419bceaf47204fa2d2c2184d29d02b Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Fri, 14 Aug 2026 21:12:42 +0200 Subject: [PATCH 31/50] fix(pangraph): reject empty genome and sequence names --- CHANGELOG.md | 2 +- .../pangraph/src/commands/build/build_run.rs | 6 +- .../pangraph/src/commands/merge/merge_run.rs | 4 +- .../src/commands/simplify/simplify_run.rs | 6 +- .../pangraph/src/pangraph/graph_merging.rs | 3 +- packages/pangraph/src/pangraph/reconstruct.rs | 97 ++++++++++++++----- packages/pangraph/tests/itest_merge.rs | 21 +++- packages/pangraph/tests/itest_reconstruct.rs | 2 +- 8 files changed, 103 insertions(+), 38 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e82ca2a..107a9f35 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. -- more strict checks on input sequences ids: duplicated or absent ids are now rejected. The `--verify` options compare the input sequence by id and not by order. +- more strict checks on input sequences ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequence by id and not by order. ## 1.3.0 diff --git a/packages/pangraph/src/commands/build/build_run.rs b/packages/pangraph/src/commands/build/build_run.rs index d22d786a..30d865fd 100644 --- a/packages/pangraph/src/commands/build/build_run.rs +++ b/packages/pangraph/src/commands/build/build_run.rs @@ -4,9 +4,7 @@ use crate::io::fasta::{FastaReader, FastaRecord}; use crate::io::json::{JsonPretty, json_write_file}; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; -use crate::pangraph::reconstruct::{ - GenomeCoverage, check_unique_sequence_names, sequences_by_name, verify_graph_sequences, -}; +use crate::pangraph::reconstruct::{GenomeCoverage, check_sequence_names, sequences_by_name, verify_graph_sequences}; use crate::pangraph::strand::Strand::Forward; use crate::representation::seq::Seq; use crate::tree::clade::postorder; @@ -57,7 +55,7 @@ pub fn build_run(args: &PangraphBuildArgs) -> Result<(), Report> { } pub fn build(fastas: Vec, args: &PangraphBuildArgs, verify: bool) -> Result { - check_unique_sequence_names(&fastas).wrap_err("When checking the names of the input sequences")?; + check_sequence_names(&fastas).wrap_err("When checking the names of the input sequences")?; // If verification is requested, keep the input sequences, keyed by genome name, to compare them // with the sequences reconstructed from the graph. Names were just checked to be unique, so no diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index b14f9394..1aff54fd 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -5,7 +5,7 @@ use crate::make_error; use crate::pangraph::graph_merging::merge_graphs; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PangraphPath; -use crate::pangraph::reconstruct::{check_unique_genome_names, verify_graph_against_graphs}; +use crate::pangraph::reconstruct::{check_genome_names, verify_graph_against_graphs}; use color_eyre::owo_colors::{AnsiColors, OwoColorize}; use color_eyre::{Help, SectionExt}; use eyre::{Report, WrapErr}; @@ -86,7 +86,7 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right // whose genomes cannot be told apart afterwards is not useful either way. Checked here rather // than at verification time so that it fails before the expensive merge. `build` enforces the // same invariant on its input FASTA records. - check_unique_genome_names(&[left, right]) + check_genome_names(&[left, right]) .wrap_err("When checking the genome names of the input graphs") .with_section(|| { format!("{}\n{}", args.left_graph.display(), args.right_graph.display()).header("Input graphs:") diff --git a/packages/pangraph/src/commands/simplify/simplify_run.rs b/packages/pangraph/src/commands/simplify/simplify_run.rs index 114464ab..817ae420 100644 --- a/packages/pangraph/src/commands/simplify/simplify_run.rs +++ b/packages/pangraph/src/commands/simplify/simplify_run.rs @@ -4,7 +4,7 @@ use crate::io::file::create_file_or_stdout; use crate::io::json::{JsonPretty, json_write}; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_path::PathId; -use crate::pangraph::reconstruct::check_unique_genome_names; +use crate::pangraph::reconstruct::check_genome_names; use eyre::{Report, WrapErr}; use std::collections::BTreeSet; @@ -23,7 +23,7 @@ pub fn simplify_run(args: PangraphSimplifyArgs) -> Result<(), Report> { fn simplify(graph: &mut Pangraph, focal_paths: &BTreeSet) -> Result<(), Report> { // Genomes are selected by name here, so an unnamed path cannot be resolved either way: without // this check the filter below would silently drop it rather than report it. - check_unique_genome_names(&[graph]).wrap_err("When checking the genome names of the input graph")?; + check_genome_names(&[graph]).wrap_err("When checking the genome names of the input graph")?; let path_ids_to_remove: Vec = graph .paths @@ -227,6 +227,6 @@ mod tests { graph.paths.get_mut(&PathId(3)).unwrap().name = None; let err = report_to_string(&simplify(&mut graph, &btreeset! {o!("pathA"), o!("pathB")}).unwrap_err()); - assert!(err.contains("without a name"), "unexpected error: {err}"); + assert!(err.contains("no name or an empty name"), "unexpected error: {err}"); } } diff --git a/packages/pangraph/src/pangraph/graph_merging.rs b/packages/pangraph/src/pangraph/graph_merging.rs index 85b2520c..165f436d 100644 --- a/packages/pangraph/src/pangraph/graph_merging.rs +++ b/packages/pangraph/src/pangraph/graph_merging.rs @@ -1,6 +1,5 @@ use crate::align::alignment::Alignment; -use crate::align::alignment_args::AlignmentArgs; -use crate::align::alignment_args::{AlignmentBackend, GraphMergeParams}; +use crate::align::alignment_args::{AlignmentArgs, AlignmentBackend, GraphMergeParams}; use crate::align::energy::alignment_energy2; use crate::align::minimap2_lib::align_with_minimap2_lib::align_with_minimap2_lib; use crate::align::mmseqs::align_with_mmseqs::align_with_mmseqs; diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 49089f3a..f6c5e47b 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -56,31 +56,33 @@ pub fn reconstruct(graph: &Pangraph) -> impl Iterator Result<(), Report> { +pub fn check_genome_names(graphs: &[&Pangraph]) -> Result<(), Report> { let unnamed = graphs .iter() .flat_map(|graph| graph.paths.iter()) - .filter(|(_, path)| path.name.is_none()) + .filter(|(_, path)| path.name.as_deref().is_none_or(|name| name.trim().is_empty())) .map(|(path_id, _)| path_id.to_string()) .collect_vec(); if !unnamed.is_empty() { return make_error!( - "Found {} genome(s) without a name (path ids: {}). Genomes are identified by name, so every path must be named.", + "Found {} genome(s) with no name or an empty name (path ids: {}). Genomes are identified by name, so every path must be named.", unnamed.len(), format_names(&unnamed) ); @@ -99,9 +101,9 @@ pub fn check_unique_genome_names(graphs: &[&Pangraph]) -> Result<(), Report> { /// Maps each genome name of the graph to the id of the path holding it. /// -/// Errors if any path is unnamed or if two paths share a name, via [`check_unique_genome_names`]. +/// Errors if any path is unnamed or if two paths share a name, via [`check_genome_names`]. pub fn path_ids_by_name(graph: &Pangraph) -> Result, Report> { - check_unique_genome_names(&[graph])?; + check_genome_names(&[graph])?; Ok( graph @@ -121,12 +123,29 @@ pub fn reconstruct_genome(graph: &Pangraph, path_id: PathId) -> Result Result<(), Report> { +/// The FASTA counterpart of [`check_genome_names`]: sequence names become genome names when the +/// records are built into a graph, so `build` enforces the invariant on its input to guarantee that +/// a graph can never carry unusable genome names into a later merge. +/// +/// Empty names are reported before duplicates, so that a file of headers that are all empty is +/// reported for the reason it actually has. +pub fn check_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { + let empty = fastas + .iter() + .filter(|fasta| fasta.seq_name.trim().is_empty()) + .map(|fasta| fasta.index.to_string()) + .collect_vec(); + + if !empty.is_empty() { + return make_error!( + "Found {} input sequence(s) with an empty name (record indices: {}). Sequences are identified by name, so every record must have one. Note that a space between '>' and the identifier makes the identifier part of the description rather than the name.", + empty.len(), + format_names(&empty) + ); + } + let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); if !duplicates.is_empty() { return make_error!( @@ -140,7 +159,7 @@ pub fn check_unique_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> /// Collects FASTA records into genome sequences keyed by name. /// -/// Names are assumed to be unique already: pass the records through [`check_unique_sequence_names`] +/// Names are assumed to be unique already: pass the records through [`check_sequence_names`] /// first, or records sharing a name will silently collapse into one entry. pub(crate) fn sequences_by_name(fastas: &[FastaRecord]) -> BTreeMap { fastas @@ -235,7 +254,7 @@ pub fn verify_graph_sequences( /// `sources` must have disjoint genome names: a name appearing in two of them means the two graphs /// describe overlapping genome sets, and is reported rather than verified twice. pub fn verify_graph_against_graphs(graph: &Pangraph, sources: &[&Pangraph]) -> Result<(), Report> { - check_unique_genome_names(sources).wrap_err("When checking the genome names of the input graphs")?; + check_genome_names(sources).wrap_err("When checking the genome names of the input graphs")?; let path_ids = path_ids_by_name(graph)?; let mut verified: BTreeSet<&str> = BTreeSet::new(); @@ -297,7 +316,7 @@ fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result id` — a space between the '>' and the identifier — parses into an empty name with the + /// identifier in the description, so this common header style used to yield a nameless genome. + /// The record index is part of the message, since the name cannot point at the offending record. + #[rstest] + #[case("")] + #[case(" ")] + fn test_check_sequence_names_rejects_empty_name(#[case] name: &str) { + let fastas = [FastaRecord { + seq_name: name.to_owned(), + desc: Some(o!("NC_000913.3 Escherichia coli")), + seq: Seq::from_str("ACGT"), + index: 3, + }]; + + let err = report_to_string(&check_sequence_names(&fastas).unwrap_err()); + assert!(err.contains("empty name"), "unexpected error: {err}"); + assert!(err.contains('3'), "unexpected error: {err}"); } #[rstest] diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index d0ddd6aa..f2c0f5f1 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -209,7 +209,10 @@ mod tests { }); let error = report_to_string(&result.unwrap_err()); - assert!(error.contains("without a name"), "unexpected error message: {error}"); + assert!( + error.contains("no name or an empty name"), + "unexpected error message: {error}" + ); assert!(!output.exists(), "a rejected merge must not write an output graph"); Ok(()) @@ -249,4 +252,20 @@ mod tests { Ok(()) } + + /// A FASTA header of the form `> id` leaves the record unnamed, with the identifier in the + /// description. Such a genome could not be addressed by name afterwards, so `build` rejects it. + #[rstest] + fn itest_build_rejects_empty_sequence_name() -> Result<(), Report> { + let (mut fastas, _) = read_and_split("../../data/ges-1.fa", 2, 2)?; + fastas[1].desc = Some(fastas[1].seq_name.clone()); + fastas[1].seq_name = String::new(); + + let result = build(fastas, &PangraphBuildArgs::default(), false); + + let error = report_to_string(&result.unwrap_err()); + assert!(error.contains("empty name"), "unexpected error message: {error}"); + + Ok(()) + } } diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs index 4ea3b8ff..c621a43c 100644 --- a/packages/pangraph/tests/itest_reconstruct.rs +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -202,7 +202,7 @@ mod tests { )?; let err = report_to_string(&reconstruct_run(&verify_args(&graph, &verify)).unwrap_err()); - assert!(err.contains("without a name"), "unexpected error: {err}"); + assert!(err.contains("no name or an empty name"), "unexpected error: {err}"); Ok(()) } From 348e51d215b44d0de72566d7dcfd4b42767ac667 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 17 Aug 2026 11:52:25 +0200 Subject: [PATCH 32/50] refactor(pangraph): derive block and node ids from genome names --- CHANGELOG.md | 2 + docs/docs/reference.md | 4 +- .../minimap2_lib/align_with_minimap2_lib.rs | 7 +- .../pangraph/src/circularize/merge_blocks.rs | 4 +- .../pangraph/src/commands/merge/merge_args.rs | 11 +- .../pangraph/src/commands/merge/merge_run.rs | 16 +- .../src/commands/simplify/simplify_run.rs | 7 +- packages/pangraph/src/pangraph/pangraph.rs | 338 +++++++++--------- .../pangraph/src/pangraph/pangraph_block.rs | 26 +- .../pangraph/src/pangraph/pangraph_node.rs | 17 +- .../pangraph/src/pangraph/pangraph_path.rs | 15 + packages/pangraph/src/pangraph/slice.rs | 7 +- packages/pangraph/tests/itest_merge.rs | 118 +++++- 13 files changed, 348 insertions(+), 224 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 107a9f35..84a8275f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,8 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. - more strict checks on input sequences ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequence by id and not by order. +- block and node identifiers are now derived from genome names instead of the order the input sequences are read in, enforcing order-independence. +- fixed `pangraph build` producing a different graph on repeated runs over the same input: alignment hits were collected in thread completion order, which decided how equally scoring merges were broken apart. ## 1.3.0 diff --git a/docs/docs/reference.md b/docs/docs/reference.md index e9572c6b..6404ff72 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -153,12 +153,12 @@ Merge two pangenome graphs into a single one * `` — Path to the first input graph, in pangraph JSON format. - This graph is treated as the base: its block, node and path identifiers are preserved in the output, while those of the second graph are renumbered. When extending an existing graph with new genomes, pass the existing graph here. + This graph is treated as the base: its identifiers are preserved in the output, and the path identifiers of the second graph are renumbered to follow them. When extending an existing graph with new genomes, pass the existing graph here. Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. The decompressor is chosen based on the file extension. * `` — Path to the second input graph, in pangraph JSON format. - Its identifiers are renumbered so that they do not clash with those of the first graph. Its genomes appear after those of the first graph in the output. + Its path identifiers are renumbered to follow those of the first graph, so its genomes appear after them in the output. Block and node identifiers are left alone: they are derived from genome names, which must be distinct across the two graphs, so they cannot clash. ###### **Options:** diff --git a/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs b/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs index 348e4fb9..feab7d53 100644 --- a/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs +++ b/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs @@ -61,8 +61,13 @@ fn align_with_minimap2_lib_impl( let idx = Minimap2Index::new(&seqs, &names, &args)?; + // Collected into a `Vec` first so that the parallel iterator is *indexed*: `par_bridge` hands + // work out in whatever order threads ask for it and collects in that order, which made the hit + // list vary between runs. `filter_matches` then broke equal-energy ties differently, so the same + // input could build different graphs. `Vec::into_par_iter` restores input order. let results: Vec = izip!(&seqs, &names) - .par_bridge() + .collect_vec() + .into_par_iter() .map_init( || Minimap2Mapper::new(&idx).unwrap(), move |mapper, (seq, name)| { diff --git a/packages/pangraph/src/circularize/merge_blocks.rs b/packages/pangraph/src/circularize/merge_blocks.rs index 7035fa5d..85bf76ee 100644 --- a/packages/pangraph/src/circularize/merge_blocks.rs +++ b/packages/pangraph/src/circularize/merge_blocks.rs @@ -51,7 +51,7 @@ fn orient_merging_edge(graph: &Pangraph, edge: &Edge) -> Edge { fn find_node_pairings(graph: &Pangraph, edge: &Edge) -> (BTreeMap, BTreeMap) { let mut node_pairings = btreemap! {}; let mut new_nodes = btreemap! {}; - for (&path_id, path) in &graph.paths { + for path in graph.paths.values() { let n = path.nodes.len(); let i = if path.circular { n } else { n - 1 }; for idx in 0..i { @@ -79,7 +79,7 @@ fn find_node_pairings(graph: &Pangraph, edge: &Edge) -> (BTreeMap Result<(), Report> { merge_cmd_preliminary_checks(args, &left, &right).wrap_err("When performing preliminary checks before merging")?; - // The two graphs were built independently, so their identifiers almost certainly collide. - // Namespace the second graph before joining them. + // Block and node ids are derived from genome names, which the checks above established are + // distinct across the two graphs, so they cannot collide. Path ids are sequential within each + // graph, so the appended graph is lifted above the first one. let right = right - .make_disjoint_from(&left) - .wrap_err("When making the identifiers of the two input graphs disjoint")?; + .renumber_paths(left.path_id_upper_bound()) + .wrap_err("When renumbering the path ids of the second input graph")?; + + // Cheap, and the alternative is `graph_join` panicking on the conflicting key. + if !right.is_id_disjoint_from(&left) { + return make_error!( + "The two input graphs share block or node identifiers, so they cannot be joined. Identifiers are derived from genome names since version 1.4; graphs written by earlier versions derive them from the order of the input sequences instead, and two such graphs collide. Rebuild the input graphs with the current version of pangraph." + ); + } info!( "=== Graph merging start: graph sizes {} + {}", diff --git a/packages/pangraph/src/commands/simplify/simplify_run.rs b/packages/pangraph/src/commands/simplify/simplify_run.rs index 817ae420..f38d5a17 100644 --- a/packages/pangraph/src/commands/simplify/simplify_run.rs +++ b/packages/pangraph/src/commands/simplify/simplify_run.rs @@ -54,8 +54,11 @@ mod tests { use maplit::{btreemap, btreeset}; use pretty_assertions::assert_eq; - const NID11: NodeId = NodeId(13172209629052542373); - const NID12: NodeId = NodeId(16864511183100055928); + /// Ids of the two nodes that concatenating the blocks of each path produces, as minted by + /// [`PangraphNode::with_derived_id`]. They are hashes of the new block id, the genome's seed and + /// the node's strand and position, so renaming `pathA`/`pathB` below changes them. + const NID11: NodeId = NodeId(7517044205545980976); + const NID12: NodeId = NodeId(8586529204744949647); fn block_a() -> PangraphBlock { // 0 1 2 3 diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index df0a6fe3..ec41c687 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -10,9 +10,8 @@ use crate::representation::seq::Seq; use crate::tree::clade::WithNewickName; use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; -use crate::{make_internal_error, make_internal_report, make_report}; +use crate::{make_internal_report, make_report}; use eyre::{Report, WrapErr}; -use log::warn; use maplit::btreemap; use schemars::JsonSchema; use serde::{Deserialize, Serialize}; @@ -30,8 +29,16 @@ pub struct Pangraph { impl Pangraph { pub fn singleton(fasta: FastaRecord, strand: Strand, circular: bool) -> Self { let tot_len = fasta.seq.len(); - let node_id = NodeId(fasta.index); - let block_id = BlockId(fasta.index); + // Block and node ids are seeded from the genome name rather than from the record index, so that + // two graphs built independently cannot collide: names are unique within a build, and `merge` + // rejects graphs that share one. Seeding from the index instead made every build start at + // `0, 1, 2, ...`, so merging two graphs required relabeling one of them first. + // + // Path ids stay sequential: they double as the ordering index of the genomes, and keying the + // `paths` map by a hash would scramble genome order in every output. + let seed = id(&fasta.seq_name); + let node_id = NodeId(seed); + let block_id = BlockId(seed); let block = PangraphBlock::from_consensus(fasta.seq, block_id, node_id); let path_id = PathId(fasta.index); let node_position = if circular { (0, 0) } else { (0, tot_len) }; // path wraps around if circular @@ -67,50 +74,35 @@ impl Pangraph { self.blocks.values().map(|block| block.consensus()) } - /// Returns this graph with every block, node and path id re-derived. + /// Returns this graph with its path ids renumbered contiguously from `offset`, preserving their + /// relative order. /// - /// Block and node ids become `id((salt, old_id))`; path ids are renumbered contiguously from - /// `path_id_offset`, preserving their relative order (path ids double as the ordering index of - /// the genomes, so they are kept small and sequential rather than hashed). + /// Blocks are untouched and node ids are preserved: only the `path_id` a node stores, and the id + /// of the path itself, are rewritten. Node ids are derived from [`PangraphPath::seed`] rather + /// than from the path id (see [`PangraphNode::with_derived_id`]), so renumbering cannot put a + /// node id out of step with the contents it was derived from. /// - /// Consensuses, edits, names, descriptions, strands and positions are moved over untouched: the - /// relabeled graph describes exactly the same sequences as the original one. The graph is taken - /// by value so that the sequences can be moved rather than copied. - pub fn relabel(self, salt: usize, path_id_offset: usize) -> Result { + /// Used by `merge` to lift one graph's genomes above the other's: path ids are sequential within + /// each graph, so two graphs built independently always collide on them. The graph is taken by + /// value so that the sequences can be moved rather than copied. + /// + /// Errors if a node refers to a path the graph does not contain, which can only happen in a graph + /// that pangraph did not write. + pub fn renumber_paths(self, offset: usize) -> Result { let Self { paths, blocks, nodes } = self; - let (n_blocks, n_nodes, n_paths) = (blocks.len(), nodes.len(), paths.len()); - - let block_map: BTreeMap = blocks.keys().map(|&bid| (bid, BlockId(id((salt, bid))))).collect(); - - let node_map: BTreeMap = nodes.keys().map(|&nid| (nid, NodeId(id((salt, nid))))).collect(); let path_map: BTreeMap = paths .keys() .enumerate() - .map(|(rank, &pid)| (pid, PathId(path_id_offset + rank))) + .map(|(rank, &pid)| (pid, PathId(offset + rank))) .collect(); - // The three maps above are keyed by the *map keys* of this graph, so looking an entity's own new - // id up by its key cannot miss. Ids taken from an entity's *contents* are a different matter: - // they can dangle, or disagree with the key they are stored under, in a graph that pangraph did - // not write, so those lookups are fallible. - let blocks: BTreeMap = blocks - .into_iter() - .map(|(bid, block)| { - let block = block.relabel(block_map[&bid], &node_map)?; - Ok((block.id(), block)) - }) - .collect::>()?; - + // `path_map` is keyed by the map keys of this graph, so looking a path's own new id up by its + // key cannot miss. The path id read off a node's *contents* is a different matter: it can + // dangle in a graph that pangraph did not write, so that lookup is fallible. let nodes: BTreeMap = nodes .into_iter() .map(|(nid, node)| { - let block_id = *block_map.get(&node.block_id()).ok_or_else(|| { - make_report!( - "Node {nid} refers to block {}, which the graph does not contain", - node.block_id() - ) - })?; let path_id = *path_map.get(&node.path_id()).ok_or_else(|| { make_report!( "Node {nid} refers to path {}, which the graph does not contain", @@ -118,8 +110,8 @@ impl Pangraph { ) })?; - let node = PangraphNode::new(Some(node_map[&nid]), block_id, path_id, node.strand(), node.position()); - Ok((node.id(), node)) + let node = PangraphNode::new(Some(nid), node.block_id(), path_id, node.strand(), node.position()); + Ok((nid, node)) }) .collect::>()?; @@ -127,73 +119,32 @@ impl Pangraph { .into_iter() .map(|(pid, mut path)| { path.id = path_map[&pid]; - for nid in &mut path.nodes { - let old = *nid; - *nid = *node_map - .get(&old) - .ok_or_else(|| make_report!("Path {pid} refers to node {old}, which the graph does not contain"))?; - } - Ok((path.id, path)) + (path.id, path) }) - .collect::>()?; - - // An injective relabeling cannot change the number of entities. If it did, two distinct ids - // were mapped onto the same one and entities were silently dropped. - if (blocks.len(), nodes.len(), paths.len()) != (n_blocks, n_nodes, n_paths) { - return make_internal_error!( - "When relabeling graph ids: expected {n_blocks} blocks, {n_nodes} nodes and {n_paths} paths, but got {} blocks, {} nodes and {} paths", - blocks.len(), - nodes.len(), - paths.len(), - ); - } + .collect(); Ok(Self { paths, blocks, nodes }) } /// Returns true if this graph shares no block, node or path id with `other`. + /// + /// Block and node ids are derived from genome names, which `merge` requires to be distinct across + /// the two graphs, so this holds by construction for graphs written by pangraph 1.4 or later. + /// Graphs written by earlier versions derive their ids from the input order instead, and two of + /// those do collide, which is what `merge` uses this to detect. pub fn is_id_disjoint_from(&self, other: &Self) -> bool { self.blocks.keys().all(|bid| !other.blocks.contains_key(bid)) && self.nodes.keys().all(|nid| !other.nodes.contains_key(nid)) && self.paths.keys().all(|pid| !other.paths.contains_key(pid)) } - /// Relabels this graph so that it shares no identifier with `other`, which is left untouched. + /// Returns the smallest path id that is above every path id of this graph. /// - /// Two graphs built independently always collide: `Pangraph::singleton` labels the first genome - /// of every build with path, block and node id `0`, and blocks that never merge keep that id all - /// the way to the final graph. Merging therefore requires namespacing one of the two graphs - /// first. - pub fn make_disjoint_from(self, other: &Self) -> Result { - const MAX_ATTEMPTS: usize = 8; - - // Path ids are assigned above every path id of `other`, so they cannot collide by construction. - let path_id_offset = other.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0); - - // The salt must differ from the one used by any earlier merge whose relabeled ids survive into - // `other`, otherwise those ids get re-derived a second time and land on themselves. The path id - // offset provides that: a merger salted with `offset(P)` produces a graph with at least one more - // genome than `P`, so a later merge with that graph on the left uses a strictly larger offset. - let mut relabeled = self; - for attempt in 0..MAX_ATTEMPTS { - relabeled = relabeled.relabel(id((path_id_offset, attempt)), path_id_offset)?; - - if relabeled.is_id_disjoint_from(other) { - return Ok(relabeled); - } - - // Relabeling is a composition of injective maps, so retrying on top of the previous attempt is - // safe, and path ids are assigned by rank rather than derived from the previous id, so they do - // not drift. Reaching this point needs either a genuine hash collision, or an operation that - // breaks the monotonicity of the offset: `simplify` drops paths without renumbering, so - // `build -> merge -> merge -> simplify -> merge` can bring the offset back to a value already - // used as a salt. - warn!("Identifier collision when relabeling graph ids (attempt {attempt}); retrying"); - } - - make_internal_error!( - "When making graphs id-disjoint: no collision-free relabeling of block and node ids found after {MAX_ATTEMPTS} attempts" - ) + /// The offset `merge` renumbers the appended graph from, so that the two sets of path ids cannot + /// overlap. Not simply the number of paths: `simplify` drops paths without renumbering, so path + /// ids are not necessarily contiguous. + pub fn path_id_upper_bound(&self) -> usize { + self.paths.keys().map(|pid| pid.0 + 1).max().unwrap_or(0) } pub fn update(&mut self, u: &GraphUpdate) { @@ -615,77 +566,51 @@ mod tests { assert_eq!(g.newick_name(), expected); } - /// Builds a two-genome graph whose ids are the small sequential integers that `build` assigns to - /// blocks that never merge. Two such graphs collide on every single id. - fn colliding_graph(names: [&str; 2]) -> Pangraph { + /// Builds a two-genome graph the way `build` would, with block and node ids seeded from the + /// genome names. Two such graphs collide only on their path ids. + fn two_genome_graph(names: [&str; 2]) -> Pangraph { + let seeds = names.map(|name| id(name.to_owned())); + let (b0, b1) = (BlockId(seeds[0]), BlockId(seeds[1])); + let (n0, n1) = (NodeId(seeds[0]), NodeId(seeds[1])); + let blocks = btreemap! { - BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap!{ NodeId(0) => Edit::empty() }), - BlockId(1) => PangraphBlock::new(BlockId(1), "TTTTGGGG", btreemap!{ NodeId(1) => Edit::empty() }), + b0 => PangraphBlock::new(b0, "ACGTACGT", btreemap!{ n0 => Edit::empty() }), + b1 => PangraphBlock::new(b1, "TTTTGGGG", btreemap!{ n1 => Edit::empty() }), }; let nodes = btreemap! { - NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Reverse, (0, 8)), + n0 => PangraphNode::new(Some(n0), b0, PathId(0), Forward, (0, 8)), + n1 => PangraphNode::new(Some(n1), b1, PathId(1), Reverse, (0, 8)), }; let paths = btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, Some(names[0].to_owned()), None), - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 8, false, Some(names[1].to_owned()), None), + PathId(0) => PangraphPath::new(Some(PathId(0)), [n0], 8, false, Some(names[0].to_owned()), None), + PathId(1) => PangraphPath::new(Some(PathId(1)), [n1], 8, false, Some(names[1].to_owned()), None), }; Pangraph { paths, blocks, nodes } } /// A graph read from JSON is keyed by the object key, while each entity also stores its own id; - /// serde never checks that the two agree. `relabel` resolves entities by key, so a disagreement - /// used to abort the process with a bare "no entry found for key" panic and no indication of - /// which file was at fault. It must be a reportable error instead. - #[rstest] - fn test_relabel_reports_block_stored_under_a_mismatched_id() { - let mut graph = colliding_graph(["a", "b"]); - let block = graph.blocks.remove(&BlockId(0)).unwrap(); - graph.blocks.insert(BlockId(42), block); // key 42, but the block still reports id 0 - - let err = report_to_string(&graph.clone().relabel(1, 7).unwrap_err()); - assert!(err.contains("does not contain"), "unexpected error: {err}"); - - // `sanity_check` never compared key against stored id, so it used to pass this graph through. - let err = report_to_string(&graph.sanity_check().unwrap_err()); - assert!(err.contains("stored under id 42"), "unexpected error: {err}"); - assert!(err.contains("reports id 0"), "unexpected error: {err}"); - } - - #[rstest] - fn test_relabel_reports_path_referring_to_a_missing_node() { - let mut graph = colliding_graph(["a", "b"]); - graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; - - let err = report_to_string(&graph.relabel(1, 7).unwrap_err()); - assert!( - err.contains("Path 0 refers to node 99, which the graph does not contain"), - "unexpected error: {err}" - ); - } - + /// serde never checks that the two agree. `renumber_paths` resolves paths by key, so a + /// disagreement would otherwise abort the process with a bare "no entry found for key" panic and + /// no indication of which file was at fault. #[rstest] - fn test_relabel_reports_node_referring_to_a_missing_block() { - let mut graph = colliding_graph(["a", "b"]); - let node = graph.nodes.get_mut(&NodeId(0)).unwrap(); - *node = PangraphNode::new( - Some(NodeId(0)), - BlockId(99), - node.path_id(), - node.strand(), - node.position(), - ); - - let err = report_to_string(&graph.relabel(1, 7).unwrap_err()); + fn test_renumber_paths_reports_node_referring_to_a_missing_path() { + let mut graph = two_genome_graph(["a", "b"]); + let nid = graph.node_ids().next().unwrap(); + let node = &graph.nodes[&nid]; + let detached = PangraphNode::new(Some(nid), node.block_id(), PathId(99), node.strand(), node.position()); + graph.nodes.insert(nid, detached); + + let err = report_to_string(&graph.renumber_paths(7).unwrap_err()); assert!( - err.contains("Node 0 refers to block 99, which the graph does not contain"), + err.contains("refers to path 99, which the graph does not contain"), "unexpected error: {err}" ); } #[rstest] - fn test_relabel_keeps_graph_consistent() { - let graph = colliding_graph(["a", "b"]).relabel(1, 7).unwrap(); + fn test_renumber_paths_keeps_graph_consistent() { + let original = two_genome_graph(["a", "b"]); + let graph = original.clone().renumber_paths(7).unwrap(); graph.sanity_check().unwrap(); assert_eq!(graph.blocks.len(), 2); @@ -699,15 +624,20 @@ mod tests { vec![Some(o!("a")), Some(o!("b"))] ); - // block and node ids are re-derived, and no longer the original small integers - assert!(graph.block_ids().all(|bid| bid.0 > 1)); - assert!(graph.node_ids().all(|nid| nid.0 > 1)); + // blocks and node ids are untouched: only the path id a node stores is rewritten + assert_eq!(graph.blocks, original.blocks); + assert_eq!(graph.node_ids().collect_vec(), original.node_ids().collect_vec()); + for path in graph.paths.values() { + for nid in path.nodes() { + assert_eq!(graph.nodes[nid].path_id(), path.id()); + } + } } #[rstest] - fn test_relabel_preserves_sequences() { - let original = colliding_graph(["a", "b"]); - let relabeled = original.clone().relabel(3, 0).unwrap(); + fn test_renumber_paths_preserves_sequences() { + let original = two_genome_graph(["a", "b"]); + let renumbered = original.clone().renumber_paths(5).unwrap(); let seqs = |g: &Pangraph| { reconstruct(g) @@ -716,34 +646,47 @@ mod tests { .unwrap() }; - assert_eq!(seqs(&original), seqs(&relabeled)); + assert_eq!(seqs(&original), seqs(&renumbered)); } + /// `simplify` drops paths without renumbering the survivors, so the offset cannot be taken to be + /// the number of paths. #[rstest] - fn test_relabel_is_deterministic() { - let first = colliding_graph(["a", "b"]).relabel(2, 5).unwrap(); - let second = colliding_graph(["a", "b"]).relabel(2, 5).unwrap(); - assert_eq!(first, second); + fn test_path_id_upper_bound_clears_sparse_path_ids() { + let mut graph = two_genome_graph(["a", "b"]); + assert_eq!(graph.path_id_upper_bound(), 2); + + graph.paths.remove(&PathId(0)); + assert_eq!(graph.paths.len(), 1); + assert_eq!(graph.path_id_upper_bound(), 2); } #[rstest] - fn test_make_disjoint_from() { - let left = colliding_graph(["a", "b"]); - let right = colliding_graph(["c", "d"]); - - // the two graphs share every single id before relabeling - assert!(!right.is_id_disjoint_from(&left)); + fn test_path_id_upper_bound_of_an_empty_graph() { + let graph = Pangraph { + paths: btreemap! {}, + blocks: btreemap! {}, + nodes: btreemap! {}, + }; + assert_eq!(graph.path_id_upper_bound(), 0); + } - let right = right.make_disjoint_from(&left).unwrap(); + /// The property the whole identifier model rests on: two graphs built from differently named + /// genomes share no block or node id, so `merge` can join them without relabeling either. + #[rstest] + fn test_graphs_with_distinct_genome_names_are_id_disjoint() { + let left = two_genome_graph(["a", "b"]); + let right = two_genome_graph(["c", "d"]) + .renumber_paths(left.path_id_upper_bound()) + .unwrap(); assert!(right.is_id_disjoint_from(&left)); right.sanity_check().unwrap(); - // the left graph is untouched, and the right graph's genomes follow it assert_eq!(left.path_ids().collect_vec(), vec![PathId(0), PathId(1)]); assert_eq!(right.path_ids().collect_vec(), vec![PathId(2), PathId(3)]); - // joining the two no longer conflicts + // joining the two does not conflict let joined = crate::pangraph::graph_merging::graph_join(&left, &right); joined.sanity_check().unwrap(); assert_eq!(joined.paths.len(), 4); @@ -751,17 +694,20 @@ mod tests { assert_eq!(joined.nodes.len(), 4); } - /// Appending to a graph that already absorbed a relabeled graph. With a constant salt the ids of - /// the third graph were re-derived exactly onto those the second one left behind, so the second - /// append always failed. Every graph here carries the same small ids, which is what `build` - /// assigns to blocks and nodes that never merge. + /// Appending to a graph that already absorbed another one. This needed a varying relabeling salt + /// back when block and node ids were re-derived on merge; deriving them from genome names instead + /// makes it hold with no bookkeeping at all. #[rstest] - fn test_make_disjoint_from_after_a_previous_merge() { - let first = colliding_graph(["a", "b"]); - let second = colliding_graph(["c", "d"]).make_disjoint_from(&first).unwrap(); + fn test_appending_to_an_already_merged_graph() { + let first = two_genome_graph(["a", "b"]); + let second = two_genome_graph(["c", "d"]) + .renumber_paths(first.path_id_upper_bound()) + .unwrap(); let joined = crate::pangraph::graph_merging::graph_join(&first, &second); - let third = colliding_graph(["e", "f"]).make_disjoint_from(&joined).unwrap(); + let third = two_genome_graph(["e", "f"]) + .renumber_paths(joined.path_id_upper_bound()) + .unwrap(); assert!(third.is_id_disjoint_from(&joined)); third.sanity_check().unwrap(); @@ -774,4 +720,48 @@ mod tests { assert_eq!(joined.blocks.len(), 6); assert_eq!(joined.nodes.len(), 6); } + + /// Two graphs holding the same genome do collide, which is why `merge` rejects them by name + /// before it ever gets as far as joining them. + #[rstest] + fn test_graphs_sharing_a_genome_name_are_not_id_disjoint() { + let left = two_genome_graph(["a", "b"]); + let right = two_genome_graph(["a", "c"]) + .renumber_paths(left.path_id_upper_bound()) + .unwrap(); + + assert!(!right.is_id_disjoint_from(&left)); + } + + /// Ids no longer depend on the order the input sequences were read in, only on their names. + #[rstest] + fn test_singleton_ids_are_seeded_from_the_name_not_the_index() { + let singleton = |name: &str, index: usize| { + Pangraph::singleton( + FastaRecord { + seq_name: name.to_owned(), + desc: None, + seq: Seq::from_str("ACGTACGT"), + index, + }, + Forward, + false, + ) + }; + + // the same genome read at a different position in the input gets the same block and node ids + let first = singleton("a", 0); + let shifted = singleton("a", 7); + assert_eq!(first.block_ids().collect_vec(), shifted.block_ids().collect_vec()); + assert_eq!(first.node_ids().collect_vec(), shifted.node_ids().collect_vec()); + + // but the path id still records where it was read, so genome order survives + assert_eq!(first.path_ids().collect_vec(), vec![PathId(0)]); + assert_eq!(shifted.path_ids().collect_vec(), vec![PathId(7)]); + + // and a different genome read at the same position gets different block and node ids + let other = singleton("b", 0); + assert_ne!(first.block_ids().collect_vec(), other.block_ids().collect_vec()); + assert_ne!(first.node_ids().collect_vec(), other.node_ids().collect_vec()); + } } diff --git a/packages/pangraph/src/pangraph/pangraph_block.rs b/packages/pangraph/src/pangraph/pangraph_block.rs index 81f838ac..384bbf0e 100644 --- a/packages/pangraph/src/pangraph/pangraph_block.rs +++ b/packages/pangraph/src/pangraph/pangraph_block.rs @@ -3,6 +3,7 @@ use crate::align::map_variations::{BandParameters, map_variations}; use crate::io::fasta::FastaRecord; use crate::io::json::{JsonPretty, json_write_str}; use crate::io::seq::reverse_complement; +use crate::make_internal_error; use crate::pangraph::edits::{Del, Edit, Ins, Sub}; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::pangraph_node::NodeId; @@ -10,7 +11,6 @@ use crate::pangraph::pangraph_path::PathId; use crate::representation::seq::Seq; use crate::utils::collections::has_duplicates; use crate::utils::interval::positions_to_intervals; -use crate::{make_internal_error, make_report}; use derive_more::{Display, From}; use eyre::{Report, WrapErr}; use getset::{CopyGetters, Getters}; @@ -60,30 +60,6 @@ impl PangraphBlock { } } - /// Returns this block with a new id and its alignment keys rewritten through `node_map`. - /// The consensus and the edits are moved over unchanged. - /// - /// Errors if an alignment is keyed by a node the graph does not contain, which can only happen in - /// a graph that pangraph did not write. - pub fn relabel(self, id: BlockId, node_map: &BTreeMap) -> Result { - let alignments = self - .alignments - .into_iter() - .map(|(nid, edit)| { - let new_nid = *node_map - .get(&nid) - .ok_or_else(|| make_report!("Block {id} aligns node {nid}, which the graph does not contain"))?; - Ok((new_nid, edit)) - }) - .collect::>()?; - - Ok(Self { - id, - consensus: self.consensus, - alignments, - }) - } - pub fn reverse_complement(&self) -> Result { let rev_cons = reverse_complement(&self.consensus)?; diff --git a/packages/pangraph/src/pangraph/pangraph_node.rs b/packages/pangraph/src/pangraph/pangraph_node.rs index 225a040d..91498ce9 100644 --- a/packages/pangraph/src/pangraph/pangraph_node.rs +++ b/packages/pangraph/src/pangraph/pangraph_node.rs @@ -1,5 +1,5 @@ use crate::pangraph::pangraph_block::BlockId; -use crate::pangraph::pangraph_path::PathId; +use crate::pangraph::pangraph_path::{PangraphPath, PathId}; use crate::pangraph::strand::Strand; use crate::utils::id::id; use derive_more::{Display, From}; @@ -52,6 +52,21 @@ impl PangraphNode { } } + /// Creates a node placing `block_id` on `path`, with an id derived from its contents. + /// + /// Genomes are discriminated by [`PangraphPath::seed`] rather than by their path id, so the + /// resulting id does not depend on the order in which the input sequences were read. The path id + /// is still what the node stores. + pub fn with_derived_id(block_id: BlockId, path: &PangraphPath, strand: Strand, position: (usize, usize)) -> Self { + Self { + id: id((&block_id, &path.seed(), &strand, &position)), + block_id, + path_id: path.id(), + strand, + position, + } + } + // this is almost equivalent to checking if the node is empty // except for an edge case: when a circular path contains only // one node. In this case even if the node is not empty, the diff --git a/packages/pangraph/src/pangraph/pangraph_path.rs b/packages/pangraph/src/pangraph/pangraph_path.rs index 0c19a10a..6c5e55a6 100644 --- a/packages/pangraph/src/pangraph/pangraph_path.rs +++ b/packages/pangraph/src/pangraph/pangraph_path.rs @@ -51,4 +51,19 @@ impl PangraphPath { desc, } } + + /// Order-independent identity of the genome on this path, used as the discriminator when deriving + /// node ids. + /// + /// Taken from the genome name rather than from the path id, so that node ids do not depend on the + /// order in which the input sequences were read, and do not change when `renumber_paths` shifts + /// path ids during a merge. Genome names are unique within a graph, and `merge` rejects graphs + /// that share one, so the seed identifies a genome as well as the path id does. + /// + /// Recomputed on demand rather than stored: a stored field would have to be kept out of the JSON + /// and then recomputed on load anyway, and silently defaults to the same value for every path if + /// that is forgotten. Falls back to the path id for unnamed paths, which pangraph never writes. + pub fn seed(&self) -> usize { + self.name.as_ref().map_or(self.id.0, id) + } } diff --git a/packages/pangraph/src/pangraph/slice.rs b/packages/pangraph/src/pangraph/slice.rs index 620f9458..98e8fd50 100644 --- a/packages/pangraph/src/pangraph/slice.rs +++ b/packages/pangraph/src/pangraph/slice.rs @@ -167,16 +167,17 @@ pub fn block_slice( old_strandedness }; - let path_L = G.paths[&old_node.path_id()].tot_len; + let path = &G.paths[&old_node.path_id()]; + let path_L = path.tot_len; let node_coords = interval_node_coords(i, edits, block_L); - let circular = G.paths[&old_node.path_id()].circular(); + let circular = path.circular(); let new_pos = if circular { new_position_circular(old_node.position(), node_coords, path_L, old_strandedness) } else { new_position_non_circular(old_node.position(), node_coords, old_strandedness) }; - let new_node = PangraphNode::new(None, i.new_block_id, old_node.path_id(), new_strand, new_pos); + let new_node = PangraphNode::with_derived_id(i.new_block_id, path, new_strand, new_pos); // extract edits for the slice let new_edits = slice_edits(i, edits, block_L); diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index f2c0f5f1..7b3bc9cf 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -12,7 +12,11 @@ mod tests { use pangraph::io::fasta::{FastaReader, FastaRecord}; use pangraph::io::json::{JsonPretty, json_write_file}; use pangraph::pangraph::pangraph::Pangraph; + use pangraph::pangraph::pangraph_block::{BlockId, PangraphBlock}; + use pangraph::pangraph::pangraph_node::{NodeId, PangraphNode}; + use pangraph::pangraph::pangraph_path::{PangraphPath, PathId}; use pangraph::pangraph::reconstruct::reconstruct; + use pangraph::pangraph::strand::Strand::Forward; use pangraph::representation::seq::Seq; use pangraph::utils::error::report_to_string; use pretty_assertions::assert_eq; @@ -141,11 +145,12 @@ mod tests { Ok(()) } - /// Appending to a graph that is itself the result of an earlier merge. Identifiers relabeled by - /// the first merge survive into its output whenever a block or node finds no homologue, and a - /// constant relabeling salt then re-derived the third graph's identifiers onto exactly those - /// values, so the second append always failed. The three genomes here are mutually unrelated, so - /// nothing aligns and every identifier survives; homologous appends never hit this. + /// Appending to a graph that is itself the result of an earlier merge. Identifiers minted by the + /// first merge survive into its output whenever a block or node finds no homologue, so this is + /// the case that used to need a varying relabeling salt to keep the third graph's identifiers off + /// them. Deriving identifiers from genome names removes the problem at the source. The three + /// genomes here are mutually unrelated, so nothing aligns and every identifier survives; + /// homologous appends never exercised this. #[rstest] fn itest_merge_appends_to_an_already_merged_graph() -> Result<(), Report> { let dir = tempdir()?; @@ -167,6 +172,109 @@ mod tests { Ok(()) } + /// Maps each genome name to the ids of the blocks its path walks through, in order. Comparing the + /// id *sets* of two graphs would not do: `build` used to label singleton blocks `0, 1, 2, ...` + /// whatever the genome, so the sets matched even when every genome held a different block. + fn blocks_by_genome(graph: &Pangraph) -> BTreeMap> { + graph + .paths + .values() + .map(|path| { + let blocks = path.nodes().iter().map(|nid| graph.nodes[nid].block_id()).collect_vec(); + (path.name().clone().unwrap_or_default(), blocks) + }) + .collect() + } + + /// Block and node identifiers are derived from genome names rather than from the order the input + /// sequences were read in, so the same genomes under the same guide tree produce the same graph + /// whichever order they arrive in. Only path ids, which deliberately record the input order, are + /// expected to differ. + #[rstest] + fn itest_build_ids_do_not_depend_on_input_order() -> Result<(), Report> { + let dir = tempdir()?; + let fastas = read_records("../../data/ges-1.fa", 4)?; + let names = fastas.iter().map(|f| f.seq_name.clone()).collect_vec(); + + // Pinned, so that the input order is the only thing that varies between the two builds. + let newick = dir.path().join("guide.nwk"); + std::fs::write( + &newick, + format!("(({},{}),({},{}));", names[0], names[1], names[2], names[3]), + )?; + + let build_in_order = |fastas: Vec| -> Result { + let args = PangraphBuildArgs { + circular: false, + guide_tree: Some(newick.clone()), + ..PangraphBuildArgs::default() + }; + build(fastas, &args, true) + }; + + // Reversed *and* re-indexed, which is what reading the same genomes from a reordered file does. + let mut backwards = fastas.clone(); + backwards.reverse(); + for (index, record) in backwards.iter_mut().enumerate() { + record.index = index; + } + + let forward = build_in_order(fastas)?; + let reversed = build_in_order(backwards)?; + + assert_eq!(blocks_by_genome(&forward), blocks_by_genome(&reversed)); + assert_eq!( + forward.node_ids().collect::>(), + reversed.node_ids().collect::>() + ); + assert_eq!(sequences(&forward)?, sequences(&reversed)?); + + // path ids still record the input order, so the genomes come back in the order they were read + assert_eq!(forward.path_names().flatten().collect_vec(), names); + assert_eq!( + reversed.path_names().flatten().collect_vec(), + names.iter().rev().collect_vec() + ); + + Ok(()) + } + + /// A graph written by pangraph 1.3 or earlier labels its first genome with block, node and path + /// id `0`, whatever that genome is called, so two of them collide even though their names differ. + /// `merge` has to say so: without the check, `graph_join` panics on the conflicting key. + #[rstest] + fn itest_merge_rejects_graphs_with_colliding_identifiers() -> Result<(), Report> { + let dir = tempdir()?; + + // A singleton graph in the pre-1.4 identifier scheme, where ids came from the record index. + let legacy_graph = |name: &str, seq: &str| { + let (bid, nid, pid) = (BlockId(0), NodeId(0), PathId(0)); + Pangraph { + blocks: BTreeMap::from([(bid, PangraphBlock::from_consensus(seq, bid, nid))]), + nodes: BTreeMap::from([(nid, PangraphNode::new(Some(nid), bid, pid, Forward, (0, seq.len())))]), + paths: BTreeMap::from([( + pid, + PangraphPath::new(Some(pid), [nid], seq.len(), false, Some(name.to_owned()), None), + )]), + } + }; + + let left = dir.path().join("left.json"); + let right = dir.path().join("right.json"); + json_write_file(&left, &legacy_graph("genome_a", "ACGTACGTAC"), JsonPretty(false))?; + json_write_file(&right, &legacy_graph("genome_b", "TTTTGGGGCC"), JsonPretty(false))?; + + let result = merge_run(&merge_args(left, right, dir.path().join("out.json"))); + + let error = report_to_string(&result.unwrap_err()); + assert!( + error.contains("share block or node identifiers"), + "unexpected error message: {error}" + ); + + Ok(()) + } + /// Merging a graph with itself duplicates every genome name, and must be rejected. #[rstest] fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { From e2a26222c319b4082a1f75b758b55c6c4a1e0922 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Mon, 17 Aug 2026 13:40:46 +0200 Subject: [PATCH 33/50] docs(cli): simplify the help text of the merge input arguments --- docs/docs/reference.md | 8 ++++---- packages/pangraph/src/commands/merge/merge_args.rs | 12 +++++------- 2 files changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/docs/reference.md b/docs/docs/reference.md index 6404ff72..c13b3157 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -151,14 +151,14 @@ Merge two pangenome graphs into a single one ###### **Arguments:** -* `` — Path to the first input graph, in pangraph JSON format. - - This graph is treated as the base: its identifiers are preserved in the output, and the path identifiers of the second graph are renumbered to follow them. When extending an existing graph with new genomes, pass the existing graph here. +* `` — Path to the first input graph, in pangraph JSON format. This graph is treated as the base. Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. The decompressor is chosen based on the file extension. * `` — Path to the second input graph, in pangraph JSON format. - Its path identifiers are renumbered to follow those of the first graph, so its genomes appear after them in the output. Block and node identifiers are left alone: they are derived from genome names, which must be distinct across the two graphs, so they cannot clash. + Its path identifiers are renumbered to follow those of the first graph, so its genomes appear after them in the output. + + Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. The decompressor is chosen based on the file extension. ###### **Options:** diff --git a/packages/pangraph/src/commands/merge/merge_args.rs b/packages/pangraph/src/commands/merge/merge_args.rs index 6094c50f..a77e6175 100644 --- a/packages/pangraph/src/commands/merge/merge_args.rs +++ b/packages/pangraph/src/commands/merge/merge_args.rs @@ -7,11 +7,7 @@ use std::path::PathBuf; /// Merge two pangenome graphs into a single one #[derive(Parser, Debug, SmartDefault)] pub struct PangraphMergeArgs { - /// Path to the first input graph, in pangraph JSON format. - /// - /// This graph is treated as the base: its identifiers are preserved in the output, and the path - /// identifiers of the second graph are renumbered to follow them. When extending an existing - /// graph with new genomes, pass the existing graph here. + /// Path to the first input graph, in pangraph JSON format. This graph is treated as the base. /// /// Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. /// The decompressor is chosen based on the file extension. @@ -22,8 +18,10 @@ pub struct PangraphMergeArgs { /// Path to the second input graph, in pangraph JSON format. /// /// Its path identifiers are renumbered to follow those of the first graph, so its genomes appear - /// after them in the output. Block and node identifiers are left alone: they are derived from - /// genome names, which must be distinct across the two graphs, so they cannot clash. + /// after them in the output. + /// + /// Accepts plain or compressed files. Supported compression formats: `gz`, `bz2`, `xz`, `zstd`. + /// The decompressor is chosen based on the file extension. #[clap(value_hint = ValueHint::FilePath)] #[clap(display_order = 2)] pub right_graph: PathBuf, From a1705ec63aca28ce19ab3662c88dc70cf76c146c Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 09:18:11 +0200 Subject: [PATCH 34/50] docs: fix broken documentation link in pangraph --help --- docs/docs/reference.md | 2 +- packages/pangraph/src/commands/root_args.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/reference.md b/docs/docs/reference.md index e9572c6b..c067fd9a 100644 --- a/docs/docs/reference.md +++ b/docs/docs/reference.md @@ -45,7 +45,7 @@ Finds homology amongst large collections of closely related genomes. The core of Publication: "PanGraph: scalable bacterial pan-genome graph construction." Nicholas Noll, Marco Molari, Richard Neher. Microbial Genomics 9.6 (2023): 001034.; doi: https://doi.org/10.1099/mgen.0.001034 -Documentation: https://pangraph.readthedocs.io/en/stable/ +Documentation: https://docs.pangraph.org/ Source code: https://github.com/neherlab/pangraph diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index 284367ac..e6ee0bde 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -39,7 +39,7 @@ fn styles() -> styling::Styles { /// /// Publication: "PanGraph: scalable bacterial pan-genome graph construction." Nicholas Noll, Marco Molari, Richard Neher. Microbial Genomics 9.6 (2023): 001034.; doi: https://doi.org/10.1099/mgen.0.001034 /// -/// Documentation: https://pangraph.readthedocs.io/en/stable/ +/// Documentation: https://docs.pangraph.org/ /// /// Source code: https://github.com/neherlab/pangraph /// From 2cfcc85f879f674bee976ab423e64a917e6ca017 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 10:06:54 +0200 Subject: [PATCH 35/50] fix(pangraph): report malformed graphs on load instead of panicking --- CHANGELOG.md | 1 + .../pangraph/src/commands/merge/merge_run.rs | 7 +- .../commands/reconstruct/reconstruct_run.rs | 3 +- packages/pangraph/src/pangraph/edits.rs | 79 ++--- packages/pangraph/src/pangraph/pangraph.rs | 288 +++++++++++++++--- packages/pangraph/src/pangraph/reconstruct.rs | 56 +++- packages/pangraph/tests/itest_reconstruct.rs | 39 +++ 7 files changed, 394 insertions(+), 79 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 107a9f35..f56ab647 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. - more strict checks on input sequences ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequence by id and not by order. +- graphs read from a JSON file are now validated before use, in release builds too. A graph whose block, node or path references do not resolve, or whose node positions or alignment edits fall outside the sequences they index, is reported as an error naming the offending file, instead of aborting the process with a panic. ## 1.3.0 diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index 1aff54fd..73b9dc16 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -60,7 +60,12 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { Ok(()) } -/// Reads a pangraph from a JSON file, and checks its internal consistency in debug builds. +/// Reads a pangraph from a JSON file. +/// +/// `from_path` already rejects a graph whose ids do not resolve or whose offsets are out of range, +/// in release builds too. The extra `sanity_check` here adds the semantic invariants on top — that +/// node positions tile the genome — which indicate a bug rather than a bad file, and so are only +/// worth paying for in debug builds. fn read_graph(filepath: &Path) -> Result { let graph = Pangraph::from_path(&Some(filepath))?; diff --git a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs index d980ee95..26f17eea 100644 --- a/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs +++ b/packages/pangraph/src/commands/reconstruct/reconstruct_run.rs @@ -1,6 +1,5 @@ use crate::commands::reconstruct::reconstruct_args::PangraphReconstructArgs; use crate::io::fasta::{FastaReader, FastaRecord, FastaWriter}; -use crate::io::json::json_read_file; use crate::make_error; use crate::pangraph::pangraph::Pangraph; use crate::pangraph::reconstruct::{format_names, path_ids_by_name, reconstruct, reconstruct_genome, verify_genome}; @@ -17,7 +16,7 @@ pub fn reconstruct_run(args: &PangraphReconstructArgs) -> Result<(), Report> { verify, } = &args; - let graph: Pangraph = json_read_file(input_graph)?; + let graph = Pangraph::from_path(input_graph)?; if let Some(verify) = verify { info!("Verifying sequences reconstructed from pangenome graph"); diff --git a/packages/pangraph/src/pangraph/edits.rs b/packages/pangraph/src/pangraph/edits.rs index 8ee48ec8..692ddc1b 100644 --- a/packages/pangraph/src/pangraph/edits.rs +++ b/packages/pangraph/src/pangraph/edits.rs @@ -565,18 +565,54 @@ impl Edit { } } + /// Checks that every edit addresses a position inside a reference sequence of length `len`. + /// + /// This is the subset of [`Self::sanity_check`] that [`Self::apply`] depends on to stay inside + /// array bounds: it indexes the reference by substitution and deletion position, and + /// `Seq::insert_seq` asserts on its index, so an out-of-range edit read from a graph file aborts + /// the process instead of being reported. Kept apart from `sanity_check`, and available in + /// release builds, so that `Pangraph::validate` can run it on every loaded graph: it is linear in + /// the number of edits, whereas `sanity_check` also compares them pairwise. + pub fn check_bounds(&self, len: usize) -> Result<(), Report> { + for sub in &self.subs { + if sub.pos >= len { + return make_error!( + "Substitution position {} is out of bounds for sequence of length {len}", + sub.pos + ); + } + } + + for del in &self.dels { + // `end()` adds the two, which can overflow on values read from a file rather than computed. + let Some(end) = del.pos.checked_add(del.len) else { + return make_error!("Deletion {del:?} has a position and length whose sum overflows"); + }; + if end > len { + return make_error!("Deletion {del:?} is out of bounds for sequence of length {len}"); + } + } + + for ins in &self.inss { + if ins.pos > len { + return make_error!( + "Insertion position {} is out of bounds for sequence of length {len}", + ins.pos + ); + } + } + + Ok(()) + } + #[cfg(any(test, debug_assertions))] pub fn sanity_check(&self, len: usize) -> Result<(), Report> { - let block_interval = Interval::new(0, len); + // Every edit is in range. Shared with `Pangraph::validate`, which checks it in release builds + // too, since it is what keeps `apply` inside the reference sequence. + self.check_bounds(len)?; + // === substitution checks === for sub in &self.subs { - if !block_interval.contains(sub.pos) { - return Err(eyre!( - "Substitution position {} is out of bounds for sequence of length {}", - sub.pos, - len - )); - } if sub.alt == AsciiChar(b'-') { return Err(eyre!("Substitution with deletion character '-' is not allowed")); } @@ -609,22 +645,6 @@ impl Edit { if del.len == 0 { return Err(eyre!("Deletion {:?} has length 0", del)); } - - if !block_interval.contains(del.pos) { - return Err(eyre!( - "Deletion {:?} is out of bounds for sequence of length {}", - del, - len - )); - } - - if del.end() > len { - return Err(eyre!( - "Deletion {:?} is out of bounds for sequence of length {}", - del, - len - )); - } } // check that deletions are non-overlapping @@ -639,17 +659,6 @@ impl Edit { } } - // === insertion checks === - for ins in &self.inss { - if ins.pos > len { - return Err(eyre!( - "Insertion position {} is out of bounds for sequence of length {}", - ins.pos, - len - )); - } - } - Ok(()) } } diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index df0a6fe3..740a18ae 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -10,7 +10,7 @@ use crate::representation::seq::Seq; use crate::tree::clade::WithNewickName; use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; -use crate::{make_internal_error, make_internal_report, make_report}; +use crate::{make_error, make_internal_error, make_internal_report, make_report}; use eyre::{Report, WrapErr}; use log::warn; use maplit::btreemap; @@ -51,10 +51,21 @@ impl Pangraph { } } + /// Reads a graph from a JSON file, or from stdin when no path is given, and validates it. + /// + /// The single entry point through which every command reads a graph, and therefore the one place + /// that has both the graph and the name of the file it came from. [`Self::validate`] runs here so + /// that a malformed file is rejected up front, named, and before any work is done on it, rather + /// than surfacing much later as a failed lookup deep inside an algorithm. pub fn from_path>(filepath: &Option

) -> Result { let reader = open_file_or_stdin(filepath)?; let data = read_reader_to_string(reader).wrap_err("When reading Pangraph JSON")?; - Self::from_str(&data).wrap_err("When parsing Pangraph JSON") + let graph = Self::from_str(&data).wrap_err("When parsing Pangraph JSON")?; + graph.validate().wrap_err_with(|| match filepath { + Some(filepath) => format!("When validating the graph read from '{}'", filepath.as_ref().display()), + None => "When validating the graph read from standard input".to_owned(), + })?; + Ok(graph) } pub fn to_string_pretty(&self) -> Result { @@ -262,78 +273,123 @@ impl Pangraph { } } - #[cfg(any(test, debug_assertions))] - pub fn sanity_check(&self) -> Result<(), Report> { + /// Checks the invariants that let the rest of pangraph resolve graph entities by id without + /// error handling, and that keep sequence reconstruction inside array bounds. + /// + /// Unlike [`Self::sanity_check`] this runs in release builds, because it guards against a + /// malformed *input*: a graph read from a file was not necessarily written by pangraph, so + /// nothing guarantees that its cross-references resolve or that its offsets are in range. Every + /// graph read through [`Self::from_path`] is validated, and that is what lets the code downstream + /// index the maps directly and treat a lookup that fails anyway as an internal error. + /// + /// Deliberately limited to what a bad file can break. It does *not* check that the graph is + /// semantically coherent — that node positions tile the genome, that edits do not overlap — since + /// those are symptoms of a bug in pangraph rather than of a bad input, and are covered by + /// [`Self::sanity_check`] in debug builds. Errors are reported as ordinary user-facing errors for + /// the same reason: the offending graph came from the user. + /// + /// Costs roughly a tenth of the JSON parse it follows, so it is always worth running. + pub fn validate(&self) -> Result<(), Report> { // Each entity is stored under a map key *and* carries its own id. Serde keys the maps by the // JSON object key, so the two can disagree in a graph that pangraph did not write. Everything // that resolves an entity by key while reading its id off the entity itself depends on them // agreeing, so check it first. One integer comparison per entity. for (block_id, block) in &self.blocks { if block.id() != *block_id { - return Err(eyre::eyre!( - "Block is stored under id {block_id} but reports id {}", - block.id() - )); + return make_error!("Block is stored under id {block_id} but reports id {}", block.id()); } } for (node_id, node) in &self.nodes { if node.id() != *node_id { - return Err(eyre::eyre!( - "Node is stored under id {node_id} but reports id {}", - node.id() - )); + return make_error!("Node is stored under id {node_id} but reports id {}", node.id()); } } for (path_id, path) in &self.paths { if path.id() != *path_id { - return Err(eyre::eyre!( - "Path is stored under id {path_id} but reports id {}", - path.id() - )); + return make_error!("Path is stored under id {path_id} but reports id {}", path.id()); } } - for (node_id, node) in &self.nodes { - if !self.blocks.contains_key(&node.block_id()) { - return Err(eyre::eyre!("Block {} not found in graph", node.block_id())); + // Which path walks each node, as claimed by the paths. Collected in one pass because + // `path.nodes` is a `Vec`: scanning it per node would re-read a genome's entire walk once for + // every node of that genome. + let mut walked_by: BTreeMap = BTreeMap::new(); + for (path_id, path) in &self.paths { + for node_id in &path.nodes { + if !self.nodes.contains_key(node_id) { + return make_error!("Node {node_id} from path {path_id} not found in graph"); + } + // A node id may repeat within the walk of one path, which happens for empty nodes, but two + // paths sharing a node would make the node's own `path_id` ambiguous. + if let Some(previous) = walked_by.insert(*node_id, *path_id) { + if previous != *path_id { + return make_error!("Node {node_id} is walked by both path {previous} and path {path_id}"); + } + } } - let block = &self.blocks[&node.block_id()]; + } - if !self.paths.contains_key(&node.path_id()) { - return Err(eyre::eyre!("Path {} not found in graph", node.path_id())); + for (node_id, node) in &self.nodes { + let Some(block) = self.blocks.get(&node.block_id()) else { + return make_error!("Block {} of node {node_id} not found in graph", node.block_id()); + }; + if !block.alignments().contains_key(node_id) { + return make_error!("Node {node_id} not found in block {}", block.id()); } - let path = &self.paths[&node.path_id()]; - if !block.alignments().contains_key(node_id) { - return Err(eyre::eyre!("Node {} not found in block {}", node_id, block.id())); + let Some(path) = self.paths.get(&node.path_id()) else { + return make_error!("Path {} of node {node_id} not found in graph", node.path_id()); + }; + if walked_by.get(node_id) != Some(&node.path_id()) { + return make_error!("Node {node_id} is not in the walk of its path {}", node.path_id()); } - if !path.nodes.contains(node_id) { - return Err(eyre::eyre!("Node {} not found in path {}", node_id, path.id())); + // Reconstruction rotates a genome by the offset of its first node, so an offset reaching past + // the end of that genome would rotate by more than the genome's length. Both ends are + // compared, and `tot_len` itself is a valid offset: the last node of a circular path ends + // where the genome does. + let (start, end) = node.position(); + if start > path.tot_len() || end > path.tot_len() { + return make_error!( + "Node {node_id} has position ({start}, {end}), outside its path {} of total length {}", + path.id(), + path.tot_len() + ); } } for (block_id, block) in &self.blocks { - if block.alignments().is_empty() { - return Err(eyre::eyre!("Block {} has no nodes", block_id)); - } - - for node_id in block.alignments().keys() { + let consensus_len = block.consensus().len(); + for (node_id, edits) in block.alignments() { if !self.nodes.contains_key(node_id) { - return Err(eyre::eyre!("Node {} not found in graph", node_id)); + return make_error!("Node {node_id} of block {block_id} not found in graph"); } + // `Edit::apply` indexes the consensus by edit position while reconstructing this node. + edits + .check_bounds(consensus_len) + .wrap_err_with(|| format!("When checking the alignment of node {node_id} against block {block_id}"))?; } } - for (path_id, path) in &self.paths { - for node_id in &path.nodes { - if !self.nodes.contains_key(node_id) { - return Err(eyre::eyre!("Node {node_id} from path {path_id} not found in graph")); - } + Ok(()) + } + + #[cfg(any(test, debug_assertions))] + pub fn sanity_check(&self) -> Result<(), Report> { + // Referential integrity and the bounds that keep reconstruction in range. Shared with the + // release-build validation of graphs read from a file, since a graph that pangraph just built + // must satisfy at least as much as one it is willing to load. + self.validate()?; + + for (block_id, block) in &self.blocks { + if block.alignments().is_empty() { + return Err(eyre::eyre!("Block {} has no nodes", block_id)); } + } + for (path_id, path) in &self.paths { // // check that there are no duplicated node ids // // currently disabled because this could rarely happen for empty nodes // let mut seen = BTreeSet::new(); @@ -466,7 +522,7 @@ mod tests { use super::*; use crate::o; - use crate::pangraph::edits::Edit; + use crate::pangraph::edits::{Edit, Sub}; use crate::pangraph::pangraph_node::PangraphNode; use crate::pangraph::pangraph_path::PangraphPath; use crate::pangraph::reconstruct::reconstruct; @@ -774,4 +830,158 @@ mod tests { assert_eq!(joined.blocks.len(), 6); assert_eq!(joined.nodes.len(), 6); } + + /// `validate` is the contract every graph read from a file must satisfy, and the reason the code + /// downstream is allowed to resolve ids by direct indexing. Each case below is a malformation + /// that a hand-edited or third-party JSON graph can carry, and that used to reach an indexing + /// panic in a release build, where `sanity_check` is compiled out. + #[rstest] + fn test_validate_accepts_a_well_formed_graph() { + colliding_graph(["a", "b"]).validate().unwrap(); + } + + #[rstest] + fn test_validate_rejects_path_referring_to_a_missing_node() { + let mut graph = colliding_graph(["a", "b"]); + graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Node 99 from path 0 not found in graph"), + "unexpected error: {err}" + ); + } + + #[rstest] + fn test_validate_rejects_node_referring_to_a_missing_block() { + let mut graph = colliding_graph(["a", "b"]); + let node = &graph.nodes[&NodeId(0)]; + graph.nodes.insert( + NodeId(0), + PangraphNode::new( + Some(NodeId(0)), + BlockId(99), + node.path_id(), + node.strand(), + node.position(), + ), + ); + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Block 99 of node 0 not found in graph"), + "unexpected error: {err}" + ); + } + + #[rstest] + fn test_validate_rejects_node_referring_to_a_missing_path() { + let mut graph = colliding_graph(["a", "b"]); + let node = &graph.nodes[&NodeId(0)]; + graph.nodes.insert( + NodeId(0), + PangraphNode::new( + Some(NodeId(0)), + node.block_id(), + PathId(99), + node.strand(), + node.position(), + ), + ); + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Path 99 of node 0 not found in graph"), + "unexpected error: {err}" + ); + } + + /// A node whose path exists but does not walk it: reconstruction would never emit this node, and + /// its `path_id` claims a genome it is not part of. + #[rstest] + fn test_validate_rejects_node_missing_from_the_walk_of_its_path() { + let mut graph = colliding_graph(["a", "b"]); + graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![]; + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Node 0 is not in the walk of its path 0"), + "unexpected error: {err}" + ); + } + + /// Node ids are unique across the graph, so a node claimed by two walks makes its own `path_id` + /// meaningless, and would have the node reconstructed into two different genomes. + #[rstest] + fn test_validate_rejects_node_walked_by_two_paths() { + let mut graph = colliding_graph(["a", "b"]); + graph.paths.get_mut(&PathId(1)).unwrap().nodes = vec![NodeId(0), NodeId(1)]; + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Node 0 is walked by both path 0 and path 1"), + "unexpected error: {err}" + ); + } + + #[rstest] + fn test_validate_rejects_node_missing_from_the_alignments_of_its_block() { + let mut graph = colliding_graph(["a", "b"]); + graph + .blocks + .insert(BlockId(0), PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap! {})); + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!(err.contains("Node 0 not found in block 0"), "unexpected error: {err}"); + } + + /// Reconstruction rotates a genome by the offset of its first node. An offset past the end of the + /// genome used to reach `rotate_right`, which panics rather than reporting. + #[rstest] + fn test_validate_rejects_node_position_beyond_the_length_of_its_path() { + let mut graph = colliding_graph(["a", "b"]); + let node = &graph.nodes[&NodeId(0)]; + graph.nodes.insert( + NodeId(0), + PangraphNode::new( + Some(NodeId(0)), + node.block_id(), + node.path_id(), + node.strand(), + (100, 8), + ), + ); + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Node 0 has position (100, 8), outside its path 0 of total length 8"), + "unexpected error: {err}" + ); + } + + /// `Edit::apply` indexes the consensus by edit position, so an out-of-range edit used to panic + /// while reconstructing the block, even though `apply` returns a `Result`. + #[rstest] + fn test_validate_rejects_edit_beyond_the_consensus_of_its_block() { + let mut graph = colliding_graph(["a", "b"]); + let edit = Edit { + subs: vec![Sub::new(99, 'T')], + dels: vec![], + inss: vec![], + }; + graph.blocks.insert( + BlockId(0), + PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap! { NodeId(0) => edit }), + ); + + let err = report_to_string(&graph.validate().unwrap_err()); + assert!( + err.contains("Substitution position 99 is out of bounds for sequence of length 8"), + "unexpected error: {err}" + ); + assert!( + err.contains("alignment of node 0 against block 0"), + "unexpected error: {err}" + ); + } } diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index f6c5e47b..9a7920bb 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -6,7 +6,7 @@ use crate::pangraph::pangraph_path::{PangraphPath, PathId}; use crate::representation::seq::Seq; use crate::utils::collections::find_duplicates; use crate::utils::string::str_slice_safe; -use crate::{make_error, make_internal_report}; +use crate::{make_error, make_internal_error, make_internal_report}; use eyre::{Report, WrapErr}; use itertools::Itertools; use std::collections::{BTreeMap, BTreeSet}; @@ -304,9 +304,20 @@ pub(crate) fn format_names>(names: &[S]) -> String { } } +/// Reconstructs the genome of a path, by concatenating the sequences of its nodes and rotating the +/// result so that it starts where the genome does. +/// +/// Both lookups here are guaranteed by [`Pangraph::validate`], which every graph read from a file +/// passes and every graph pangraph builds satisfies, so a failure is a bug rather than bad input. +/// They are still checked, because reporting beats aborting: indexing the node map and rotating by +/// an out-of-range offset both panic. fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result { if let Some(first_node_id) = path.nodes.first() { - let first_node_pos = graph.nodes[first_node_id].position().0; + let first_node = graph + .nodes + .get(first_node_id) + .ok_or_else(|| make_internal_report!("Node {first_node_id} not found in graph"))?; + let first_node_pos = first_node.position().0; let mut genome: Seq = path .nodes @@ -323,6 +334,12 @@ fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result genome.len() { + return make_internal_error!( + "When reconstructing sequences, the first node of the genome starts at position {first_node_pos}, past the end of a genome of length {}", + genome.len() + ); + } genome.rotate_right(first_node_pos); Ok(genome) @@ -641,4 +658,39 @@ mod tests { ); assert_eq!(format_names(&names[..2]), "[g0, g1]"); } + + /// `Pangraph::validate` rejects both of these before reconstruction is ever reached, so they + /// stand for a graph assembled in-process rather than read from a file. They are still worth + /// pinning: both used to be indexing panics, which abort the process instead of being reported, + /// and `sanity_check` is compiled out of release builds. + #[rstest] + fn test_reconstruct_reports_path_referring_to_a_missing_node() { + let mut graph = graph(); + graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; + + let err = report_to_string(&reconstruct_genome(&graph, PathId(0)).unwrap_err()); + assert!(err.contains("Node 99 not found in graph"), "unexpected error: {err}"); + } + + #[rstest] + fn test_reconstruct_reports_first_node_starting_past_the_end_of_the_genome() { + let mut graph = graph(); + let node = &graph.nodes[&NodeId(0)]; + graph.nodes.insert( + NodeId(0), + PangraphNode::new( + Some(NodeId(0)), + node.block_id(), + node.path_id(), + node.strand(), + (100, 8), + ), + ); + + let err = report_to_string(&reconstruct_genome(&graph, PathId(0)).unwrap_err()); + assert!( + err.contains("starts at position 100, past the end of a genome of length 8"), + "unexpected error: {err}" + ); + } } diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs index c621a43c..96e703b1 100644 --- a/packages/pangraph/tests/itest_reconstruct.rs +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -235,4 +235,43 @@ mod tests { Ok(()) } + + /// A graph read from a file was not necessarily written by pangraph, so nothing guarantees its + /// cross-references resolve. `reconstruct` used to index the node map directly, so a path naming + /// a node the graph does not contain aborted the process with a bare "no entry found for key" + /// panic — and in a release build, where `sanity_check` is compiled out, with no indication of + /// which file was at fault. Validation now happens where the file name is still known. + #[rstest] + fn itest_reconstruct_rejects_malformed_graph_naming_the_file() -> Result<(), Report> { + let dir = tempdir()?; + + let mut graph = Pangraph { + blocks: btreemap! { + BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap! { NodeId(0) => Edit::empty() }), + }, + nodes: btreemap! { + NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + }, + paths: btreemap! { + PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, Some("a".to_owned()), None), + }, + }; + graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; + let graph = write_graph(&dir, "malformed.json", &graph)?; + + let args = PangraphReconstructArgs { + input_graph: Some(graph), + output_fasta: PathBuf::from("-"), + verify: None, + }; + let err = report_to_string(&reconstruct_run(&args).unwrap_err()); + + assert!( + err.contains("Node 99 from path 0 not found in graph"), + "unexpected error: {err}" + ); + assert!(err.contains("malformed.json"), "unexpected error: {err}"); + + Ok(()) + } } From ea1f9faaa93328df23c2718e788c267bb0a7a2d1 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 11:23:10 +0200 Subject: [PATCH 36/50] refactor(pangraph): require an explicit id in the node constructor --- .../pangraph/src/circularize/circularize.rs | 50 +++---- .../pangraph/src/circularize/merge_blocks.rs | 134 ++++++++++-------- .../src/commands/simplify/simplify_run.rs | 34 ++--- .../pangraph/src/pangraph/detach_unaligned.rs | 18 +-- packages/pangraph/src/pangraph/pangraph.rs | 44 +++--- .../pangraph/src/pangraph/pangraph_node.rs | 21 ++- packages/pangraph/src/pangraph/reconstruct.rs | 14 +- packages/pangraph/src/pangraph/reweave.rs | 22 +-- packages/pangraph/src/pangraph/slice.rs | 18 +-- .../pangraph/src/reconsensus/reconsensus.rs | 22 +-- .../pangraph/src/reconsensus/remove_nodes.rs | 10 +- packages/pangraph/tests/itest_merge.rs | 2 +- packages/pangraph/tests/itest_reconstruct.rs | 4 +- 13 files changed, 202 insertions(+), 191 deletions(-) diff --git a/packages/pangraph/src/circularize/circularize.rs b/packages/pangraph/src/circularize/circularize.rs index 0c501cad..bce68297 100644 --- a/packages/pangraph/src/circularize/circularize.rs +++ b/packages/pangraph/src/circularize/circularize.rs @@ -110,28 +110,28 @@ mod tests { #[rustfmt::skip] let nodes = btreemap! { - NodeId(10) => PangraphNode::new(Some(NodeId(10)), BlockId(1), PathId(0), Forward, (0, 0)), - NodeId(20) => PangraphNode::new(Some(NodeId(20)), BlockId(2), PathId(0), Forward, (0, 0)), - NodeId(30) => PangraphNode::new(Some(NodeId(30)), BlockId(3), PathId(0), Forward, (0, 0)), - NodeId(40) => PangraphNode::new(Some(NodeId(40)), BlockId(4), PathId(0), Forward, (0, 0)), - NodeId(11) => PangraphNode::new(Some(NodeId(11)), BlockId(1), PathId(1), Forward, (0, 0)), - NodeId(21) => PangraphNode::new(Some(NodeId(21)), BlockId(2), PathId(1), Reverse, (0, 0)), - NodeId(22) => PangraphNode::new(Some(NodeId(22)), BlockId(2), PathId(1), Forward, (0, 0)), - NodeId(31) => PangraphNode::new(Some(NodeId(31)), BlockId(3), PathId(1), Forward, (0, 0)), - NodeId(41) => PangraphNode::new(Some(NodeId(41)), BlockId(4), PathId(1), Forward, (0, 0)), - NodeId(12) => PangraphNode::new(Some(NodeId(12)), BlockId(1), PathId(2), Forward, (0, 0)), - NodeId(23) => PangraphNode::new(Some(NodeId(23)), BlockId(2), PathId(2), Forward, (0, 0)), - NodeId(32) => PangraphNode::new(Some(NodeId(32)), BlockId(3), PathId(2), Reverse, (0, 0)), - NodeId(42) => PangraphNode::new(Some(NodeId(42)), BlockId(4), PathId(2), Forward, (0, 0)), - NodeId(13) => PangraphNode::new(Some(NodeId(13)), BlockId(1), PathId(3), Forward, (0, 0)), - NodeId(33) => PangraphNode::new(Some(NodeId(33)), BlockId(3), PathId(3), Reverse, (0, 0)), - NodeId(24) => PangraphNode::new(Some(NodeId(24)), BlockId(2), PathId(3), Forward, (0, 0)), - NodeId(34) => PangraphNode::new(Some(NodeId(34)), BlockId(3), PathId(3), Reverse, (0, 0)), - NodeId(43) => PangraphNode::new(Some(NodeId(43)), BlockId(4), PathId(3), Forward, (0, 0)), - NodeId(44) => PangraphNode::new(Some(NodeId(44)), BlockId(4), PathId(4), Reverse, (0, 0)), - NodeId(35) => PangraphNode::new(Some(NodeId(35)), BlockId(3), PathId(4), Reverse, (0, 0)), - NodeId(25) => PangraphNode::new(Some(NodeId(25)), BlockId(2), PathId(4), Reverse, (0, 0)), - NodeId(14) => PangraphNode::new(Some(NodeId(14)), BlockId(1), PathId(4), Reverse, (0, 0)), + NodeId(10) => PangraphNode::new(NodeId(10), BlockId(1), PathId(0), Forward, (0, 0)), + NodeId(20) => PangraphNode::new(NodeId(20), BlockId(2), PathId(0), Forward, (0, 0)), + NodeId(30) => PangraphNode::new(NodeId(30), BlockId(3), PathId(0), Forward, (0, 0)), + NodeId(40) => PangraphNode::new(NodeId(40), BlockId(4), PathId(0), Forward, (0, 0)), + NodeId(11) => PangraphNode::new(NodeId(11), BlockId(1), PathId(1), Forward, (0, 0)), + NodeId(21) => PangraphNode::new(NodeId(21), BlockId(2), PathId(1), Reverse, (0, 0)), + NodeId(22) => PangraphNode::new(NodeId(22), BlockId(2), PathId(1), Forward, (0, 0)), + NodeId(31) => PangraphNode::new(NodeId(31), BlockId(3), PathId(1), Forward, (0, 0)), + NodeId(41) => PangraphNode::new(NodeId(41), BlockId(4), PathId(1), Forward, (0, 0)), + NodeId(12) => PangraphNode::new(NodeId(12), BlockId(1), PathId(2), Forward, (0, 0)), + NodeId(23) => PangraphNode::new(NodeId(23), BlockId(2), PathId(2), Forward, (0, 0)), + NodeId(32) => PangraphNode::new(NodeId(32), BlockId(3), PathId(2), Reverse, (0, 0)), + NodeId(42) => PangraphNode::new(NodeId(42), BlockId(4), PathId(2), Forward, (0, 0)), + NodeId(13) => PangraphNode::new(NodeId(13), BlockId(1), PathId(3), Forward, (0, 0)), + NodeId(33) => PangraphNode::new(NodeId(33), BlockId(3), PathId(3), Reverse, (0, 0)), + NodeId(24) => PangraphNode::new(NodeId(24), BlockId(2), PathId(3), Forward, (0, 0)), + NodeId(34) => PangraphNode::new(NodeId(34), BlockId(3), PathId(3), Reverse, (0, 0)), + NodeId(43) => PangraphNode::new(NodeId(43), BlockId(4), PathId(3), Forward, (0, 0)), + NodeId(44) => PangraphNode::new(NodeId(44), BlockId(4), PathId(4), Reverse, (0, 0)), + NodeId(35) => PangraphNode::new(NodeId(35), BlockId(3), PathId(4), Reverse, (0, 0)), + NodeId(25) => PangraphNode::new(NodeId(25), BlockId(2), PathId(4), Reverse, (0, 0)), + NodeId(14) => PangraphNode::new(NodeId(14), BlockId(1), PathId(4), Reverse, (0, 0)), }; #[rustfmt::skip] @@ -216,9 +216,9 @@ mod tests { BlockId(1) => block_1() }; let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Forward, (0, 32)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Forward, (0, 31)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Reverse, (0, 35)) + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Forward, (0, 32)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Forward, (0, 31)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Reverse, (0, 35)) }; Pangraph { paths, blocks, nodes } } diff --git a/packages/pangraph/src/circularize/merge_blocks.rs b/packages/pangraph/src/circularize/merge_blocks.rs index 85bf76ee..b5a5ccd9 100644 --- a/packages/pangraph/src/circularize/merge_blocks.rs +++ b/packages/pangraph/src/circularize/merge_blocks.rs @@ -240,7 +240,7 @@ mod tests { use crate::circularize::circularize::remove_transitive_edges; use crate::pangraph::edits::{Del, Edit, Ins, Sub}; use crate::pangraph::pangraph_path::{PangraphPath, PathId}; - use crate::pangraph::strand::Strand::{Forward, Reverse}; + use crate::pangraph::strand::Strand::{self, Forward, Reverse}; use itertools::Itertools; use maplit::btreemap; use pretty_assertions::assert_eq; @@ -349,14 +349,14 @@ mod tests { }; let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Forward, (0, 32)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Forward, (10, 41)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Reverse, (40, 5 )), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(1), Reverse, (32, 61)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Reverse, (41, 72)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(2), PathId(3), Forward, (5, 40)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Forward, (0, 32)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Forward, (10, 41)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Reverse, (40, 5 )), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(1), Reverse, (32, 61)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Reverse, (41, 72)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(2), PathId(3), Forward, (5, 40)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; Pangraph { paths, blocks, nodes } @@ -384,14 +384,14 @@ mod tests { }; let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Reverse, (0, 32)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Reverse, (10, 41)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Forward, (40, 5 )), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(1), Forward, (32, 61)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Forward, (41, 72)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(2), PathId(3), Reverse, (5, 40)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Reverse, (0, 32)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Reverse, (10, 41)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Forward, (40, 5 )), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(1), Forward, (32, 61)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Forward, (41, 72)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(2), PathId(3), Reverse, (5, 40)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; Pangraph { paths, blocks, nodes } @@ -419,14 +419,14 @@ mod tests { }; let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Forward, (0, 32)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Forward, (10, 41)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Reverse, (40, 5 )), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(1), Forward, (32, 61)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Forward, (41, 72)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(2), PathId(3), Reverse, (5, 40)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Forward, (0, 32)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Forward, (10, 41)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Reverse, (40, 5 )), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(1), Forward, (32, 61)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Forward, (41, 72)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(2), PathId(3), Reverse, (5, 40)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; Pangraph { paths, blocks, nodes } @@ -505,36 +505,51 @@ mod tests { assert_eq!(block_2_revcomp(), block_2().reverse_complement().unwrap()); } + /// Builds a node the way the code under test does: with its id derived from the genome that + /// walks it, looked up in `graph`, rather than from the path id. + fn derived_node( + graph: &Pangraph, + block_id: BlockId, + path_id: PathId, + strand: Strand, + position: (usize, usize), + ) -> PangraphNode { + PangraphNode::with_derived_id(block_id, &graph.paths[&path_id], strand, position) + } + fn expected_new_nodes_a() -> BTreeMap { + let g = graph_a(); btreemap! { - NodeId(1) => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - NodeId(2) => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - NodeId(3) => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), - NodeId(4) => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - NodeId(5) => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - NodeId(6) => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(1) => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + NodeId(2) => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + NodeId(3) => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(4) => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + NodeId(5) => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + NodeId(6) => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), } } fn expected_new_nodes_b() -> BTreeMap { + let g = graph_b(); btreemap! { - NodeId(1) => PangraphNode::new(None, BlockId(1), PathId(1), Reverse, (0 , 61)), - NodeId(2) => PangraphNode::new(None, BlockId(1), PathId(2), Reverse, (10, 72)), - NodeId(3) => PangraphNode::new(None, BlockId(1), PathId(3), Forward, (5, 5)), - NodeId(4) => PangraphNode::new(None, BlockId(1), PathId(1), Reverse, (0 , 61)), - NodeId(5) => PangraphNode::new(None, BlockId(1), PathId(2), Reverse, (10, 72)), - NodeId(6) => PangraphNode::new(None, BlockId(1), PathId(3), Forward, (5, 5)), + NodeId(1) => derived_node(&g, BlockId(1), PathId(1), Reverse, (0 , 61)), + NodeId(2) => derived_node(&g, BlockId(1), PathId(2), Reverse, (10, 72)), + NodeId(3) => derived_node(&g, BlockId(1), PathId(3), Forward, (5, 5)), + NodeId(4) => derived_node(&g, BlockId(1), PathId(1), Reverse, (0 , 61)), + NodeId(5) => derived_node(&g, BlockId(1), PathId(2), Reverse, (10, 72)), + NodeId(6) => derived_node(&g, BlockId(1), PathId(3), Forward, (5, 5)), } } fn expected_new_nodes_c() -> BTreeMap { + let g = graph_c(); btreemap! { - NodeId(1) => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - NodeId(2) => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - NodeId(3) => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), - NodeId(4) => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - NodeId(5) => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - NodeId(6) => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(1) => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + NodeId(2) => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + NodeId(3) => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(4) => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + NodeId(5) => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + NodeId(6) => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), } } @@ -672,6 +687,7 @@ mod tests { // (40|-----------|40) // p3) (b1-|-----------|n3) l=67 + let g = graph_a(); let new_ids = expected_new_node_ids_a(); let blocks = btreemap! { @@ -683,11 +699,11 @@ mod tests { }; let nodes = btreemap! { - new_ids[&NodeId(1)] => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - new_ids[&NodeId(2)] => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - new_ids[&NodeId(3)] => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + new_ids[&NodeId(1)] => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + new_ids[&NodeId(2)] => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + new_ids[&NodeId(3)] => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; #[rustfmt::skip] @@ -708,6 +724,7 @@ mod tests { // (40|-----------|40) // p3) (b1+|-----------|n3) l=67 + let g = graph_b(); let new_ids = expected_new_node_ids_b(); let blocks = btreemap! { @@ -719,11 +736,11 @@ mod tests { }; let nodes = btreemap! { - new_ids[&NodeId(1)] => PangraphNode::new(None, BlockId(1), PathId(1), Reverse, (0 , 61)), - new_ids[&NodeId(2)] => PangraphNode::new(None, BlockId(1), PathId(2), Reverse, (10, 72)), - new_ids[&NodeId(3)] => PangraphNode::new(None, BlockId(1), PathId(3), Forward, (5, 5)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + new_ids[&NodeId(1)] => derived_node(&g, BlockId(1), PathId(1), Reverse, (0 , 61)), + new_ids[&NodeId(2)] => derived_node(&g, BlockId(1), PathId(2), Reverse, (10, 72)), + new_ids[&NodeId(3)] => derived_node(&g, BlockId(1), PathId(3), Forward, (5, 5)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; #[rustfmt::skip] @@ -744,6 +761,7 @@ mod tests { // (40|-----------|40) // p3) (b1-|-----------|n3) l=67 + let g = graph_c(); let new_ids = expected_new_node_ids_c(); let blocks = btreemap! { @@ -755,11 +773,11 @@ mod tests { }; let nodes = btreemap! { - new_ids[&NodeId(1)] => PangraphNode::new(None, BlockId(1), PathId(1), Forward, (0 , 61)), - new_ids[&NodeId(2)] => PangraphNode::new(None, BlockId(1), PathId(2), Forward, (10, 72)), - new_ids[&NodeId(3)] => PangraphNode::new(None, BlockId(1), PathId(3), Reverse, (5, 5)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (61, 0 )), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Forward, (72, 10)), + new_ids[&NodeId(1)] => derived_node(&g, BlockId(1), PathId(1), Forward, (0 , 61)), + new_ids[&NodeId(2)] => derived_node(&g, BlockId(1), PathId(2), Forward, (10, 72)), + new_ids[&NodeId(3)] => derived_node(&g, BlockId(1), PathId(3), Reverse, (5, 5)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (61, 0 )), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Forward, (72, 10)), }; #[rustfmt::skip] diff --git a/packages/pangraph/src/commands/simplify/simplify_run.rs b/packages/pangraph/src/commands/simplify/simplify_run.rs index f38d5a17..04a6215b 100644 --- a/packages/pangraph/src/commands/simplify/simplify_run.rs +++ b/packages/pangraph/src/commands/simplify/simplify_run.rs @@ -133,14 +133,14 @@ mod tests { // n2+ -> n5+ -> n8- // n3+ -> n6- let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Forward, (0, 32)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Forward, (0, 31)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Forward, (0, 35)), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(1), Forward, (32, 64)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Forward, (31, 60)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(2), PathId(3), Forward, (35, 0)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (64, 0)), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Reverse, (60, 0)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Forward, (0, 32)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Forward, (0, 31)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Forward, (0, 35)), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(1), Forward, (32, 64)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Forward, (31, 60)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(2), PathId(3), Forward, (35, 0)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (64, 0)), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Reverse, (60, 0)), }; let blocks = btreemap! { BlockId(1) => block_a(), @@ -157,10 +157,10 @@ mod tests { fn expected_graph() -> Pangraph { let nodes = btreemap! { - NID11 => PangraphNode::new(Some(NodeId(11)), BlockId(1), PathId(1), Forward, (0, 64)), - NID12 => PangraphNode::new(Some(NodeId(12)), BlockId(1), PathId(2), Forward, (0, 60)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(1), Forward, (64, 0)), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Reverse, (60, 0)), + NID11 => PangraphNode::new(NodeId(11), BlockId(1), PathId(1), Forward, (0, 64)), + NID12 => PangraphNode::new(NodeId(12), BlockId(1), PathId(2), Forward, (0, 60)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(1), Forward, (64, 0)), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Reverse, (60, 0)), }; let blocks = btreemap! { BlockId(1) => block_ab(), @@ -186,11 +186,11 @@ mod tests { }; let expected_nodes = btreemap! { - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(2), Forward, (0, 31)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(3), Forward, (0, 35)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Forward, (31, 60)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(2), PathId(3), Forward, (35, 0)), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(2), Reverse, (60, 0)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(2), Forward, (0, 31)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(3), Forward, (0, 35)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Forward, (31, 60)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(2), PathId(3), Forward, (35, 0)), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(2), Reverse, (60, 0)), }; let expected_blocks = btreemap! { diff --git a/packages/pangraph/src/pangraph/detach_unaligned.rs b/packages/pangraph/src/pangraph/detach_unaligned.rs index ee33c6e6..e718721b 100644 --- a/packages/pangraph/src/pangraph/detach_unaligned.rs +++ b/packages/pangraph/src/pangraph/detach_unaligned.rs @@ -103,7 +103,7 @@ fn create_new_node_and_block( // Create a new PangraphNode for the unaligned node let new_node = PangraphNode::new( - Some(node_id), + node_id, new_block_id, old_node.path_id(), // same path ID as the old node Forward, // assuming the new node is always on the forward strand @@ -132,11 +132,11 @@ mod tests { let seq = Seq::from_str("ATGTTGATAG"); let old_block_id = BlockId(0); let old_path_id = PathId(0); - let old_node = PangraphNode::new(Some(node_id), old_block_id, old_path_id, Forward, (10, 20)); + let old_node = PangraphNode::new(node_id, old_block_id, old_path_id, Forward, (10, 20)); let (new_node, new_block) = create_new_node_and_block(node_id, seq.clone(), &old_node)?; - let expected_new_node = PangraphNode::new(Some(node_id), new_block.id(), old_path_id, Forward, (10, 20)); + let expected_new_node = PangraphNode::new(node_id, new_block.id(), old_path_id, Forward, (10, 20)); let expected_new_block = PangraphBlock::from_consensus(seq, new_block.id(), node_id); assert_eq!(new_node, expected_new_node); @@ -151,11 +151,11 @@ mod tests { let seq = Seq::from_str("ATGTTGATAG"); let old_block_id = BlockId(0); let old_path_id = PathId(1); - let old_node = PangraphNode::new(Some(node_id), old_block_id, old_path_id, Reverse, (5, 15)); + let old_node = PangraphNode::new(node_id, old_block_id, old_path_id, Reverse, (5, 15)); let (new_node, new_block) = create_new_node_and_block(node_id, seq.clone(), &old_node)?; - let expected_new_node = PangraphNode::new(Some(node_id), new_block.id(), old_path_id, Forward, (5, 15)); + let expected_new_node = PangraphNode::new(node_id, new_block.id(), old_path_id, Forward, (5, 15)); let expected_new_block = PangraphBlock::from_consensus(reverse_complement(&seq)?, new_block.id(), node_id); assert_eq!(new_node, expected_new_node); @@ -211,8 +211,8 @@ mod tests { }, ); let mut blocks = vec![block]; - let node1 = PangraphNode::new(Some(NodeId(1)), BlockId(0), PathId(0), Forward, (0, 16)); - let node2 = PangraphNode::new(Some(NodeId(2)), BlockId(0), PathId(1), Reverse, (0, 8)); + let node1 = PangraphNode::new(NodeId(1), BlockId(0), PathId(0), Forward, (0, 16)); + let node2 = PangraphNode::new(NodeId(2), BlockId(0), PathId(1), Reverse, (0, 8)); let mut nodes = btreemap! { NodeId(1) => node1, NodeId(2) => node2, @@ -231,8 +231,8 @@ mod tests { assert_eq!(blocks[1], expected_block2); let expected_node_dict = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(0), PathId(0), Forward, (0, 16)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), new_block_id, PathId(1), Forward, (0, 8)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(0), PathId(0), Forward, (0, 16)), + NodeId(2) => PangraphNode::new(NodeId(2), new_block_id, PathId(1), Forward, (0, 8)), }; assert_eq!(nodes, expected_node_dict); Ok(()) diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 1a7c6b1f..9b90e1d2 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -42,7 +42,7 @@ impl Pangraph { let block = PangraphBlock::from_consensus(fasta.seq, block_id, node_id); let path_id = PathId(fasta.index); let node_position = if circular { (0, 0) } else { (0, tot_len) }; // path wraps around if circular - let node = PangraphNode::new(Some(node_id), block.id(), path_id, strand, node_position); + let node = PangraphNode::new(node_id, block.id(), path_id, strand, node_position); let path = PangraphPath::new( Some(path_id), [node.id()], @@ -121,7 +121,7 @@ impl Pangraph { ) })?; - let node = PangraphNode::new(Some(nid), node.block_id(), path_id, node.strand(), node.position()); + let node = PangraphNode::new(nid, node.block_id(), path_id, node.strand(), node.position()); Ok((nid, node)) }) .collect::>()?; @@ -493,14 +493,14 @@ mod tests { // b2+ -> [b4+, b5-] let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Forward, (0, 0)), // FIXME - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(3), Forward, (0, 0)), // FIXME - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(2), PathId(1), Forward, (0, 0)), // FIXME - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(2), Forward, (0, 0)), // FIXME - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(3), Reverse, (0, 0)), // FIXME - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(3), PathId(1), Forward, (0, 0)), // FIXME - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(3), PathId(2), Forward, (0, 0)), // FIXME - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(3), PathId(3), Forward, (0, 0)) // FIXME + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Forward, (0, 0)), // FIXME + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(3), Forward, (0, 0)), // FIXME + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(2), PathId(1), Forward, (0, 0)), // FIXME + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(2), Forward, (0, 0)), // FIXME + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(3), Reverse, (0, 0)), // FIXME + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(3), PathId(1), Forward, (0, 0)), // FIXME + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(3), PathId(2), Forward, (0, 0)), // FIXME + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(3), PathId(3), Forward, (0, 0)) // FIXME }; let blocks = btreemap! { @@ -525,12 +525,12 @@ mod tests { }; let new_nodes = btreemap! { - NodeId(9) => PangraphNode::new(Some(NodeId(9)), BlockId(4), PathId(1), Forward, (0, 0)), - NodeId(10) => PangraphNode::new(Some(NodeId(10)), BlockId(5), PathId(1), Reverse, (0, 0)), - NodeId(11) => PangraphNode::new(Some(NodeId(11)), BlockId(4), PathId(2), Forward, (0, 0)), - NodeId(12) => PangraphNode::new(Some(NodeId(12)), BlockId(5), PathId(2), Reverse, (0, 0)), - NodeId(13) => PangraphNode::new(Some(NodeId(13)), BlockId(4), PathId(3), Reverse, (0, 0)), - NodeId(14) => PangraphNode::new(Some(NodeId(14)), BlockId(5), PathId(3), Forward, (0, 0)), + NodeId(9) => PangraphNode::new(NodeId(9), BlockId(4), PathId(1), Forward, (0, 0)), + NodeId(10) => PangraphNode::new(NodeId(10), BlockId(5), PathId(1), Reverse, (0, 0)), + NodeId(11) => PangraphNode::new(NodeId(11), BlockId(4), PathId(2), Forward, (0, 0)), + NodeId(12) => PangraphNode::new(NodeId(12), BlockId(5), PathId(2), Reverse, (0, 0)), + NodeId(13) => PangraphNode::new(NodeId(13), BlockId(4), PathId(3), Reverse, (0, 0)), + NodeId(14) => PangraphNode::new(NodeId(14), BlockId(5), PathId(3), Forward, (0, 0)), }; let new_blocks = btreemap! { @@ -634,8 +634,8 @@ mod tests { b1 => PangraphBlock::new(b1, "TTTTGGGG", btreemap!{ n1 => Edit::empty() }), }; let nodes = btreemap! { - n0 => PangraphNode::new(Some(n0), b0, PathId(0), Forward, (0, 8)), - n1 => PangraphNode::new(Some(n1), b1, PathId(1), Reverse, (0, 8)), + n0 => PangraphNode::new(n0, b0, PathId(0), Forward, (0, 8)), + n1 => PangraphNode::new(n1, b1, PathId(1), Reverse, (0, 8)), }; let paths = btreemap! { PathId(0) => PangraphPath::new(Some(PathId(0)), [n0], 8, false, Some(names[0].to_owned()), None), @@ -653,7 +653,7 @@ mod tests { let mut graph = two_genome_graph(["a", "b"]); let nid = graph.node_ids().next().unwrap(); let node = &graph.nodes[&nid]; - let detached = PangraphNode::new(Some(nid), node.block_id(), PathId(99), node.strand(), node.position()); + let detached = PangraphNode::new(nid, node.block_id(), PathId(99), node.strand(), node.position()); graph.nodes.insert(nid, detached); let err = report_to_string(&graph.renumber_paths(7).unwrap_err()); @@ -849,7 +849,7 @@ mod tests { let node = &graph.nodes[&nid]; graph.nodes.insert( nid, - PangraphNode::new(Some(nid), BlockId(99), node.path_id(), node.strand(), node.position()), + PangraphNode::new(nid, BlockId(99), node.path_id(), node.strand(), node.position()), ); let err = report_to_string(&graph.validate().unwrap_err()); @@ -866,7 +866,7 @@ mod tests { let node = &graph.nodes[&nid]; graph.nodes.insert( nid, - PangraphNode::new(Some(nid), node.block_id(), PathId(99), node.strand(), node.position()), + PangraphNode::new(nid, node.block_id(), PathId(99), node.strand(), node.position()), ); let err = report_to_string(&graph.validate().unwrap_err()); @@ -931,7 +931,7 @@ mod tests { let node = &graph.nodes[&nid]; graph.nodes.insert( nid, - PangraphNode::new(Some(nid), node.block_id(), node.path_id(), node.strand(), (100, 8)), + PangraphNode::new(nid, node.block_id(), node.path_id(), node.strand(), (100, 8)), ); let err = report_to_string(&graph.validate().unwrap_err()); diff --git a/packages/pangraph/src/pangraph/pangraph_node.rs b/packages/pangraph/src/pangraph/pangraph_node.rs index 91498ce9..55419867 100644 --- a/packages/pangraph/src/pangraph/pangraph_node.rs +++ b/packages/pangraph/src/pangraph/pangraph_node.rs @@ -35,14 +35,13 @@ impl NodeId { } impl PangraphNode { - pub fn new( - node_id: Option, - block_id: BlockId, - path_id: PathId, - strand: Strand, - position: (usize, usize), - ) -> Self { - let id = node_id.unwrap_or_else(|| id((&block_id, &path_id, &strand, &position))); + /// Creates a node with an explicit id. + /// + /// Node ids are content-derived, and [`PangraphNode::with_derived_id`] is the one place that + /// derives them. This constructor takes the id it is given, so a caller that has an id already — + /// because the node is being rewritten rather than created — cannot accidentally seed a second, + /// divergent derivation scheme. + pub fn new(id: NodeId, block_id: BlockId, path_id: PathId, strand: Strand, position: (usize, usize)) -> Self { Self { id, block_id, @@ -54,9 +53,9 @@ impl PangraphNode { /// Creates a node placing `block_id` on `path`, with an id derived from its contents. /// - /// Genomes are discriminated by [`PangraphPath::seed`] rather than by their path id, so the - /// resulting id does not depend on the order in which the input sequences were read. The path id - /// is still what the node stores. + /// This is the only place a node id is derived. Genomes are discriminated by + /// [`PangraphPath::seed`] rather than by their path id, so the resulting id does not depend on + /// the order in which the input sequences were read. The path id is still what the node stores. pub fn with_derived_id(block_id: BlockId, path: &PangraphPath, strand: Strand, position: (usize, usize)) -> Self { Self { id: id((&block_id, &path.seed(), &strand, &position)), diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 9a7920bb..3c0b8921 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -394,8 +394,8 @@ mod tests { BlockId(1) => PangraphBlock::new(BlockId(1), "TTTTGGGG", btreemap!{ NodeId(1) => Edit::empty() }), }; let nodes = btreemap! { - NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(1), Reverse, (0, 8)), + NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), Forward, (0, 8)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Reverse, (0, 8)), }; let paths = btreemap! { PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, names[0].map(String::from), None), @@ -413,7 +413,7 @@ mod tests { BlockId(0) => PangraphBlock::new(BlockId(0), consensus, btreemap!{ NodeId(0) => Edit::empty() }), }; let nodes = btreemap! { - NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), strand, (0, len)), + NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), strand, (0, len)), }; let paths = btreemap! { PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], len, false, Some(name.to_owned()), None), @@ -678,13 +678,7 @@ mod tests { let node = &graph.nodes[&NodeId(0)]; graph.nodes.insert( NodeId(0), - PangraphNode::new( - Some(NodeId(0)), - node.block_id(), - node.path_id(), - node.strand(), - (100, 8), - ), + PangraphNode::new(NodeId(0), node.block_id(), node.path_id(), node.strand(), (100, 8)), ); let err = report_to_string(&reconstruct_genome(&graph, PathId(0)).unwrap_err()); diff --git a/packages/pangraph/src/pangraph/reweave.rs b/packages/pangraph/src/pangraph/reweave.rs index 8bc07213..74856317 100644 --- a/packages/pangraph/src/pangraph/reweave.rs +++ b/packages/pangraph/src/pangraph/reweave.rs @@ -704,9 +704,9 @@ mod tests { let nid2 = NodeId(2000); let nid3 = NodeId(3000); - let n1 = PangraphNode::new(Some(nid1), bid, PathId(100), Forward, (100, 230)); - let n2 = PangraphNode::new(Some(nid2), bid, PathId(200), Reverse, (1000, 1130)); - let n3 = PangraphNode::new(Some(nid3), bid, PathId(300), Reverse, (180, 110)); + let n1 = PangraphNode::new(nid1, bid, PathId(100), Forward, (100, 230)); + let n2 = PangraphNode::new(nid2, bid, PathId(200), Reverse, (1000, 1130)); + let n3 = PangraphNode::new(nid3, bid, PathId(300), Reverse, (180, 110)); let b1 = PangraphBlock::new( bid, @@ -885,14 +885,14 @@ mod tests { fn generate_example() -> (Pangraph, Vec) { let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(10), PathId(100), Forward, (700, 885)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(30), PathId(100), Forward, (885, 988)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(30), PathId(200), Reverse, (100, 180)), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(20), PathId(200), Reverse, (180, 555)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(10), PathId(200), Reverse, (555, 735)), - NodeId(6) => PangraphNode::new(Some(NodeId(6)), BlockId(40), PathId(300), Forward, (600, 100)), - NodeId(7) => PangraphNode::new(Some(NodeId(7)), BlockId(50), PathId(300), Forward, (100, 325)), - NodeId(8) => PangraphNode::new(Some(NodeId(8)), BlockId(50), PathId(300), Reverse, (325, 580)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(10), PathId(100), Forward, (700, 885)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(30), PathId(100), Forward, (885, 988)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(30), PathId(200), Reverse, (100, 180)), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(20), PathId(200), Reverse, (180, 555)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(10), PathId(200), Reverse, (555, 735)), + NodeId(6) => PangraphNode::new(NodeId(6), BlockId(40), PathId(300), Forward, (600, 100)), + NodeId(7) => PangraphNode::new(NodeId(7), BlockId(50), PathId(300), Forward, (100, 325)), + NodeId(8) => PangraphNode::new(NodeId(8), BlockId(50), PathId(300), Reverse, (325, 580)), }; let paths = btreemap! { diff --git a/packages/pangraph/src/pangraph/slice.rs b/packages/pangraph/src/pangraph/slice.rs index 98e8fd50..25bb23a8 100644 --- a/packages/pangraph/src/pangraph/slice.rs +++ b/packages/pangraph/src/pangraph/slice.rs @@ -468,15 +468,15 @@ mod tests { let (new_b, new_nodes) = block_slice(&b, &i, &G); assert_eq!(new_b.consensus(), "TATATTTATC"); - let nn1 = PangraphNode::new(None, new_bid, PathId(1), Forward, (111, 120)); + let nn1 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(1)], Forward, (111, 120)); let nn1_slice = new_nodes[&NodeId(1)].as_ref().unwrap(); assert_eq!(nn1, *nn1_slice); - let nn2 = PangraphNode::new(None, new_bid, PathId(2), Reverse, (1008, 1017)); + let nn2 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(2)], Reverse, (1008, 1017)); let nn2_slice = new_nodes[&NodeId(2)].as_ref().unwrap(); assert_eq!(nn2, *nn2_slice); - let nn3 = PangraphNode::new(None, new_bid, PathId(3), Reverse, (96, 4)); + let nn3 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(3)], Reverse, (96, 4)); let nn3_slice = new_nodes[&NodeId(3)].as_ref().unwrap(); assert_eq!(nn3, *nn3_slice); @@ -533,9 +533,9 @@ mod tests { inss: vec![Ins::new(20, "T")], }; - let n1 = PangraphNode::new(Some(NodeId(1)), bid, PathId(1), Forward, (100, 125)); - let n2 = PangraphNode::new(Some(NodeId(2)), bid, PathId(2), Reverse, (1000, 1025)); - let n3 = PangraphNode::new(Some(NodeId(3)), bid, PathId(3), Reverse, (90, 9)); + let n1 = PangraphNode::new(NodeId(1), bid, PathId(1), Forward, (100, 125)); + let n2 = PangraphNode::new(NodeId(2), bid, PathId(2), Reverse, (1000, 1025)); + let n3 = PangraphNode::new(NodeId(3), bid, PathId(3), Reverse, (90, 9)); let p1 = PangraphPath::new( Some(PathId(1)), @@ -610,15 +610,15 @@ mod tests { assert_eq!(new_b.consensus(), "TATATTTATC"); - let nn1 = PangraphNode::new(None, new_bid, PathId(1), Reverse, (111, 120)); + let nn1 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(1)], Reverse, (111, 120)); let nn1_slice = new_nodes[&NodeId(1)].as_ref().unwrap(); assert_eq!(nn1, *nn1_slice); - let nn2 = PangraphNode::new(None, new_bid, PathId(2), Forward, (1008, 1017)); + let nn2 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(2)], Forward, (1008, 1017)); let nn2_slice = new_nodes[&NodeId(2)].as_ref().unwrap(); assert_eq!(nn2, *nn2_slice); - let nn3 = PangraphNode::new(None, new_bid, PathId(3), Forward, (96, 4)); + let nn3 = PangraphNode::with_derived_id(new_bid, &G.paths[&PathId(3)], Forward, (96, 4)); let nn3_slice = new_nodes[&NodeId(3)].as_ref().unwrap(); assert_eq!(nn3, *nn3_slice); diff --git a/packages/pangraph/src/reconsensus/reconsensus.rs b/packages/pangraph/src/reconsensus/reconsensus.rs index 1a53bee2..1666770c 100644 --- a/packages/pangraph/src/reconsensus/reconsensus.rs +++ b/packages/pangraph/src/reconsensus/reconsensus.rs @@ -436,11 +436,11 @@ mod tests { let expected_block = block_1_reconsensus(); let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), block.id(), PathId(1), Forward, (0, 23)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), block.id(), PathId(2), Forward, (0, 23)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), block.id(), PathId(3), Forward, (0, 23)), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), block.id(), PathId(4), Forward, (0, 23)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), block.id(), PathId(5), Forward, (0, 23)), + NodeId(1) => PangraphNode::new(NodeId(1), block.id(), PathId(1), Forward, (0, 23)), + NodeId(2) => PangraphNode::new(NodeId(2), block.id(), PathId(2), Forward, (0, 23)), + NodeId(3) => PangraphNode::new(NodeId(3), block.id(), PathId(3), Forward, (0, 23)), + NodeId(4) => PangraphNode::new(NodeId(4), block.id(), PathId(4), Forward, (0, 23)), + NodeId(5) => PangraphNode::new(NodeId(5), block.id(), PathId(5), Forward, (0, 23)), }; let paths = btreemap! { PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 23, false, None, None), @@ -510,11 +510,11 @@ mod tests { // Create nodes for the block with lengths reflecting actual sequence lengths let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), initial_block.id(), PathId(1), Reverse, (0, 10)), // 50 - 40 = 9 (deletes positions 0-39) - NodeId(2) => PangraphNode::new(Some(NodeId(2)), initial_block.id(), PathId(2), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) - NodeId(3) => PangraphNode::new(Some(NodeId(3)), initial_block.id(), PathId(3), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) - NodeId(4) => PangraphNode::new(Some(NodeId(4)), initial_block.id(), PathId(4), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) - NodeId(5) => PangraphNode::new(Some(NodeId(5)), initial_block.id(), PathId(5), Forward, (0, 49)), // no deletions + NodeId(1) => PangraphNode::new(NodeId(1), initial_block.id(), PathId(1), Reverse, (0, 10)), // 50 - 40 = 9 (deletes positions 0-39) + NodeId(2) => PangraphNode::new(NodeId(2), initial_block.id(), PathId(2), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) + NodeId(3) => PangraphNode::new(NodeId(3), initial_block.id(), PathId(3), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) + NodeId(4) => PangraphNode::new(NodeId(4), initial_block.id(), PathId(4), Forward, (0, 35)), // 50 - 15 = 35 (deletes positions 35-49) + NodeId(5) => PangraphNode::new(NodeId(5), initial_block.id(), PathId(5), Forward, (0, 49)), // no deletions }; // Create paths @@ -554,7 +554,7 @@ mod tests { // check that the node was updated correctly, flipping the strandedness let new_node1 = &graph.nodes[&NodeId(1)]; - let expected_node1 = PangraphNode::new(Some(NodeId(1)), singleton_block_exp.id(), PathId(1), Forward, (0, 10)); + let expected_node1 = PangraphNode::new(NodeId(1), singleton_block_exp.id(), PathId(1), Forward, (0, 10)); assert_eq!(new_node1, &expected_node1); } } diff --git a/packages/pangraph/src/reconsensus/remove_nodes.rs b/packages/pangraph/src/reconsensus/remove_nodes.rs index cf9b88b3..5ed3baaa 100644 --- a/packages/pangraph/src/reconsensus/remove_nodes.rs +++ b/packages/pangraph/src/reconsensus/remove_nodes.rs @@ -87,11 +87,11 @@ mod tests { fn create_input_graph() -> Pangraph { let nodes = btreemap! { - NodeId(1) => PangraphNode::new(Some(NodeId(1)), BlockId(1), PathId(0), Forward, (0, 10)), - NodeId(2) => PangraphNode::new(Some(NodeId(2)), BlockId(1), PathId(1), Forward, (0, 10)), - NodeId(3) => PangraphNode::new(Some(NodeId(3)), BlockId(1), PathId(2), Reverse, (0, 0)), - NodeId(4) => PangraphNode::new(Some(NodeId(4)), BlockId(2), PathId(0), Forward, (10, 20)), - NodeId(5) => PangraphNode::new(Some(NodeId(5)), BlockId(2), PathId(2), Forward, (0, 10)), + NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(0), Forward, (0, 10)), + NodeId(2) => PangraphNode::new(NodeId(2), BlockId(1), PathId(1), Forward, (0, 10)), + NodeId(3) => PangraphNode::new(NodeId(3), BlockId(1), PathId(2), Reverse, (0, 0)), + NodeId(4) => PangraphNode::new(NodeId(4), BlockId(2), PathId(0), Forward, (10, 20)), + NodeId(5) => PangraphNode::new(NodeId(5), BlockId(2), PathId(2), Forward, (0, 10)), }; let paths = btreemap! { diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 7b3bc9cf..13d96d39 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -251,7 +251,7 @@ mod tests { let (bid, nid, pid) = (BlockId(0), NodeId(0), PathId(0)); Pangraph { blocks: BTreeMap::from([(bid, PangraphBlock::from_consensus(seq, bid, nid))]), - nodes: BTreeMap::from([(nid, PangraphNode::new(Some(nid), bid, pid, Forward, (0, seq.len())))]), + nodes: BTreeMap::from([(nid, PangraphNode::new(nid, bid, pid, Forward, (0, seq.len())))]), paths: BTreeMap::from([( pid, PangraphPath::new(Some(pid), [nid], seq.len(), false, Some(name.to_owned()), None), diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs index 96e703b1..10134fb3 100644 --- a/packages/pangraph/tests/itest_reconstruct.rs +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -182,7 +182,7 @@ mod tests { BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap!{ NodeId(0) => Edit::empty() }), }, nodes: btreemap! { - NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), Forward, (0, 8)), }, paths: btreemap! { PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, None, None), @@ -250,7 +250,7 @@ mod tests { BlockId(0) => PangraphBlock::new(BlockId(0), "ACGTACGT", btreemap! { NodeId(0) => Edit::empty() }), }, nodes: btreemap! { - NodeId(0) => PangraphNode::new(Some(NodeId(0)), BlockId(0), PathId(0), Forward, (0, 8)), + NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), Forward, (0, 8)), }, paths: btreemap! { PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, Some("a".to_owned()), None), From 3c64030924fbbb9febd49b12c661302581a51e3e Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 11:29:39 +0200 Subject: [PATCH 37/50] docs: improve error message for identifier collisions in input graphs --- packages/pangraph/src/commands/merge/merge_run.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index a9bb317d..edb8dd86 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -29,7 +29,7 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { // Cheap, and the alternative is `graph_join` panicking on the conflicting key. if !right.is_id_disjoint_from(&left) { return make_error!( - "The two input graphs share block or node identifiers, so they cannot be joined. Identifiers are derived from genome names since version 1.4; graphs written by earlier versions derive them from the order of the input sequences instead, and two such graphs collide. Rebuild the input graphs with the current version of pangraph." + "The two input graphs share block or node identifiers, so they cannot be joined. Identifier collision avoidance is implemented since v1.4.0. If you built your graphs with a previous version of pangraph try rebuilding them. If the error persists please submit an issue." ); } From 7c6a77001cf731311708a95180567f1e1bd8ec70 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 11:43:01 +0200 Subject: [PATCH 38/50] fix(pangraph): reject genome names with leading or trailing whitespace --- packages/pangraph/src/pangraph/reconstruct.rs | 86 ++++++++++++++++++- 1 file changed, 84 insertions(+), 2 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 3c0b8921..236f81d0 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -63,7 +63,10 @@ pub fn reconstruct(graph: &Pangraph) -> impl Iterator Result<(), Report> { ); } + let padded = graphs + .iter() + .flat_map(|graph| graph.path_names().flatten()) + .filter(|name| name.trim() != *name) + .map(|name| format!("{name:?}")) + .collect_vec(); + + if !padded.is_empty() { + return make_error!( + "Found {} genome name(s) with leading or trailing whitespace: {}. Genome names are compared exactly, so surrounding whitespace makes a genome impossible to tell apart from, and impossible to address as, the name it appears to have.", + padded.len(), + format_names(&padded) + ); + } + let duplicates = find_duplicates(graphs.iter().flat_map(|graph| graph.path_names().flatten())); if !duplicates.is_empty() { return make_error!( @@ -130,7 +148,8 @@ pub fn reconstruct_genome(graph: &Pangraph, path_id: PathId) -> Result Result<(), Report> { let empty = fastas .iter() @@ -146,6 +165,20 @@ pub fn check_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { ); } + let padded = fastas + .iter() + .filter(|fasta| fasta.seq_name.trim() != fasta.seq_name) + .map(|fasta| format!("{:?}", fasta.seq_name)) + .collect_vec(); + + if !padded.is_empty() { + return make_error!( + "Found {} input sequence(s) whose name has leading or trailing whitespace: {}. Sequence names are compared exactly, so surrounding whitespace makes a genome impossible to tell apart from the name it appears to have. Note that whitespace between '>' and the identifier becomes part of the name, unless it is a plain space, which makes the identifier part of the description instead.", + padded.len(), + format_names(&padded) + ); + } + let duplicates = find_duplicates(fastas.iter().map(|fasta| fasta.seq_name.as_str())); if !duplicates.is_empty() { return make_error!( @@ -454,6 +487,34 @@ mod tests { assert!(err.contains("no name or an empty name"), "unexpected error: {err}"); } + /// Every comparison on genome names is exact, so " a" would be a genome that `--strains a` + /// cannot address, that does not collide with the "a" of the graph it is merged with, and that + /// round-trips to a FASTA header with an invisible space in it. + #[rstest] + #[case(" a")] + #[case("a ")] + #[case("\ta")] + fn test_path_ids_by_name_rejects_padded_path_name(#[case] name: &str) { + let graph = two_genome_graph([Some("b"), Some(name)]); + let err = report_to_string(&path_ids_by_name(&graph).unwrap_err()); + assert!( + err.contains("leading or trailing whitespace"), + "unexpected error: {err}" + ); + } + + /// The pair a padded name is most likely to be mistaken for: without the check they are two + /// distinct genomes, since duplicates are detected on the name as given. + #[rstest] + fn test_path_ids_by_name_rejects_a_name_that_differs_only_by_padding() { + let graph = two_genome_graph([Some("a"), Some("a ")]); + let err = report_to_string(&path_ids_by_name(&graph).unwrap_err()); + assert!( + err.contains("leading or trailing whitespace"), + "unexpected error: {err}" + ); + } + #[rstest] fn test_path_ids_by_name_rejects_duplicate_names() { let graph = two_genome_graph([Some("a"), Some("a")]); @@ -505,6 +566,27 @@ mod tests { assert!(err.contains('3'), "unexpected error: {err}"); } + /// A header indented with anything other than a plain space — `>\tid` — keeps the whitespace in + /// the name, which is invisible in every place the name is later shown. + #[rstest] + #[case(" a")] + #[case("a ")] + #[case("\ta")] + fn test_check_sequence_names_rejects_padded_name(#[case] name: &str) { + let fastas = [FastaRecord { + seq_name: name.to_owned(), + desc: None, + seq: Seq::from_str("ACGT"), + index: 3, + }]; + + let err = report_to_string(&check_sequence_names(&fastas).unwrap_err()); + assert!( + err.contains("leading or trailing whitespace"), + "unexpected error: {err}" + ); + } + #[rstest] fn test_verify_graph_sequences_accepts_exact_match() { verify_graph_sequences(&graph(), &expected_genomes(), GenomeCoverage::Complete).unwrap(); From fa876df8c6b3af49c7183b7bcc6d5810ecad5d7c Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 11:58:42 +0200 Subject: [PATCH 39/50] style: use plain ascii punctuation in comments and docs --- CHANGELOG.md | 2 +- docs/docs/tutorial/t04b-merging-two-graphs.md | 2 +- packages/pangraph/src/commands/merge/merge_run.rs | 8 ++++---- packages/pangraph/src/commands/root_args.rs | 2 +- packages/pangraph/src/pangraph/pangraph.rs | 2 +- packages/pangraph/src/pangraph/pangraph_node.rs | 4 ++-- packages/pangraph/src/pangraph/reconstruct.rs | 10 +++++----- packages/pangraph/tests/itest_reconstruct.rs | 2 +- 8 files changed, 16 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 1238f94a..9d595cd5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. - `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. -- more strict checks on input sequences ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequence by id and not by order. +- more strict checks on input sequence ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequences by id and not by order. - added more graph validation at loading. - block and node identifiers are now derived from genome names instead of the order the input sequences are read in, enforcing order-independence. - fixed a race condition in `pangraph build` that could produce slightly different graphs on repeated runs over the same input. diff --git a/docs/docs/tutorial/t04b-merging-two-graphs.md b/docs/docs/tutorial/t04b-merging-two-graphs.md index 76138759..091e7df3 100644 --- a/docs/docs/tutorial/t04b-merging-two-graphs.md +++ b/docs/docs/tutorial/t04b-merging-two-graphs.md @@ -30,7 +30,7 @@ The two graphs can then be merged: pangraph merge graph.json k12.json -o graph_11.json ``` -The resulting `graph_11.json` contains 11 paths: the 10 genomes of `graph.json`, in their original order, followed by K-12. Adding the eleventh chromosome created comparatively few new blocks (2896 → 2943): most of it was absorbed into blocks that already existed. +The resulting `graph_11.json` contains 11 paths: the 10 genomes of `graph.json`, in their original order, followed by K-12. Adding the eleventh chromosome created comparatively few new blocks (2896 -> 2943): most of it was absorbed into blocks that already existed. On a consumer laptop the merge takes around 20 seconds, against the roughly 3.5 minutes needed to build the 10-genome graph in the first place. diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index edb8dd86..fc89502c 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -71,9 +71,9 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { /// Reads a pangraph from a JSON file. /// /// `from_path` already rejects a graph whose ids do not resolve or whose offsets are out of range, -/// in release builds too. The extra `sanity_check` here adds the semantic invariants on top — that -/// node positions tile the genome — which indicate a bug rather than a bad file, and so are only -/// worth paying for in debug builds. +/// in release builds too. The extra `sanity_check` here adds the semantic invariants on top, such +/// as node positions tiling the genome. Those indicate a bug rather than a bad file, and so are +/// only worth paying for in debug builds. fn read_graph(filepath: &Path) -> Result { let graph = Pangraph::from_path(&Some(filepath))?; @@ -110,7 +110,7 @@ fn merge_cmd_preliminary_checks(args: &PangraphMergeArgs, left: &Pangraph, right .header("Suggestion:") })?; - // Circularity is a per-path property, so mixing is structurally fine. It is however most often a + // Circularity is a per-path property, so mixing is structurally fine, but it is most often a // mistake, since `build --circular` applies to all genomes of a graph at once. if circularity(left) != circularity(right) { warn!( diff --git a/packages/pangraph/src/commands/root_args.rs b/packages/pangraph/src/commands/root_args.rs index e6ee0bde..07ef3a1d 100644 --- a/packages/pangraph/src/commands/root_args.rs +++ b/packages/pangraph/src/commands/root_args.rs @@ -187,7 +187,7 @@ mod tests { /// This pins the assignment so that mistake is a test failure. An argument appended after a /// flattened group shows up in one of these sets; an argument added before it does not, so there /// are no false alarms. The escape hatch, if a trailing argument really is needed, is to set - /// `#[clap(help_heading = ...)]` on the argument itself — that wins over the cursor. + /// `#[clap(help_heading = ...)]` on the argument itself, which wins over the cursor. #[rstest] fn test_arguments_are_filed_under_the_expected_help_section() { // `jobs` is declared after the `Verbosity` flatten and used to be swept into it. diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 9b90e1d2..f8761052 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -234,7 +234,7 @@ impl Pangraph { /// index the maps directly and treat a lookup that fails anyway as an internal error. /// /// Deliberately limited to what a bad file can break. It does *not* check that the graph is - /// semantically coherent — that node positions tile the genome, that edits do not overlap — since + /// semantically coherent (that node positions tile the genome, that edits do not overlap), since /// those are symptoms of a bug in pangraph rather than of a bad input, and are covered by /// [`Self::sanity_check`] in debug builds. Errors are reported as ordinary user-facing errors for /// the same reason: the offending graph came from the user. diff --git a/packages/pangraph/src/pangraph/pangraph_node.rs b/packages/pangraph/src/pangraph/pangraph_node.rs index 55419867..6a9b7207 100644 --- a/packages/pangraph/src/pangraph/pangraph_node.rs +++ b/packages/pangraph/src/pangraph/pangraph_node.rs @@ -38,8 +38,8 @@ impl PangraphNode { /// Creates a node with an explicit id. /// /// Node ids are content-derived, and [`PangraphNode::with_derived_id`] is the one place that - /// derives them. This constructor takes the id it is given, so a caller that has an id already — - /// because the node is being rewritten rather than created — cannot accidentally seed a second, + /// derives them. This constructor takes the id it is given, so a caller that already has an id + /// (because the node is being rewritten rather than created) cannot accidentally seed a second, /// divergent derivation scheme. pub fn new(id: NodeId, block_id: BlockId, path_id: PathId, strand: Strand, position: (usize, usize)) -> Self { Self { diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 236f81d0..3ca4caa9 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -279,8 +279,8 @@ pub fn verify_graph_sequences( /// Used to verify a merged graph against the graphs it was built from. Both sides of every /// comparison are reconstructed on demand and dropped again, so this holds two genomes at a time /// instead of the whole sequence content of `sources`. The saving is proportional to the total -/// genome length and modest in practice — peak usage during a merge is dominated by the graphs -/// themselves — but it keeps verification consistent with `reconstruct --verify`, which streams for +/// genome length and modest in practice (peak usage during a merge is dominated by the graphs +/// themselves), but it keeps verification consistent with `reconstruct --verify`, which streams for /// the same reason. /// /// Every genome of `sources` must appear in `graph`, and `graph` must contain nothing else. The @@ -547,7 +547,7 @@ mod tests { assert!(report_to_string(&check_sequence_names(&fastas).unwrap_err()).contains("Duplicate sequence names")); } - /// `> id` — a space between the '>' and the identifier — parses into an empty name with the + /// `> id`, with a space between the '>' and the identifier, parses into an empty name with the /// identifier in the description, so this common header style used to yield a nameless genome. /// The record index is part of the message, since the name cannot point at the offending record. #[rstest] @@ -566,8 +566,8 @@ mod tests { assert!(err.contains('3'), "unexpected error: {err}"); } - /// A header indented with anything other than a plain space — `>\tid` — keeps the whitespace in - /// the name, which is invisible in every place the name is later shown. + /// A header indented with anything other than a plain space, such as `>\tid`, keeps the whitespace + /// in the name, which is invisible in every place the name is later shown. #[rstest] #[case(" a")] #[case("a ")] diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs index 10134fb3..b837f2f6 100644 --- a/packages/pangraph/tests/itest_reconstruct.rs +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -239,7 +239,7 @@ mod tests { /// A graph read from a file was not necessarily written by pangraph, so nothing guarantees its /// cross-references resolve. `reconstruct` used to index the node map directly, so a path naming /// a node the graph does not contain aborted the process with a bare "no entry found for key" - /// panic — and in a release build, where `sanity_check` is compiled out, with no indication of + /// panic, and in a release build, where `sanity_check` is compiled out, with no indication of /// which file was at fault. Validation now happens where the file name is still known. #[rstest] fn itest_reconstruct_rejects_malformed_graph_naming_the_file() -> Result<(), Report> { From 51f99d751777443e9e6ecaa83a0cf3c09404fb72 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:00:49 +0200 Subject: [PATCH 40/50] refactor(pangraph): require an explicit id in the path constructor --- .../pangraph/src/circularize/circularize.rs | 16 ++++----- .../pangraph/src/circularize/merge_blocks.rs | 36 +++++++++---------- .../src/commands/simplify/simplify_run.rs | 14 ++++---- packages/pangraph/src/pangraph/pangraph.rs | 27 ++++++-------- .../pangraph/src/pangraph/pangraph_path.rs | 12 ++++--- packages/pangraph/src/pangraph/reconstruct.rs | 6 ++-- packages/pangraph/src/pangraph/reweave.rs | 12 +++---- packages/pangraph/src/pangraph/slice.rs | 27 ++------------ .../pangraph/src/reconsensus/reconsensus.rs | 20 +++++------ .../pangraph/src/reconsensus/remove_nodes.rs | 6 ++-- packages/pangraph/tests/itest_merge.rs | 2 +- packages/pangraph/tests/itest_reconstruct.rs | 4 +-- 12 files changed, 79 insertions(+), 103 deletions(-) diff --git a/packages/pangraph/src/circularize/circularize.rs b/packages/pangraph/src/circularize/circularize.rs index bce68297..d7806e34 100644 --- a/packages/pangraph/src/circularize/circularize.rs +++ b/packages/pangraph/src/circularize/circularize.rs @@ -101,11 +101,11 @@ mod tests { #[rustfmt::skip] let paths = btreemap!{ - PathId(0) => PangraphPath::new(Some(PathId(0)), [10, 20, 30, 40].into_iter().map(NodeId).collect_vec(), 0, true, None, None), - PathId(1) => PangraphPath::new(Some(PathId(1)), [11, 21, 22, 31, 41].into_iter().map(NodeId).collect_vec(), 0, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [12, 23, 32, 42].into_iter().map(NodeId).collect_vec(), 0, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [13, 33, 24, 34, 43].into_iter().map(NodeId).collect_vec(), 0, true, None, None), - PathId(4) => PangraphPath::new(Some(PathId(4)), [44, 35, 25, 14].into_iter().map(NodeId).collect_vec(), 0, true, None, None), + PathId(0) => PangraphPath::new(PathId(0), [10, 20, 30, 40].into_iter().map(NodeId).collect_vec(), 0, true, None, None), + PathId(1) => PangraphPath::new(PathId(1), [11, 21, 22, 31, 41].into_iter().map(NodeId).collect_vec(), 0, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [12, 23, 32, 42].into_iter().map(NodeId).collect_vec(), 0, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [13, 33, 24, 34, 43].into_iter().map(NodeId).collect_vec(), 0, true, None, None), + PathId(4) => PangraphPath::new(PathId(4), [44, 35, 25, 14].into_iter().map(NodeId).collect_vec(), 0, true, None, None), }; #[rustfmt::skip] @@ -208,9 +208,9 @@ mod tests { // (0|35) // p3) (b1-|n3) l=35 let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), vec![NodeId(1)], 32, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), vec![NodeId(2)], 31, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), vec![NodeId(3)], 35, true, None, None) + PathId(1) => PangraphPath::new(PathId(1), vec![NodeId(1)], 32, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), vec![NodeId(2)], 31, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), vec![NodeId(3)], 35, true, None, None) }; let blocks = btreemap! { BlockId(1) => block_1() diff --git a/packages/pangraph/src/circularize/merge_blocks.rs b/packages/pangraph/src/circularize/merge_blocks.rs index b5a5ccd9..f8cbf3d2 100644 --- a/packages/pangraph/src/circularize/merge_blocks.rs +++ b/packages/pangraph/src/circularize/merge_blocks.rs @@ -337,9 +337,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), + PathId(1) => PangraphPath::new(PathId(1), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), }; let blocks = btreemap! { @@ -372,9 +372,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), + PathId(1) => PangraphPath::new(PathId(1), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), }; let blocks = btreemap! { @@ -407,9 +407,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), + PathId(1) => PangraphPath::new(PathId(1), [1, 4, 7].into_iter().map(NodeId).collect_vec(), 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [2, 5, 8].into_iter().map(NodeId).collect_vec(), 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [6, 3].into_iter().map(NodeId).collect_vec(), 67, true, None, None), }; let blocks = btreemap! { @@ -708,9 +708,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [new_ids[&NodeId(3)]], 67, true, None, None) + PathId(1) => PangraphPath::new(PathId(1), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [new_ids[&NodeId(3)]], 67, true, None, None) }; Pangraph { paths, blocks, nodes } @@ -745,9 +745,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [new_ids[&NodeId(3)]], 67, true, None, None) + PathId(1) => PangraphPath::new(PathId(1), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [new_ids[&NodeId(3)]], 67, true, None, None) }; Pangraph { paths, blocks, nodes } @@ -782,9 +782,9 @@ mod tests { #[rustfmt::skip] let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [new_ids[&NodeId(3)]], 67, true, None, None) + PathId(1) => PangraphPath::new(PathId(1), [new_ids[&NodeId(1)], NodeId(7)], 80, true, None, None), + PathId(2) => PangraphPath::new(PathId(2), [new_ids[&NodeId(2)], NodeId(8)], 83, true, None, None), + PathId(3) => PangraphPath::new(PathId(3), [new_ids[&NodeId(3)]], 67, true, None, None) }; Pangraph { paths, blocks, nodes } diff --git a/packages/pangraph/src/commands/simplify/simplify_run.rs b/packages/pangraph/src/commands/simplify/simplify_run.rs index 04a6215b..0a720feb 100644 --- a/packages/pangraph/src/commands/simplify/simplify_run.rs +++ b/packages/pangraph/src/commands/simplify/simplify_run.rs @@ -148,9 +148,9 @@ mod tests { BlockId(3) => block_c(), }; let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), vec![NodeId(1), NodeId(4), NodeId(7)], 81, true, Some(o!("pathA")), None), - PathId(2) => PangraphPath::new(Some(PathId(2)), vec![NodeId(2), NodeId(5), NodeId(8)], 77, true, Some(o!("pathB")), None), - PathId(3) => PangraphPath::new(Some(PathId(3)), vec![NodeId(3), NodeId(6)], 70, true, Some(o!("pathC")), None), + PathId(1) => PangraphPath::new(PathId(1), vec![NodeId(1), NodeId(4), NodeId(7)], 81, true, Some(o!("pathA")), None), + PathId(2) => PangraphPath::new(PathId(2), vec![NodeId(2), NodeId(5), NodeId(8)], 77, true, Some(o!("pathB")), None), + PathId(3) => PangraphPath::new(PathId(3), vec![NodeId(3), NodeId(6)], 70, true, Some(o!("pathC")), None), }; Pangraph { paths, blocks, nodes } } @@ -167,8 +167,8 @@ mod tests { BlockId(3) => block_c(), }; let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), vec![NID11, NodeId(7)], 81, true, Some(o!("pathA")), None), - PathId(2) => PangraphPath::new(Some(PathId(2)), vec![NID12, NodeId(8)], 77, true, Some(o!("pathB")), None), + PathId(1) => PangraphPath::new(PathId(1), vec![NID11, NodeId(7)], 81, true, Some(o!("pathA")), None), + PathId(2) => PangraphPath::new(PathId(2), vec![NID12, NodeId(8)], 77, true, Some(o!("pathB")), None), }; Pangraph { paths, blocks, nodes } @@ -181,8 +181,8 @@ mod tests { graph.remove_path(PathId(1)); let expected_paths = btreemap! { - PathId(2) => PangraphPath::new(Some(PathId(2)), vec![NodeId(2), NodeId(5), NodeId(8)], 77, true, Some(o!("pathB")), None), - PathId(3) => PangraphPath::new(Some(PathId(3)), vec![NodeId(3), NodeId(6)], 70, true, Some(o!("pathC")), None), + PathId(2) => PangraphPath::new(PathId(2), vec![NodeId(2), NodeId(5), NodeId(8)], 77, true, Some(o!("pathB")), None), + PathId(3) => PangraphPath::new(PathId(3), vec![NodeId(3), NodeId(6)], 70, true, Some(o!("pathC")), None), }; let expected_nodes = btreemap! { diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index f8761052..0eafdefd 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -44,7 +44,7 @@ impl Pangraph { let node_position = if circular { (0, 0) } else { (0, tot_len) }; // path wraps around if circular let node = PangraphNode::new(node_id, block.id(), path_id, strand, node_position); let path = PangraphPath::new( - Some(path_id), + path_id, [node.id()], tot_len, circular, @@ -513,9 +513,9 @@ mod tests { }; let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1), NodeId(3), NodeId(6)], 0, false, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [NodeId(4), NodeId(7) ], 0, false, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [NodeId(2), NodeId(5), NodeId(8)], 0, false, None, None), + PathId(1) => PangraphPath::new(PathId(1), [NodeId(1), NodeId(3), NodeId(6)], 0, false, None, None), + PathId(2) => PangraphPath::new(PathId(2), [NodeId(4), NodeId(7) ], 0, false, None, None), + PathId(3) => PangraphPath::new(PathId(3), [NodeId(2), NodeId(5), NodeId(8)], 0, false, None, None), }; let mut G = Pangraph { @@ -559,9 +559,9 @@ mod tests { assert_eq!(G.blocks, expected_blocks); let expected_paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1), NodeId(9), NodeId(10), NodeId(6)], 0, false, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [NodeId(11), NodeId(12), NodeId(7) ], 0, false, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [NodeId(2), NodeId(14), NodeId(13), NodeId(8)], 0, false, None, None), + PathId(1) => PangraphPath::new(PathId(1), [NodeId(1), NodeId(9), NodeId(10), NodeId(6)], 0, false, None, None), + PathId(2) => PangraphPath::new(PathId(2), [NodeId(11), NodeId(12), NodeId(7) ], 0, false, None, None), + PathId(3) => PangraphPath::new(PathId(3), [NodeId(2), NodeId(14), NodeId(13), NodeId(8)], 0, false, None, None), }; assert_eq!(G.paths, expected_paths); @@ -588,14 +588,7 @@ mod tests { .iter() .enumerate() .map(|(i, name)| { - let path = PangraphPath::new( - Some(PathId(i)), - Vec::::new(), - 0, - false, - name.map(String::from), - None, - ); + let path = PangraphPath::new(PathId(i), Vec::::new(), 0, false, name.map(String::from), None); (path.id, path) }) .collect::>(); @@ -638,8 +631,8 @@ mod tests { n1 => PangraphNode::new(n1, b1, PathId(1), Reverse, (0, 8)), }; let paths = btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [n0], 8, false, Some(names[0].to_owned()), None), - PathId(1) => PangraphPath::new(Some(PathId(1)), [n1], 8, false, Some(names[1].to_owned()), None), + PathId(0) => PangraphPath::new(PathId(0), [n0], 8, false, Some(names[0].to_owned()), None), + PathId(1) => PangraphPath::new(PathId(1), [n1], 8, false, Some(names[1].to_owned()), None), }; Pangraph { paths, blocks, nodes } } diff --git a/packages/pangraph/src/pangraph/pangraph_path.rs b/packages/pangraph/src/pangraph/pangraph_path.rs index 6c5e55a6..cb6db3e9 100644 --- a/packages/pangraph/src/pangraph/pangraph_path.rs +++ b/packages/pangraph/src/pangraph/pangraph_path.rs @@ -32,19 +32,23 @@ pub struct PangraphPath { } impl PangraphPath { + /// Creates a path with an explicit id. + /// + /// Path ids are sequential rather than content-derived: they double as the ordering index of the + /// genomes, so `build` assigns them from the input order and `renumber_paths` shifts them during + /// a merge. The id is therefore always the caller's to supply, and there is no fallback that + /// could seed a second, divergent numbering. pub fn new( - path_id: Option, + id: PathId, nodes: impl Into>, tot_len: usize, circular: bool, name: Option, desc: Option, ) -> Self { - let nodes = nodes.into(); - let id = path_id.unwrap_or_else(|| id((&nodes, &tot_len, &circular, &desc, &name))); Self { id, - nodes, + nodes: nodes.into(), tot_len, circular, name, diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 3ca4caa9..961b36be 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -431,8 +431,8 @@ mod tests { NodeId(1) => PangraphNode::new(NodeId(1), BlockId(1), PathId(1), Reverse, (0, 8)), }; let paths = btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, names[0].map(String::from), None), - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 8, false, names[1].map(String::from), None), + PathId(0) => PangraphPath::new(PathId(0), [NodeId(0)], 8, false, names[0].map(String::from), None), + PathId(1) => PangraphPath::new(PathId(1), [NodeId(1)], 8, false, names[1].map(String::from), None), }; Pangraph { paths, blocks, nodes } } @@ -449,7 +449,7 @@ mod tests { NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), strand, (0, len)), }; let paths = btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], len, false, Some(name.to_owned()), None), + PathId(0) => PangraphPath::new(PathId(0), [NodeId(0)], len, false, Some(name.to_owned()), None), }; Pangraph { paths, blocks, nodes } } diff --git a/packages/pangraph/src/pangraph/reweave.rs b/packages/pangraph/src/pangraph/reweave.rs index 74856317..8c83cf42 100644 --- a/packages/pangraph/src/pangraph/reweave.rs +++ b/packages/pangraph/src/pangraph/reweave.rs @@ -718,9 +718,9 @@ mod tests { }, ); - let p1 = PangraphPath::new(Some(PathId(100)), [nid1], 2000, true, None, None); - let p2 = PangraphPath::new(Some(PathId(200)), [nid2], 2000, true, None, None); - let p3 = PangraphPath::new(Some(PathId(300)), [nid3], 200, true, None, None); + let p1 = PangraphPath::new(PathId(100), [nid1], 2000, true, None, None); + let p2 = PangraphPath::new(PathId(200), [nid2], 2000, true, None, None); + let p3 = PangraphPath::new(PathId(300), [nid3], 200, true, None, None); let G = Pangraph { paths: btreemap! { @@ -896,9 +896,9 @@ mod tests { }; let paths = btreemap! { - PathId(100) => PangraphPath::new(Some(PathId(100)), [NodeId(1), NodeId(2)], 1000, true, None, None), - PathId(200) => PangraphPath::new(Some(PathId(200)), [NodeId(3), NodeId(4), NodeId(5)], 1000, true, None, None), - PathId(300) => PangraphPath::new(Some(PathId(300)), [NodeId(6), NodeId(7), NodeId(8)], 1000, true, None, None), + PathId(100) => PangraphPath::new(PathId(100), [NodeId(1), NodeId(2)], 1000, true, None, None), + PathId(200) => PangraphPath::new(PathId(200), [NodeId(3), NodeId(4), NodeId(5)], 1000, true, None, None), + PathId(300) => PangraphPath::new(PathId(300), [NodeId(6), NodeId(7), NodeId(8)], 1000, true, None, None), }; #[rustfmt::skip] diff --git a/packages/pangraph/src/pangraph/slice.rs b/packages/pangraph/src/pangraph/slice.rs index 25bb23a8..21143142 100644 --- a/packages/pangraph/src/pangraph/slice.rs +++ b/packages/pangraph/src/pangraph/slice.rs @@ -537,30 +537,9 @@ mod tests { let n2 = PangraphNode::new(NodeId(2), bid, PathId(2), Reverse, (1000, 1025)); let n3 = PangraphNode::new(NodeId(3), bid, PathId(3), Reverse, (90, 9)); - let p1 = PangraphPath::new( - Some(PathId(1)), - /*"p1"*/ [NodeId(1), NodeId(4)], - 2000, - true, - None, - None, - ); - let p2 = PangraphPath::new( - Some(PathId(2)), - /*"p2"*/ [NodeId(2), NodeId(5)], - 2000, - true, - None, - None, - ); - let p3 = PangraphPath::new( - Some(PathId(3)), - /*"p3"*/ [NodeId(3), NodeId(6)], - 100, - true, - None, - None, - ); + let p1 = PangraphPath::new(PathId(1), /*"p1"*/ [NodeId(1), NodeId(4)], 2000, true, None, None); + let p2 = PangraphPath::new(PathId(2), /*"p2"*/ [NodeId(2), NodeId(5)], 2000, true, None, None); + let p3 = PangraphPath::new(PathId(3), /*"p3"*/ [NodeId(3), NodeId(6)], 100, true, None, None); let b1 = PangraphBlock::new( bid, diff --git a/packages/pangraph/src/reconsensus/reconsensus.rs b/packages/pangraph/src/reconsensus/reconsensus.rs index 1666770c..e7d4afce 100644 --- a/packages/pangraph/src/reconsensus/reconsensus.rs +++ b/packages/pangraph/src/reconsensus/reconsensus.rs @@ -443,11 +443,11 @@ mod tests { NodeId(5) => PangraphNode::new(NodeId(5), block.id(), PathId(5), Forward, (0, 23)), }; let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 23, false, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [NodeId(2)], 23, false, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [NodeId(3)], 23, false, None, None), - PathId(4) => PangraphPath::new(Some(PathId(4)), [NodeId(4)], 23, false, None, None), - PathId(5) => PangraphPath::new(Some(PathId(5)), [NodeId(5)], 23, false, None, None), + PathId(1) => PangraphPath::new(PathId(1), [NodeId(1)], 23, false, None, None), + PathId(2) => PangraphPath::new(PathId(2), [NodeId(2)], 23, false, None, None), + PathId(3) => PangraphPath::new(PathId(3), [NodeId(3)], 23, false, None, None), + PathId(4) => PangraphPath::new(PathId(4), [NodeId(4)], 23, false, None, None), + PathId(5) => PangraphPath::new(PathId(5), [NodeId(5)], 23, false, None, None), }; let mut graph = Pangraph { blocks: btreemap! { @@ -519,11 +519,11 @@ mod tests { // Create paths let paths = btreemap! { - PathId(1) => PangraphPath::new(Some(PathId(1)), [NodeId(1)], 49, false, None, None), - PathId(2) => PangraphPath::new(Some(PathId(2)), [NodeId(2)], 49, false, None, None), - PathId(3) => PangraphPath::new(Some(PathId(3)), [NodeId(3)], 49, false, None, None), - PathId(4) => PangraphPath::new(Some(PathId(4)), [NodeId(4)], 49, false, None, None), - PathId(5) => PangraphPath::new(Some(PathId(5)), [NodeId(5)], 49, false, None, None), + PathId(1) => PangraphPath::new(PathId(1), [NodeId(1)], 49, false, None, None), + PathId(2) => PangraphPath::new(PathId(2), [NodeId(2)], 49, false, None, None), + PathId(3) => PangraphPath::new(PathId(3), [NodeId(3)], 49, false, None, None), + PathId(4) => PangraphPath::new(PathId(4), [NodeId(4)], 49, false, None, None), + PathId(5) => PangraphPath::new(PathId(5), [NodeId(5)], 49, false, None, None), }; // Create blocks map diff --git a/packages/pangraph/src/reconsensus/remove_nodes.rs b/packages/pangraph/src/reconsensus/remove_nodes.rs index 5ed3baaa..929c5f18 100644 --- a/packages/pangraph/src/reconsensus/remove_nodes.rs +++ b/packages/pangraph/src/reconsensus/remove_nodes.rs @@ -95,9 +95,9 @@ mod tests { }; let paths = btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(1)), vec![NodeId(1), NodeId(4)], 20, false, None, None), - PathId(1) => PangraphPath::new(Some(PathId(2)), vec![NodeId(2)], 10, false, None, None), - PathId(2) => PangraphPath::new(Some(PathId(3)), vec![NodeId(3), NodeId(5)], 10, false, None, None), + PathId(0) => PangraphPath::new(PathId(1), vec![NodeId(1), NodeId(4)], 20, false, None, None), + PathId(1) => PangraphPath::new(PathId(2), vec![NodeId(2)], 10, false, None, None), + PathId(2) => PangraphPath::new(PathId(3), vec![NodeId(3), NodeId(5)], 10, false, None, None), }; let blocks = btreemap! { diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 13d96d39..4ae07ee9 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -254,7 +254,7 @@ mod tests { nodes: BTreeMap::from([(nid, PangraphNode::new(nid, bid, pid, Forward, (0, seq.len())))]), paths: BTreeMap::from([( pid, - PangraphPath::new(Some(pid), [nid], seq.len(), false, Some(name.to_owned()), None), + PangraphPath::new(pid, [nid], seq.len(), false, Some(name.to_owned()), None), )]), } }; diff --git a/packages/pangraph/tests/itest_reconstruct.rs b/packages/pangraph/tests/itest_reconstruct.rs index b837f2f6..2be595f2 100644 --- a/packages/pangraph/tests/itest_reconstruct.rs +++ b/packages/pangraph/tests/itest_reconstruct.rs @@ -185,7 +185,7 @@ mod tests { NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), Forward, (0, 8)), }, paths: btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, None, None), + PathId(0) => PangraphPath::new(PathId(0), [NodeId(0)], 8, false, None, None), }, }; @@ -253,7 +253,7 @@ mod tests { NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), Forward, (0, 8)), }, paths: btreemap! { - PathId(0) => PangraphPath::new(Some(PathId(0)), [NodeId(0)], 8, false, Some("a".to_owned()), None), + PathId(0) => PangraphPath::new(PathId(0), [NodeId(0)], 8, false, Some("a".to_owned()), None), }, }; graph.paths.get_mut(&PathId(0)).unwrap().nodes = vec![NodeId(99)]; From 330316d0460103d8b655a515824b1ef115901f6c Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:00:55 +0200 Subject: [PATCH 41/50] refactor(pangraph): derive the genome seed in a single place --- packages/pangraph/src/pangraph/pangraph.rs | 7 +++---- packages/pangraph/src/pangraph/pangraph_path.rs | 10 +++++++++- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 0eafdefd..451f9013 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -4,11 +4,10 @@ use crate::io::fs::read_reader_to_string; use crate::io::json::json_read_str; use crate::pangraph::pangraph_block::{BlockId, PangraphBlock}; use crate::pangraph::pangraph_node::{NodeId, PangraphNode}; -use crate::pangraph::pangraph_path::{PangraphPath, PathId}; +use crate::pangraph::pangraph_path::{PangraphPath, PathId, genome_seed}; use crate::pangraph::strand::Strand; use crate::representation::seq::Seq; use crate::tree::clade::WithNewickName; -use crate::utils::id::id; use crate::utils::map_merge::{ConflictResolution, map_merge}; use crate::{make_error, make_internal_report, make_report}; use eyre::{Report, WrapErr}; @@ -36,7 +35,7 @@ impl Pangraph { // // Path ids stay sequential: they double as the ordering index of the genomes, and keying the // `paths` map by a hash would scramble genome order in every output. - let seed = id(&fasta.seq_name); + let seed = genome_seed(&fasta.seq_name); let node_id = NodeId(seed); let block_id = BlockId(seed); let block = PangraphBlock::from_consensus(fasta.seq, block_id, node_id); @@ -618,7 +617,7 @@ mod tests { /// Builds a two-genome graph the way `build` would, with block and node ids seeded from the /// genome names. Two such graphs collide only on their path ids. fn two_genome_graph(names: [&str; 2]) -> Pangraph { - let seeds = names.map(|name| id(name.to_owned())); + let seeds = names.map(genome_seed); let (b0, b1) = (BlockId(seeds[0]), BlockId(seeds[1])); let (n0, n1) = (NodeId(seeds[0]), NodeId(seeds[1])); diff --git a/packages/pangraph/src/pangraph/pangraph_path.rs b/packages/pangraph/src/pangraph/pangraph_path.rs index cb6db3e9..2d35035c 100644 --- a/packages/pangraph/src/pangraph/pangraph_path.rs +++ b/packages/pangraph/src/pangraph/pangraph_path.rs @@ -68,6 +68,14 @@ impl PangraphPath { /// and then recomputed on load anyway, and silently defaults to the same value for every path if /// that is forgotten. Falls back to the path id for unnamed paths, which pangraph never writes. pub fn seed(&self) -> usize { - self.name.as_ref().map_or(self.id.0, id) + self.name.as_deref().map_or(self.id.0, genome_seed) } } + +/// Derives the identifier seed of a genome from its name. +/// +/// The single definition of the seeding rule, so that [`PangraphPath::seed`] and +/// `Pangraph::singleton`, which needs the seed before it has a path to ask, cannot drift apart. +pub fn genome_seed(name: &str) -> usize { + id(name) +} From 8dfb00f48f14ebee98a3b2a2e23e31ec8d9ea68a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:00:55 +0200 Subject: [PATCH 42/50] perf(pangraph): renumber path ids without rebuilding the node map --- packages/pangraph/src/pangraph/pangraph.rs | 32 ++++++++++--------- .../pangraph/src/pangraph/pangraph_node.rs | 9 ++++++ 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/packages/pangraph/src/pangraph/pangraph.rs b/packages/pangraph/src/pangraph/pangraph.rs index 451f9013..d8195278 100644 --- a/packages/pangraph/src/pangraph/pangraph.rs +++ b/packages/pangraph/src/pangraph/pangraph.rs @@ -99,7 +99,11 @@ impl Pangraph { /// Errors if a node refers to a path the graph does not contain, which can only happen in a graph /// that pangraph did not write. pub fn renumber_paths(self, offset: usize) -> Result { - let Self { paths, blocks, nodes } = self; + let Self { + paths, + blocks, + mut nodes, + } = self; let path_map: BTreeMap = paths .keys() @@ -107,23 +111,21 @@ impl Pangraph { .map(|(rank, &pid)| (pid, PathId(offset + rank))) .collect(); + // Node ids do not change, so the map is updated in place rather than rebuilt: only the path a + // node points at moves. + // // `path_map` is keyed by the map keys of this graph, so looking a path's own new id up by its // key cannot miss. The path id read off a node's *contents* is a different matter: it can // dangle in a graph that pangraph did not write, so that lookup is fallible. - let nodes: BTreeMap = nodes - .into_iter() - .map(|(nid, node)| { - let path_id = *path_map.get(&node.path_id()).ok_or_else(|| { - make_report!( - "Node {nid} refers to path {}, which the graph does not contain", - node.path_id() - ) - })?; - - let node = PangraphNode::new(nid, node.block_id(), path_id, node.strand(), node.position()); - Ok((nid, node)) - }) - .collect::>()?; + for (nid, node) in &mut nodes { + let path_id = *path_map.get(&node.path_id()).ok_or_else(|| { + make_report!( + "Node {nid} refers to path {}, which the graph does not contain", + node.path_id() + ) + })?; + node.set_path_id(path_id); + } let paths: BTreeMap = paths .into_iter() diff --git a/packages/pangraph/src/pangraph/pangraph_node.rs b/packages/pangraph/src/pangraph/pangraph_node.rs index 6a9b7207..bc35593f 100644 --- a/packages/pangraph/src/pangraph/pangraph_node.rs +++ b/packages/pangraph/src/pangraph/pangraph_node.rs @@ -66,6 +66,15 @@ impl PangraphNode { } } + /// Moves this node onto a different path. + /// + /// The node id derives from the block, the genome *seed*, the strand and the position, none of + /// which this touches, so renumbering a path leaves every node id intact. That is why this is the + /// one field a node may have rewritten in place; everything else goes through a constructor. + pub(crate) fn set_path_id(&mut self, path_id: PathId) { + self.path_id = path_id; + } + // this is almost equivalent to checking if the node is empty // except for an edge case: when a circular path contains only // one node. In this case even if the node is not empty, the From c2ea5b59c49f5ae4592738a44409a5854d8d2a1a Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:01 +0200 Subject: [PATCH 43/50] refactor(pangraph): share the whitespace check between the name validators --- packages/pangraph/src/pangraph/reconstruct.rs | 29 ++++++++++++------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 961b36be..920c210d 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -91,12 +91,7 @@ pub fn check_genome_names(graphs: &[&Pangraph]) -> Result<(), Report> { ); } - let padded = graphs - .iter() - .flat_map(|graph| graph.path_names().flatten()) - .filter(|name| name.trim() != *name) - .map(|name| format!("{name:?}")) - .collect_vec(); + let padded = padded_names(graphs.iter().flat_map(|graph| graph.path_names().flatten())); if !padded.is_empty() { return make_error!( @@ -165,11 +160,7 @@ pub fn check_sequence_names(fastas: &[FastaRecord]) -> Result<(), Report> { ); } - let padded = fastas - .iter() - .filter(|fasta| fasta.seq_name.trim() != fasta.seq_name) - .map(|fasta| format!("{:?}", fasta.seq_name)) - .collect_vec(); + let padded = padded_names(fastas.iter().map(|fasta| fasta.seq_name.as_str())); if !padded.is_empty() { return make_error!( @@ -327,6 +318,22 @@ pub fn verify_graph_against_graphs(graph: &Pangraph, sources: &[&Pangraph]) -> R Ok(()) } +/// Collects the names that carry leading or trailing whitespace, quoted so that the padding is +/// visible in an error message. +/// +/// Shared by [`check_genome_names`] and [`check_sequence_names`]: both reject a padded name for the +/// same reason, so what counts as padding is decided in one place. The two keep their own error +/// messages, which name the container the offending records came from. +fn padded_names>(names: impl Iterator) -> Vec { + names + .filter(|name| { + let name = name.as_ref(); + name.trim() != name + }) + .map(|name| format!("{:?}", name.as_ref())) + .collect_vec() +} + /// Formats a list of genome names for an error message, eliding all but the first few. pub(crate) fn format_names>(names: &[S]) -> String { let shown = names.iter().take(MAX_NAMES_IN_ERROR).map(AsRef::as_ref).join(", "); From a201dc586c92a0a8c8c19dcb3faea9d36314f323 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:01 +0200 Subject: [PATCH 44/50] refactor(reconstruct): flatten path reconstruction with a let-else guard --- packages/pangraph/src/pangraph/reconstruct.rs | 60 +++++++++---------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index 920c210d..f66f9051 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -352,40 +352,40 @@ pub(crate) fn format_names>(names: &[S]) -> String { /// They are still checked, because reporting beats aborting: indexing the node map and rotating by /// an out-of-range offset both panic. fn reconstruct_path_sequence(graph: &Pangraph, path: &PangraphPath) -> Result { - if let Some(first_node_id) = path.nodes.first() { - let first_node = graph - .nodes - .get(first_node_id) - .ok_or_else(|| make_internal_report!("Node {first_node_id} not found in graph"))?; - let first_node_pos = first_node.position().0; - - let mut genome: Seq = path - .nodes - .iter() - .map(|node_id| reconstruct_block_sequence(graph, *node_id)) - .collect::>()?; + let Some(first_node_id) = path.nodes.first() else { + return Ok(Seq::new()); + }; - let genome_len = path.tot_len(); - if genome.len() != genome_len { - return make_error!( - "When reconstructing sequences, genome length mismatch: computed length {} expected {}", - genome.len(), - genome_len - ); - } + let first_node = graph + .nodes + .get(first_node_id) + .ok_or_else(|| make_internal_report!("Node {first_node_id} not found in graph"))?; + let first_node_pos = first_node.position().0; - if first_node_pos > genome.len() { - return make_internal_error!( - "When reconstructing sequences, the first node of the genome starts at position {first_node_pos}, past the end of a genome of length {}", - genome.len() - ); - } - genome.rotate_right(first_node_pos); + let mut genome: Seq = path + .nodes + .iter() + .map(|node_id| reconstruct_block_sequence(graph, *node_id)) + .collect::>()?; - Ok(genome) - } else { - Ok(Seq::new()) + let genome_len = path.tot_len(); + if genome.len() != genome_len { + return make_error!( + "When reconstructing sequences, genome length mismatch: computed length {} expected {}", + genome.len(), + genome_len + ); } + + if first_node_pos > genome.len() { + return make_internal_error!( + "When reconstructing sequences, the first node of the genome starts at position {first_node_pos}, past the end of a genome of length {}", + genome.len() + ); + } + genome.rotate_right(first_node_pos); + + Ok(genome) } fn reconstruct_block_sequence(graph: &Pangraph, node_id: NodeId) -> Result { From 5bce759123cacab93092fef6cf506227e1997f4b Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:01 +0200 Subject: [PATCH 45/50] perf(pangraph): avoid cloning genome names for an error message --- packages/pangraph/src/pangraph/reconstruct.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index f66f9051..b7c82cbe 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -250,7 +250,7 @@ pub fn verify_graph_sequences( let missing = expected .keys() .filter(|name| !path_ids.contains_key(name.as_str())) - .cloned() + .map(String::as_str) .collect_vec(); if !missing.is_empty() { From e12359b173533d29ddef40c849bb5b1f48b180cc Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:01 +0200 Subject: [PATCH 46/50] test(pangraph): build the single-genome fixture with Pangraph::singleton --- packages/pangraph/src/pangraph/reconstruct.rs | 22 ++++++++----------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/packages/pangraph/src/pangraph/reconstruct.rs b/packages/pangraph/src/pangraph/reconstruct.rs index b7c82cbe..7ab2425e 100644 --- a/packages/pangraph/src/pangraph/reconstruct.rs +++ b/packages/pangraph/src/pangraph/reconstruct.rs @@ -444,21 +444,17 @@ mod tests { Pangraph { paths, blocks, nodes } } - /// A one-genome graph, so that a pair of them stands in for the two inputs of a merge. Every id - /// is `0`, exactly as `Pangraph::singleton` assigns them, which is also what makes two of these - /// indistinguishable by id. + /// A one-genome graph, so that a pair of them stands in for the two inputs of a merge. Built + /// through `Pangraph::singleton`, so these are exactly the graphs `build` starts from and they + /// track its id derivation instead of restating it. fn one_genome_graph(name: &str, consensus: &str, strand: Strand) -> Pangraph { - let len = consensus.len(); - let blocks = btreemap! { - BlockId(0) => PangraphBlock::new(BlockId(0), consensus, btreemap!{ NodeId(0) => Edit::empty() }), - }; - let nodes = btreemap! { - NodeId(0) => PangraphNode::new(NodeId(0), BlockId(0), PathId(0), strand, (0, len)), - }; - let paths = btreemap! { - PathId(0) => PangraphPath::new(PathId(0), [NodeId(0)], len, false, Some(name.to_owned()), None), + let fasta = FastaRecord { + seq_name: name.to_owned(), + desc: None, + seq: Seq::from_str(consensus), + index: 0, }; - Pangraph { paths, blocks, nodes } + Pangraph::singleton(fasta, strand, false) } /// The two single-genome graphs whose merger `graph()` stands for. From e4d98b7ca8377d5129fff587e0cbd988760ab377 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:07 +0200 Subject: [PATCH 47/50] perf(merge): drop the duplicate sanity check of the merged graph --- packages/pangraph/src/commands/merge/merge_run.rs | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/packages/pangraph/src/commands/merge/merge_run.rs b/packages/pangraph/src/commands/merge/merge_run.rs index fc89502c..b6130d37 100644 --- a/packages/pangraph/src/commands/merge/merge_run.rs +++ b/packages/pangraph/src/commands/merge/merge_run.rs @@ -50,11 +50,10 @@ pub fn merge_run(args: &PangraphMergeArgs) -> Result<(), Report> { ); if args.verify { - #[cfg(debug_assertions)] - merged.sanity_check().wrap_err("When checking the merged graph")?; - - // Compared against the inputs one genome at a time: reconstructing both graphs up front would - // hold their entire sequence content in memory for the duration of the check. + // `merge_graphs` already sanity-checks the graph it returns in debug builds, so only the + // sequences are compared here. Compared against the inputs one genome at a time: reconstructing + // both graphs up front would hold their entire sequence content in memory for the duration of + // the check. verify_graph_against_graphs(&merged, &[&left, &right]) .wrap_err("When verifying the sequences of the merged graph")?; info!( From 04bd3c148a290ee245b6608ccfc64bcafd0941a3 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:07 +0200 Subject: [PATCH 48/50] perf(align): align sequence pairs without materializing them first --- .../minimap2_lib/align_with_minimap2_lib.rs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs b/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs index feab7d53..cbc01cf2 100644 --- a/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs +++ b/packages/pangraph/src/align/minimap2_lib/align_with_minimap2_lib.rs @@ -4,7 +4,7 @@ use crate::pangraph::pangraph_block::{BlockId, PangraphBlock}; use crate::pangraph::strand::Strand; use crate::{make_error, make_internal_error}; use eyre::{Report, WrapErr}; -use itertools::{Itertools, izip}; +use itertools::Itertools; use minimap2::{Minimap2Args, Minimap2Index, Minimap2Mapper, Minimap2Preset, Minimap2Result}; use noodles::sam::record::Cigar; use num_traits::clamp_min; @@ -61,13 +61,14 @@ fn align_with_minimap2_lib_impl( let idx = Minimap2Index::new(&seqs, &names, &args)?; - // Collected into a `Vec` first so that the parallel iterator is *indexed*: `par_bridge` hands - // work out in whatever order threads ask for it and collects in that order, which made the hit - // list vary between runs. `filter_matches` then broke equal-energy ties differently, so the same - // input could build different graphs. `Vec::into_par_iter` restores input order. - let results: Vec = izip!(&seqs, &names) - .collect_vec() - .into_par_iter() + // Driven by an *indexed* parallel iterator: `par_bridge` hands work out in whatever order threads + // ask for it and collects in that order, which made the hit list vary between runs. + // `filter_matches` then broke equal-energy ties differently, so the same input could build + // different graphs. `par_iter` over a slice is indexed and `zip` preserves that, so input order + // is restored without materializing the pairs first. + let results: Vec = seqs + .par_iter() + .zip(names.par_iter()) .map_init( || Minimap2Mapper::new(&idx).unwrap(), move |mapper, (seq, name)| { From a46fa46891a343de4a6457ccf1ebcc29c900bfee Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Tue, 18 Aug 2026 14:01:07 +0200 Subject: [PATCH 49/50] test(merge): define read_and_split in terms of read_records --- packages/pangraph/tests/itest_merge.rs | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/packages/pangraph/tests/itest_merge.rs b/packages/pangraph/tests/itest_merge.rs index 4ae07ee9..b5d6e465 100644 --- a/packages/pangraph/tests/itest_merge.rs +++ b/packages/pangraph/tests/itest_merge.rs @@ -25,15 +25,6 @@ mod tests { use std::path::{Path, PathBuf}; use tempfile::{TempDir, tempdir}; - /// Reads the first `n_total` records of a FASTA file and splits them in two groups, so that the - /// two resulting graphs are built from disjoint but homologous sets of genomes. - fn read_and_split(path: &str, n_left: usize, n_total: usize) -> Result<(Vec, Vec), Report> { - let mut fastas = FastaReader::from_paths(&[PathBuf::from(path)])?.read_many()?; - fastas.truncate(n_total); - let right = fastas.split_off(n_left); - Ok((fastas, right)) - } - /// Reads the first `n` records of a FASTA file. fn read_records(path: &str, n: usize) -> Result, Report> { let mut fastas = FastaReader::from_paths(&[PathBuf::from(path)])?.read_many()?; @@ -41,6 +32,14 @@ mod tests { Ok(fastas) } + /// Reads the first `n_total` records of a FASTA file and splits them in two groups, so that the + /// two resulting graphs are built from disjoint but homologous sets of genomes. + fn read_and_split(path: &str, n_left: usize, n_total: usize) -> Result<(Vec, Vec), Report> { + let mut left = read_records(path, n_total)?; + let right = left.split_off(n_left); + Ok((left, right)) + } + /// Builds a graph out of the given records and writes it to a JSON file in `dir`. fn build_graph_file(dir: &TempDir, name: &str, fastas: Vec) -> Result { let args = PangraphBuildArgs { @@ -279,7 +278,7 @@ mod tests { #[rstest] fn itest_merge_rejects_duplicate_genome_names() -> Result<(), Report> { let dir = tempdir()?; - let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; + let fastas = read_records("../../data/ges-1.fa", 3)?; let graph = build_graph_file(&dir, "graph.json", fastas)?; let output = dir.path().join("merged.json"); @@ -301,7 +300,7 @@ mod tests { #[rstest] fn itest_merge_rejects_unnamed_genomes_without_verify() -> Result<(), Report> { let dir = tempdir()?; - let (fastas, _) = read_and_split("../../data/ges-1.fa", 3, 3)?; + let fastas = read_records("../../data/ges-1.fa", 3)?; let mut graph = read_graph(&build_graph_file(&dir, "named.json", fastas)?)?; for path in graph.paths.values_mut() { @@ -347,7 +346,7 @@ mod tests { /// can never carry duplicate genome names into a later merge. #[rstest] fn itest_build_rejects_duplicate_sequence_names() -> Result<(), Report> { - let (mut fastas, _) = read_and_split("../../data/ges-1.fa", 2, 2)?; + let mut fastas = read_records("../../data/ges-1.fa", 2)?; fastas[1].seq_name = fastas[0].seq_name.clone(); let result = build(fastas, &PangraphBuildArgs::default(), false); @@ -365,7 +364,7 @@ mod tests { /// description. Such a genome could not be addressed by name afterwards, so `build` rejects it. #[rstest] fn itest_build_rejects_empty_sequence_name() -> Result<(), Report> { - let (mut fastas, _) = read_and_split("../../data/ges-1.fa", 2, 2)?; + let mut fastas = read_records("../../data/ges-1.fa", 2)?; fastas[1].desc = Some(fastas[1].seq_name.clone()); fastas[1].seq_name = String::new(); From 84e6f9f697021daa61845f61bd55825be716b7f8 Mon Sep 17 00:00:00 2001 From: Marco Molari Date: Wed, 19 Aug 2026 09:02:29 +0200 Subject: [PATCH 50/50] docs: update changelog --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d595cd5..7d132058 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,9 @@ ## Unreleased - added `pangraph merge` command, to combine two existing pangenome graphs into a single one, see #197. -- `pangraph build` now accepts a single input sequence, and rejects inputs with duplicate genome names. +- `pangraph build` now accepts a single input sequence. - more strict checks on input sequence ids: duplicated or empty ids are now rejected. The `--verify` options compare the input sequences by id and not by order. -- added more graph validation at loading. +- more strict graph validation at loading. - block and node identifiers are now derived from genome names instead of the order the input sequences are read in, enforcing order-independence. - fixed a race condition in `pangraph build` that could produce slightly different graphs on repeated runs over the same input.