Repository navigation
v0.7.5: validated MAF output, streaming I/O, Parquet, STDIN - #4
Merged
Merged
Conversation
- Fix indel alleles: strip shared VCF anchor base per MAF spec (e.g., VCF A→ATCG now produces MAF ref="-", alt="TCG") - Fix deletion start position off-by-one (now points to first deleted base) - Fix frameshift classification: distinguish Frame_Shift_Ins vs Frame_Shift_Del - Add defensive guard against u64 underflow on malformed VCF indels - Reduce ~30 unnecessary .clone() calls using Cow<str>, as_str(), move semantics - Optimize from_vcf_line: zero clones for common single-record case - Replace parsed_lines[7].clone() with std::mem::take() - Eliminate remaining_info_map.clone() via control-flow restructuring - Add 4 new tests (5'UTR ins/del, 3'UTR ins, frameshift ins vs del) - Update 5 existing tests with correct MAF expectations - Update README for v0.4.0 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add indexmap for ordered HashMaps, chrono for timestamps, and arrow/parquet behind the parquet_out feature flag for upcoming summary and parquet output features. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New src/summary.rs with SummaryStats struct, chromosome counting from VCF data lines, natural sort order (1-22, X, Y, M/MT), and formatted text file output. Includes 5 unit tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New src/parquet_writer.rs behind parquet_out feature flag. Converts ReformattedVcfRecords and MafRecords to parquet with Snappy compression. Numeric columns (POS, QUAL, depth, VAF) use native Arrow types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --summary and --parquet CLI flags. --parquet validates against --compress conflict. Summary and parquet writing wired into both TSV and MAF output paths. Parquet replaces text output when enabled. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- 3 row count conservation tests (TSV first, MAF first, split transcripts) - 4 summary correctness tests (counts sum, chromosome order, expansion ratio, file output) - 3 parquet equivalence tests behind parquet_out feature flag (TSV row count, column names, MAF row count) Total: 93 tests without parquet, 96 with parquet_out feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add summary reports and parquet output to features table, CLI options, and usage examples. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_report_html_snpeff_shows_impact_metric_not_sift asserted the
positive half (Impact/HIGH/LOW present) but never checked the negative
half implied by its own name. Add !html.contains("SIFT") so the test
actually verifies VEP-only metrics are excluded when only SnpEff/ANN
data is present.
Pure function converting HGVSp protein-change strings (e.g. "p.Val600Glu") to their short 1-letter form ("p.V600E"), using vcf2maf's own %aa3to1 table so output matches the reference tool exactly.
Extracts vcf2maf's Exon_Number source: VEP's CSQ_EXON, falling back to SnpEff's ANN_Rank. Both are formatted <rank>/<total>. Follows the same get_annotation_field wrapper pattern as get_transcript_id and get_protein_position. Not yet wired into from_reformatted_record (later task) -- dead_code warning is expected until then. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Extends extract_depth_from_sample_data to also return sample-level AD[0] (ref depth) as a fallback, mirroring the existing INFO AO / sample AD[1] pattern for tumor depth. Return type changes from (Option<u32>, Option<u32>) to (Option<u32>, Option<u32>, Option<u32>) (total, ref, alt). Updates the one call site in from_reformatted_record to thread the new ref-depth value into a tumor_ref_depth local (compile-correctness only — Task 4 wires it into the MafRecord struct). Part of the vcf2maf column parity plan, Task 3 of 8. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Struct: renamed depth -> t_alt_count, total_depth -> t_depth; added t_ref_count, hgvsp_short, exon_number. Field order and grouping (core fields, then trailing custom columns) matches the plan brief exactly. from_reformatted_record: wires in Task 1-3's helpers (hgvsp_to_short, get_exon_number, extract_ref_depth + the 3-tuple extract_depth_from_sample_data). from_reformatted_record_multi: updated the two allele-specific-depth field references to the new names, logic unchanged. get_maf_headers/to_tsv_line: rewritten to the exact 50-column vcf2maf-parity layout (46 core columns + 4 trailing custom columns: FILTER, QUAL, VAF, Protein_Position), with literal "." at the columns this tool has no backing data for (matched-normal/validation/UUID/ n_* columns, all_effects). Added two new unit tests (header layout, tsv-line dot-fill) to essentials_fields.rs's own test module, TDD'd red->green. Added a local test-only equivalent of tests/test.rs's create_test_maf_record helper since that one lives in a separate integration-test crate and isn't reachable from here. Does not touch tests/test.rs (Task 6) or src/parquet_writer.rs (Task 5) - both still reference the old depth/total_depth field names and are expected to fail to build until those tasks land. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Replace build_maf_schema/build_maf_batch's ~190-line hand-rolled 30-column Arrow builder copy of the old MAF layout (which no longer compiled after Task 4's MafRecord rewrite, and had independently carried the same t_depth/t_alt_count mislabel Task 4 fixed in the TSV/MAF-text path) with a short version driven directly by MafRecord::get_maf_headers()/to_tsv_line(). Schema and data can no longer drift apart since both come from the same source of truth. Added test_parquet_maf_columns_match_get_maf_headers (TDD: written first, confirmed failing on the pre-fix compile error, passes after the rewrite). build_tsv_schema/build_tsv_batch/get_tsv_string_value (unrelated TSV Parquet path) untouched; their Float64Builder/UInt64Builder/StringBuilder imports still used there, left in place.
- Clarify transcript-handling mode descriptions and warnings (`main.rs`).
- Preserve VCF chromosome naming instead of normalizing to a bare number.
- Add tumor and normal sample-specific depth columns (t/n_ref_count, t/n_alt_count, n_depth).
- Strip reference-sequence accession from HGVS notations (`essentials_fields.rs`).
- Generate synthetic `p.X{codon}_splice` annotations for splice sites.
…ary.rs report added
- Implement `merge_damage_breakdowns` to process damage counts incrementally. - Replace intermediate record storage with per-chunk breakdown folding in `convert_to_maf_records`. - Update HTML report generation to use precomputed breakdowns. - Introduce the `compute_damage` flag to skip unnecessary computation when not generating reports.
- Introduce `ParquetSink` for chunk-wise Parquet writing. - Replace one-shot Parquet writes with `ParquetSink` API. - Migrate TSV processing to chunk-based streaming. - Refactor VCF reading to decouple metadata and streaming lines. - Enhance memory efficiency with configurable stream chunk size (`STREAM_CHUNK`).
- Introduce `count_malformed_annotation_entries` to detect truncated CSQ/ANN entries. - Add `is_symbolic_allele` to skip symbolic alleles in MAF outputs. - Warn on malformed annotations with contextual details in stderr. - Update CLI to clarify empty column handling for optional MAF fields. - Validate MAF correctness for symbolic alleles and annotation integrity in test cases. - Drop unused `comfy-table` dependency for simpler builds. - Update README with revised performance metrics, MAF validation details, and Parquet improvements. - Version bump from 0.4.0 to 0.7.5.
- Add new edge case VCF fixtures for testing (`no_samples.vcf`, `no_annotation.vcf`, `crlf.vcf`, `star_alt.vcf`). - Introduce `compare_maf_columns.py` for comprehensive MAF column comparison between tools. - Add `validate_release.sh` script for end-to-end release validation matrix.
…ndling - Enable `--report html,txt` for generating both HTML and TXT reports in a single run. - Introduce `--report-dir` option to separate report files from data output. - Update headers to ensure all declared VCF fields are included, even if absent in the first record. - Improve test coverage for reporting and header generation logic. - Update README to document new features and usage examples.
…ocumentation - Introduce QEMU and Docker Hub login steps to GitHub Actions `release.yml` for multi-arch Docker image builds. - Switch Docker image namespace to `thefericcio` and enable pushing (`push: true`). - Extend README with updated output formats (TSV, Parquet, MAF), step-by-step Docker/Apptainer usage, and new CLI options like `--report html,txt`. - Minor wording improvements and corrections in documentation/reports.
There was a problem hiding this comment.
🟡 Changes recommended
Critical correctness, MSRV, annotation, and API issues remain, along with output and validation gaps.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
v0.7.5 adds streaming VCF conversion, validated MAF output, Parquet support, stdin handling, reports, validation tooling, and release infrastructure.
Changes:
- Adds chunked I/O, sample metadata flags, and gzip/stdin support.
- Adds typed Parquet output and HTML/text reports.
- Expands fixtures, documentation, packaging, and multi-platform releases.
File summaries
| File | Summary |
|---|---|
tests/scripts/validate_vep_vs_vcf2maf.sh |
vcf2maf comparison helper; mismatches do not currently fail the script. |
tests/scripts/validate_release.sh |
Release validation matrix; timing and NULL/header-only checks need tightening. |
tests/scripts/maf_header.expected |
Expected MAF schema. |
tests/scripts/compare_maf_columns.py |
MAF column comparison utility. |
tests/fixtures/edge/symbolic_alt.vcf |
Symbolic ALT edge-case fixture. |
tests/fixtures/edge/star_alt.vcf |
Star ALT edge-case fixture. |
tests/fixtures/edge/no_samples.vcf |
No-sample edge-case fixture. |
tests/fixtures/edge/no_annotation.vcf |
No-annotation edge-case fixture. |
tests/fixtures/edge/malformed_csq.vcf |
Malformed CSQ edge-case fixture. |
tests/fixtures/edge/empty.vcf |
Empty-input fixture. |
tests/fixtures/edge/crlf.vcf |
CRLF input fixture. |
src/summary.rs |
Streaming summary aggregation; severity categories need stable ordering. |
src/reformat_vcf.rs |
Streaming annotation conversion; schema discovery, annotation selection, and public API compatibility issues remain. |
src/read_vcf_gz.rs |
Plain, gzip, and stdin reading; short reads can misclassify gzip input. |
src/parquet_writer.rs |
Typed Parquet writing; missing fields can be emitted as . instead of NULL. |
src/main.rs |
CLI orchestration; empty-input schemas, sample validation, report errors, and version reporting need correction. |
src/lib.rs |
Module exports. |
src/html_report.rs |
HTML report rendering; chart values lack keyboard and screen-reader access. |
src/get_info_from_header.rs |
Annotation header parsing. |
src/extract_sample_info.rs |
Sample and FORMAT metadata extraction. |
src/extract_csq_and_csq_names.rs |
CSQ extraction; tag matching needs field boundaries. |
src/extract_ann_and_ann_names.rs |
ANN extraction. |
README.md |
Usage, release, Parquet, and validation documentation. |
Dockerfile |
Container image build. |
Cargo.toml |
Package version, dependencies, and release configuration. |
.gitignore |
Generated validation output exclusions. |
.github/workflows/release.yml |
Multi-platform release publishing. |
Review details
Suppressed comments (7)
src/html_report.rs:134
- The damage chart exposes per-segment category and count only through mouse-hover handlers on SVG
rectelements. These elements are neither focusable nor given an accessible name/role, so keyboard and screen-reader users cannot discover the values; add focus/keyboard support with accessible labels or provide an equivalent textual table.
svg += '<rect class="seg" x="' + x + '" y="' + y + '" width="' + segWidth + '" height="' + barHeight + '" fill="' + data.colors[ci] + '"'
+ ' data-chrom="' + escapeHtml(row.chrom) + '" data-cat="' + escapeHtml(data.categories[ci]) + '"'
+ ' data-count="' + count + '" data-pct="' + pct.toFixed(1) + '"></rect>';
src/main.rs:1073
- The startup banner reports
v0.4.0even though this release/package is0.7.5, making the user-visible version misleading. Use the package version at compile time instead of a hardcoded old value.
println!("🧬 VCF REFORMATTER v0.4.0");
src/main.rs:181
--tumor-idand--normal-idare documented as exact sample names, but neither is validated against#CHROM. A typo is accepted and produces successful output with empty tumor depths; a nonexistent normal is still written toMatched_Norm_Sample_Barcode. Validate both names once fromcolumns_titleand fail with a clear error.
/// Name of the tumor sample, exactly as it appears in the VCF's #CHROM line.
///
/// The t_depth / t_ref_count / t_alt_count columns are read from this sample. Without it
/// the first sample declaring DP is used, which is a guess — set this on any multi-sample
/// VCF, where guessing can report the wrong sample's read counts.
#[arg(long)]
tumor_id: Option<String>,
src/main.rs:1194
- Report-generation errors are reduced to warnings, so a requested
--report htmlortxtcan leave the command with exit status 0 and no report (for example, an unwritable--report-dir). This makes successful processing look like a complete release artifact; propagate the error and terminate non-zero, consistently with the data and Parquet write paths.
if let Err(e) =
html_report::write_html_report(&summary_stats, &damage_breakdowns, &summary_file)
{
eprintln!("Warning: Could not write summary report: {}", e);
tests/scripts/validate_release.sh:275
cell_mismatchestreats both""and"."as valid text for a Parquet NULL, andnull_cellsis only recorded as INFO. The check therefore passes even if missing string fields are encoded as literal.and does not validate the promised NULL-vs-source-dot distinction. Add assertions for missing declared INFO/FORMAT fields and literal source dots.
eq parquet "$ann/$fmt/cell_mismatches" "$(g cell_mismatches)" 0 \
"cell-by-cell, keyed by column name"
eq parquet "$ann/$fmt/schema_violations" "$(g schema_violations)" 0 \
"positions/depths uint64, QUAL/VAF double, rest string — $(g schema_detail)"
row parquet "$ann/$fmt/null_cells" "$(g null_cells)" INFO "empty text cell maps to NULL"
tests/scripts/validate_release.sh:593
- This check explicitly records a valid header-only VCF producing zero TSV lines as INFO, so the release matrix accepts the missing output schema instead of catching it. A header-only VCF should still produce one TSV header line; make this an assertion so the empty-input regression cannot be reported as a successful validation.
# A header-only VCF: exits 0, and the output carries no header row either, because
# columns are derived from record #1 and there is none. Recorded, not asserted — the
# plan expected a header-only output file.
row edge "empty/tsv_lines" "$(wc -l < "$WORK/edge_empty_tsv_reformatted.tsv" | tr -d ' ')" \
INFO "0 = no column header either; columns come from record #1, which does not exist"
tests/scripts/validate_vep_vs_vcf2maf.sh:152
- This helper only prints the mismatch counters and then exits successfully. A regression in Hugo/Variant/End_Position or unmatched rows therefore cannot fail a CI or manual validation run; return non-zero when any unexpected counter is nonzero while preserving cleanup.
print "Hugo_Symbol diffs on matched rows: " hs_diff " / " matched
print "Variant_Classification diffs on matched rows: " vc_diff " / " matched
print "Variant_Type diffs on matched rows: " vt_diff " / " matched
print "End_Position diffs on matched rows: " ep_diff " / " matched
}'
- Files reviewed: 26/30 changed files
- Comments generated: 12
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| .as_ref()? | ||
| .samples | ||
| .iter() | ||
| .filter(|s| sample.is_none_or(|name| s.sample_name == name)) |
Comment on lines
568
to
572
| ref_depth = Some(r); | ||
| } | ||
| if depths.len() >= 2 { | ||
| if let Ok(alt_d) = depths[1].parse::<u32>() { | ||
| alt_depth = Some(alt_d); |
| if let Some(captures) = re.captures(info_field) { | ||
| if let Some(csq_match) = captures.get(1) { | ||
| let csq_value = csq_match.as_str().to_string(); | ||
| let re = Regex::new(&format!(r"{tag}=([^;]+)")).ok()?; |
| pub filter: String, | ||
| pub info_fields: HashMap<String, String>, | ||
| pub format_sample_data: Option<ParsedFormatSample>, | ||
| pub annotation_field_type: AnnotationFieldType, |
Comment on lines
244
to
+248
| info: &str, | ||
| csq_field_names: &Option<Vec<String>>, | ||
| ann_field_names: &Option<Vec<String>>, | ||
| transcript_handling: TranscriptHandling, | ||
| ) -> std::result::Result<Vec<HashMap<String, String>>, Box<dyn std::error::Error>> { | ||
| ) -> Result<ParsedInfoField, Box<dyn std::error::Error>> { |
Comment on lines
648
to
652
| if headers.is_empty() && !records.is_empty() { | ||
| headers = chunk_headers; | ||
| if let Err(e) = reformat_vcf::write_tsv_header(&mut writer, &headers) { | ||
| eprintln!("❌ Error writing file: {e}"); | ||
| std::process::exit(1); |
Comment on lines
+138
to
+140
| let value = get_tsv_string_value(r, header); | ||
| builder.append_value(value.as_ref()); | ||
| } |
Comment on lines
+57
to
+59
| let mut magic = [0u8; 2]; | ||
| let bytes_read = source.read(&mut magic)?; | ||
| let chained = Cursor::new(magic[..bytes_read].to_vec()).chain(source); |
Comment on lines
+316
to
+319
| for cat in incoming.categories { | ||
| if !existing.categories.contains(&cat) { | ||
| existing.categories.push(cat); | ||
| } |
Comment on lines
+35
to
+37
| peak_rss() { | ||
| /usr/bin/time -l "$@" >/dev/null 2> "$WORK/rss.err" || true | ||
| awk '/maximum resident set size/{print int($1/1048576); exit}' "$WORK/rss.err" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3.
cc @aalnawfal1992 @tetedange13 as promised.
The Parquet notes are in their own section below,
could you run the Polars/DuckDB check? If you have time to point your real VCFs at it, that would be great.
This is the v0.7.5 release: 38 commits covering:
What is in it
MAF output validated against vcf2maf. 46 core columns in vcf2maf's own order and naming, plus 4 of ours (
FILTER,QUAL,VAF,Protein_Position). Checked column by column on three real VEP files (12,239 / 34,415 / 92,216 variants) in all three transcript modes: 0 unexplained differences. Every remaining difference sits in a named, counted bucket.Streaming I/O. The reader hands back an iterator and both output paths convert and write in 10,000-line chunks. Peak RSS on the 92k-variant file went 2,638 MB to 333 MB for TSV and 2,930 MB to 437 MB for MAF with
--parquet. Verified flat by running one file against 8 copies of itself: TSV 232 MB to 238 MB.New flags:
--tumor-id,--normal-id,--mutation-status,--sequence-source,--report html|txt|none,--parquet, and-for stdin.Validation harness:
tests/scripts/validate_release.sh, sectionsmaf|parquet|stdin|reports|flags|bench|edge|memory. Latest run: 163 PASS / 130 INFO / 0 FAIL over 293 checks. It exits non-zero if any check fails.Container images. Releases now publish
thefericcio/vcf-reformatterto Docker Hub forlinux/amd64andlinux/arm64. An Apptainer.sifconverts from it directly on a login node, no daemon and no root.Parquet (#3)
ArrowWritera chunk at a time.uint64, notint64.QUALandVAFaredouble, everything else is a string.--output-format.--parquetwrites the text file and a Parquet copy next to it. Your--output-format parquetwould make them exclusive..becomes NULL, with one distinction. An unpopulated cell is a real NULL. A.that came from the VCF itself (FILTER, theIDfallback, a missing FORMAT field) stays the string..--parquetrejects-c/--compress.arrow's unusedprettyprintfeature is dropped. Yourdefault-features = falsesuggestion is untried. Happy to discuss it for a next release.Parquet is verified lossless against the sibling text output: 0 mismatches across 35.9M cells. The schema is asserted too, not only the values.
@tetedange13, on ANNOVAR: I will open a separate issue for it.
Behaviour changes worth knowing
., matching vcf2maf's rendering of a tumor-only MAF.t_depthcomes from the tumor sample'sFORMAT/DP, notINFO/DP.Strandis always+. Per the MAF spec the column is genomic. The transcript strand is still in the TSV'sCSQ_STRAND.Mutation_StatusandSequence_Sourceare no longer hardcoded toSomaticandWXS. Nothing in a VCF establishes either, so they are empty unless you pass the flags.chrprefix passes through unchanged rather than being stripped.#CHROMline now exits non-zero. Empty stdin used to exit 0 over a 1-byte file.<DEL>, breakends) produce no MAF row. They used to come out typedINS, an insertion of the five literal characters<DEL>. The TSV path still carries them verbatim.##INFO/##FORMATdeclarations rather than the first record, so a field absent from early variants is no longer dropped from the whole file.tests/.includewas pulling in gitignored files.Tests
98 unit + 99 integration + 1 doctest, 0 failures.
cargo fmt --all -- --checkandcargo clippy --all-targets -- -D warningsclean.cargo packageverification build passes.