Skip to content

v0.7.5: validated MAF output, streaming I/O, Parquet, STDIN - #4

Merged
flalom merged 40 commits into
mainfrom
feat/streaming-io
Sep 12, 2026
Merged

flalom merged 40 commits into
mainfrom
feat/streaming-io

Conversation

@flalom

@flalom flalom commented Sep 12, 2026

Copy link
Copy Markdown
Owner

Closes #3.

cc @aalnawfal1992 @tetedange13 as promised.
The Parquet notes are in their own section below,
could you run the Polars/DuckDB check? If you have time to point your real VCFs at it, that would be great.

This is the v0.7.5 release: 38 commits covering:

  • MAF correctness
  • streaming I/O
  • Parquet output
  • an HTML/txt report,
  • validation harness.

What is in it

MAF output validated against vcf2maf. 46 core columns in vcf2maf's own order and naming, plus 4 of ours (FILTER, QUAL, VAF, Protein_Position). Checked column by column on three real VEP files (12,239 / 34,415 / 92,216 variants) in all three transcript modes: 0 unexplained differences. Every remaining difference sits in a named, counted bucket.

Streaming I/O. The reader hands back an iterator and both output paths convert and write in 10,000-line chunks. Peak RSS on the 92k-variant file went 2,638 MB to 333 MB for TSV and 2,930 MB to 437 MB for MAF with --parquet. Verified flat by running one file against 8 copies of itself: TSV 232 MB to 238 MB.

New flags: --tumor-id, --normal-id, --mutation-status, --sequence-source, --report html|txt|none, --parquet, and - for stdin.

Validation harness: tests/scripts/validate_release.sh, sections maf|parquet|stdin|reports|flags|bench|edge|memory. Latest run: 163 PASS / 130 INFO / 0 FAIL over 293 checks. It exits non-zero if any check fails.

Container images. Releases now publish thefericcio/vcf-reformatter to Docker Hub for linux/amd64 and linux/arm64. An Apptainer .sif converts from it directly on a login node, no daemon and no root.

Parquet (#3)

  • Bounded row groups: agreed and implemented. One row group per 65,536 rows, fed by an open ArrowWriter a chunk at a time.
  • Positions are uint64, not int64. QUAL and VAF are double, everything else is a string.
  • A flag --output-format. --parquet writes the text file and a Parquet copy next to it. Your --output-format parquet would make them exclusive.
  • . becomes NULL, with one distinction. An unpopulated cell is a real NULL. A . that came from the VCF itself (FILTER, the ID fallback, a missing FORMAT field) stays the string ..
  • ZSTD rather than snappy, 20 to 32% smaller at no write cost. --parquet rejects -c/--compress.
  • Dependencies: arrow's unused prettyprint feature is dropped. Your default-features = false suggestion is untried. Happy to discuss it for a next release.

Parquet is verified lossless against the sibling text output: 0 mismatches across 35.9M cells. The schema is asserted too, not only the values.

@tetedange13, on ANNOVAR: I will open a separate issue for it.

Behaviour changes worth knowing

  • Unpopulated MAF columns are now empty rather than ., matching vcf2maf's rendering of a tumor-only MAF.
  • t_depth comes from the tumor sample's FORMAT/DP, not INFO/DP.
  • Strand is always +. Per the MAF spec the column is genomic. The transcript strand is still in the TSV's CSQ_STRAND.
  • Mutation_Status and Sequence_Source are no longer hardcoded to Somatic and WXS. Nothing in a VCF establishes either, so they are empty unless you pass the flags.
  • The chr prefix passes through unchanged rather than being stripped.
  • Input with no #CHROM line now exits non-zero. Empty stdin used to exit 0 over a 1-byte file.
  • Symbolic ALTs (<DEL>, breakends) produce no MAF row. They used to come out typed INS, an insertion of the five literal characters <DEL>. The TSV path still carries them verbatim.
  • TSV columns come from the VCF header's ##INFO / ##FORMAT declarations rather than the first record, so a field absent from early variants is no longer dropped from the whole file.
  • The published crate no longer ships tests/. include was pulling in gitignored files.

Tests

98 unit + 99 integration + 1 doctest, 0 failures. cargo fmt --all -- --check and cargo clippy --all-targets -- -D warnings clean. cargo package verification build passes.

flalom and others added 30 commits March 31, 2026 17:11
- Fix indel alleles: strip shared VCF anchor base per MAF spec
  (e.g., VCF A→ATCG now produces MAF ref="-", alt="TCG")
- Fix deletion start position off-by-one (now points to first deleted base)
- Fix frameshift classification: distinguish Frame_Shift_Ins vs Frame_Shift_Del
- Add defensive guard against u64 underflow on malformed VCF indels
- Reduce ~30 unnecessary .clone() calls using Cow<str>, as_str(), move semantics
- Optimize from_vcf_line: zero clones for common single-record case
- Replace parsed_lines[7].clone() with std::mem::take()
- Eliminate remaining_info_map.clone() via control-flow restructuring
- Add 4 new tests (5'UTR ins/del, 3'UTR ins, frameshift ins vs del)
- Update 5 existing tests with correct MAF expectations
- Update README for v0.4.0

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add indexmap for ordered HashMaps, chrono for timestamps, and
arrow/parquet behind the parquet_out feature flag for upcoming
summary and parquet output features.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New src/summary.rs with SummaryStats struct, chromosome counting
from VCF data lines, natural sort order (1-22, X, Y, M/MT), and
formatted text file output. Includes 5 unit tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New src/parquet_writer.rs behind parquet_out feature flag. Converts
ReformattedVcfRecords and MafRecords to parquet with Snappy compression.
Numeric columns (POS, QUAL, depth, VAF) use native Arrow types.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add --summary and --parquet CLI flags. --parquet validates against
--compress conflict. Summary and parquet writing wired into both
TSV and MAF output paths. Parquet replaces text output when enabled.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- 3 row count conservation tests (TSV first, MAF first, split transcripts)
- 4 summary correctness tests (counts sum, chromosome order, expansion ratio, file output)
- 3 parquet equivalence tests behind parquet_out feature flag (TSV row count, column names, MAF row count)

Total: 93 tests without parquet, 96 with parquet_out feature.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add summary reports and parquet output to features table, CLI options,
and usage examples.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
test_report_html_snpeff_shows_impact_metric_not_sift asserted the
positive half (Impact/HIGH/LOW present) but never checked the negative
half implied by its own name. Add !html.contains("SIFT") so the test
actually verifies VEP-only metrics are excluded when only SnpEff/ANN
data is present.
Pure function converting HGVSp protein-change strings (e.g. "p.Val600Glu") to their short 1-letter form ("p.V600E"), using vcf2maf's own %aa3to1 table so output matches the reference tool exactly.
Extracts vcf2maf's Exon_Number source: VEP's CSQ_EXON, falling back
to SnpEff's ANN_Rank. Both are formatted <rank>/<total>. Follows the
same get_annotation_field wrapper pattern as get_transcript_id and
get_protein_position. Not yet wired into from_reformatted_record
(later task) -- dead_code warning is expected until then.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Extends extract_depth_from_sample_data to also return sample-level
AD[0] (ref depth) as a fallback, mirroring the existing INFO AO /
sample AD[1] pattern for tumor depth. Return type changes from
(Option<u32>, Option<u32>) to (Option<u32>, Option<u32>, Option<u32>)
(total, ref, alt).

Updates the one call site in from_reformatted_record to thread the
new ref-depth value into a tumor_ref_depth local (compile-correctness
only — Task 4 wires it into the MafRecord struct).

Part of the vcf2maf column parity plan, Task 3 of 8.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Struct: renamed depth -> t_alt_count, total_depth -> t_depth; added
t_ref_count, hgvsp_short, exon_number. Field order and grouping (core
fields, then trailing custom columns) matches the plan brief exactly.

from_reformatted_record: wires in Task 1-3's helpers
(hgvsp_to_short, get_exon_number, extract_ref_depth + the 3-tuple
extract_depth_from_sample_data).

from_reformatted_record_multi: updated the two allele-specific-depth
field references to the new names, logic unchanged.

get_maf_headers/to_tsv_line: rewritten to the exact 50-column
vcf2maf-parity layout (46 core columns + 4 trailing custom columns:
FILTER, QUAL, VAF, Protein_Position), with literal "." at the columns
this tool has no backing data for (matched-normal/validation/UUID/
n_* columns, all_effects).

Added two new unit tests (header layout, tsv-line dot-fill) to
essentials_fields.rs's own test module, TDD'd red->green. Added a
local test-only equivalent of tests/test.rs's create_test_maf_record
helper since that one lives in a separate integration-test crate and
isn't reachable from here.

Does not touch tests/test.rs (Task 6) or src/parquet_writer.rs (Task
5) - both still reference the old depth/total_depth field names and
are expected to fail to build until those tasks land.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4PwQ4o6RBP9xaj1hkR4NA
Replace build_maf_schema/build_maf_batch's ~190-line hand-rolled
30-column Arrow builder copy of the old MAF layout (which no longer
compiled after Task 4's MafRecord rewrite, and had independently
carried the same t_depth/t_alt_count mislabel Task 4 fixed in the
TSV/MAF-text path) with a short version driven directly by
MafRecord::get_maf_headers()/to_tsv_line(). Schema and data can no
longer drift apart since both come from the same source of truth.

Added test_parquet_maf_columns_match_get_maf_headers (TDD: written
first, confirmed failing on the pre-fix compile error, passes after
the rewrite).

build_tsv_schema/build_tsv_batch/get_tsv_string_value (unrelated TSV
Parquet path) untouched; their Float64Builder/UInt64Builder/StringBuilder
imports still used there, left in place.
- Clarify transcript-handling mode descriptions and warnings (`main.rs`).
- Preserve VCF chromosome naming instead of normalizing to a bare number.
- Add tumor and normal sample-specific depth columns (t/n_ref_count, t/n_alt_count, n_depth).
- Strip reference-sequence accession from HGVS notations (`essentials_fields.rs`).
- Generate synthetic `p.X{codon}_splice` annotations for splice sites.
- Implement `merge_damage_breakdowns` to process damage counts incrementally.
- Replace intermediate record storage with per-chunk breakdown folding in `convert_to_maf_records`.
- Update HTML report generation to use precomputed breakdowns.
- Introduce the `compute_damage` flag to skip unnecessary computation when not generating reports.
- Introduce `ParquetSink` for chunk-wise Parquet writing.
- Replace one-shot Parquet writes with `ParquetSink` API.
- Migrate TSV processing to chunk-based streaming.
- Refactor VCF reading to decouple metadata and streaming lines.
- Enhance memory efficiency with configurable stream chunk size (`STREAM_CHUNK`).
- Introduce `count_malformed_annotation_entries` to detect truncated CSQ/ANN entries.
- Add `is_symbolic_allele` to skip symbolic alleles in MAF outputs.
- Warn on malformed annotations with contextual details in stderr.
- Update CLI to clarify empty column handling for optional MAF fields.
- Validate MAF correctness for symbolic alleles and annotation integrity in test cases.
- Drop unused `comfy-table` dependency for simpler builds.
- Update README with revised performance metrics, MAF validation details, and Parquet improvements.
- Version bump from 0.4.0 to 0.7.5.
- Add new edge case VCF fixtures for testing (`no_samples.vcf`, `no_annotation.vcf`, `crlf.vcf`, `star_alt.vcf`).
- Introduce `compare_maf_columns.py` for comprehensive MAF column comparison between tools.
- Add `validate_release.sh` script for end-to-end release validation matrix.
…ndling

- Enable `--report html,txt` for generating both HTML and TXT reports in a single run.
- Introduce `--report-dir` option to separate report files from data output.
- Update headers to ensure all declared VCF fields are included, even if absent in the first record.
- Improve test coverage for reporting and header generation logic.
- Update README to document new features and usage examples.
…ocumentation

- Introduce QEMU and Docker Hub login steps to GitHub Actions `release.yml` for multi-arch Docker image builds.
- Switch Docker image namespace to `thefericcio` and enable pushing (`push: true`).
- Extend README with updated output formats (TSV, Parquet, MAF), step-by-step Docker/Apptainer usage, and new CLI options like `--report html,txt`.
- Minor wording improvements and corrections in documentation/reports.
Copilot AI lite review requested due to automatic review settings September 12, 2026 12:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical correctness, MSRV, annotation, and API issues remain, along with output and validation gaps.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

v0.7.5 adds streaming VCF conversion, validated MAF output, Parquet support, stdin handling, reports, validation tooling, and release infrastructure.

Changes:

  • Adds chunked I/O, sample metadata flags, and gzip/stdin support.
  • Adds typed Parquet output and HTML/text reports.
  • Expands fixtures, documentation, packaging, and multi-platform releases.
File summaries
File Summary
tests/scripts/validate_vep_vs_vcf2maf.sh vcf2maf comparison helper; mismatches do not currently fail the script.
tests/scripts/validate_release.sh Release validation matrix; timing and NULL/header-only checks need tightening.
tests/scripts/maf_header.expected Expected MAF schema.
tests/scripts/compare_maf_columns.py MAF column comparison utility.
tests/fixtures/edge/symbolic_alt.vcf Symbolic ALT edge-case fixture.
tests/fixtures/edge/star_alt.vcf Star ALT edge-case fixture.
tests/fixtures/edge/no_samples.vcf No-sample edge-case fixture.
tests/fixtures/edge/no_annotation.vcf No-annotation edge-case fixture.
tests/fixtures/edge/malformed_csq.vcf Malformed CSQ edge-case fixture.
tests/fixtures/edge/empty.vcf Empty-input fixture.
tests/fixtures/edge/crlf.vcf CRLF input fixture.
src/summary.rs Streaming summary aggregation; severity categories need stable ordering.
src/reformat_vcf.rs Streaming annotation conversion; schema discovery, annotation selection, and public API compatibility issues remain.
src/read_vcf_gz.rs Plain, gzip, and stdin reading; short reads can misclassify gzip input.
src/parquet_writer.rs Typed Parquet writing; missing fields can be emitted as . instead of NULL.
src/main.rs CLI orchestration; empty-input schemas, sample validation, report errors, and version reporting need correction.
src/lib.rs Module exports.
src/html_report.rs HTML report rendering; chart values lack keyboard and screen-reader access.
src/get_info_from_header.rs Annotation header parsing.
src/extract_sample_info.rs Sample and FORMAT metadata extraction.
src/extract_csq_and_csq_names.rs CSQ extraction; tag matching needs field boundaries.
src/extract_ann_and_ann_names.rs ANN extraction.
README.md Usage, release, Parquet, and validation documentation.
Dockerfile Container image build.
Cargo.toml Package version, dependencies, and release configuration.
.gitignore Generated validation output exclusions.
.github/workflows/release.yml Multi-platform release publishing.
Review details

Suppressed comments (7)

src/html_report.rs:134

  • The damage chart exposes per-segment category and count only through mouse-hover handlers on SVG rect elements. These elements are neither focusable nor given an accessible name/role, so keyboard and screen-reader users cannot discover the values; add focus/keyboard support with accessible labels or provide an equivalent textual table.
            svg += '<rect class="seg" x="' + x + '" y="' + y + '" width="' + segWidth + '" height="' + barHeight + '" fill="' + data.colors[ci] + '"'
                 + ' data-chrom="' + escapeHtml(row.chrom) + '" data-cat="' + escapeHtml(data.categories[ci]) + '"'
                 + ' data-count="' + count + '" data-pct="' + pct.toFixed(1) + '"></rect>';

src/main.rs:1073

  • The startup banner reports v0.4.0 even though this release/package is 0.7.5, making the user-visible version misleading. Use the package version at compile time instead of a hardcoded old value.
    println!("🧬 VCF REFORMATTER v0.4.0");

src/main.rs:181

  • --tumor-id and --normal-id are documented as exact sample names, but neither is validated against #CHROM. A typo is accepted and produces successful output with empty tumor depths; a nonexistent normal is still written to Matched_Norm_Sample_Barcode. Validate both names once from columns_title and fail with a clear error.
    /// Name of the tumor sample, exactly as it appears in the VCF's #CHROM line.
    ///
    /// The t_depth / t_ref_count / t_alt_count columns are read from this sample. Without it
    /// the first sample declaring DP is used, which is a guess — set this on any multi-sample
    /// VCF, where guessing can report the wrong sample's read counts.
    #[arg(long)]
    tumor_id: Option<String>,

src/main.rs:1194

  • Report-generation errors are reduced to warnings, so a requested --report html or txt can leave the command with exit status 0 and no report (for example, an unwritable --report-dir). This makes successful processing look like a complete release artifact; propagate the error and terminate non-zero, consistently with the data and Parquet write paths.
        if let Err(e) =
            html_report::write_html_report(&summary_stats, &damage_breakdowns, &summary_file)
        {
            eprintln!("Warning: Could not write summary report: {}", e);

tests/scripts/validate_release.sh:275

  • cell_mismatches treats both "" and "." as valid text for a Parquet NULL, and null_cells is only recorded as INFO. The check therefore passes even if missing string fields are encoded as literal . and does not validate the promised NULL-vs-source-dot distinction. Add assertions for missing declared INFO/FORMAT fields and literal source dots.
        eq parquet "$ann/$fmt/cell_mismatches" "$(g cell_mismatches)" 0 \
            "cell-by-cell, keyed by column name"
        eq parquet "$ann/$fmt/schema_violations" "$(g schema_violations)" 0 \
            "positions/depths uint64, QUAL/VAF double, rest string — $(g schema_detail)"
        row parquet "$ann/$fmt/null_cells" "$(g null_cells)" INFO "empty text cell maps to NULL"

tests/scripts/validate_release.sh:593

  • This check explicitly records a valid header-only VCF producing zero TSV lines as INFO, so the release matrix accepts the missing output schema instead of catching it. A header-only VCF should still produce one TSV header line; make this an assertion so the empty-input regression cannot be reported as a successful validation.
    # A header-only VCF: exits 0, and the output carries no header row either, because
    # columns are derived from record #1 and there is none. Recorded, not asserted — the
    # plan expected a header-only output file.
    row edge "empty/tsv_lines" "$(wc -l < "$WORK/edge_empty_tsv_reformatted.tsv" | tr -d ' ')" \
        INFO "0 = no column header either; columns come from record #1, which does not exist"

tests/scripts/validate_vep_vs_vcf2maf.sh:152

  • This helper only prints the mismatch counters and then exits successfully. A regression in Hugo/Variant/End_Position or unmatched rows therefore cannot fail a CI or manual validation run; return non-zero when any unexpected counter is nonzero while preserving cleanup.
    print "Hugo_Symbol diffs on matched rows: " hs_diff " / " matched
    print "Variant_Classification diffs on matched rows: " vc_diff " / " matched
    print "Variant_Type diffs on matched rows: " vt_diff " / " matched
    print "End_Position diffs on matched rows: " ep_diff " / " matched
}'
  • Files reviewed: 26/30 changed files
  • Comments generated: 12
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/essentials_fields.rs
.as_ref()?
.samples
.iter()
.filter(|s| sample.is_none_or(|name| s.sample_name == name))
Comment thread src/essentials_fields.rs Outdated
Comment on lines 568 to 572
ref_depth = Some(r);
}
if depths.len() >= 2 {
if let Ok(alt_d) = depths[1].parse::<u32>() {
alt_depth = Some(alt_d);
Comment thread src/extract_csq_and_csq_names.rs Outdated
if let Some(captures) = re.captures(info_field) {
if let Some(csq_match) = captures.get(1) {
let csq_value = csq_match.as_str().to_string();
let re = Regex::new(&format!(r"{tag}=([^;]+)")).ok()?;
Comment thread src/reformat_vcf.rs
pub filter: String,
pub info_fields: HashMap<String, String>,
pub format_sample_data: Option<ParsedFormatSample>,
pub annotation_field_type: AnnotationFieldType,
Comment thread src/reformat_vcf.rs
Comment on lines 244 to +248
info: &str,
csq_field_names: &Option<Vec<String>>,
ann_field_names: &Option<Vec<String>>,
transcript_handling: TranscriptHandling,
) -> std::result::Result<Vec<HashMap<String, String>>, Box<dyn std::error::Error>> {
) -> Result<ParsedInfoField, Box<dyn std::error::Error>> {
Comment thread src/main.rs Outdated
Comment on lines 648 to 652
if headers.is_empty() && !records.is_empty() {
headers = chunk_headers;
if let Err(e) = reformat_vcf::write_tsv_header(&mut writer, &headers) {
eprintln!("❌ Error writing file: {e}");
std::process::exit(1);
Comment thread src/parquet_writer.rs
Comment on lines +138 to +140
let value = get_tsv_string_value(r, header);
builder.append_value(value.as_ref());
}
Comment thread src/read_vcf_gz.rs
Comment on lines +57 to +59
let mut magic = [0u8; 2];
let bytes_read = source.read(&mut magic)?;
let chained = Cursor::new(magic[..bytes_read].to_vec()).chain(source);
Comment thread src/summary.rs
Comment on lines +316 to +319
for cat in incoming.categories {
if !existing.categories.contains(&cat) {
existing.categories.push(cat);
}
Comment thread tests/scripts/validate_release.sh Outdated
Comment on lines +35 to +37
peak_rss() {
/usr/bin/time -l "$@" >/dev/null 2> "$WORK/rss.err" || true
awk '/maximum resident set size/{print int($1/1048576); exit}' "$WORK/rss.err"
@flalom flalom added enhancement New feature or request feature Something new labels Sep 12, 2026
@flalom
flalom merged commit 0a295b4 into main Sep 12, 2026
4 checks passed
@flalom
flalom deleted the feat/streaming-io branch September 12, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request feature Something new

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Output as parquet ?

2 participants