You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanks for the kind words - and great feature request!
Parquet support makes total sense for vcf-reformatter. Columnar storage + compression would be an awesome feature for large variant datasets and downstream tools.
This is definitely on my radar now. If you (or anyone) wants to contribute, I'd welcome the help! Otherwise I'll prioritize it for an upcoming release⌛️.
I would be happy to work on Parquet output since I would also find it to be very helpful. I see it's assigned to you, are you already on it or would it be okay for me to have a go at it?
If it's free, here is how I was generally thinking, so you can guide me early if you would prefer it to go a different way.
--output-format parquet option alongside the existing tsv and maf, writing the same columns as the TSV output
POS as INT64 and QUAL as DOUBLE, the rest as string
VCF's . stored as a real NULL
-c/--compress selecting the gzip codec instead of wrapping the file, since Parquet already compresses per column (output stays .parquet, snappy by default)
Rows written in bounded row groups to keep memory flat
Using the parquet + arrow crates, with default-features to false to keep the dependency tree lean
I would follow the existing module layout and test style, and make sure the Parquet output matches the TSV output.
I’m happy to change any of this to suit your preferences. Please let me know if you want me to proceed.
I'm very close to shipping a number of new features, including Parquet, so I'm working on this issue already.
I'll tag you on the PR when I open it (probably in the next few days). It would be great if you could review it and test the output against your real-world use case, especially with Polars/DuckDB downstream.
And if you'd like to contribute code, help would be very welcome elsewhere, for example adding ANNOVAR annotation support.
I'm very close to shipping a number of new features, including Parquet, so I'm working on this issue already.
I'll tag you on the PR when I open it (probably in the next few days). It would be great if you could review it and test the output against your real-world use case, especially with Polars/DuckDB downstream.
And if you'd like to contribute code, help would be very welcome elsewhere, for example adding ANNOVAR annotation support.
Thanks again, i appreciate the nice design doc 😀
Best, Flavio
That great news thanks for the quick reply. Happy to hold off then.
Please do tag me on the PR. I'll test it against my real VCFs and check the Polars/DuckDB path specifically. ANNOVAR support sounds interesting I can't promise a timeline right now but if you open an issue for it I'll take a proper look and let you know if I can pick it up.
As promised, here is the PR: #4. Parquet output is in it, along with the rest of the v0.7.5 work.
@aalnawfal1992, your design sketch was close to what I ended up shipping.
The differences, so you can see them before testing:
Bounded row groups: agreed, and implemented. One row group per 65,536 rows.
Positions are uint64, not int64. A genomic coordinate should never be negative :). QUAL and VAF are double, everything else is a string. If uint64 causes issues with Polars, please let me know.
A flag rather than a third --output-format.--parquet writes the text file and a Parquet copy with it, instead of replacing it. I went additive so existing pipelines keep their TSV.
. becomes NULL, with one distinction. An unpopulated cell is a real NULL. A . that came from the VCF itself (FILTER, the ID fallback, a missing FORMAT field) stays the string ..
ZSTD instead of snappy, 20 to 32% smaller with no measurable write cost on disk/memory. --parquet rejects -c/--compress.
Dependencies: I dropped arrow's unused prettyprint feature. Your default-features = false suggestion is untried, and we could discuss it.
Parquet is verified lossless against the text output, cell by cell keyed on column name, across both output formats and both annotators: 0 mismatches over 35.9M cells.
The schema is asserted as well, not only the values.
Could you run the Polars/DuckDB check? If you have time to point your real VCFs at it, that would be great.
@tetedange13, thank you for the pointers to natir/vcf2parquet and variantplaner.
I will open a separate issue for ANNOVAR support so it has its own home, but this will be in v0.9, solidifying MAF for snpEff is my next step.
Activity
flalom commentedon Nov 5, 2025
Hi Felix 👋
Thanks for the kind words - and great feature request!
Parquet support makes total sense for vcf-reformatter. Columnar storage + compression would be an awesome feature for large variant datasets and downstream tools.
This is definitely on my radar now. If you (or anyone) wants to contribute, I'd welcome the help! Otherwise I'll prioritize it for an upcoming release⌛️.
Best,
Flavio
aalnawfal1992 commentedon Aug 26, 2026
Hi @flalom
I would be happy to work on Parquet output since I would also find it to be very helpful. I see it's assigned to you, are you already on it or would it be okay for me to have a go at it?
If it's free, here is how I was generally thinking, so you can guide me early if you would prefer it to go a different way.
--output-format parquetoption alongside the existing tsv and maf, writing the same columns as the TSV outputPOSasINT64andQUALasDOUBLE, the rest as string.stored as a real NULL-c/--compressselecting the gzip codec instead of wrapping the file, since Parquet already compresses per column (output stays.parquet, snappy by default)parquet+arrowcrates, withdefault-featuresto false to keep the dependency tree leanI would follow the existing module layout and test style, and make sure the Parquet output matches the TSV output.
I’m happy to change any of this to suit your preferences. Please let me know if you want me to proceed.
Thank you for the tool, it is really fast 🚀
flalom commentedon Aug 26, 2026
Hi @aalnawfal1992 👋
Thank you for such a well-thought-out proposal!
I'm very close to shipping a number of new features, including Parquet, so I'm working on this issue already.
I'll tag you on the PR when I open it (probably in the next few days). It would be great if you could review it and test the output against your real-world use case, especially with Polars/DuckDB downstream.
And if you'd like to contribute code, help would be very welcome elsewhere, for example adding ANNOVAR annotation support.
Thanks again, i appreciate the nice design doc 😀
Best,
Flavio
tetedange13 commentedon Aug 26, 2026
Because ANNOVAR annotations are not nested, I personally use https://github.com/natir/vcf2parquet or equivalent subcommand from variant-planer : https://seqoia-it.github.io/variantplaner/usage/#clinvar-annotations
Thanks!
Best,
Felix.
aalnawfal1992 commentedon Aug 26, 2026
That great news thanks for the quick reply. Happy to hold off then.
Please do tag me on the PR. I'll test it against my real VCFs and check the Polars/DuckDB path specifically. ANNOVAR support sounds interesting I can't promise a timeline right now but if you open an issue for it I'll take a proper look and let you know if I can pick it up.
Best,
AM
flalom commentedon Sep 12, 2026
Hi @aalnawfal1992 and @tetedange13 👋
As promised, here is the PR: #4. Parquet output is in it, along with the rest of the v0.7.5 work.
@aalnawfal1992, your design sketch was close to what I ended up shipping.
The differences, so you can see them before testing:
uint64, notint64. A genomic coordinate should never be negative :).QUALandVAFaredouble, everything else is a string. Ifuint64causes issues with Polars, please let me know.--output-format.--parquetwrites the text file and a Parquet copy with it, instead of replacing it. I went additive so existing pipelines keep their TSV..becomes NULL, with one distinction. An unpopulated cell is a real NULL. A.that came from the VCF itself (FILTER, theIDfallback, a missing FORMAT field) stays the string..--parquetrejects-c/--compress.arrow's unusedprettyprintfeature. Yourdefault-features = falsesuggestion is untried, and we could discuss it.Parquet is verified lossless against the text output, cell by cell keyed on column name, across both output formats and both annotators: 0 mismatches over 35.9M cells.
The schema is asserted as well, not only the values.
Could you run the Polars/DuckDB check? If you have time to point your real VCFs at it, that would be great.
@tetedange13, thank you for the pointers to
natir/vcf2parquetandvariantplaner.I will open a separate issue for ANNOVAR support so it has its own home, but this will be in v0.9, solidifying MAF for snpEff is my next step.
Best,
Flavio
v0.7.5: validated MAF output, streaming I/O, Parquet, stdin