Skip to content

Stream dists to reduce max memory - #81

Merged
johnlees merged 10 commits into
mainfrom
stream_dists
Jul 31, 2026
Merged

johnlees merged 10 commits into
mainfrom
stream_dists

Conversation

@johnlees

@johnlees johnlees commented Jul 29, 2026 •

Copy link
Copy Markdown
Member

Merge after #80

Closes #18

Uses formatted strings in a buffer to send to a writer thread
Keeps old functions which don't steam for e.g. PopPUNK, but CLI will use these by default

@codecov

codecov Bot commented Jul 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.37751% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.44%. Comparing base (4072a47) to head (7c5ee0d).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
src/sketch/mod.rs 85.05% 13 Missing ⚠️
src/inverted.rs 61.53% 5 Missing ⚠️
src/sketch/multisketch.rs 82.75% 5 Missing ⚠️
src/lib.rs 87.09% 4 Missing ⚠️
src/hashing/aahash_iterator.rs 66.66% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
+ Coverage   90.65%   91.44%   +0.78%     
==========================================
  Files          17       17              
  Lines        3232     3623     +391     
==========================================
+ Hits         2930     3313     +383     
- Misses        302      310       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@johnlees

Copy link
Copy Markdown
Member Author

On 10k samples, ~21Mb sketches (~18Mb in v0.3.0), 1.4Gb output file
v0.4.0 28Mb max rss, 1.1s
v0.3.0 217Mb max rss, 4.6s

Seems to work nicely, and also a bit of a performance boost as the writing out is done a lot more quickly.

@johnlees
johnlees marked this pull request as ready for review July 31, 2026 15:49
@johnlees
johnlees merged commit 81e05e3 into main Jul 31, 2026
13 checks passed
@johnlees
johnlees deleted the stream_dists branch July 31, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stream distances

1 participant