Skip to content

Update README with streaming support, benchmark numbers, and improved charts - #12

Merged
bhope merged 1 commit into
mainfrom
update-readme
Apr 15, 2026
Merged

bhope merged 1 commit into
mainfrom
update-readme

Conversation

@bhope

@bhope bhope commented Apr 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • Added streaming / LLM inference section explaining TTFH vs TTFT signal distinction and how hedge handles prefill disaggregated architectures
  • Updated benchmark table with concrete numbers from 50k-request runs (p99: 64ms → 17ms, 74% reduction, ~9% overhead)
  • Added no-hedging baseline to streaming latency chart for meaningful comparison
  • Regenerated both eval.png and new eval_streaming.png charts using matplotlib with consistent styling
  • Minor fixes: observability API, streaming table numbers, chart title

@bhope
bhope merged commit 1d168b1 into main Apr 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant