Open source LLM evaluation: accuracy + cost + hallucination for flow evaluation

[vigneshwar (vignesh2027)]

Hey PromptFlow community!

PromptFlow is excellent for building and testing LLM workflows. For teams who need systematic benchmarking of the LLM nodes inside their flows, I built a complementary evaluation framework.

LLM Evaluation Framework measures 5 metrics per model:

• Accuracy: 4-strategy cascade (exact, normalized, MC letter, fuzzy)
• Latency p95: tail latency is what fails your SLA
• Cost per 1K tokens: flow runs can be expensive, know the cost before you scale
• Hallucination Rate: detect overconfident wrong outputs before they propagate through your flow
• Reasoning Quality: CoT depth scoring for multi-step flows

Works with any LiteLLM-compatible model so it covers all the providers PromptFlow supports.

Live demo (no API key): https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework

71 tests, 82% coverage. Open source, free forever. Would love feedback on integrating with PromptFlow evaluation workflows!