Does promptflow support logging LLM calls and promptflow evaluation metrics to azure app insights?

[Sugandh Mishra (meghdivya)]

We want following features:

• Ingesting promptflow logs in app insights
• Ingesting metrics in app insights (in case of a evaluation)
• Add custom dimension into the logs and metrics.
• Log OpenAI / other model call details in app insights: total token counts, prompt token counts, output token counts and latency.
• Log logprobs details (in case of completion model) and additional responses from OpenAI

[Collin (0mza987)]

As customers can have really different demands on tracing/logging, we encourage customers to follow OTEL protocols and write their own collectors. Please refer to below guides:

• promptflow/examples/tutorials/tracing/custom-otlp-collector/otlp-trace-collector.ipynb at main · microsoft/promptflow
• Microsoft OpenTelemetry exporter for Azure Monitor | Microsoft Learn