Skip to content

fix: identify instance traces by Kubernetes Pod IP - #1584

Merged
robocanic merged 2 commits into
apache:developfrom
dodjdnh:fix/instance-trace-kubernetes-identity
Oct 17, 2026
Merged

robocanic merged 2 commits into
apache:developfrom
dodjdnh:fix/instance-trace-kubernetes-identity

Conversation

@dodjdnh

@dodjdnh dodjdnh commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

fix: identify instance traces by Kubernetes Pod IP

Summary

Align Instance Trace with Kubernetes Pod identity across Dubbo Admin, Grafana, and the tracing pipeline.

Dubbo Admin passes the selected instance’s bare Pod IP to Grafana, sample traces are enriched with Kubernetes Pod metadata by an OpenTelemetry Collector, and Grafana filters Instance Trace with k8s.pod.ip.

These changes form a single instance-identity contract; changing only one layer would leave Instance Trace inconsistent.

Motivation

The previous combination of IP:QoSPort and net.peer.name could not reliably identify the Kubernetes Pod that owns the current span. With multiple replicas of the same application, the selected instance could therefore return traces that did not correspond to that Pod.

Changes

  • Keep Instance Metric identity unchanged as IP:QoSPort.
  • Pass AppName and the selected instance’s bare Pod IP to Instance Trace.
  • Change the Instance Trace dashboard filter from net.peer.name=${instance} to k8s.pod.ip=${instance}, while preserving service=${application}.
  • Route sample OTLP traces through an OpenTelemetry Collector before Jaeger.
  • Use k8sattributes to enrich traces with Pod IP, UID, name, and namespace.
  • Add a dedicated Collector ServiceAccount and namespace-scoped, read-only Pod RBAC.
  • Add Go regression coverage for Instance Metric and Instance Trace URL/variable behavior.

Verification

Area Check Result
Unit go test ./pkg/console/service ./pkg/console/handler PASS
Static Instance identity, Collector manifests, dashboard semantics, YAML/JSON parsing, and protected-dashboard regression checks PASS
Static git diff --check PASS
Runtime Two shop-user replicas generated fresh traces whose Pod IP, UID, name, and namespace matched the Kubernetes API PASS
Runtime Querying Pod A by IP returned A and excluded B; querying Pod B returned B and excluded A PASS
Regression Instance Metric remained IP:QoSPort; Instance Trace used bare Pod IP PASS
Regression Service Trace continued to pass var-application and var-service PASS
Minimal Kind runtime coverage Dynamic checks for Application Metric, Service Metric, and Application Trace NOT VERIFIED
Runtime Disposable Kind cluster cleanup PASS

The runtime fixture used a tracing-capable shop-user image rebuilt locally from the current apache/dubbo-samples source. The image currently referenced by the deployment manifest does not emit traces with this tracing configuration. A separate follow-up change in apache/dubbo-samples will address the tracing-capable sample image. This PR does not modify or publish that image.

Review Focus

  1. Is bare Pod IP the appropriate identity exposed by Admin for Instance Trace?
  2. Does k8sattributes reliably associate each trace with its own Kubernetes Pod?
  3. Is the Collector RBAC limited to the minimum namespace-scoped, read-only Pod access required for enrichment?
  4. Are Instance Metric, Service Trace, and the other existing dashboard semantics preserved?

Compatibility / Impact

  • Instance Metric identity remains IP:QoSPort.
  • Service Trace continues to use var-application and var-service.
  • The other five protected dashboard semantics remain unchanged.
  • No Log, cross-signal navigation, or Prometheus behavior is changed.
  • The sample tracing path changes from direct-to-Jaeger to workload → OpenTelemetry Collector → Jaeger.

Scope

This PR does not include:

  • Rebuilding, publishing, or updating the currently referenced shop-user image.
  • Logs, Trace → Log, Log → Trace, Trace → Metric, or Prometheus changes.
  • Disposable Kind harness files or local E2E image artifacts.

Related Issue

Fixes #1583

Dependency

Built on the changes merged from PR #1569 and PR #1573.

@sonarqubecloud

Copy link
Copy Markdown

@robocanic robocanic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@robocanic
robocanic merged commit 1b60b72 into apache:develop Oct 17, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Instance Trace cannot reliably identify the selected Kubernetes Pod

2 participants