Repository navigation
Description
[Bug] Instance Trace cannot reliably identify the selected Kubernetes Pod
Motivation / Problem
Dubbo Admin, the Grafana Instance Trace dashboard, and the trace data use inconsistent meanings for an “instance”.
Dubbo Admin passes IP:QoSPort, while the Instance Trace dashboard filters spans with net.peer.name=${instance}. net.peer.name describes the remote peer rather than the Kubernetes Pod that owns the current span. With multiple replicas of the same application, selecting an instance therefore cannot reliably isolate that Pod’s traces.
Current State
| Area | Current behavior |
|---|---|
| Instance Metric | Uses IP:QoSPort as the instance identity |
| Instance Trace URL | Also passes IP:QoSPort |
| Grafana Instance Trace | Filters with service=${application} and net.peer.name=${instance} |
| Trace data | Does not provide a reliable own-Pod identity for this query |
Desired Behavior
Instance Metric should keep its existing identity:
var-instance=IP:QoSPort
Instance Trace should use the selected Kubernetes Pod identity:
var-application=<AppName>var-instance=<bare Pod IP>
The Instance Trace dashboard should query:
service=${application}k8s.pod.ip=${instance}
Traces should also carry Pod metadata that can be checked against the Kubernetes API:
k8s.pod.ipk8s.pod.uidk8s.pod.namek8s.namespace.name
The bare Pod IP is used as the query key because the selected Admin instance already exposes its Pod IP directly. Pod UID, name, and namespace remain supporting identity metadata and runtime verification evidence.
Scope
Included
- Align the Instance Trace identity used by Dubbo Admin and Grafana.
- Route sample traces through an OpenTelemetry Collector before Jaeger.
- Enrich traces with own-Pod Kubernetes metadata using
k8sattributes. - Add the namespace-scoped, read-only Pod access required by the Collector.
- Preserve existing Instance Metric and Service Trace semantics.
Not Included
- Rebuilding, publishing, or updating the currently referenced
shop-userimage. - Log observability or cross-signal navigation such as Trace → Log, Log → Trace, or Trace → Metric.
- Prometheus behavior or metric pipeline changes.
- Query semantic changes to the other five existing dashboards.
- The disposable Kind test harness or local E2E image artifacts.
Acceptance Criteria
- Instance Trace receives the selected instance’s
AppNameand bare Pod IP. - Instance Metric continues to use
IP:QoSPort. - The Instance Trace dashboard filters by
service=${application}andk8s.pod.ip=${instance}. - Traces contain Pod IP, UID, name, and namespace values matching Kubernetes truth.
- For two replicas of the same application, querying Pod A by IP returns Pod A traces and excludes Pod B; the reverse query behaves the same way.
- Existing Service Trace behavior remains unchanged.
- The other five dashboard semantics remain unchanged.
- Relevant Go tests, manifest/dashboard validation, and static regression checks pass.
Context / Design Notes
The Admin variable, Grafana filter, and span resource metadata form one instance-identity contract. Changing only one layer would leave Instance Trace semantics inconsistent, so they must be verified together.
Historical introduction
This issue was formed by two historical observability PRs:
- PR feat: add deploy manifests and fix metric and trace dashboard bugs #1387 (
2eb4409) introduced the Grafana Instance Trace dashboard withnet.peer.name=${instance}and configured sample traces to go directly to Jaeger. - PR fix: console bugs and ci problems #1397 (
d4d7a8f) set the Admin-side Instance Dashboard variable toIP:QoSPort.
Together, these two layers made the Admin instance value unreliable as a Kubernetes Pod identity in multi-replica deployments. PR #1302 (37db89e) only migrated the existing code into this repository and is not the main functional source of this issue. PR #1569 and PR #1573 are prerequisite Service Trace and observability provisioning changes, not the introducing PRs for this issue.
The prerequisite behavior has since been merged into upstream/develop through PR #1569 and PR #1573.
Activity