Skip to content

[Bug] Instance Trace cannot reliably identify the selected Kubernetes Pod #1583

Closed

Description

@dodjdnh

[Bug] Instance Trace cannot reliably identify the selected Kubernetes Pod

Motivation / Problem

Dubbo Admin, the Grafana Instance Trace dashboard, and the trace data use inconsistent meanings for an “instance”.

Dubbo Admin passes IP:QoSPort, while the Instance Trace dashboard filters spans with net.peer.name=${instance}. net.peer.name describes the remote peer rather than the Kubernetes Pod that owns the current span. With multiple replicas of the same application, selecting an instance therefore cannot reliably isolate that Pod’s traces.

Current State

Area Current behavior
Instance Metric Uses IP:QoSPort as the instance identity
Instance Trace URL Also passes IP:QoSPort
Grafana Instance Trace Filters with service=${application} and net.peer.name=${instance}
Trace data Does not provide a reliable own-Pod identity for this query

Desired Behavior

Instance Metric should keep its existing identity:

  • var-instance=IP:QoSPort

Instance Trace should use the selected Kubernetes Pod identity:

  • var-application=<AppName>
  • var-instance=<bare Pod IP>

The Instance Trace dashboard should query:

  • service=${application}
  • k8s.pod.ip=${instance}

Traces should also carry Pod metadata that can be checked against the Kubernetes API:

  • k8s.pod.ip
  • k8s.pod.uid
  • k8s.pod.name
  • k8s.namespace.name

The bare Pod IP is used as the query key because the selected Admin instance already exposes its Pod IP directly. Pod UID, name, and namespace remain supporting identity metadata and runtime verification evidence.

Scope

Included

  • Align the Instance Trace identity used by Dubbo Admin and Grafana.
  • Route sample traces through an OpenTelemetry Collector before Jaeger.
  • Enrich traces with own-Pod Kubernetes metadata using k8sattributes.
  • Add the namespace-scoped, read-only Pod access required by the Collector.
  • Preserve existing Instance Metric and Service Trace semantics.

Not Included

  • Rebuilding, publishing, or updating the currently referenced shop-user image.
  • Log observability or cross-signal navigation such as Trace → Log, Log → Trace, or Trace → Metric.
  • Prometheus behavior or metric pipeline changes.
  • Query semantic changes to the other five existing dashboards.
  • The disposable Kind test harness or local E2E image artifacts.

Acceptance Criteria

  • Instance Trace receives the selected instance’s AppName and bare Pod IP.
  • Instance Metric continues to use IP:QoSPort.
  • The Instance Trace dashboard filters by service=${application} and k8s.pod.ip=${instance}.
  • Traces contain Pod IP, UID, name, and namespace values matching Kubernetes truth.
  • For two replicas of the same application, querying Pod A by IP returns Pod A traces and excludes Pod B; the reverse query behaves the same way.
  • Existing Service Trace behavior remains unchanged.
  • The other five dashboard semantics remain unchanged.
  • Relevant Go tests, manifest/dashboard validation, and static regression checks pass.

Context / Design Notes

The Admin variable, Grafana filter, and span resource metadata form one instance-identity contract. Changing only one layer would leave Instance Trace semantics inconsistent, so they must be verified together.

Historical introduction

This issue was formed by two historical observability PRs:

Together, these two layers made the Admin instance value unreliable as a Kubernetes Pod identity in multi-replica deployments. PR #1302 (37db89e) only migrated the existing code into this repository and is not the main functional source of this issue. PR #1569 and PR #1573 are prerequisite Service Trace and observability provisioning changes, not the introducing PRs for this issue.

The prerequisite behavior has since been merged into upstream/develop through PR #1569 and PR #1573.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions