Repository navigation
fix(cel): rebuild the resource binding per evaluation and surface cancelled prometheusMetrics queries - #213
Conversation
…celled prometheusMetrics queries A reused variables map kept the first evaluation's context in its resource binding, so a later evaluation could fail on an earlier cancellation or ignore its own deadline. The binding is now rebuilt on every evaluation. A prometheusMetrics query cancelled by its context returned an error map; it now returns an error wrapping the context error. Other query errors keep the error map. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approved
No active review issues are recorded, although tests were not run during this review.
No open issues 🎉
What changed in this PR
This PR fixes context handling in OttoFlow’s CEL evaluator so reused variables honor the current evaluation’s context and cancelled Prometheus queries surface as errors.
Changes:
- Rebuilds the
resourcebinding on every evaluation to avoid retaining an earlier context. - Returns a CEL error when a Prometheus query error wraps the query context’s cancellation error, while preserving error maps for unrelated failures.
- Adds regression tests for reused-context deadlines and Prometheus cancellation handling.
Tests were not run during this review.
| File | Description |
|---|---|
internal/workflow/executor/resource_macros.go |
Propagates context-related Prometheus query failures as CEL errors. |
internal/workflow/executor/cel.go |
Refreshes the resource binding for each evaluation. |
internal/workflow/executor/cel_ctx_binding_test.go |
Tests deadline handling and cancellation error classification. |
Review effort: Balanced
patelrit
left a comment
There was a problem hiding this comment.
The code looks like a faithful port of ottoflow-enterprise#250. The resource binding is rebuilt every call (cel.go:471-478), and the prometheusMetrics cancel check uses errors.Is on the query ctx (resource_macros.go:550-554), so there's no string matching and the Prometheus URL isn't leaked. The tests cover T13/T14/T14b/T14c and fail on revert. I also checked that dropping the vars["resource"] skip doesn't break resourceQuery outputs, since BuildVariableMap never passes "resource" through.
Two things before I can approve:
- CI never got past compile on this head. It fails on
undefined: runIsTerminal, which is the main breakage #211 fixed, and the branch is 8 commits behind main. Can you update from main so CI actually runs? - There's no e2e (kind) run on this head. Please start the E2E workflow (workflow_dispatch is fine) on the updated SHA.
Not blocking: T12 (flat resource.List cancel attribution) doesn't apply here because OSS doesn't have ContextEval/cancelRec yet. Worth a follow-up issue to port the rest of #250's interruptibility work.
patelrit
left a comment
There was a problem hiding this comment.
Both things I asked for at 6d148a0 are done on 169eeab.
- CI is green on 169eeab (run 42056971970). lint, build, test, verify-codegen and validate-samples all pass, so the
runIsTerminalcompile break is gone after the merge from main. CodeQL passes too. - E2E workflow_dispatch run 42056990993 on 169eeab passed, and the e2e job actually ran.
- 169eeab is only a merge from main. It doesn't touch
cel.go,resource_macros.goorcel_ctx_binding_test.go, so what I said at 6d148a0 still holds. The resource binding is rebuilt on every call (cel.go:471-478), and the cancel check useserrors.Ison the query ctx (resource_macros.go:550-554). The new tests fail on revert. - Source PR ottoflow-enterprise#250 is merged and approved.
No open threads, and Copilot is 🟢 on this head. Approving. I'd still like a follow-up issue to port the rest of #250's interruptibility work (T12 / ContextEval).
Bring in main 038e331: #174, #192, #193, #211, #213, #216, #217, #218, #219, #220, #221, #222, #223, #224, #225, #228, #229 and #230. Resolve the conflict in internal/workflow/controller/controller_unit_test.go: both sides added tests at the same spot. Keep both: this branch's TestBuildWorkflowRunnerJob_ShutdownGraceAndHardStop, and main's TestWorkflowRunReconciler_EnsureRunnerAccess_StaleCacheKeepsReplacementClusterRoleBinding and TestWorkflowRunReconciler_EnsureRunnerConfigMapWriter_StaleCacheKeepsReplacement from #192, with their helpers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
What was wrong
Two cancellation/correctness bugs in CEL evaluation:
EvaluateExpressionbuilt theresourcebinding only whenvars["resource"]was unset. A reused variables map therefore kept the first evaluation's context: after that context was cancelled, laterresource.Get/resource.Listcalls on a live context failed withcontext canceled, and a later call's own deadline was ignored.prometheusMetricsquery cancelled by its context was returned as an error map instead of an error.What changed
cel.go: theresourcebinding is rebuilt on every evaluation, as the other bindings already are.resource_macros.go: when the query context has ended and the query error wraps that context error,prometheusMetricsreturnsprometheus query cancelled: %w. The check iserrors.Is, so other query errors keep the error map.Tests
cel_ctx_binding_test.go:TestCELReusedVarsHonourLaterDeadline: one vars map, a later evaluation's own deadline is honoured. Fails if thevars["resource"]skip is restored.TestCELPrometheusMetricsCancelledQueryReturnsError: a cancelledprometheusMetricsquery (real HTTP client) returns an error satisfyingerrors.Is(err, context.DeadlineExceeded). Fails with the new check disabled.TestCELPrometheusMetricsUnrelatedErrorOnCancelledCtxKeepsErrorMap: an unrelated query error with the textupstream: context canceledon an already cancelled context keeps the error map. Passes with or without the fix; fails if the check is loosened toqctx.Err() != nilalone or to an error-text match.TestCELPrometheusMetricsDeadlineErrorOnLiveCtxKeepsErrorMap: a dial timeout wrappingcontext.DeadlineExceededon a live context keeps the error map. Fails if the check ignores the query context and matches on the error alone.go test -race,go vetand golangci-lint v2.11.4 pass oninternal/workflow/executorandinternal/metrics.Note on #113
Until #113 merges, a cancelled
prometheusMetricsinside anoutputs:value evaluated throughEvaluateOutputValueyields the literal expression string, because that method falls back to the literal on any evaluation error. The in-cluster runner's root context iscontext.Background()today, so there this needs #113. A localottoflow runreaches it when Ctrl-C or--timeoutends the context during the query; before this change that path stored the error map. Either merge order works.🤖 Generated with Claude Code