Skip to content

feat!: consume contextual bandit payload - #174

Open
vazarkevych wants to merge 9 commits into
mainfrom
feat/contextual-bandits
Open

vazarkevych wants to merge 9 commits into
mainfrom
feat/contextual-bandits

Conversation

@vazarkevych

@vazarkevych vazarkevych commented Aug 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds consumption of the contextual-bandit payload, so a feature rule referencing a contextual bandit is evaluated with the per-segment variation weights the backend computed instead of being skipped.

Today such a rule is silently ignored: its variations ship under contextualVariations, and the evaluator skips on variations == null. Graceful degradation works as designed, but bandits do not function at all.

All the optimization stays server-side. The SDK only consumes ready weights: resolve the referenced definition, pick the leaf whose condition matches the user, apply its weights, and bucket through the normal mechanism.

Diff size note: 7.5k added lines look alarming, but only 389 lines land under lib/. The bulk is the shared contextualBandit conformance block embedded in test/test_cases/test_case.dart.

Scope

  • Parse contextualBandits and encryptedContextualBandits from the feature payload.
  • Resolve a rule's contextualBanditRef against the payload definitions.
  • Leaf selection: evaluate each context's condition against user attributes, first match wins.
  • Fallback: no matching leaf, or a missing/unparseable definition → leafId = -1 with the experiment's existing or equal weights.
  • Surface leafId, variationWeights, and banditVersion on GBExperimentResult, but only when the user was actually hash-bucketed in (hashUsed && inExperiment) — a force-assigned exposure must not carry bandit metadata.
  • New models: ContextualBandit, ContextualBanditContext, ContextualBanditDefinition, exported from the package entry point.

Two fixes found along the way

Both predate this work; bandits only surfaced them.

featuresAPIModelSuccessfully never fired for encrypted payloads. handleValidFeatures invoked the delegate only inside the plaintext branch, so encrypted deployments silently skipped refreshStickyBucketService(model) — sticky-bucket documents were never refreshed against a new payload. Now mirrored in the encrypted branch after a successful decrypt.

Sticky-bucket identifier scan skipped bandit rules. deriveStickyBucketIdentifierAttributes only collected a rule's hash/fallback attributes when it had a plain variations list, so bandit rules contributed nothing and sticky bucketing had no attribute to key on.

Reported weights match the assignment

The bandit learns from the reported weights, so they must be the ones the user was actually bucketed with. Beyond the JS SDK, and as in the Kotlin SDK:

  • a rule with explicit ranges, which take precedence over weights in the bucketer, reports no bandit metadata;
  • a leaf without a leafId, or whose weights the bucketer would replace (wrong length, sum outside 0.99–1.01, negative or non-finite), falls back to leafId: -1;
  • the fallback reports the weights actually used: the rule's weights if valid, otherwise equal weights.

8 new tests in contextual_bandit_test.dart; 7 fail without this change.

Test plan

  • New test/common_test/contextual_bandit_test.dart — 7 unit tests over the evaluator state matrix: matching leaf applies its weights and records metadata; no matching leaf falls back to -1 with equal weights; a ref absent from the payload leaks no metadata; first match wins when several leaves match; a malformed definition is swallowed and treated as no-bandit; contextualVariations without a ref still runs the experiment; a zero-coverage rule attributes no metadata to a non-bucketed exposure.
  • New test/common_test/contextual_bandit_conformance_test.dart — runs the 35 shared conformance cases with a per-field diff, so a failure names the exact mismatch rather than just failing. All 35 pass.
  • Regression tests for the encrypted-payload fix: the delegate fires exactly once with the original model in both the plaintext and encrypted branches, and stays silent when decryption fails.
  • flutter test — 312 pass. One pre-existing flake in caching_manager_test (concurrent test files race on the shared temp cache directory); it passes in isolation 3/3 and also fails on current main, where 2 tests fail.
  • dart analyze --fatal-infos — clean.
  • dart format --output=none --set-exit-if-changed . — clean.

BREAKING CHANGE: feature rules with a contextualBanditRef were skipped before and now enroll users, so the values of those features can change after upgrading.

Introduce data types for the contextual-bandit payload:

- ContextualBanditContext — one leaf: `leafId`, `condition`, `weights`.
- ContextualBanditDefinition — server payload: `banditVersion`, ordered
  `contexts`. Deserialized on demand during evaluation.
- ContextualBandit — evaluator output attached to GBExperiment: the
  selected `leafId`, applied `variationWeights`, and `banditVersion`.

Also extends existing models with the fields the algorithm needs:

- GBFeatureRule.contextualBanditRef / contextualVariations
- GBExperiment.contextualBandit (transient, not serialized)
- GBExperimentResult.leafId / variationWeights / banditVersion
- FeaturedDataModel.contextualBandits / encryptedContextualBandits
- GBContext.contextualBandits (in-memory bandit definitions)

No behavior change yet — subsequent commits port the evaluator.

Ports growthbook-java-internal PR #79.
Wires the bandit algorithm into the feature evaluation path:

- FeatureEvaluator prefers `contextualVariations` over `variations` when
  present so newer SDKs pick up bandit variations while older SDKs
  gracefully skip (see cross-SDK spec change).
- Before evaluating an experiment, if the rule carries a
  `contextualBanditRef`, resolve the definition from the payload,
  iterate leaves top-to-bottom, and apply the first matching leaf's
  weights to the experiment.
- Fallback (no ref, unparseable definition, or no matching leaf):
  leave weights untouched; when a definition exists but no leaf
  matches, record `leafId = -1` with the experiment's aggregate or
  equal weights.
- After the experiment runs, clear the bandit selection unless the
  user was actually hash-bucketed in (`hashUsed && inExperiment`) so
  force-assigned exposures don't ship bandit metadata.
- ExperimentEvaluator copies the surviving `leafId`,
  `variationWeights`, and `banditVersion` onto GBExperimentResult.

Wiring:

- GBSDKBuilderApp.featuresAPIModelSuccessfully now populates
  `context.contextualBandits` from either the plaintext or encrypted
  section of the payload (via a new
  `Crypto.getContextualBanditsFromEncrypted`).
- GBUtils.initializeEvalContext propagates the definitions through
  to `GlobalContext` so the evaluator can read them.

All 300 existing tests still pass. Dedicated bandit tests land in the
next commit.
Seven unit tests against FeatureEvaluator with a hand-built
EvaluationContext, covering the state matrix from the ported algorithm:

- Matching leaf → leaf's weights are applied and leafId /
  variationWeights / banditVersion are on the result.
- No matching leaf → fallback marker (-1) with equal weights.
- Ref not present in payload → no bandit metadata leaks.
- Multiple matching leaves → first match wins (top-down order).
- Malformed definition (contexts is not a list) → evaluator swallows
  the parse error and treats it as "no bandit for this ref".
- contextualVariations without a ref → graceful degradation shape:
  new SDKs run the experiment, older SDKs skip on `variations == null`.
- Zero-coverage rule → user is filtered out, no bandit metadata is
  attributed to a non-bucketed exposure.

Cases.json conformance is not ported: the Java PR later removed its
own fabricated bandit cases (commits 7e0e351, 3b82c4b) pending an
upstream shared suite. These direct unit tests replace that
placeholder coverage until the shared cases.json ships.

All 307 tests pass locally (one pre-existing flaky
caching_manager_test unrelated to this change).
…tifier scan

deriveStickyBucketIdentifierAttributes only added a rule's hash/fallback
attributes when it had a plain 'variations' list. Bandit rules ship
their variations under 'contextualVariations' and would be silently
dropped from the identifier set — sticky bucketing then had nothing to
key on for those experiments.

Widen the check to include contextualVariations so bandit rules also
contribute their attributes.
handleValidFeatures only invoked the 'payload ready' delegate hook
inside the plaintext branch (data.features != null &&
data.encryptedFeatures == null). Encrypted deployments therefore
silently skipped:

  - refreshStickyBucketService(model), so sticky-bucket docs were never
    refreshed against a new payload — a pre-existing gap that predates
    the bandit work.
  - _applyContextualBandits(model), making the encrypted branch of
    Crypto.getContextualBanditsFromEncrypted dead code.

Mirror the plaintext call site: after a successful decrypt in the
encrypted branch, invoke delegate.featuresAPIModelSuccessfully(data)
with the original FeaturedDataModel. Both refreshes now run for
encrypted feature responses too.

Regression coverage in features_view_model_extra_test:

  - encrypted success → hook fires exactly once with the original model
  - encrypted failure → hook does not fire
  - plaintext success → hook fires exactly once (guards against
    accidental changes to the existing behavior)

The delegate mock is extended with apiModelSuccessCount / lastApiModel
so assertions can verify both the fire count and the payload identity.
Merge the 'contextualBandit' block from the shared cases.json into the
embedded gbTestCases blob and drive it through FeatureEvaluator via a
new conformance runner. The block covers the state matrix the spec
pins down:

  - single catch-all leaf, ordered leaves with first-match-wins,
    condition operator combinations ($in, $gte, missing attribute)
  - outer rule condition interactions (skipped when condition fails,
    passes when it matches)
  - coverage filtering both at the rule level and within a leaf
  - hashAttribute variants (empty / null / missing / custom)
  - forcedVariations override, querystring override, QA mode
  - globally disabled context
  - JSON variation values
  - dangling ref, empty contexts array, omitted banditVersion
  - required attribute present but no leaf matches → fallback -1

Adds:

  - GBTestHelper.getContextualBanditData() reader
  - GBContextTest.contextualBandits field so the input side maps into
    GBContext
  - GBExperimentResultTest.leafId / variationWeights / banditVersion
    for expected-result comparison
  - test/common_test/contextual_bandit_conformance_test.dart runner
    with a per-field diff so failures point at the exact mismatch
    (value, source, on, experimentResult.{leafId,variationWeights,
    banditVersion,key,variationId,inExperiment} and
    experiment.contextualBandit.*)

All 35 cases pass against the current implementation. Test-only
change; no runtime code was touched by this commit.
Brings the branch up to 4.4.0 — it was cut before the error-aware
refresh handler and the pointycastle 4 upgrade landed. No conflicts;
the bandit changes and the 4.4.0 features are disjoint.
@andywhite37
andywhite37 self-requested a review October 8, 2026 21:30
@andywhite37 andywhite37 added effort:large Substantial review and testing, often across APIs, components, or environments. needs-human-input A specific decision or manual task blocks progress; see the queue entry. priority:normal Regular queue priority; no evidence of urgency or low value. risk:high Could substantially affect correctness, isolation, compatibility, or lifecycle behavior. labels Oct 19, 2026
@andywhite37

Copy link
Copy Markdown
Contributor

Please confirm contextual-bandit support and the proposed public models and exposure metadata are in scope for Flutter. This changes assignment behavior and also alters encrypted-payload callbacks. Record approval of that scope, or identify changes to split out, so review can proceed.

@vazarkevych vazarkevych changed the title feat: consume contextual bandit payload feat!: consume contextual bandit payload Oct 21, 2026
@vazarkevych

Copy link
Copy Markdown
Collaborator Author

Please confirm contextual-bandit support and the proposed public models and exposure metadata are in scope for Flutter. This changes assignment behavior and also alters encrypted-payload callbacks. Record approval of that scope, or identify changes to split out, so review can proceed.

Decisions:

  • Scope: in scope. The JS SDK consumes contextual-bandit payloads (buildContextualBanditExperiment in core.ts), and the shared spec 0.9.0 has a contextualBandit section. This PR follows the JS behavior:
    • first matching leaf wins;
    • no matching leaf → leafId: -1 with the rule's weights (or equal weights);
    • a missing ref → aggregate weights and no metadata;
    • metadata only when the user was hash-bucketed into the experiment.
  • Public models and exposure metadata: ContextualBandit, ContextualBanditContext, ContextualBanditDefinition, and leafId / variationWeights / banditVersion on GBExperimentResult, matching the JS field names. Carrying the metadata in the tracking plugin's event is left out, as in JS.
  • Hardening beyond JS (new commit, mirroring the Kotlin SDK): the reported weights must be the ones the user was actually bucketed with, since the bandit learns from them. So:
    • a rule with explicit ranges (which override weights in the bucketer) reports no bandit metadata;
    • a leaf without a leafId, or with weights the bucketer would replace (wrong length, sum outside 0.99–1.01, negative or non-finite), falls back to -1 with the weights actually used.
  • Assignment behavior: rules with a contextualBanditRef were silently skipped before; now they enroll users. Users of those features can therefore change value after upgrading, so this ships in 5.0.0 as feat!: (see Release plan #192).
  • Encrypted-payload callbacks: kept in this PR. featuresAPIModelSuccessfully never fired for encrypted payloads, so sticky-bucket documents were not refreshed against a new encrypted payload. Bandit rules surfaced it, and it ships with them.
  • Spec: this branch's test_case.dart is spec 0.7.2 plus the contextualBandit section; feat!: support saved group references v2 ($savedGroup) and align condition operators with the JS SDK #178 brings 0.9.0 without it. Whichever merges second takes the full cases.json 0.9.0. Both PRs together pass it in full, including the four bandit feature cases missing here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort:large Substantial review and testing, often across APIs, components, or environments. needs-human-input A specific decision or manual task blocks progress; see the queue entry. priority:normal Regular queue priority; no evidence of urgency or low value. risk:high Could substantially affect correctness, isolation, compatibility, or lifecycle behavior.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants