Repository navigation
feat!: consume contextual bandit payload - #174
Open
vazarkevych wants to merge 9 commits into
Open
vazarkevych wants to merge 9 commits into
vazarkevych wants to merge 9 commits into
Conversation
Introduce data types for the contextual-bandit payload: - ContextualBanditContext — one leaf: `leafId`, `condition`, `weights`. - ContextualBanditDefinition — server payload: `banditVersion`, ordered `contexts`. Deserialized on demand during evaluation. - ContextualBandit — evaluator output attached to GBExperiment: the selected `leafId`, applied `variationWeights`, and `banditVersion`. Also extends existing models with the fields the algorithm needs: - GBFeatureRule.contextualBanditRef / contextualVariations - GBExperiment.contextualBandit (transient, not serialized) - GBExperimentResult.leafId / variationWeights / banditVersion - FeaturedDataModel.contextualBandits / encryptedContextualBandits - GBContext.contextualBandits (in-memory bandit definitions) No behavior change yet — subsequent commits port the evaluator. Ports growthbook-java-internal PR #79.
Wires the bandit algorithm into the feature evaluation path: - FeatureEvaluator prefers `contextualVariations` over `variations` when present so newer SDKs pick up bandit variations while older SDKs gracefully skip (see cross-SDK spec change). - Before evaluating an experiment, if the rule carries a `contextualBanditRef`, resolve the definition from the payload, iterate leaves top-to-bottom, and apply the first matching leaf's weights to the experiment. - Fallback (no ref, unparseable definition, or no matching leaf): leave weights untouched; when a definition exists but no leaf matches, record `leafId = -1` with the experiment's aggregate or equal weights. - After the experiment runs, clear the bandit selection unless the user was actually hash-bucketed in (`hashUsed && inExperiment`) so force-assigned exposures don't ship bandit metadata. - ExperimentEvaluator copies the surviving `leafId`, `variationWeights`, and `banditVersion` onto GBExperimentResult. Wiring: - GBSDKBuilderApp.featuresAPIModelSuccessfully now populates `context.contextualBandits` from either the plaintext or encrypted section of the payload (via a new `Crypto.getContextualBanditsFromEncrypted`). - GBUtils.initializeEvalContext propagates the definitions through to `GlobalContext` so the evaluator can read them. All 300 existing tests still pass. Dedicated bandit tests land in the next commit.
Seven unit tests against FeatureEvaluator with a hand-built EvaluationContext, covering the state matrix from the ported algorithm: - Matching leaf → leaf's weights are applied and leafId / variationWeights / banditVersion are on the result. - No matching leaf → fallback marker (-1) with equal weights. - Ref not present in payload → no bandit metadata leaks. - Multiple matching leaves → first match wins (top-down order). - Malformed definition (contexts is not a list) → evaluator swallows the parse error and treats it as "no bandit for this ref". - contextualVariations without a ref → graceful degradation shape: new SDKs run the experiment, older SDKs skip on `variations == null`. - Zero-coverage rule → user is filtered out, no bandit metadata is attributed to a non-bucketed exposure. Cases.json conformance is not ported: the Java PR later removed its own fabricated bandit cases (commits 7e0e351, 3b82c4b) pending an upstream shared suite. These direct unit tests replace that placeholder coverage until the shared cases.json ships. All 307 tests pass locally (one pre-existing flaky caching_manager_test unrelated to this change).
…tifier scan deriveStickyBucketIdentifierAttributes only added a rule's hash/fallback attributes when it had a plain 'variations' list. Bandit rules ship their variations under 'contextualVariations' and would be silently dropped from the identifier set — sticky bucketing then had nothing to key on for those experiments. Widen the check to include contextualVariations so bandit rules also contribute their attributes.
handleValidFeatures only invoked the 'payload ready' delegate hook
inside the plaintext branch (data.features != null &&
data.encryptedFeatures == null). Encrypted deployments therefore
silently skipped:
- refreshStickyBucketService(model), so sticky-bucket docs were never
refreshed against a new payload — a pre-existing gap that predates
the bandit work.
- _applyContextualBandits(model), making the encrypted branch of
Crypto.getContextualBanditsFromEncrypted dead code.
Mirror the plaintext call site: after a successful decrypt in the
encrypted branch, invoke delegate.featuresAPIModelSuccessfully(data)
with the original FeaturedDataModel. Both refreshes now run for
encrypted feature responses too.
Regression coverage in features_view_model_extra_test:
- encrypted success → hook fires exactly once with the original model
- encrypted failure → hook does not fire
- plaintext success → hook fires exactly once (guards against
accidental changes to the existing behavior)
The delegate mock is extended with apiModelSuccessCount / lastApiModel
so assertions can verify both the fire count and the payload identity.
Merge the 'contextualBandit' block from the shared cases.json into the
embedded gbTestCases blob and drive it through FeatureEvaluator via a
new conformance runner. The block covers the state matrix the spec
pins down:
- single catch-all leaf, ordered leaves with first-match-wins,
condition operator combinations ($in, $gte, missing attribute)
- outer rule condition interactions (skipped when condition fails,
passes when it matches)
- coverage filtering both at the rule level and within a leaf
- hashAttribute variants (empty / null / missing / custom)
- forcedVariations override, querystring override, QA mode
- globally disabled context
- JSON variation values
- dangling ref, empty contexts array, omitted banditVersion
- required attribute present but no leaf matches → fallback -1
Adds:
- GBTestHelper.getContextualBanditData() reader
- GBContextTest.contextualBandits field so the input side maps into
GBContext
- GBExperimentResultTest.leafId / variationWeights / banditVersion
for expected-result comparison
- test/common_test/contextual_bandit_conformance_test.dart runner
with a per-field diff so failures point at the exact mismatch
(value, source, on, experimentResult.{leafId,variationWeights,
banditVersion,key,variationId,inExperiment} and
experiment.contextualBandit.*)
All 35 cases pass against the current implementation. Test-only
change; no runtime code was touched by this commit.
Brings the branch up to 4.4.0 — it was cut before the error-aware refresh handler and the pointycastle 4 upgrade landed. No conflicts; the bandit changes and the 4.4.0 features are disjoint.
andywhite37
self-requested a review
October 8, 2026 21:30
Contributor
|
Please confirm contextual-bandit support and the proposed public models and exposure metadata are in scope for Flutter. This changes assignment behavior and also alters encrypted-payload callbacks. Record approval of that scope, or identify changes to split out, so review can proceed. |
This was referenced Oct 20, 2026
Open
Collaborator
Author
Decisions:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds consumption of the contextual-bandit payload, so a feature rule referencing a contextual bandit is evaluated with the per-segment variation weights the backend computed instead of being skipped.
Today such a rule is silently ignored: its variations ship under
contextualVariations, and the evaluator skips onvariations == null. Graceful degradation works as designed, but bandits do not function at all.All the optimization stays server-side. The SDK only consumes ready weights: resolve the referenced definition, pick the leaf whose condition matches the user, apply its weights, and bucket through the normal mechanism.
Diff size note: 7.5k added lines look alarming, but only 389 lines land under
lib/. The bulk is the sharedcontextualBanditconformance block embedded intest/test_cases/test_case.dart.Scope
contextualBanditsandencryptedContextualBanditsfrom the feature payload.contextualBanditRefagainst the payload definitions.leafId = -1with the experiment's existing or equal weights.leafId,variationWeights, andbanditVersiononGBExperimentResult, but only when the user was actually hash-bucketed in (hashUsed && inExperiment) — a force-assigned exposure must not carry bandit metadata.ContextualBandit,ContextualBanditContext,ContextualBanditDefinition, exported from the package entry point.Two fixes found along the way
Both predate this work; bandits only surfaced them.
featuresAPIModelSuccessfullynever fired for encrypted payloads.handleValidFeaturesinvoked the delegate only inside the plaintext branch, so encrypted deployments silently skippedrefreshStickyBucketService(model)— sticky-bucket documents were never refreshed against a new payload. Now mirrored in the encrypted branch after a successful decrypt.Sticky-bucket identifier scan skipped bandit rules.
deriveStickyBucketIdentifierAttributesonly collected a rule's hash/fallback attributes when it had a plainvariationslist, so bandit rules contributed nothing and sticky bucketing had no attribute to key on.Reported weights match the assignment
The bandit learns from the reported weights, so they must be the ones the user was actually bucketed with. Beyond the JS SDK, and as in the Kotlin SDK:
ranges, which take precedence over weights in the bucketer, reports no bandit metadata;leafId, or whose weights the bucketer would replace (wrong length, sum outside 0.99–1.01, negative or non-finite), falls back toleafId: -1;8 new tests in
contextual_bandit_test.dart; 7 fail without this change.Test plan
test/common_test/contextual_bandit_test.dart— 7 unit tests over the evaluator state matrix: matching leaf applies its weights and records metadata; no matching leaf falls back to-1with equal weights; a ref absent from the payload leaks no metadata; first match wins when several leaves match; a malformed definition is swallowed and treated as no-bandit;contextualVariationswithout a ref still runs the experiment; a zero-coverage rule attributes no metadata to a non-bucketed exposure.test/common_test/contextual_bandit_conformance_test.dart— runs the 35 shared conformance cases with a per-field diff, so a failure names the exact mismatch rather than just failing. All 35 pass.flutter test— 312 pass. One pre-existing flake incaching_manager_test(concurrent test files race on the shared temp cache directory); it passes in isolation 3/3 and also fails on currentmain, where 2 tests fail.dart analyze --fatal-infos— clean.dart format --output=none --set-exit-if-changed .— clean.BREAKING CHANGE: feature rules with a
contextualBanditRefwere skipped before and now enroll users, so the values of those features can change after upgrading.