Repository navigation
Add per-domain RewardAdapter #199
Description
Context
A pack's TaskFamily.check_success(...) returns a structured
EpisodeResult — {"success": <bool>, "subgoals": {...}, "reason": <str>}.
Training loops want a scalar (or multi-objective vector) reward.
Today, the only mapping is "did success: true → 1.0, else 0.0", and
that's hardcoded in examples/codex_eval.py's post-processing.
That's fine for a binary cyber task. It's wrong for trading
(continuous P&L), robotics (geometric distance + smoothness +
energy), and most other domains.
What this issue tracks
Add a RewardAdapter ABC and per-pack implementations. Each pack
ships its natural one; training loops consume them.
- Cyber:
success → 1.0, plus partial credit from the
EpisodeResult.subgoalsthe pentest check already returns (closer
chains → higher) - Trading: P&L → Sharpe / drawdown / risk-adjusted return
- Robotics (when it lands): geometric task success + smoothness +
energy
Why it matters
Without this, every training script has to write its own
result-to-reward mapping, and the mapping leaks domain
knowledge into the trainer. With it, the pack owns the reward
shape and the trainer is domain-agnostic.
Where to start
openrange-pack-sdk—_types.pydefinesEpisodeResult
({success, subgoals, reason}) andFeasibilityVerdict;_protocols.py
TaskFamily.check_successreturns the former.RewardAdapteris
downstream ofEpisodeResult.src/openrange/core/episode.py—EpisodeReport.episode_result
(anEpisodeResult) is what an adapter consumes.packs/cyber_webapp/cyber_webapp/families/pentest.py— the cyber
check_success, for reference on thesubgoalsstructure.examples/codex_eval.py— the current ad-hoc mapping; that's
what an adapter would replace.
Suggested shape
class RewardAdapter(ABC):
@abstractmethod
def reward(self, report: EpisodeReport) -> float | Sequence[float]: ...Pack ships a default (Pack.default_reward_adapter() or similar);
training loops can override.
Acceptance
RewardAdapterABC inopenrange-pack-sdk(packs import only the
SDK, notopenrange), with aPack.default_reward_adapter()hook.- Cyber pack ships a default that returns 1.0/0.0 plus a richer
signal fromEpisodeResult.subgoals(closer chains → higher
partial credit). - One runnable example showing the adapter consumed by a training
script.
Notes
Keep the v0 cyber adapter simple — binary success is fine until a
training run actually needs partial credit. The point of this issue
is the seam, not the perfect reward function.
Activity
[-][roadmap] Per-domain reward adapters[/-][+]Add per-domain RewardAdapter (verifier result → scalar / vector reward)[/+][-]Add per-domain RewardAdapter (verifier result → scalar / vector reward)[/-][+]Add per-domain RewardAdapter[/+]snakefood3232 commentedon May 4, 2026
alijahak commentedon May 10, 2026
larstalian commentedon Oct 12, 2026
No longer involved with this project, so closing my open issues. Feel free to reopen anything you still want to track.