Skip to content

Add per-domain RewardAdapter #199

Closed as not planned
Closed as not planned

Description

@larstalian

Context

A pack's TaskFamily.check_success(...) returns a structured
EpisodeResult — {"success": <bool>, "subgoals": {...}, "reason": <str>}.
Training loops want a scalar (or multi-objective vector) reward.
Today, the only mapping is "did success: true → 1.0, else 0.0", and
that's hardcoded in examples/codex_eval.py's post-processing.

That's fine for a binary cyber task. It's wrong for trading
(continuous P&L), robotics (geometric distance + smoothness +
energy), and most other domains.

What this issue tracks

Add a RewardAdapter ABC and per-pack implementations. Each pack
ships its natural one; training loops consume them.

  • Cyber: success → 1.0, plus partial credit from the
    EpisodeResult.subgoals the pentest check already returns (closer
    chains → higher)
  • Trading: P&L → Sharpe / drawdown / risk-adjusted return
  • Robotics (when it lands): geometric task success + smoothness +
    energy

Why it matters

Without this, every training script has to write its own
result-to-reward mapping, and the mapping leaks domain
knowledge into the trainer. With it, the pack owns the reward
shape and the trainer is domain-agnostic.

Where to start

  • openrange-pack-sdk — _types.py defines EpisodeResult
    ({success, subgoals, reason}) and FeasibilityVerdict; _protocols.py
    TaskFamily.check_success returns the former. RewardAdapter is
    downstream of EpisodeResult.
  • src/openrange/core/episode.py — EpisodeReport.episode_result
    (an EpisodeResult) is what an adapter consumes.
  • packs/cyber_webapp/cyber_webapp/families/pentest.py — the cyber
    check_success, for reference on the subgoals structure.
  • examples/codex_eval.py — the current ad-hoc mapping; that's
    what an adapter would replace.

Suggested shape

class RewardAdapter(ABC):
    @abstractmethod
    def reward(self, report: EpisodeReport) -> float | Sequence[float]: ...

Pack ships a default (Pack.default_reward_adapter() or similar);
training loops can override.

Acceptance

  • RewardAdapter ABC in openrange-pack-sdk (packs import only the
    SDK, not openrange), with a Pack.default_reward_adapter() hook.
  • Cyber pack ships a default that returns 1.0/0.0 plus a richer
    signal from EpisodeResult.subgoals (closer chains → higher
    partial credit).
  • One runnable example showing the adapter consumed by a training
    script.

Notes

Keep the v0 cyber adapter simple — binary success is fine until a
training run actually needs partial credit. The point of this issue
is the seam, not the perfect reward function.

Activity

  1. changed the title [-][roadmap] Per-domain reward adapters[/-] [+]Add per-domain RewardAdapter (verifier result → scalar / vector reward)[/+] on May 4, 2026
  2. changed the title [-]Add per-domain RewardAdapter (verifier result → scalar / vector reward)[/-] [+]Add per-domain RewardAdapter[/+] on May 4, 2026
  3. snakefood3232 commented on May 4, 2026

    @snakefood3232
  4. alijahak commented on May 10, 2026

    @alijahak
  5. larstalian commented on Oct 12, 2026

    @larstalian
    ContributorAuthor

    No longer involved with this project, so closing my open issues. Feel free to reopen anything you still want to track.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is neededroadmapTracked on the public roadmaptrainingTraining pipeline

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions