refactor(proto): separate persistence-only messages from public API packages

[drew]

User Story

As an API maintainer, I want database storage messages separated from public RPC contracts, so that public API evolution and storage migration can be reviewed and versioned independently.

Problem Statement

Persistence decodes database payloads directly into protobuf messages, including types also returned by public RPCs. Several messages named Stored* live inside the public openshell.v1 package even though they are not used in RPC signatures. Generated clients and descriptor consumers therefore see storage-only types, while changes to shared public/persisted types also become database migrations.

Impact / Why This Matters

The beta compatibility promise would accidentally include internal storage messages. A public field cleanup can break stored data, and a storage-only change can appear to break the public package. Maintainers cannot reason about wire compatibility and durable-data compatibility as separate concerns.

Proposed Design

Move storage-only messages into an explicitly internal, versioned storage protobuf package and exclude them from public SDK exports where practical. For resource messages used both on the wire and in persistence, decide whether to introduce internal storage twins with explicit conversion at the persistence boundary.

If storage twins are deferred, document which public messages are also durable storage formats, permanently prohibit unsafe tag reuse, and require migration review for every change to those messages.

Acceptance Criteria

• Every persisted protobuf message and every public RPC message is inventoried, including overlaps.
• All Stored* messages move out of public API packages without changing their durable wire encoding.
• Generated public SDK surfaces no longer advertise storage-only types.
• A reviewed decision is recorded for each dual-purpose public/persisted resource.
• If storage twins are introduced, conversions deep-copy data and preserve unknown/legacy fields as required.
• Existing databases upgrade without data loss and have regression fixtures from earlier versions.
• Compatibility policy distinguishes public wire changes from storage migrations.
• Tests fail when a storage-format change lacks the required migration handling.

Alternatives Considered

Leave dual-purpose types in place and rely on field-number discipline. This is the lowest-cost option but permanently couples every API change to storage. Move only the pure Stored* messages. This is a useful first step and may be selected if storage twins are not justified before 0.1.0.

Agent Investigation

crates/openshell-server/src/persistence/mod.rs decodes stored payloads into protobuf messages. Multiple Stored* declarations remain in proto/openshell.proto, alongside resource messages used both for persistence and public responses.

Related: #2565. Source audit: https://gist.github.com/mrunalp/e80942c1544a0225ee588796a41ab30b.

[varshaprasad96]

📋 triage-agent

Triage Assessment

Classification: validated-feature

Summary

The reported API/storage coupling is confirmed on current main. Five persistence-only protobuf messages live in the public openshell.v1 package even though none appears in an RPC request or response. Moving these records behind an internal, versioned protobuf boundary is technically feasible and would separate public API compatibility from durable-storage compatibility.

Investigation

• proto/openshell.proto currently declares StoredProviderCredentialRefreshState, StoredRefreshMaterialDeletion, StoredProviderProfile, StoredPolicyRevision, and StoredDraftChunk in openshell.v1.
• The server uses these types in persistence, provider refresh, provider profiles, and policy storage. Database routing uses stable application-owned object-type strings rather than protobuf fully-qualified names.
• Generated consumers currently receive the storage types because they share openshell.proto. The committed Go binding contains all five, and the Rust raw SDK re-exports the complete openshell_core::proto module. Python and TypeScript generation inputs also include this public proto file.
• Existing payloads can remain decodable after a package move because ordinary protobuf message bytes encode field tags and wire values, not the package name. Compatibility requires preserving every field number, wire type, map-entry shape, optional-presence behavior, and existing object-type string.
• Seven public resource types are also persisted directly: Sandbox, Provider, Workspace, WorkspaceMember, SshSession, ServiceEndpoint, and InferenceRoute. The issue already permits documenting this overlap and deferring storage twins; implementing twins is not required to validate the proposal.
• Backward-compatibility coverage should use checked-in bytes emitted by the old schema and decode them with the relocated types. A new-schema round trip alone would not prove database compatibility.
• A package move by itself will not satisfy the Rust SDK boundary while raw re-exports all of openshell_core::proto. Implementation must define a selective raw export, server-owned generated module, or equivalent explicit internal boundary.
• No open or closed canonical duplicate or implementation PR was found. Stabilize public API, SDK, and extension contracts for 0.1.0 #2565 is the parent stabilization effort, not a duplicate. Latest release reviewed: v0.0.116; current main retains the reported layout.

Impact Signals

• Affected users/scope: API and SDK maintainers, generated-client consumers, and operators carrying durable gateway state across upgrades.
• Regression: No; this is existing architectural coupling before the 0.1.0 compatibility boundary.
• Workaround: Manual field-number discipline and treating generated Stored* symbols as unofficial; this does not remove SDK exposure or independently version storage changes.
• Evidence quality: High; declarations, persistence callers, SDK exports, code-generation inputs, and compatibility mechanics are directly verifiable in the repository.

Human Decision Required

Decide whether OpenShell should address this issue. If yes, apply state:accepted, associate it with a roadmap item, or do both, and decide whether the work remains human-owned. Either action records acceptance; roadmap placement additionally records sequencing. To queue investigation or planning for an unattended agent, also apply agent:plan-requested. You can instead directly ask an agent to use build-from-issue on this issue; the agent will warn about missing expected workflow labels and continue without changing them. If no, close it as not planned and record the rationale.

[varshaprasad96]

@drew I can open up a PR to fix the public APIs.

[varshaprasad96]

🏗️ build-plan

Implementation Plan

Issue type: refactor
Complexity: High
Confidence: High — the boundary and compatibility path are clear

Summary

Move all seven persistence-only protobuf messages from openshell.v1 into a server-private, versioned openshell.storage.v1 package without changing their durable wire schemas. Keep the eight public resource types currently persisted directly, document their durable-schema coupling, and add descriptor/fixture gates that separate public API compatibility from storage migration review.

Scope

• proto/openshell.proto and server-private storage proto: relocate StoredProviderCredentialRefreshState, StoredRefreshMaterialDeletion, StoredProviderProfile, PolicyRevisionPayload, DraftChunkPayload, StoredPolicyRevision, and StoredDraftChunk while preserving tags, wire types, presence, maps, enums, secret annotations, and database object-type strings.
• crates/openshell-core: keep the public generated module and descriptor free of storage-only schemas; remove metadata implementations for relocated types.
• crates/openshell-server: generate the storage package privately, relocate metadata implementations, and update persistence consumers.
• Public SDK/codegen surfaces: regenerate Go bindings and verify Rust, Go, TypeScript, Python, and public descriptors do not expose storage-only symbols.
• Compatibility coverage and architecture/gateway.md: inventory public RPC and durable roots/overlap, record the twin decision, preserve legacy payload fixtures, and enforce a frozen v1 schema/migration gate.

Implementation Steps

1. Introduce the private openshell.storage.v1 schema and server-local code generation, moving all seven storage-only messages verbatim.
2. Update server consumers and metadata traits while preserving stable database routing strings and payload encoding.
3. Regenerate committed Go bindings and assert public descriptors/generation closures exclude storage-only messages.
4. Add legacy encoded-payload fixtures and descriptor-derived tests proving old data decodes after relocation.
5. Add a durable-schema inventory and frozen-v1 compatibility gate so format changes require a versioned migration path.
6. Document the eight dual-purpose resource roots (Sandbox, SandboxWorkloadTemplate, Provider, Workspace, WorkspaceMember, SshSession, ServiceEndpoint, and InferenceRoute), their transitive durable dependencies, the decision to defer storage twins, and separate API/storage compatibility policies.

Test Plan

• Unit tests: storage schema matches the frozen v1 manifest; legacy bytes decode semantically; public descriptors exclude storage types; durable/public inventories remain classified.
• Integration tests: current persistence loaders decode representative pre-move payloads and preserve stable object routing; SDK codegen checks remain green.
• E2E tests: N/A — no sandbox runtime or infrastructure behavior changes.

Risks & Open Questions

• Prost drops unknown fields on decode/re-encode, so mixed-version writers remain unsupported and will be documented.
• Package relocation is byte-compatible for ordinary protobuf payloads; the audited paths do not persist type URLs or fully-qualified descriptor names.
• Storage twins are deferred to avoid a broad conversion layer before 0.1.0; each persisted public root and its transitive dependencies remain governed as durable formats.
• No LSM impact: this does not touch process identity, /proc, execution, or inter-process visibility.

Documentation Impact

• Update architecture/gateway.md with the canonical inventory, per-root twin decisions, and compatibility/migration policy.
• No gateway TOML/default/Helm or published user-workflow documentation changes are expected.

---

Revision 1 — initial plan