[tchaton]

Add LitData and Pipeline skills that take agents from a dataset or workflow request to verified outputs. Prioritize compact dataset preparation and composing jobs through lightning_storage.

• LitData keeps existing JPEG/PNG encoding, starts with 64 MB chunks and two workers, and measures chunks plus indexes against source bytes. Common image tasks use the entrypoint; advanced tasks load only relevant sections from 15 unchanged, pinned upstream guides.
• Pipelines create managed storage before submission, show explicit prepare → consume dependencies, capture the actual run ID, and verify downloaded outputs. The skill also covers Studio workflows, schedules, logs and cleanup.
• Register both skills in the plugin and README. Keep entrypoints compact: LitData 192 lines; Pipelines 230 lines. Line counts describe entrypoint size, not prompt-cache behavior.

Baseline vs optimized skills

Fresh GPT-6 Luna agents, medium reasoning, used the same task prompts and inputs. Baseline is the original skill at 3c90141; optimized is the revised skill. Measurements come from actual agent transcripts; outputs were checked independently.

LitData: prepare a compact photo dataset

Task: optimize 48 JPEGs, preserve image content/captions/labels/splits, read decoded training batches with two workers, and report storage sizes.

Metric	Baseline skill	Optimized skill	Observed change
Task outcome	Passed	Passed	All 48 images and metadata preserved
Tool calls	13	10	23% fewer
Accumulated tokens	613,817	444,275	28% fewer
Uncached input tokens	51,406	15,785	Cache-sensitive; not a standalone efficiency claim
Elapsed time	77.08 s	58.53 s	24% less

What helped: keep the common image workflow in the entrypoint, load media references only for other modalities/custom encoding, and use the public dataset length instead of guessing the private index schema. The final agent read no additional reference files and had no failed commands.

Both variants already produced compact data: approximately 683 KB including indexes, versus 678 KB of source JPEGs and 9.44 MB of decoded RGB. Existing JPEG bytes and every decoded pixel were preserved. The storage saving is versus decoded pixels, not versus the baseline skill or already compressed JPEGs.

An intermediate revision regressed to 19 calls, 880,530 tokens and 101.23 s. We inspected that trace and revised the guidance again; the report retains the regression. These are single trials, not an established average speedup. Accumulated tokens include cached/repeated context across calls and are not a billable-cost estimate.

Pipelines: prepare → consume through lightning_storage

Task: two separate CPU jobs exchange 100 records and a runtime-generated marker, verify values, download the summary/logs, and clean up test resources in production lightning-ai/general.

Metric	Baseline skill	Optimized skill
Workflow outcome	Failed: consumer could not find producer files; recovery stopped before consumer ran	Passed: both jobs completed and the downloaded summary matched the producer marker
Pipeline submissions	2	1
Tool calls	93	41
Accumulated tokens	8,129,362	2,418,241
Elapsed time	814.94 s	491.87 s
Cleanup	Pipeline/jobs/files removed; empty folder registration pending	Pipeline/jobs/files removed; empty folder registration pending

What helped: explicitly create managed storage before submitting jobs, show dependencies and shared paths together, capture the submitted run ID, and provide exact download/cleanup commands. mkdir inside a job alone had left the baseline producer's files on its local disk.

This demonstrates a correctness improvement. Do not interpret the pipeline numbers as a controlled speedup: the baseline failed, and it also upgraded the SDK environment while the optimized trial started with that version installed. Two empty managed-folder registrations remain after accepted deletion requests; all test jobs and files were removed. The final cleanup guidance also distinguishes deleting files from deferred managed-folder removal.

Repeated reliability pilot: text and float32 embeddings

Task: prepare 1,024 text/embedding records for repeated training, preserve exact values and the 896/128 split, compare uncompressed vs zstd including indexes, and load with two workers. Three genuine no-skill controls and three frozen-skill trials, fresh GPT-6 Luna medium agents, identical inputs/environment. The exploratory original-skill run is excluded. A trial is a complete rollout; internal recovery is allowed and counted in its usage.

Metric	No skill	Frozen candidate (e682e78)
Passed / attempted	3 / 3	3 / 3
pass@1	100%	100%
95% Wilson CI for pass@1	43.9–100%	43.9–100%
pass^3 (all three succeed)	100%	100%
Median tool calls	29	29
Median accumulated tokens	1,534,501	1,570,406
Median elapsed time	157.25 s	213.08 s
Selected storage, bytes incl. indexes	20,763–21,065	14,477–16,723

No demonstrated reliability or efficiency improvement on this task. Candidate median tokens increased 2.3% and elapsed time 35.5%; candidate trial 2 regressed to 73 calls / 3,858,894 tokens / 322.34 s. Smaller files are an observed outcome, not an isolated compression benchmark: agents chose different serializers and chunk settings. The fixture is deliberately repetitive, so its compression ratio does not generalize to arbitrary embeddings.

All six persisted outputs passed independent checks of every ID/text/label/float32 value/split, unchanged source bytes, and full two-worker iteration. Candidate 1's example checked a first training batch; the independent grader checked complete coverage. Recoveries included multiprocessing setup, a NumPy-versus-Torch verifier assumption and macOS worker behavior. The candidate was not edited between scored trials. The exploratory ordering failure produced one narrow skill change: verify by stable ID and reject duplicates.

Use pass@1 = c/n and pass^3 = choose(c,3)/choose(n,3), following tau-bench's reliability definition. Intervals use the Wilson method. With just three trials, the pass^3 estimate is extremely uncertain: cubing the Wilson bounds gives an approximate 8.4–100% interval under independent, identically distributed trials. This is one synthetic task, not a repository-wide reliability claim. Runs share a host and some overlap; elapsed time is descriptive. Counts include repeated/cached context, not a billable-cost estimate. Full per-trial evidence stays in the local progress report.

Live fan-out follow-up: propagation was not accounted for

Using the existing managed folder and SDK 2026.10.13, a fresh Luna agent ran two independent producers → one consumer. Both attempts verified 200 distinct IDs and both runtime markers through persisted producer files. All three jobs completed in each attempt, but the consumer summary was absent from managed storage in both. The second attempt used an explicit consumer subdirectory; that did not resolve it. Downloaded logs are evidence of execution, not a substitute for the missing artifact.

This single rollout did not meet its download acceptance criterion and used 2 submissions, 58 calls, 4,006,944 accumulated tokens and 1,219.64 s, including provisioning and cleanup. It is separate from the local pass@1/pass^3 estimates. Six job logs and failure/readback evidence were retained; both pipelines, all six jobs and both run prefixes were removed. The protected sibling file remained byte-identical. After that check, the parent harness removed its own sentinel, verified the folder empty and requested registration deletion; removal remains pending. Default storage propagates asynchronously and does not use an R2 FUSE mount (maintainer clarification). Optimized storage uses a distributed filesystem. The test did not establish a sufficient propagation deadline before cleanup, so it does not prove persistence failure or data loss. The skill now distinguishes storage types and requires bounded visibility checks; an appropriate propagation-aware rerun remains outstanding.

Agent task checklist

LitData:

• Compare uncompressed/zstd text and float32 embeddings with three no-skill and three frozen-skill trials; report pass@1, pass^3, confidence intervals and all rollout measurements.
• Optimize 48 JPEGs while preserving encoded bytes, decoded pixels, captions, labels and train/validation splits.
• Read training batches with two workers and measure source, optimized and decoded RGB storage.
• Compare original and revised skills with fresh GPT-6 Luna agents; inspect actual conversations, failures, tool calls, token usage and elapsed time.
• Secondary coverage: checkpoint/resume in a fresh process, with an original-skill and no-skill comparison.

Pipelines:

• Execute fan-out/fan-in with 200 records, two runtime markers and a protected sibling file. Artifact-download acceptance was not met on either submission: all jobs completed and producer data was readable, but neither consumer summary could be downloaded. All six logs were saved; jobs/pipelines/run prefixes removed; sibling file preserved and independently compared byte-for-byte.
• Rerun fan-out with explicit storage-type verification and a propagation-aware observation window. Prior checks did not establish data loss; changing output directories alone did not establish a fix.
• Run two separate CPU jobs in production lightning-ai/general; exchange 100 records and a runtime-generated marker through lightning_storage.
• Download the summary and both logs; verify the consumer checked all values and received the producer's marker.
• Remove the disposable pipelines, jobs and storage files; read back absence/empty prefixes.
• Verify eventual removal of three empty managed-folder registrations (two earlier trials plus the new fan-out fixture). Files were cleared and DELETE accepted; the latest fixture still reads CREATED. Managed-folder deletion can be deferred by retention.
• Earlier SDK PR #201 validation: repeated schedule replacement and an actual scheduled child ran the updated version.

Validation and detailed evidence

Detailed progress notes, task fixtures, raw traces and machine-readable measurements are maintained locally outside this PR. The key comparisons and remaining work are recorded above. Following the Agent Skills best practices, revisions address observed agent friction and keep detailed references conditional.

Local checks pass: skill format validation, plugin validation, README/manifest consistency, snippet syntax, local links, changed-file pre-commit hooks and whitespace. Advanced LitData cloud/multi-node/media examples, activation accuracy and compaction recovery were not exhaustively tested. These are fresh contexts/workspaces, not OS-enforced sandboxes.

_{— GPT-6, on direct instruction; text not reviewed}

[codexceed]

The managed-storage guidance fixes the producer/consumer failure the trials surfaced (create the folder before the jobs, or the consumer finds nothing), the SDK calls in the examples match the released SDK, and the 15 LitData references are byte-identical to litData at the pinned commit. One runnable example needs a fix before merge: the Studio pipeline in studio-workflows.md fails before submission whenever the Studio lives outside the teamspace's default cloud (inline). Two optional notes inline.

Checked against SDK source and signatures (lightning-sdk 2026.9.18.post1); no live pipeline or end-to-end LitData run.

1 DO · 0 TRY · 2 SUGGEST