cellular-soaks.md

Cellular overnight explorations

Two funded, resumable conditions give the new cellular substrate a longer horizon. They use the same Julia implementation as the local arena. The cellular lifecycle, contacts and resource pool currently run on CPU; a GPU on the host does not make these jobs GPU simulations.

Profile	Physical environment	Development and junctions
experiments/arena/cellular-soak-broad.toml (seed 401)	32 × 24 periodic domain, one broad complementary resource stripe; diffusivity 0.06	Response 0.8, adaptation 6, junction conductance 0.8; plastic decay time 50
experiments/arena/cellular-soak-mosaic.toml (seed 402)	Same domain, complementary cosine-product patches with wavelengths 16 × 8; diffusivity 0.02	Response 1.2, adaptation 24, junction conductance 0.25; plastic decay time 200

Both start with 64 founders, physical exchange radius 1, spacing 0.5 and the same metabolic machinery. niche_waves = [kx, ky, kz] specifies integer cycles around the periodic domain; positive axes multiply their cosines. The default [1,0,0] preserves the original stripe. Masks are complementary and each has mean one for these resolved recipes. Contrast 0.9 leaves both resources locally available.

The broad landscape allows movement and connected bodies to encounter a long chemical gradient. The mosaic offers shorter, intersecting gradients, slower external mixing, and developmental memory that lasts across more encounters. Slower junction exchange leaves room for intracellular differences to persist. These are analytical design choices, not claims that a particular organization will evolve. Different seeds and several changed parameters mean the pair is an exploration of two conditions, not a causal ablation or fitness ranking.

Read-only inspection

Run the compact inspector on hbox without installing or changing anything there:

ssh hbox python3 - \
  /othersys/tmp/intmat-soaks-20261015/broad \
  /othersys/tmp/intmat-soaks-20261015/mosaic \
  /othersys/tmp/intmat-soaks-optimized-20261015/continuation \
  < scripts/inspect_cellular_soaks.py > outputs/observatory/inspection-TIMESTAMP.json

Use a fresh timestamped destination. The JSON records process identity, affinity, cgroup memory controls, a stable checkpoint pointer/hash, complete metric rows, partial tails, artifact bytes and all files against each source manifest. A partial last metric row can be an append in progress; a malformed complete row is reported separately. Large per-cell diagnostic arrays stay in the original logs. Read inspection_error and mismatch fields; successful transport alone is not a healthy-run verdict. No checkpoint is deserialized or resumed, and the original driver failure caveat below still applies.

Supply and population headroom

Initial precursor concentration is 0.45 per species. Continuing supply is 0.0025 per species per unit area per unit time: 3.84 total mass per model time. At chemical potential 4 and conversion efficiency 0.65, that provides at most 9.984 work per time. Basal maintenance alone at 0.035 per biomass implies about 285 biomass in a steady supply-only balance, or about 1,141 cells if every cell were at the minimum living mass 0.25. Neural operation, motion and synthesis also consume work.

This calculation motivates a guard of 4,096 cells; it does not predict carrying capacity. Initial and accumulated resource stores can support overshoots, bodies can become large, and a time-varying population need not be in steady balance. Inspect actual population, biomass, stored resource, births/deaths and blocked births. Sustained blocking means the computational guard is affecting the run; it is not evidence of higher fitness. There is no fixed-age death.

Each precursor has a finite source budget of one million mass, enough for roughly 520,833 model time at the nominal source rate. Waste export is budgeted separately and counted by the open-system ledger. It prevents inert waste accumulating indefinitely; the model has no waste toxicity reward or recycling into fuel.

Execution and retention

The intended hbox allocation is two jobs, each with two Julia threads, one GC thread, one BLAS thread, an 8 GiB memory limit, no swap, low CPU priority and a 12-hour wall bound. Use disjoint allowed CPU pairs 2–3 and 4–5 under the existing shared build controls. Each job has a fresh directory and immutable source, project and profile snapshot; the older running arenas remain separate.

The runtime lane owns deployment and records the actual host commands and paths. Retain 20-second longitudinal metrics, 90-second rotating restart checkpoints and 10-minute full-state archives, with a 4 GB artifact budget per run. End by saving a checkpoint; resume with the same pinned source and Julia environment. A budget limit should pause collection or stop cleanly, never delete another run's files. If a population becomes empty, preserve the terminal state; do not silently reseed it and join that trajectory to the original run.

Useful first readings are spatial chemistry, body and bond organization, fragmentation/reconnection, hereditary and developmental diversity, age and lineage histories, resource turnover, and actual junction flux. Longitudinal checkpoints allow later assays without changing the overnight trajectory. A negative or quiet outcome concerns these founders and conditions; it cannot establish that the substrate lacks the corresponding potential.

Running hbox jobs (15 October 2026)

Both 12-hour jobs run from the immutable source at /othersys/tmp/intmat-soaks-20261015/source. Its thirty Julia source files match commit 606ca2e; the copied profiles, driver and launcher have independent SHA256 fingerprints in source/provenance.json and environment.json. The resolved manifest retains the portable Interior path . and uses Julia 1.12.4. All new package writes use the isolated depot/, with the existing verified probe depot available as a read fallback.

Condition	Output	Julia PID at launch	CPU affinity
Broad	/othersys/tmp/intmat-soaks-20261015/broad	3736521	2,3
Mosaic	/othersys/tmp/intmat-soaks-20261015/mosaic	3736525	4,5

Both actual cgroups enforce 8 GiB memory, zero swap and nice 15. The driver saves at its 12-hour wall deadline, with an external interrupt/kill guard 15 minutes later. status.json gives the actual deadline and progress; checkpoint.json names the latest completed restart slot. metrics-SESSION-NNNN.jsonl chunks keep longitudinal metrics. state-STEP-SESSION.jls files retain complete future state every ten minutes. latest.json is the bounded current observation. The 4 GB artifact cap reserves room for final checkpoints and stops without deleting history. A STOP file in a run directory requests a checkpointed stop between steps; rename that operator-created file before intentionally resuming.

A short real hbox soak completed at step 415, then an independent Julia process strictly restored it, checked resource/grid aliases and advanced to 416. This validates restart machinery rather than any long-term ecological conclusion.

To resume an interrupted broad job on hbox, first confirm its PID is absent, then read the current pointer and remaining wall budget:

root=/othersys/tmp/intmat-soaks-20261015
checkpoint=$(python3 -c 'import json;from pathlib import Path;p=Path("/othersys/tmp/intmat-soaks-20261015/broad");print(p/json.loads((p/"checkpoint.json").read_text())["file"])')
remaining=$(python3 -c 'import json,time;from pathlib import Path;p=Path("/othersys/tmp/intmat-soaks-20261015/broad/status.json");print(max(0,int(json.loads(p.read_text())["deadline_unix"]-time.time())))')
test "$remaining" -gt 0 && SOAK_DEPOT="$root/depot" "$root/source/scripts/launch_cellular_soak.sh" "$root/source" "$root/source/experiments/arena/cellular-soak-broad.toml" "$root/broad" 2,3 "$remaining" "$checkpoint"

For mosaic, replace both occurrences of broad in the Python paths, the profile and output names, and use CPUs 4,5. The launcher accepts an explicit seconds argument for a separately authorized extension. Strict source validation remains mandatory; there is no automatic source-migration or reseeding path. A live writer is refused by the output lock. The existing local arenas and their observers are separate and were left running.

Recovery boundary for the currently pinned jobs

The two jobs launched above retain their original driver hash cba3c8f41102aee0ebcd8dd4a2aa774dbb70377d0961bc57f37c92682e452aad. Their normal wall-deadline and STOP shutdown paths save between completed steps. That first driver also attempts a final checkpoint after an exception. If a job fails or is interrupted during a physics step, do not automatically resume its final checkpoint pointer: inspect the log and use a rotating checkpoint or full-state archive completed before the error. Preserve the questionable state for diagnosis. This caveat does not require restarting a normally running job.

The repository driver has since been hardened for future launches. It rejects a resume profile that differs from the stored configuration, and tracks complete step boundaries. A mid-step failure leaves the prior checkpoint and observation untouched and writes unsafe_in_memory_state=true failure metadata; it does not serialize the partially mutated state. The standalone fault regression is julia --project test/cellular_soak_faults.jl (it deliberately replaces a method inside that test process, so it is excluded from the shared test harness).

Explicit optimized continuations

The future driver accepts --throughput to omit the profile's batch sleep. This changes wall-clock pacing rather than the integration timestep or model rules. Session metadata records the mode, actual delay, initial step and elapsed wall time, allowing sustained runner throughput to include observation/checkpoint costs. The original jobs above retain their original pacing and source.

--allow-source-change requires --resume and a fresh empty output directory. It creates a separate continuation: the exact input bytes are retained as parent-checkpoint.jls; the session records their SHA256, parent and loaded source hashes, and initial step. Provided profile values must still match the stored configuration. Ordinary resume remains strict. This flag cannot overwrite an old run or silently introduce a different recipe.

One optimized long continuation now runs from integrated commit 361d0a9 on CPU pair 6,7, with two Julia threads, one GC/BLAS thread, an enforced 8 GiB cgroup limit, no swap, nice 15, a 12-hour wall bound and the same 4 GB artifact cap. The original CPU2–5 jobs continue unchanged. A bounded migrated trial and strict restoration of its completed checkpoint preceded launch; a faster simulation clock does not establish a different ecological result.

For a reviewed final snapshot at $optimized/source, an immutable mature parent at $parent, and a new output $optimized/continuation, the launcher protocol is:

SOAK_DEPOT="$optimized/depot" SOAK_THROUGHPUT=1 SOAK_ALLOW_SOURCE_CHANGE=1 \
  "$optimized/source/scripts/launch_cellular_soak.sh" \
  "$optimized/source" "$optimized/source/experiments/arena/cellular-soak-broad.toml" \
  "$optimized/continuation" 6,7 43200 "$parent"

Use the parent's matching condition/profile. The launcher translates the two explicit environment switches into recorded driver flags. Both switches default to zero. Subsequent ordinary resumes of the new continuation use its pinned source and strict validation, without the migration switch. No older pin or running process needs to be modified.

Verified optimized run

Completed: this continuation ended normally at its original deadline. See the terminal record for final hashes, metrics and resource-control evidence. The launch and early verification details below are retained as history.

The long Julia process is PID 3828298, with output /othersys/tmp/intmat-soaks-optimized-20261015/continuation and pinned project /othersys/tmp/intmat-soaks-optimized-20261015/source. It started at step 206665 from the immutable, 207-cell broad-condition parent, SHA256 d7f81c4cbbf21b05ce1918a142764eb0d22e1d6b68565c012c66cd759a6f01de. The session preserves that image and both source maps. It advanced to step 210875 within its first published 20.8 seconds. Its wall deadline is 15 October 2026, 21:54:05 UTC (17:54:05 America/New_York); normal end saves a checkpoint.

The integrated source reproduced the published 100-step sparse-state SHA256 5ecc693f9ebd1415a607dde567c1e198a544814439c5655aea3c6719da47dd75 when using the profiler's deepcopy fixture. A direct-restored replay initially had a different serialization hash, but recursive comparison found identical scalar values, bitwise numeric arrays, types and 89 mutable alias pairs across 2,206 visited values. Its only difference was dictionary iteration order at arena.config[cellular]. Both diagnostic reports are retained; serialized-byte parity requires the same copy fixture.

The actual driver trial advanced 206665 → 224935 in 60.0256 wall seconds: 304.37 steps/s including startup, observations and checkpoints, and approximately 359 steps/s over the later 20–60 second window. An independent process strictly restored its completed checkpoint and advanced 224935 → 224936. Those evolving runner rates describe this trace; the separate matched-state speedups are in docs/cellular-performance-results.json.

Remote evidence beside the run includes integrated-deepcopy-parity.json, recursive-comparison.json, trial60-strict-restore.json, long-runtime.json, and the source's per-file provenance.json. The throughput-trial60 directory retains the bounded trial and its input image. These are separate from both original soaks.

The original recipe and remaining source budgets are unchanged. Its nominal precursor funding covers about 520,833 model time in total; this parent begins at 20,666.5. At the measured runner rate, exhaustion could occur in roughly four wall hours, well before the 12-hour ceiling. Treat that as the funded experiment's boundary. Retain resource inventories, source/withdrawal ledgers, terminal state and checkpoint; do not silently refuel, reseed, or interpret fuel exhaustion as a limit on the substrate's potential.

The first advanced long-run checkpoint was independently strict-restored at step 234995 (199 cells). Resource/grid aliases and all 31 loaded source fingerprints matched the pin. At that checkpoint the remaining precursor budgets were 954881.2693 and 954881.2505 mass; material and chemical-energy residuals were 0.000339 and 0.000806. Full dissolved/intracellular inventories, biomass, supply and removal ledgers are retained in long-checkpoint-verification.json alongside the run. These are early recorded states, not final endpoints.