bug: GPU sandboxes miss filesystem access for CUDA workloads

[elezar]

Related Issue

• Related to GPU-accelerated Sandbox Support #1444

Agent Diagnostic

• Loaded and used the openshell-cli skill for a real sandbox lifecycle check, then used the create-github-issue workflow to file this diagnostic.
• Context: follow-up from PR test(e2e): add GPU workload image artifacts #1484, which adds local GPU workload images including cuda-basic.
• Related prior detection: feat(onboard): use OpenShell Docker GPU sandboxes NemoClaw#3001 hit the same GPU-enabled Docker driver onboarding problem and added regression coverage proving direct sandbox GPU access with nvidia-smi, /proc/$$/task/$$/comm writes, and cuInit(0): feat(onboard): use OpenShell Docker GPU sandboxes NemoClaw#3001
• Direct Docker/CDI validation of the image succeeds on this host:
  • docker run --rm --device nvidia.com/gpu=all localhost/openshell/gpu-workload-cuda-basic:785872b4
  • deviceQuery detected one NVIDIA L4, printed Result = PASS.
  • vectorAdd printed Test PASSED.
• OpenShell Docker-backed GPU sandbox validation with the same image fails under the default sandbox policy:
  • Command shape:
    OPENSHELL_E2E_DOCKER_GPU=1 e2e/with-docker-gateway.sh target/debug/openshell sandbox create --from localhost/openshell/gpu-workload-cuda-basic:785872b4 --gpu --no-tty --no-keep -- /usr/local/bin/openshell-gpu-workload
  • Failure:
    cudaGetDeviceCount returned 304
-> OS call failed or operation not supported on this OS
Result = FAIL
• The Docker driver does attach the GPU correctly to the outer sandbox container:
  • docker inspect showed HostConfig.DeviceRequests=[{"Driver":"cdi","DeviceIDs":["nvidia.com/gpu=all"],...}].
  • docker exec <outer-container> sh -lc 'ls -l /dev/nvidia*' showed /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, and /dev/nvidia-modeset.
• OpenShell-executed processes inside the sandbox do not see the same GPU device nodes under the default policy:
  • openshell sandbox exec --name cuda-basic-inspect --no-tty -- sh -lc 'ls -l /dev/nvidia* /dev/nvidia-caps/* 2>&1 || true'
  • Output: ls: cannot access '/dev/nvidia*': No such file or directory.
• The same diagnostic showed /proc write denial from inside the OpenShell-executed process:
  • echo openshell-proc-test > /proc/$$/task/$$/comm
  • Output: Permission denied.
• This matters because crates/openshell-sandbox/src/lib.rs already documents that CUDA writes to /proc/<pid>/task/<tid>/comm during cuInit() and can return error 304 without writable /proc.
• A narrow explicit policy that made /proc and the existing NVIDIA device nodes read-write made the same OpenShell sandbox workload pass:
  • deviceQuery printed Result = PASS.
  • vectorAdd printed Test PASSED.
  • Workload printed OPENSHELL_GPU_WORKLOAD_SUCCESS cuda-basic.
• Follow-up call-boundary diagnosis confirmed the failure is at CUDA driver initialization, not image startup or device discovery:
  • A minimal C probe using dlopen("libcuda.so.1") and dlsym() succeeds through cuDriverGetVersion() when NVIDIA device nodes are read-write but /proc remains read-only.
  • Under that policy, cuInit(0) returns 304 (CUDA_ERROR_OPERATING_SYSTEM) and the following cuDeviceGetCount() returns 3 (CUDA_ERROR_NOT_INITIALIZED).
  • With the same policy except /proc moved to read-write, the same probe reports cuInit(0) -> CUDA_SUCCESS and cuDeviceGetCount() -> CUDA_SUCCESS.
  • A syscall trace with explicit call markers showed the procfs access occurs after MARK before cuInit and before RESULT cuInit=0: openat(..., "/proc/self/task/<tid>/comm", O_WRONLY|O_CREAT|O_TRUNC, 0666) followed by write(..., "cuda00001400006", 15).
  • This behavior is confirmed on the host driver stack tested here: NVIDIA driver 580.126.09, nvidia-smi CUDA version 13.0, and cuDriverGetVersion() -> 13000. Other driver branches may behave differently and should be validated before treating this as universal.

Relevant code findings:

• crates/openshell-policy/src/lib.rs: restrictive_default_policy() includes /proc in filesystem.read_only and does not include GPU device nodes in read_write.
• crates/openshell-sandbox/src/lib.rs: GPU_BASELINE_READ_WRITE already contains /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset, /dev/dxg, and /proc, and enumerate_gpu_device_nodes() handles /dev/nvidia0, etc.
• crates/openshell-sandbox/src/lib.rs: enrich_proto_baseline_paths() returns early when network_policies is empty. A default no-network GPU sandbox can therefore miss GPU filesystem enrichment even when --gpu is requested and Docker CDI injected devices into the container.

Historical PR context:

• fix(sandbox/bootstrap): GPU Landlock baseline paths and CDI spec missing diagnosis #710 originally added the GPU Landlock baseline paths, including /proc as read-write for CUDA workloads because CUDA writes /proc/<pid>/task/<tid>/comm during cuInit(): fix(sandbox/bootstrap): GPU Landlock baseline paths and CDI spec missing diagnosis #710
• fix(sandbox): preserve explicit read-only baseline paths #910 later changed baseline enrichment precedence so an explicit read_only entry prevents system-injected baseline read_write promotion. This did not remove /proc from GPU_BASELINE_READ_WRITE, but it can negate the fix(sandbox/bootstrap): GPU Landlock baseline paths and CDI spec missing diagnosis #710 behavior for policies where /proc is already listed in read_only, including the restrictive default policy: fix(sandbox): preserve explicit read-only baseline paths #910
• Separate current wrinkle: the no-network default path can also miss enrichment entirely because enrich_proto_baseline_paths() returns early when network_policies is empty. The eventual fix should account for both the fix(sandbox): preserve explicit read-only baseline paths #910 precedence behavior and the no-network enrichment path.

Security/design considerations:

• Do not blindly promote all of /proc to read-write for every sandbox. That widens the Landlock filesystem surface beyond CUDA's specific need.
• The concrete CUDA write observed here is to process/thread-specific procfs paths such as /proc/<pid>/task/<tid>/comm.
• Narrower-looking paths such as /proc/self/task, /proc/thread-self/comm, or /proc/<pid>/task are not straightforward with the current Landlock implementation. Rules are prepared by the supervisor before the workload child runs, and procfs self, pid, and tid paths resolve to process-specific inodes. Landlock also does not support glob rules like /proc/*/task/*/comm.
• A safer fix should be scoped to GPU-requested sandboxes and should not silently override explicit custom policies without a clear diagnostic. For example, the default GPU policy path could add the required GPU device nodes and handle the /proc requirement, while custom policies that keep /proc read-only should fail with an actionable CUDA/GPU policy diagnostic.
• If we want to avoid whole-/proc read-write, investigate a more invasive runtime design where child-local Landlock rules are prepared after fork and before exec, then test whether a narrower procfs rule actually covers CUDA-created threads.
• The implementation should document the threat tradeoff: CUDA may require writable procfs thread-name paths, but granting AccessFs::from_all() on /proc is broader than that specific operation.

Description

Actual behavior: A Docker-backed OpenShell sandbox created with --gpu from the cuda-basic workload image fails CUDA initialization under the default sandbox policy. Docker CDI attaches the GPU to the outer sandbox container, but OpenShell-executed processes cannot see /dev/nvidia* and cannot write /proc/<pid>/task/<tid>/comm.

Expected behavior: A sandbox created with --gpu should have the minimal filesystem policy needed to use the GPU resources exposed by the driver. The cuda-basic workload should pass with the default policy when Docker CDI is configured and the same image passes under plain docker run --device nvidia.com/gpu=all.

Reproduction Steps

1. Build the local GPU workload images from PR test(e2e): add GPU workload image artifacts #1484:
   mise run e2e:gpu:images:build
2. Verify the CUDA image works directly with Docker CDI:
   docker run --rm --device nvidia.com/gpu=all localhost/openshell/gpu-workload-cuda-basic:<tag>
3. Run a Docker-backed OpenShell gateway with GPU mode and create a GPU sandbox from the same image:
   OPENSHELL_E2E_DOCKER_GPU=1 e2e/with-docker-gateway.sh target/debug/openshell sandbox create --from localhost/openshell/gpu-workload-cuda-basic:<tag> --gpu --no-tty --no-keep -- /usr/local/bin/openshell-gpu-workload
4. Observe deviceQuery fail with CUDA error 304.
5. Keep a sandbox alive and compare device visibility:
   • docker exec <outer-container> sh -lc 'ls -l /dev/nvidia*' shows NVIDIA device nodes.
   • openshell sandbox exec --name <sandbox> --no-tty -- sh -lc 'ls -l /dev/nvidia* 2>&1 || true' does not show them.
6. Run again with an explicit policy that marks /proc, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset, and /dev/nvidia0 read-write. The workload passes.

Environment

• OS: Ubuntu Linux on AWS, kernel 6.17.0-1012-aws, x86_64
• GPU: NVIDIA L4
• NVIDIA driver: 580.126.09
• nvidia-smi CUDA version: 13.0
• CUDA driver API reported by minimal probe: cuDriverGetVersion() -> 13000
• Docker: Docker version 29.4.1, build 055a478
• Docker CDI: CDISpecDirs=["/etc/cdi","/var/run/cdi"]
• Docker discovered CDI devices: nvidia.com/gpu=0, nvidia.com/gpu=GPU-0a548031-a324-9815-5a41-c5a89bd22f07, nvidia.com/gpu=all
• OpenShell branch/commit tested: feat/1476-gpu-workload-images/elezar, commit 785872b41279978e5d95c721b6785c776287702d
• OpenShell CLI version from existing local binary: openshell 0.0.46-dev.5+gbdaa08fb

Logs

running CUDA sample: deviceQuery
/usr/local/lib/openshell-gpu-workload/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 304
-> OS call failed or operation not supported on this OS
Result = FAIL
OPENSHELL_GPU_WORKLOAD_FAILURE deviceQuery exited non-zero
Error:   × ssh exited with status exit status: 1

device_requests=[{"Driver":"cdi","Count":0,"DeviceIDs":["nvidia.com/gpu=all"],"Capabilities":null,"Options":null}]

outer_container_devices
crw-rw-rw- 1 root root 195, 254 /dev/nvidia-modeset
crw-rw-rw- 1 root root 511,   0 /dev/nvidia-uvm
crw-rw-rw- 1 root root 511,   1 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 /dev/nvidiactl

openshell_exec_devices
ls: cannot access '/dev/nvidia*': No such file or directory
ls: cannot access '/dev/nvidia-caps/*': No such file or directory

sh: 1: cannot create /proc/74/task/74/comm: Permission denied

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why