Local R2 shared by several Workers: concurrent writes fail with SQLITE_BUSY

[drwpwrs]

alchemy dev runs each Worker in a workerd of its own, and each runtime's R2 simulator (R2BucketObject) opens the same {storage}/r2/cloudflare-runtime-R2BucketObject/<id>.sqlite for a bucket the Workers share. When two Workers write that bucket at the same moment, one write fails:

Error: database is locked: SQLITE_BUSY
    at putRow (bindings/r2-bucket/R2Bucket.worker.mjs)
→ put: Unspecified error (0)

The simulator itself says it assumes "a single runtime instance operating on a blob store". We hit it with three Workers sharing one bucket, one writing small objects while another wrote others, in about one run in four of a test that does both.

Our patch, applied to the built worker (dist/core/workers/bindings/r2-bucket/R2Bucket.worker.mjs), retries a write that finds the database locked, after yielding, since time doesn't advance within a synchronous turn:

async function unlocked(write) {
  for (let attempt = 1; ; attempt++)
    try {
      return write();
    } catch (error) {
      if (attempt === 50 || !String(error).includes("SQLITE_BUSY")) throw error;
      await scheduler.wait(attempt * 2);
    }
}
// #put:    oldBlobIds = await unlocked(() => this.#stmts.put(row, opts.onlyIf));
// #delete: const oldBlobIds = await unlocked(() => this.#stmts.deleteByKeys(keys));

It doesn't address the blob store's in-process locking, which is also per runtime.

Ask: host a shared simulator in one runtime, with the other Workers reaching it through their binding, or give the simulator's database a busy timeout.

Versions: @alchemy.run/cloudflare-runtime 2.0.0-beta.79, workerd 1.20260921.1, macOS arm64.

[brandoncarl]

The D1 simulator has the same failure. Each runtime.start(worker) runs its own workerd with its own D1.worker.mjs Durable Object, and all of them open the same {storage}/d1 file. Concurrent writes from two Workers fail:

Error: D1_ERROR: database is locked: SQLITE_BUSY

Our real case: a People Worker and a publisher Worker share one CONTENT_DB. Their writes overlap during an end-to-end test, and the read path also hits SQLITE_BUSY_SNAPSHOT.

Minimal reproduction: start two Workers in one Runtime with the same module and binding, D1.local({ binding: "DB", id: "shared" }), then request both at once.

export default {
  async fetch(request: Request, env: { DB: D1Database }) {
    const who = new URL(request.url).searchParams.get("who");
    await env.DB.prepare("CREATE TABLE IF NOT EXISTS t (who TEXT, n INTEGER)").run();
    const errors: string[] = [];
    await Promise.all(
      Array.from({ length: 200 }, (_, n) =>
        env.DB.prepare("INSERT INTO t VALUES (?, ?)").bind(who, n).run()
          .catch((e) => { errors.push(String(e)); }),
      ),
    );
    return Response.json({ errors: errors.length, first: errors[0] });
  },
};

Result in three of three runs (error counts out of 200 writes per Worker):

[{"errors":72,...},{"errors":59,...}]
[{"errors":43,...},{"errors":161,...}]
[{"errors":56,...},{"errors":144,...}]

The same two Workers under Miniflare (one workerd, one D1 owner) report zero errors. That is how we run this test for now.

The fix we'd prefer covers both R2 and D1: host each shared local resource's simulator in one runtime, and route the other Workers to it through their bindings. Retrying on SQLITE_BUSY would hide the error, but it would not give D1 transactions or snapshot reads one owner.

Versions: @alchemy.run/cloudflare-runtime 2.0.0-beta.80, macOS arm64.

[ryanleecode]

I'd like to take this: one runtime owns each simulator namespace, and the other runtimes reach it through their bindings, as suggested above. A PR with tests will follow, linked here.

Evidence that it's broader than R2 data files, and worse than an error the Worker sees:

It isn't limited to Workers that share a database or bucket. Each simulator's Durable Object namespace keeps a metadata.sqlite beside its per-object files, and every runtime opens it. Two Workers that bind different D1 databases still share it, and so does every Worker's Cache simulator:

{storage}/d1/cloudflare-runtime-D1DatabaseObject/metadata.sqlite
{storage}/cache/cloudflare-runtime-CacheObject/metadata.sqlite

We sampled /proc/<pid>/fd for every workerd while an alchemy dev stack of two Workers ran (each binds its own D1.local database, and both use caches.default). In every stack, two workerd processes held both files open at once.

workerd aborts. A locked metadata.sqlite doesn't come back as a query error:

*** Fatal uncaught kj::Exception: workerd/util/sqlite.c++:873: failed: SQLite failed; dbErrorMessage(prepareResult, db) = database is locked: SQLITE_BUSY (extended: SQLITE_BUSY_RECOVERY)
*** Fatal uncaught kj::Exception: workerd/util/sqlite.c++:1716: failed: SQLite failed; dbErrorMessage(err, db) = database is locked: SQLITE_BUSY

Runtime.start replaces the dead process on the same ports, so the dev session survives. But requests in flight on the old process are reset (ECONNRESET at the client) or held until the client gives up. Three restarts within 60 seconds stop the Worker for the rest of the session. At startup, the D1 migration gateway (PlatformProxy.open over the same storage directory) races the Worker it migrates for, and the stack fails with "The Workers runtime failed to start".

On an 8-core Linux host under load, with stderr teed from every workerd child:

• one test-suite run: 473 workerd processes spawned, 24 exited with code 1 on the lines above;
• one larger run: 490 spawned, 4 exited this way.

The other exits were Alchemy's own SIGKILLs on teardown, and code 0.

So D1 and Cache fail the same way as R2, at two levels: the per-database file when Workers share a resource, and the per-namespace metadata.sqlite even when they don't. A busy timeout inside a simulator wouldn't cover the second, because workerd itself opens that file, outside the simulator's code.

Versions: alchemy 2.0.0-beta.80, @alchemy.run/cloudflare-runtime 2.0.0-beta.80, workerd 1.20260918.1, Linux x64 and arm64.