feat: queuing of calls to python process by doc-han · Pull Request #718 · OpenFn/apollo

doc-han · 2026-10-13T09:05:57Z

Short Description

Caps how many python child processes bridge.ts runs at once
(APOLLO_MAX_CONCURRENT_PYTHON, default 10). Requests past the cap queue in
FIFO order in Node instead of spawning immediately, so a burst of traffic
can't pile an unbounded number of python interpreters on top of each other.

Fixes #

Implementation Details

Today every request to a python service spawns (or forks, if
APOLLO_FORK_SERVER is on) a process immediately, with no limit on how many
run at once. A burst of concurrent requests has no backpressure before it
hits the host's CPU/memory limits.

platform/src/util/python-queue.ts (new) — a small hand-rolled FIFO
semaphore: acquireSlot(scriptName, signal) grants immediately while under
APOLLO_MAX_CONCURRENT_PYTHON, otherwise queues the caller. If signal
aborts while still queued (the client went away before its turn), the wait
rejects and the slot is never consumed - the request never spawns a process
at all. releaseSlot() hands a freed slot straight to the next waiter.
platform/src/bridge.ts - run() now calls acquireSlot() right
before doing any real work, and releases exactly once (guarded against
double-release) from whichever path ends the run first. This applies
uniformly to both execution paths: forking from the warm master
(USE_FORK_SERVER) and spawning a fresh interpreter - either way it's one
python process doing a service's worth of real work, so both are capped the
same way.
.env.example - documents APOLLO_MAX_CONCURRENT_PYTHON (default 10).

Testing:

platform/test/util/python-queue.test.ts - pure unit tests of the queue
logic: immediate grant under the cap, FIFO ordering past it, a queued wait
rejecting on abort without consuming a slot, and an already-aborted signal
rejecting immediately when the cap is full.
platform/test/bridge-queue.test.ts - real-subprocess integration tests via
the existing _cancel_probe fixture: two run() calls serialize under a
cap of 1, and a call cancelled while queued never spawns a process.
Verified live with bun dev: under APOLLO_MAX_CONCURRENT_PYTHON=1/2,
concurrent requests to /services/echo correctly queue
(queueing echo: 1/1 python processes running, 1 waiting) and complete
serially; repeated with APOLLO_FORK_SERVER=true to confirm the same
queueing holds for the fork path.
bun test platform/test: 122 pass, 7 fail (pre-existing, unrelated -
live-Postgres tests with no DB available locally).

AI Usage

Please disclose whether you've used AI in this work (it's cool, we just want to
know!):

Yes, I have used AI
No, I have not used AI

You can read more details in our
Responsible AI Policy

josephjclark · 2026-10-16T09:21:05Z

Joe to do: write a script to send a bunch of concurrent requests and see how the queue behaves

josephjclark

Just some initial thoughts, shooting from the hip

josephjclark · 2026-10-16T08:45:03Z

+// FIFO semaphore capping how many python processes bridge.ts runs at once.
+// Requests past the cap wait here instead of spawning.
+
+const DEFAULT_MAX_CONCURRENT = 10;


IS 10 a random number?

Do we have any idea of the memory overhead per fork?

yeah 10 is a random number. yet to be validated

josephjclark · 2026-10-16T08:52:11Z

+    : DEFAULT_MAX_CONCURRENT;
+};
+
+let active = 0;


So this is module scoped. Only one queue per process.

Is that bad for testing? For re-use? Should there be a class or at least some instance based here?

Open question, maybe this is fine. Just sticks out at me a little as a weak architecture

josephjclark · 2026-10-16T08:53:31Z

+let active = 0;
+const queue: (() => void)[] = [];
+
+export const activeCount = (): number => active;


Are either of these actually neccessary? You could just export active and queue?

Looks like they're both just used by unit tests anyway

used by the test only.

josephjclark · 2026-10-16T09:24:22Z

+
+/** Releases a slot acquireSlot() granted, straight to the next waiter. Call
+ *  exactly once per resolved acquireSlot(). */
+export const releaseSlot = (): void => {


Should this be guarded to be a bit more idempotent? Or are we absolutely sure it's super tight?

Or what if every slot has its own release functuon, like with the cancel function?

josephjclark

Oh and we mentioned raising a sentry alert whenever an item is added to the queue? Or perhaps if more than N items are added

josephjclark · 2026-10-19T16:49:24Z

+  let started = false;
+  const { promise, cancel } = enqueue(scriptName, () => {
+    started = true;
+    return runFn(scriptName, port, inputPath, outputPath, onLog, onEvent, signal);


This is nice - much cleaner control logic I'd say?

josephjclark

Looking good. I'd still prefer to see the queue instanced but I'm happy to accept this.

I'll run some manual tests tomorrow morning.

josephjclark · 2026-10-23T14:48:43Z

Rebased with capacity test included

josephjclark · 2026-10-23T15:10:15Z

After a bit of finger trouble I've run the capacity tests with 512 memory in the container.

With no queuing, it'll blow up with just a couple of concurrent calls (1 is fine, 5 breaks it)

With the queue enabled I've just thrown 100 calls at the server (at the same time) and it's calmly processed them all in 114 seconds.

Well done @doc-han!

I am confident in saying that the server can handle a high number of concurrent requests 🎉

But I'm not going to guarantee we'll never blow the memory limit right now.

I'd keep the concurrency value quite low in prod. Maybe 5? I wonder if we can work out the typical process size of a typical global chat call, then work out a number from container size - some overhead for the sever (300mb?) / averagePythonSize.

As further work, we might also consider working out of we can constrain the python proccess size to, say, 300mb (or whatever we need). Hey @doc-han ever heard of a thing called cgroups?

Anyway @doc-han @elias-ba @hanna-paasivirta this can be released on Monday. Please set the production APOLLO_CAPACITY env var to a low value. I'm going to guess 5, but if anyone can work out a more scientific starting point I'd welcome it.

doc-han changed the base branch from main to forked-python-process October 13, 2026 09:22

doc-han changed the base branch from forked-python-process to main October 14, 2026 10:56

doc-han force-pushed the queue-python-process-calls branch from e7cb05e to 7443f35 Compare October 14, 2026 13:04

doc-han marked this pull request as ready for review October 14, 2026 13:30

josephjclark reviewed Oct 16, 2026

View reviewed changes

Comment thread platform/src/util/python-queue.ts Resolved

josephjclark reviewed Oct 19, 2026

View reviewed changes

josephjclark approved these changes Oct 19, 2026

View reviewed changes

josephjclark and others added 7 commits October 23, 2026 14:56

docs

845c65e

update docs and better reporting

0e40d9b

feat: queuing of python process

b157fd2

chore: cleanup comments

1ba4fb5

fix: resolve changes

5991760

test: resolve test

46fc368

chore: update changeset

515b259

josephjclark force-pushed the queue-python-process-calls branch from 1912dbd to 515b259 Compare October 23, 2026 14:48

josephjclark changed the base branch from main to release/3-5-2 October 27, 2026 10:53

josephjclark merged commit 26a82bc into release/3-5-2 Oct 27, 2026
3 checks passed

josephjclark deleted the queue-python-process-calls branch October 27, 2026 10:53

Conversation

doc-han commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short Description

Implementation Details

AI Usage

Uh oh!

josephjclark commented Oct 16, 2026

Uh oh!

josephjclark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josephjclark Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

doc-han Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josephjclark Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

josephjclark Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

doc-han Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

josephjclark Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josephjclark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josephjclark Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

josephjclark left a comment

Choose a reason for hiding this comment

Uh oh!

josephjclark commented Oct 23, 2026

Uh oh!

josephjclark commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

doc-han commented Oct 13, 2026 •

edited

Loading