Skip to content

Fix the GOPACS handler and redispatch poller lifecycle - #100

Merged
wborn merged 17 commits into
mainfrom
fix/gopacs-client-leak
Sep 17, 2026
Merged

wborn merged 17 commits into
mainfrom
fix/gopacs-client-leak

Conversation

@Miggets7

@Miggets7 Miggets7 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Part of #97 (GOPACS half). The Distro Energy status attributes from the same issue land in #96.

Client and task leaks

GOPACSHandler.undeploy() now closes the ResteasyClient. Every update of an EmsGOPACSAsset did stop-then-start, so each edit leaked the previous client and its connection pool. Closing is safe with the shared executor: createClient(ExecutorService) goes through ResteasyClientBuilder.executorService, which leaves Container.EXECUTOR untouched. Same fix as f878f18 for DistroEnergyHandler.

Both GOPACS constructors close the client if client.target(...) throws after createClient, since no reference escapes a constructor that threw and the teardown method could never run.

All scheduling goes through one schedule() helper that records every ScheduledFuture. Before, only the futures from processRawMessage were tracked, so the power updates from schedulePowerUpdate survived undeploy(). Teardown takes the endpoint down first, then sets a flag and cancels under the scheduledFutureList monitor, and schedule() checks that flag under the same monitor. A request already inside processRawMessage when teardown starts cannot queue work that outlives the handler.

EmsOptimisationService.stop() now undeploys gopacsHandlerMap. It only handled the redispatch pollers and optimisation timers before.

Both registries are ConcurrentHashMap. They are read and written from the persistence route, the attribute event subscription and the container stop thread, and this branch iterates them while removing entries.

Registering a handler under an EAN already in use stops the handler it displaces. Both maps are keyed by EAN and the put went straight over any live entry, leaving it with its endpoint and client open and no key to stop it by.

Deployment lifecycle

deploy() moved out of the constructor into a public no-arg method, called by EmsOptimisationService after the handler is registered in its map. The JAX-RS resource routes straight to processRawMessage, so the handler was reachable from request threads before construction finished. Same shape as DistroEnergyHandler.deploy() and GOPACSRedispatchHandler.startPolling().

CREATE and UPDATE now run one reconcile per handler, since creating an asset is the case where nothing is deployed yet:

  • A save that leaves the EAN pointing at the same asset keeps the handler. Every save used to tear it down and build a new one, so a rename dropped the participant cache and cancelled any UFTP conversation waiting on a delayed response or FlexOffer. The handler reads only the EAN, realm and asset id from the asset, everything else from container config.
  • An EAN edit re-keys the handler and the poller. processAssetChange stopped by the EAN on the updated entity, which carries the new value, so an EAN edit through the asset editor was a silent no-op stop: the handler under the old EAN kept its endpoint and client alive next to the new one. The attribute-event path already used the old value, but an asset merge emits no attribute events. Stopping now goes by asset id, on DELETE as well.
  • Clearing the EAN on a save stops the handler. It used to be ignored.
  • redispatchEnabled toggled through the asset editor takes effect, which previously needed an attribute event or a restart. A poller already running under the current EAN is left alone, because its announcement bookkeeping is in memory only and history entries are not deduplicated against the asset, so a restart would re-record every open announcement.

Tests

GOPACSHandlerTest gains five cases: undeploy on a test-support handler is a no-op, undeploy cancels every pending future exactly once, completed futures are pruned on the next schedule call, and scheduling after undeploy reaches the executor neither directly nor through a processed FlexRequest. GOPACSHandlerHttpTest calls deploy() after constructing the handler.

New EmsOptimisationServiceGopacsTest covers CREATE, DELETE by asset id, UPDATE with the same EAN, UPDATE with a changed EAN, UPDATE that clears the EAN, redispatchEnabled toggled on save, displacing a handler registered under an EAN already in use, and stop() for both the GOPACS handler and the redispatch poller. It builds recording subclasses through the real constructors, with a fake Container carrying the OAuth config and a temp private-key file. Same pattern as the Distro Energy service test in #96. Each behaviour case fails without its fix.

./gradlew :ems:test against the local OR stack: 55 tests, 0 failures.

GOPACSHandler kept its ResteasyClient in a field but undeploy() only
cancelled the scheduled futures and undeployed the web service. Every
update of an EmsGOPACSAsset does stop-then-start, so each edit left the
previous client and its connection pool allocated. Closing is safe with
the shared executor: createClient(ExecutorService) goes through
ResteasyClientBuilder.executorService, which leaves Container.EXECUTOR
untouched.

The constructor now closes the client if client.target(...) throws
between createClient and the end of construction, since no reference
escapes a constructor that threw and undeploy() could never run.

Scheduling now goes through a single schedule() helper that records
every ScheduledFuture. Previously only the futures from processRawMessage
were tracked; the power updates scheduled by schedulePowerUpdate were
not, so they survived undeploy(). The list is synchronized because tasks
are scheduled from Undertow request threads and cancelled from the
service thread, and completed futures are pruned on each call so the
list does not grow for the handler's lifetime.

Tests cover: undeploy on a test-support handler is a no-op, undeploy
cancels every pending future exactly once, and completed futures are
pruned on the next schedule call.
GOPACSHandler deployed its JAX-RS application as the last step of its
constructor. The deployed resource routes straight to processRawMessage,
so the handler was reachable from Undertow request threads before the
constructor had returned, and the Java memory model gives no guarantee
about what those threads see of an object whose reference escaped
mid-construction.

deploy() is now a public no-arg method called after construction, the
same shape DistroEnergyHandler and GOPACSRedispatchHandler.startPolling
already use. EmsOptimisationService registers the handler in its map
before calling deploy(), so a deployment that throws still leaves the
handler reachable from stop() and its client gets closed.
EmsOptimisationService.stop() cancelled the redispatch pollers and the
optimisation timers but never touched gopacsHandlerMap, so the GOPACS
handlers kept their JAX-RS deployment and REST client across a service
stop. Undeploy and clear them alongside the other handler maps.
@Miggets7 Miggets7 added the Bug Error or issue in the system label Sep 16, 2026
Covers processAssetChange for EmsGOPACSAsset CREATE, UPDATE and DELETE
and the new stop() teardown, through a recording GOPACSHandler subclass
built by the real constructor (fake Container with OAuth config and a
temporary private-key file) whose deploy()/undeploy() only count calls.
Same pattern as EmsOptimisationServiceDistroEnergyTest; no mocking of
the concrete class and no extra test dependency.
processAssetChange stopped the handler on UPDATE with the EAN from the
updated entity, but gopacsHandlerMap is keyed by EAN and the entity
carries the new value. When the contracted EAN was edited through the
asset editor the stop was a silent no-op, so the handler registered
under the old EAN kept its JAX-RS deployment and REST client alive next
to the new one. The attribute-event path already handles this case with
the old value, but an asset merge does not emit attribute events.

Undeploy every handler deployed for the asset id instead, whichever EAN
it is registered under. GOPACSHandler exposes the asset id it was
deployed for; the field name (electricitySupplierAssetId) predates the
EmsGOPACSAsset type and is left alone.

The lifecycle test gains the changed-EAN case, which failed before this
change.
processAssetChange left the redispatch poller untouched on UPDATE, on
the grounds that it is managed through attribute events. An asset merge
emits none, so editing the contracted EAN through the asset editor left
the poller running under the old EAN: it kept polling and resolving EAN
effectivity for a congestion point the asset no longer has, and there
was no key it could be stopped by afterwards.

Unlike the GOPACS handler, the poller is not simply restarted on every
save. Its announcement bookkeeping (lastProcessedAnnouncementId and the
set of recorded announcement ids) is in memory only and history entries
are not deduplicated against the asset, so a restart re-records every
open announcement. UPDATE therefore only stops a poller registered for
the asset under an EAN other than the current one, and starts a new
one under the current EAN if redispatch is still enabled. A poller
already running under the current EAN is left alone.

GOPACSRedispatchHandler exposes the asset id it was started for. The
lifecycle test gains CREATE, DELETE, same-EAN UPDATE, changed-EAN UPDATE
(enabled and disabled) and stop() cases for the poller; the two
changed-EAN cases failed before this change.
Toggling redispatchEnabled through the asset editor had no effect until
an attribute event or a service restart: processAssetChange only acted
on the poller when the EAN changed, and an asset merge emits no
attribute events.

The UPDATE branch now reconciles the poller with the saved asset state:
exactly one poller under the current EAN when redispatch is enabled,
none otherwise. A poller already running under the current EAN is still
left alone, for the same reason as before: its announcement bookkeeping
is in memory only and a restart would re-record history.

Replaces the stale-EAN-only helper from the previous commit. The
lifecycle test gains the enable-on-save and disable-on-save cases, which
failed before this change.

@wborn wborn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an initial AI-assisted review.

The REST client cleanup, explicit deployment lifecycle, and service-stop cleanup are good improvements. One blocking concurrency issue remains in the scheduled-task teardown: an in-flight request can still schedule new work after undeploy() has cancelled and cleared the currently tracked futures. See the inline comment.

Comment thread ems/src/main/java/org/openremote/extension/ems/manager/gopacs/GOPACSHandler.java Outdated
A request already inside processRawMessage could schedule work after
undeploy() had cancelled and cleared the tracked futures. That task
outlived the handler and later ran against a closed client.

The endpoint is now undeployed first, so no further request can reach
processRawMessage, and schedule() refuses under the same monitor the
cancel loop holds. Either a task is tracked and then cancelled, or it is
never scheduled at all.

Also drops the trailing comments that the formatter had split across the
schedule() calls and the statements after them. They named fixed 10s and
30s delays while both values come from configuration.
Both registries are touched from the persistence route, the attribute
event subscription and the container stop thread. Stopping handlers by
asset id and reconciling the redispatch poller iterate them with
entrySet().removeIf while stop() walks them with forEach, so a
concurrent put or remove on a HashMap could throw
ConcurrentModificationException or corrupt the table.
Both registries are keyed by contracted EAN, and registering a new
handler put straight over any entry already under that key. The
displaced handler kept its JAX-RS endpoint and REST client open with no
key left to stop it by, so two assets sharing an EAN leaked exactly what
this branch set out to stop leaking.

Stopping the current entry first covers every caller, including the
redispatch reconcile, which could reach this case for a poller owned by
another asset.
DELETE stopped by the EAN on the deleted entity while UPDATE stops by
asset id. The entity carries the EAN as saved, which is not necessarily
the key the running handler is registered under, so the same silent
no-op that UPDATE had applied here.

The asset-scoped stop for the poller is now a helper shared with the
redispatch reconcile, which was doing the same removeIf inline.
GOPACSRedispatchHandler creates its client and then builds two proxies
from it. If client.target(...) throws, no reference escapes the
constructor, so stopPolling() can never run and the client is left open.
Same guard the GOPACS handler already got.
Every save of the asset tore the handler down and built a new one, so a
rename dropped the participant cache and cancelled any UFTP conversation
waiting on a delayed response or FlexOffer. The redispatch poller was
already spared for the same reason. The handler reads only the EAN, the
realm and the asset id from the asset, everything else from container
config, so leaving it alone when the EAN still maps to this asset is
safe.

CREATE and UPDATE now run the same reconcile, since creating an asset is
the case where nothing is deployed yet. That drops the branch that
duplicated the start calls, and with stopping done by asset id the EAN no
longer has to be unwrapped before choosing what to do, so clearing the
EAN on a save now stops the handler instead of being ignored.

Also drops the keepEan parameter from the poller's asset-scoped stop.
Deciding which poller to leave alone belongs in the reconcile, which now
returns early, and the helper matches its GOPACS counterpart again.
@Miggets7 Miggets7 changed the title Close the GOPACS REST client on undeploy Fix the GOPACS handler and redispatch poller lifecycle Sep 17, 2026

@wborn wborn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous scheduled-task teardown race is fixed, and the latest reconciliation changes avoid unnecessarily rebuilding a handler when the EAN is unchanged.

One lifecycle issue remains in EmsOptimisationService: asset persistence changes can also generate attribute events, so the GOPACS and redispatch lifecycle can be processed independently by both paths. This can cause redundant teardown/recreation and, when the paths overlap, can lose ownership of a live handler. See the inline comment.

This review was AI-assisted.

A merge does raise attribute events. AssetStorageService.publishModificationEvents()
publishes one per created attribute on CREATE, and on UPDATE one per added or
modified attribute, or one per attribute outright when the save touched anything
besides the attributes. So every save of a GOPACS asset reached this service twice:
once through the persistence route, once through the attribute event subscription,
which ran their own separate stop/start pairs.

That undid the previous commit. Renaming the asset republishes an unchanged
contractedEan, and the attribute path tore the handler down and built a new one
anyway, losing the participant cache and every UFTP conversation waiting on a
delayed response or FlexOffer.

The two paths also run on different threads: PersistenceTopic is a seda endpoint
with multipleConsumers, and the subscriber dispatch route hands events to its own
executor. Interleaved, both stops could find nothing to stop and both puts land,
leaving the first handler registered over by the second with its endpoint and
client still open.

Both paths now call the same reconcile, so whichever arrives second finds the
handler already in its target state and leaves it alone, and the whole transition
is held under a lifecycle lock so the two cannot interleave. Container start and
stop take the same route, which drops the duplicated start calls in start().

The tests play back a save as the manager performs it, the persistence event
followed by the attribute events raised from it, rather than the persistence event
on its own.
schedule() was widened to protected and pendingTaskCount() added purely so
GOPACSHandlerTest could reach them. Neither belongs in the production API:
schedule() is called only from within the handler, and the pending count is
internal bookkeeping no caller has any business reading.

schedule() goes back to private and pendingTaskCount() is gone. The test reaches
both reflectively, which it has to do because Groovy dispatches to the recording
subclass and will not find a private member of the superclass.
save() already replayed the persistence event and its attribute events
for an EAN change and a rename, but clearing the EAN and toggling
redispatch were only tested through processAssetChange() directly. Both
are handled by the same nonAttributeChange republish as a rename, so
they are exactly the case the lifecycle-lock fix in the previous commit
is meant to cover.

save() now takes an optional oldRedispatchEnabled, alongside the
existing oldEan, so a test can state the toggle's previous value when
the republished attribute event needs one.
# Conflicts:
#	ems/src/main/java/org/openremote/extension/ems/manager/EmsOptimisationService.java
@Miggets7
Miggets7 requested a review from wborn September 17, 2026 08:44

@wborn wborn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM now!

@wborn
wborn added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit 82797dc Sep 17, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug Error or issue in the system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants