Skip to content

Terminate all processes on error in parallel_run_from_cli - #624

Open
yguclu wants to merge 2 commits into
develfrom
yguclu-abort-parallel-run
Open

yguclu wants to merge 2 commits into
develfrom
yguclu-abort-parallel-run

Conversation

@yguclu

@yguclu yguclu commented Oct 11, 2026

Copy link
Copy Markdown
Member

Terminate all MPI processes when one of them raises an uncaught exception in psydac.utilities.parallel_utils.parallel_run_from_cli (fixes #621).

Without this, the other processes hang as soon as they wait for a message from the failing one, and a job on a cluster keeps its allocation until it reaches the wall-time limit. The same mechanism as in the launcher python -m mpi4py is used: the exception is passed to mpi4py.run.set_abort_status and re-raised, hence its traceback is printed and the failing process is cleaned up, then MPI_Abort is called at exit. This works with any MPI launcher (e.g. mpirun, mpiexec, or srun in a batch job), and does not require running scripts with python -m mpi4py.

  • In parallel_run_from_cli, abort all processes at exit in case of an uncaught exception in main, and document this behavior in the docstring
  • Add section "Running examples in parallel" to the Examples page of the documentation (docs/source/examples.rst), which very strongly recommends running every parallel script through parallel_run_from_cli
  • New module psydac.utilities.tests.test_parallel_utils, which tests parallel_run_from_cli in serial with and without an exception in main (the call to MPI_Abort is mocked)

The abort was also checked manually with a script failing on one of 4 processes: with the fix, the job terminates with exit code 1 after printing the traceback; without it, the job hangs.

@codacy-production

codacy-production Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 7 complexity · 0 duplication

Metric Results
Complexity 7
Duplication 0

View in Codacy

🟢 Coverage 100.00% diff coverage · +0.05% coverage variation

Metric Results
Coverage variation ✅ +0.05% coverage variation
Diff coverage ✅ 100.00% diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (f2da0cd) 51844 33084 63.81%
Head commit (5943240) 103746 (+51902) 66252 (+33168) 63.86% (+0.05%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#624) 30 30 100.00%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@yguclu
yguclu requested a review from a team October 11, 2026 15:29
@yguclu
yguclu marked this pull request as ready for review October 11, 2026 15:29
@yguclu yguclu added the Next Release Must be in next release label Oct 11, 2026
@yguclu yguclu assigned yguclu and campospinto and unassigned yguclu Oct 15, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Next Release Must be in next release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Terminate all processes when one of them fails in parallel_run_from_cli

2 participants