Skip to content

Fix task polling deadlines in async and sync clients - #1468

Open
agocharbhatia wants to merge 1 commit into
opsmill:developfrom
agocharbhatia:fix/task-polling-deadline
Open

agocharbhatia wants to merge 1 commit into
opsmill:developfrom
agocharbhatia:fix/task-polling-deadline

Conversation

@agocharbhatia

@agocharbhatia agocharbhatia commented Oct 31, 2026 •

Copy link
Copy Markdown

Fixes #1311.

Change

Both clients check task status immediately, including when the interval exceeds the timeout. Subsequent polling uses a monotonic deadline, includes request time in the budget, and caps each sleep to the remaining time. No new poll begins after the deadline.

Terminal states, lookup errors and async cancellation retain their existing behavior. Negative timeouts, nonpositive intervals and non-finite values are rejected before HTTP access. Public signatures and dependencies are unchanged.

Verification

The same 41 native-client regression cases give 32 failures and 9 passes on the pinned develop base, then 41 passes patched. Mocked GraphQL responses and a deterministic clock cover initial checks, request-time accounting, capped sleeps, errors and cancellation.

The full native unit command passed under Python 3.12.13 in independent replay: 2087 passed, 1 expected failure. Formatting, Ruff, ty, mypy, documentation generation and generated-document validation passed.

Earlier local launcher runs hit Python subprocess pre-exec SIGSEGV failures. Their cause remains unresolved; the successful native replay used default paths and no test exclusions. This is not a hosted-CI claim.

Limits

The deadline bounds polling and sleeps, not an already-running request. Existing client transport/retry settings still govern that request. No live Infrahub service, stable-branch backport or cross-version matrix was tested. This targets develop, where the issue was reproduced.


Summary by cubic

Fixes task polling deadlines in both async and sync clients so waits no longer overrun or reject valid conditions.

  • Both clients now check task status immediately, even when the polling interval exceeds the timeout; an already-final task is returned instead of raising TaskNotCompletedError.
  • Polling uses a monotonic deadline that includes request time in the budget, and each sleep is capped to the remaining time, so no new request starts after the deadline.
  • Invalid timing values (negative timeouts, nonpositive or non-finite intervals) are rejected with ValueError before any HTTP request.
  • Terminal states, lookup errors, and async cancellation behave as before; public signatures and dependencies are unchanged.

Written for commit 571e082. Summary will update on new commits.

View guided diff

@agocharbhatia
agocharbhatia requested a review from a team as a code owner October 31, 2026 21:02
@polmichel
polmichel self-requested a review November 13, 2026 13:55

@polmichel polmichel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution @agocharbhatia.

I’ve left a few comments, mostly on the test suite and one on the data validation layer.

Could you target the stable branch please? It’s the appropriate branch for improvements like this one.

I’ll iterate quickly once you’ve addressed the comments

Comment on lines +78 to +86
@pytest.mark.parametrize(
("state", "interval", "timeout"),
[
(TaskState.COMPLETED, 90, 60),
(TaskState.FAILED, 1, 0),
(TaskState.CANCELLED, 1, 60),
(TaskState.CRASHED, 7, 60),
],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For parameterized tests, we often use dataclasses, which we find make the test cases easier to read. Here is how it could look here:

@dataclass
class TerminalStateCase:
    name: str
    state: TaskState
    interval: int
    timeout: int


TERMINAL_STATE_CASES = [
    TerminalStateCase(name="completed-interval-exceeds-timeout", state=TaskState.COMPLETED, interval=90, timeout=60),
    TerminalStateCase(name="failed-zero-timeout", state=TaskState.FAILED, interval=1, timeout=0),
    TerminalStateCase(name="cancelled-default-window", state=TaskState.CANCELLED, interval=1, timeout=60),
    TerminalStateCase(name="crashed-uneven-interval", state=TaskState.CRASHED, interval=7, timeout=60),
]

This piece of code is just a suggestion obviously

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually you can find the rule inside our guidelines here .agents/rules/python-testing.md

Comment on lines +107 to +110
@pytest.mark.parametrize(
("interval", "timeout", "query_duration", "elapsed"),
[(90, 60, 0, 60), (7, 60, 0, 60), (7, 60, 2, 60), (1, 5, 2, 5), (1, 5, 9, 9)],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another example of dataclasses usage here:

PollingBudgetCase(
        name="final-sleep-capped-to-deadline",
        interval=7,
        timeout=60,
        query_duration=0,
        expected_request_starts=[0, 7, 14, 21, 28, 35, 42, 49, 56],
        expected_elapsed=60,
    ),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

await _wait(clients, client_type, interval, timeout)

assert request_times
assert all(start < timeout for start in request_times)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better assertion here would be to assert the exact request_times

Comment on lines +166 to +179
@pytest.mark.parametrize(
("interval", "timeout"),
[
(0, 60),
(-1, 60),
(1, -1),
(float("nan"), 60),
(float("inf"), 60),
(-float("inf"), 60),
(1, float("nan")),
(1, float("inf")),
(1, -float("inf")),
],
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same rule related to parameterized tests and data classes

Comment on lines +25 to +48
class PollClock:
"""Expose elapsed polling time without changing the event loop's clock."""

elapsed: float = 0.0
sleeps: list[float] = field(default_factory=list)

def monotonic(self) -> float:
return self.elapsed

def sleep(self, delay: float) -> None:
assert delay > 0
self.sleeps.append(delay)
self.elapsed += delay

async def asleep(self, delay: float) -> None:
self.sleep(delay)


@pytest.fixture
def poll_clock(monkeypatch: pytest.MonkeyPatch) -> PollClock:
clock = PollClock()
monkeypatch.setattr(manager, "time", SimpleNamespace(monotonic=clock.monotonic, sleep=clock.sleep))
monkeypatch.setattr(manager, "asyncio", SimpleNamespace(sleep=clock.asleep))
return clock

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A deterministic clock is a good idea. I can see a potential issue regarding how the fixture installs it: it rebinds time and asyncio inside manager to objects that only expose monotonic and sleep. If manager.py later calls any other function from these modules, the tests will fail for a non-valid reason.

tests/unit/sdk/test_retry.py already has a FakeClock and a _patch_sleep helper that do the same job by replacing only the functions involved. Is this possible that you reuse them here instead of PollClock?

Internal thought: move these helpers to a shared test module in a follow-up


"""
for _ in range(timeout // interval):
if not math.isfinite(interval) or interval <= 0:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An integer too large for a float (for example timeout=10**400) makes math.isfinite raise OverflowError before the check runs. This applies to both interval and timeout. Converting both with float() first and turning an OverflowError into a ValueError would match the docstring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants