Repository navigation
Fix task polling deadlines in async and sync clients - #1468
agocharbhatia wants to merge 1 commit into
Conversation
polmichel
left a comment
There was a problem hiding this comment.
Thanks for your contribution @agocharbhatia.
I’ve left a few comments, mostly on the test suite and one on the data validation layer.
Could you target the stable branch please? It’s the appropriate branch for improvements like this one.
I’ll iterate quickly once you’ve addressed the comments
| @pytest.mark.parametrize( | ||
| ("state", "interval", "timeout"), | ||
| [ | ||
| (TaskState.COMPLETED, 90, 60), | ||
| (TaskState.FAILED, 1, 0), | ||
| (TaskState.CANCELLED, 1, 60), | ||
| (TaskState.CRASHED, 7, 60), | ||
| ], | ||
| ) |
There was a problem hiding this comment.
For parameterized tests, we often use dataclasses, which we find make the test cases easier to read. Here is how it could look here:
@dataclass
class TerminalStateCase:
name: str
state: TaskState
interval: int
timeout: int
TERMINAL_STATE_CASES = [
TerminalStateCase(name="completed-interval-exceeds-timeout", state=TaskState.COMPLETED, interval=90, timeout=60),
TerminalStateCase(name="failed-zero-timeout", state=TaskState.FAILED, interval=1, timeout=0),
TerminalStateCase(name="cancelled-default-window", state=TaskState.CANCELLED, interval=1, timeout=60),
TerminalStateCase(name="crashed-uneven-interval", state=TaskState.CRASHED, interval=7, timeout=60),
]
This piece of code is just a suggestion obviously
There was a problem hiding this comment.
Actually you can find the rule inside our guidelines here .agents/rules/python-testing.md
| @pytest.mark.parametrize( | ||
| ("interval", "timeout", "query_duration", "elapsed"), | ||
| [(90, 60, 0, 60), (7, 60, 0, 60), (7, 60, 2, 60), (1, 5, 2, 5), (1, 5, 9, 9)], | ||
| ) |
There was a problem hiding this comment.
Another example of dataclasses usage here:
PollingBudgetCase(
name="final-sleep-capped-to-deadline",
interval=7,
timeout=60,
query_duration=0,
expected_request_starts=[0, 7, 14, 21, 28, 35, 42, 49, 56],
expected_elapsed=60,
),
There was a problem hiding this comment.
expected_request_starts is tied to this comment https://github.com/opsmill/infrahub-sdk-python/pull/1468/changes#r4564400344
| await _wait(clients, client_type, interval, timeout) | ||
|
|
||
| assert request_times | ||
| assert all(start < timeout for start in request_times) |
There was a problem hiding this comment.
A better assertion here would be to assert the exact request_times
| @pytest.mark.parametrize( | ||
| ("interval", "timeout"), | ||
| [ | ||
| (0, 60), | ||
| (-1, 60), | ||
| (1, -1), | ||
| (float("nan"), 60), | ||
| (float("inf"), 60), | ||
| (-float("inf"), 60), | ||
| (1, float("nan")), | ||
| (1, float("inf")), | ||
| (1, -float("inf")), | ||
| ], | ||
| ) |
There was a problem hiding this comment.
Same rule related to parameterized tests and data classes
| class PollClock: | ||
| """Expose elapsed polling time without changing the event loop's clock.""" | ||
|
|
||
| elapsed: float = 0.0 | ||
| sleeps: list[float] = field(default_factory=list) | ||
|
|
||
| def monotonic(self) -> float: | ||
| return self.elapsed | ||
|
|
||
| def sleep(self, delay: float) -> None: | ||
| assert delay > 0 | ||
| self.sleeps.append(delay) | ||
| self.elapsed += delay | ||
|
|
||
| async def asleep(self, delay: float) -> None: | ||
| self.sleep(delay) | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def poll_clock(monkeypatch: pytest.MonkeyPatch) -> PollClock: | ||
| clock = PollClock() | ||
| monkeypatch.setattr(manager, "time", SimpleNamespace(monotonic=clock.monotonic, sleep=clock.sleep)) | ||
| monkeypatch.setattr(manager, "asyncio", SimpleNamespace(sleep=clock.asleep)) | ||
| return clock |
There was a problem hiding this comment.
A deterministic clock is a good idea. I can see a potential issue regarding how the fixture installs it: it rebinds time and asyncio inside manager to objects that only expose monotonic and sleep. If manager.py later calls any other function from these modules, the tests will fail for a non-valid reason.
tests/unit/sdk/test_retry.py already has a FakeClock and a _patch_sleep helper that do the same job by replacing only the functions involved. Is this possible that you reuse them here instead of PollClock?
Internal thought: move these helpers to a shared test module in a follow-up
|
|
||
| """ | ||
| for _ in range(timeout // interval): | ||
| if not math.isfinite(interval) or interval <= 0: |
There was a problem hiding this comment.
An integer too large for a float (for example timeout=10**400) makes math.isfinite raise OverflowError before the check runs. This applies to both interval and timeout. Converting both with float() first and turning an OverflowError into a ValueError would match the docstring.
Fixes #1311.
Change
Both clients check task status immediately, including when the interval exceeds the timeout. Subsequent polling uses a monotonic deadline, includes request time in the budget, and caps each sleep to the remaining time. No new poll begins after the deadline.
Terminal states, lookup errors and async cancellation retain their existing behavior. Negative timeouts, nonpositive intervals and non-finite values are rejected before HTTP access. Public signatures and dependencies are unchanged.
Verification
The same 41 native-client regression cases give 32 failures and 9 passes on the pinned develop base, then 41 passes patched. Mocked GraphQL responses and a deterministic clock cover initial checks, request-time accounting, capped sleeps, errors and cancellation.
The full native unit command passed under Python 3.12.13 in independent replay: 2087 passed, 1 expected failure. Formatting, Ruff, ty, mypy, documentation generation and generated-document validation passed.
Earlier local launcher runs hit Python subprocess pre-exec SIGSEGV failures. Their cause remains unresolved; the successful native replay used default paths and no test exclusions. This is not a hosted-CI claim.
Limits
The deadline bounds polling and sleeps, not an already-running request. Existing client transport/retry settings still govern that request. No live Infrahub service, stable-branch backport or cross-version matrix was tested. This targets develop, where the issue was reproduced.
Summary by cubic
Fixes task polling deadlines in both async and sync clients so waits no longer overrun or reject valid conditions.
TaskNotCompletedError.ValueErrorbefore any HTTP request.Written for commit 571e082. Summary will update on new commits.