feat: plan runs in resumable pages of ids

[ccebrecos]

Why

Planning a large run took hours, and a worker restart in the middle of it left the run stuck forever.

• Slow: Dripper walked the scope with in_batches and plucked each batch again, two queries per batch. It created every batch in one transaction, so no child could start until the last batch existed. For ProjectManagement::BackfillCostRecordsLegacy in the monorepo (~16M rows, batch_size: 100) that is ~320k queries, and planning takes hours.
• Not restart-safe: the run was marked running before planning, and the dripper returned early on anything that was not enqueued. When a deploy killed the worker mid-plan, the retried job did nothing. The run sat in running with 0 batches forever (prod runs 9459–9652, 10205 and 10210).

What

• Paged planning: the dripper reads the scope's ids 10k per query (WHERE id > last planned id ORDER BY id LIMIT 10000) and cuts each page into batches of batch_size. Batches, total_count and processed_count stay exact, and sparse scopes never get empty batches. Plain min/max id ranges were ruled out: a company-scoped run would have become ~160k mostly empty batches.
• Children start immediately: each page is committed on its own, so its children are enqueued while the rest of the scope is planned. max_parallel_batches is respected as before.
• Resumable: the cursor is the last planned batch's finish_id, read under a row lock on the run. A retried dripper resumes after it and never plans the same ids twice. No schema change.
• BackfillRun#planning?: a run is still planning while it is running without a total_count, which is written when planning ends. finalize_if_batches_finished! leaves a planning run alone, and the dripper finalizes it at the end.
• amount_of_elements: caps the planned records at the first N matching ids, across pages.
• Retry failed batches: also resumes planning for a run that failed before it was fully planned.
• README section "How a run is planned" + CHANGELOG entries.

Tests

bundle exec rspec: 321 examples, 0 failures. rubocop: clean.

New examples cover:

• large scopes (one query per page)
• sparse scopes
• children enqueued page by page
• amount_of_elements across pages
• a worker killed mid-plan: the retry resumes without duplicates, the run cannot settle early, and it completes after resuming
• a run stopped mid-plan
• planning?
• the controller's retry path

Removing the planning guard fails 3 of them.

CI

The Rails 7.2 and 8.0 builds have been red on main since json 3.0 shipped: it dropped the quirks_mode option ActiveSupport 7.2/8.0 still passes, so loading the test schema aborted before any example ran. A separate commit pins json < 3 in those two gemfiles.

Follow-up

Bump data_drip in the monorepo, then re-run BackfillCostRecordsLegacy. Stuck runs 10205/10210 count as planning after the bump, so DataDrip::Dripper.perform_later(run) resumes them (or stop them).

Jira: FCT-65262