Crawler: fix lane getting stuck; atomic lane lock, always released, Run Manually takes over idle lanes

[qtwrk]

Problem

The crawler lane lock (wp-content/litespeed/crawler/meta.data.pid) can stay held indefinitely. Every later run, including Run Manually, then stops at ⚠️ lane in use and nothing gets crawled. The only way out has been to delete the file by hand or run wp litespeed-crawler run. There are several causes:

• The lane is released only on explicit success or abort paths. A fatal error, a timeout, a worker kill, or any exception from _engine_start() leaks it.
• The 1-hour stale fallback never fires when the lane file's mtime is in the future, because time() - filemtime() is negative.
• An empty or unreadable lane file is treated as free, but it can never be taken over.
• Take-over is check-then-write, so two requests can both take the same free lane.
• Run Manually is blocked by the lane gate before $manually_run is consulted.
• ⚠️ lane in use gives no context, so support cannot tell a live crawler from a dead lock.

Changes

Only the lane logic in src/crawler.cls.php changes, plus one line in cli/crawler.cls.php.

Commit 1: atomic lane, always released

• The lane is acquired atomically with fopen( $lane_file, 'x' ). The crawler/ directory is created first, because otherwise it is only created later by save_summary().
• Release_lane( $force = false, $expected_owner = null ) deletes the lane only for its owner, unless forced. A stale reclaim only deletes the lane it actually judged stale.
• The lane is released in finally and through register_shutdown_function. Both are ownership-checked, so neither can delete a newer crawler's lane.
• The mid-run strict check _check_valid_lane( true ) only verifies ownership and never releases.
• Empty, unreadable, more than 1 h old, or more than 60 s future-dated lane files now expire.
• wp litespeed-crawler run uses Release_lane( true ), so the CLI force-release keeps working.

Commit 2: lifecycle hardening and Run Manually take-over

• _touch_lane() rewrites the lane only while this request owns it, and never recreates a released lane file. The old touch() could recreate it as an empty file that then blocked the lane.
• Short-write cleanup deletes only the file this request created. It checks the inode while the handle is still open, and uses the bytes actually written as the expected owner.
• Release_lane() tolerates a lane that vanished and logs genuine unlink failures.
• Run Manually can take over a lane that has been idle for min( 3600, max( 120, MAP_TIMEOUT + 2 * TIMEOUT + 60 ) ) seconds, which is 300 s with the defaults. Cron and CLI keep the 1 h rule.
• _terminate_running() does not save when another crawler owns the lane, so a displaced crawler cannot overwrite the new owner's summary. Exceptions from _engine_start() now terminate with end_reason = 'exception'.
• ⚠️ lane in use now logs the lane file path, the owner hash and the age.

Testing

Isolated harness. The lane methods were extracted from the real file and each scenario ran in its own child process:

• 51 lane scenarios, plus 8 terminate/exception scenarios.
• Covered: acquire races, stale, empty, unreadable and future lanes, clock skew, shutdown and foreign release, short writes, and manual versus cron thresholds.
• All pass on PHP 7.4.33 and 8.3.
• 30 mutants of the new guards were used to check that the tests detect breakage. All are caught except two known equivalent or out-of-scope ones.

End-to-end on LiteSpeed Enterprise 6.3.7 with cPanel and PHP 8.2 (plugin 7.9.1 with these two files swapped in). Cron runs were triggered with Task::async_call( 'crawler' ), and Run Manually with crawler_force:

Scenario	Result
Normal cron / manual run	Takes the lane, then releases it
Crawl in progress	Lane keeps the same owner, age stays 1–3 s
Cron / manual while crawling	lane in use [file] … [owner] "sLl4paiL" [age] 1s
wp litespeed-crawler run mid-crawl	New crawler takes over. The old one stops at the next chunk (lane_invalid), skips terminating, and does not release the new lane
kill -9 mid-crawl	Lane remains, as expected, since no shutdown runs
Leaked lane, 32 s old, manual	Refused
Leaked lane, 400 s old	Cron refused; manual reclaims it (Lane file is stale (age 405s))
Empty lane: 2 h old / 10 s old	Reclaimed / refused
Lane mtime +120 s / +30 s	Reclaimed / refused
Foreign lane, 4000 s old, cron	Reclaimed
crawler/ dir missing	Created, crawler starts

On that host the crawler worker was killed after about 36 s every run, without any log entry. Before this change, each kill leaked the lane for an hour. With this change the shutdown release frees it, and the next run resumes from last_pos.

Known limitations, not addressed here

• On a very slow sitemap or sitemap index, map generation can outlast the manual take-over window. In that case two map generations can overlap, and the result self-heals on the next generation. The existing "Refresh Crawler Map" button already has the same behaviour.
• A displaced crawler can still save the summary a few times before its first chunk, which rolls back the progress display until the new owner saves again.
• fopen( 'x' ) is not atomic on NFSv2.
• Separate issue: the async request hash in Task::async_call() is not bound to its type, so a captured crawler hash can be replayed as crawler_force.