Conversation
Part of apache#19524. Remove the changes job and every needs and step-level if that depended on it; the docs-only skip is a trigger-level paths filter, evaluated before any job exists. validate-source and validate-ci-baseline move verbatim to validate_source.yml, which is never path filtered because both are required status checks. The Flink, bundle-validation and integration-test jobs move verbatim to java_ci_engines.yml so each workflow stays under the ASF target of 15 concurrent jobs. The concurrency group carries the workflow name so the three workflows cancel only their own superseded runs. Review follow-ups: .github/** added to the positive paths list, a note in .asf.yaml on why only always-on workflows may contribute required contexts, and a shorter Codecov range comment.
Part of apache#19524, on top of the flattened job graph. - Surefire runs with hudi.surefire.forkCount=0.5C: two forks on a 4-core GitHub runner, one on a 2-core Azure agent. The two fixed test ports, HiveServer2 and the test ZooKeeper, are picked per JVM so forks do not collide; forks share one JaCoCo exec file. - TestHoodieDeltaStreamer is split into four classes by area, with bodies moved verbatim, so a second fork can help the utilities job. Three helpers move to HoodieDeltaStreamerTestBase. - The bloom filter benchmark stays out of unit runs. - The Spark workflow is resharded by measured time: Java UT split into the functional package and the rest, the three fat classes sharing a job with FTC, FTA and FTB one job each, Scala DML and Scala other split by package, utilities split into the DeltaStreamer classes and the rest. 14 jobs, under the ASF target of 15. Measured: 615 to 577 runner-minutes per run, longest job 77 to 41 minutes.
Part of apache#19524. Workflow files only. - flink-2's failsafe filter -Dit.test=!ITTestHoodieDataSource dropped failsafe's IT* include and ran 170 unit-test classes the docker job already runs. New filter: ITTest*,!ITTestHoodieDataSource. - The docker job built the whole reactor, 7.8 minutes, of which the 15 bundles and the examples are 6. The build, UT and FT reactors now skip the bundles, the examples and hudi-cli, each of which is built, tested or validated in the job that owns it. - hudi-examples-spark is built only in the jobs that run its quickstart. Measured over three runs: 577 to a mean of 548 runner-minutes, flink-2 39 to 23 minutes, the docker job 37 to about 30.
Closes apache#19528. Part of apache#19524. scripts/ci/test_timing_summary.py walks every surefire-reports directory, which surefire and the scalatest plugin both fill with JUnit XML, and prints the slowest classes and the slowest tests. Every job that runs tests, 14 in bot.yml and 3 in java_ci_engines.yml, gets one step after its test steps with if: always(), so the tables appear on the run summary for green and red jobs alike. Under a second of runner time per job.
Part of apache#19524. IT_MODULES names the four modules with integration-test classes outside hudi-flink: hudi-cli, hudi-integ-test, hudi-aws and hudi-azure. The build passes them with -am, which Maven resolves to 30 modules; the IT step verifies just the four instead of the whole reactor minus hudi-flink. Every module no longer built here is built, and where a bundle shaded and run, in the job that owns it. Measured: the job from 37 to 28 minutes, same 25 test classes.
The nine Java 17 Spark test jobs had no timeout-minutes, so a job whose surefire fork stops answering runs to GitHub's six hour default. That happened on a rerun of apache#19928: hudi-common finished all 184 test classes in 26 seconds, the fork never reported completion, and the job sat idle for 76 minutes until it was cancelled by hand. 75 minutes is what test-spark-client-and-hadoop-common already uses. The longest of these nine jobs measured 44 minutes on a slow runner, so the cap only fires on a hang.
The first run of this branch hung in test-flink-1's integration test step for the full six hour default before GitHub cancelled it. The five jobs of the engines workflow (flink-1, flink-2, the two bundle validations, integration-tests) had no timeout either; they measure 23 to 34 minutes, so 75 only fires on a hang.
test-hudi-hadoop-mr-and-hudi-java-client ran 9 minutes, 5 of them checkout and build, on the same JDK 11 lane as test-utilities-rest (18 to 22 minutes). Its test step moves into that job, whose build now also covers hudi-hadoop-mr and hudi-client/hudi-java-client, and its Codecov flag rides on the same upload. One job fewer per run, one build fewer, and the merged job stays well under the run's longest jobs at 33 to 36 minutes.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19933 +/- ##
============================================
- Coverage 80.20% 80.19% -0.01%
- Complexity 34661 34670 +9
============================================
Files 2546 2546
Lines 142452 142608 +156
Branches 17330 17484 +154
============================================
+ Hits 114260 114371 +111
- Misses 20281 20285 +4
- Partials 7911 7952 +41
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Collaborator
This was referenced Sep 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Part of #19524, stacked on #19931. Every Java CI job pays about five minutes of checkout and build before its tests.
test-hudi-hadoop-mr-and-hudi-java-clientran 9 minutes in total, andtest-utilities-rest18 to 22, on the same JDK 11 lane, so the small job's tests fit inside the other with room under the run's longest jobs.Summary and Changelog
test-utilities-restbecomestest-utilities-rest-hadoop-mr-java-client: its build also covershudi-hadoop-mrandhudi-client/hudi-java-client, and their unit-test step runs after the utilities steps, verbatim.utilities,hadoop-mr-java-client), so the flag views keep their history.test-hudi-hadoop-mr-and-hudi-java-clientis removed. Java CI goes from 14 jobs to 13.Measurement, three samples: the merged job's duration against 18 to 22 plus 9 before, the run's runner-minutes, and the run's longest job (which should not move).
Impact
CI configuration only.
Risk Level
low. Same commands, same lane; only the job boundary moves.
Documentation Update
none
Contributor's checklist