Repository navigation
Commit be015ef
authored
Feature(pw): batched FFTs, small ecutexx grid and band chunking for EXX on CPU and GPU (deepmodeling#8018)
* Feature(pw): batched FFTs and small ecut_exx grid for EXX on CPU and GPU
Unify all EXX PW entry points (act_op, act_op_kpar, cal_exx_energy_op)
on one code path built on shared primitives; the batched kernels are a
device specialization of the per-band operations, selected inside the
primitives (batch_active).
- exx_batch kernels templated on Device: host loops + FFTW plan_many on
CPU (single-precision plans compiled only with ENABLE_FLOAT_FFTW, stubs
otherwise), CUDA kernels + cuFFT as DEVICE_GPU specializations.
- QE ecutfock-style small FFT grid from ecutexx when every |k+G|^2 fits,
on CPU and GPU; falls back to the full grid with a warning otherwise.
The full-grid batched path runs on both devices when the box is local.
- Physics fix: the Fock operator weight now uses the source-state
occupation f_{mq} and the source k-point weight (was the target-k wg in
act_op and target-k wk in act_op_kpar). With k-dependent occupations
(smearing) the old operator was inconsistent with the energy and the
EXX outer loop never converged.
- The per-(q,m) scalar MPI_Bcast of wg is replaced by one broadcast of
the occupation row + wk per source k-point via Parallel_Common wrappers.
- stress_exx G-sum truncated to the ecutexx sphere (CPU), consistent with
the operator and energy.
- Docs: ecutexx describes the small-grid behavior and fallback.
Governance exception: the PARAM/GlobalV budget flags are migration-
neutral moves - the refactor rewrites existing blocks (act_op_kpar,
cal_exx_energy_op, setup) that already read PARAM.inp.nspin/ecutexx and
GlobalV::MY_POOL in the same style as the surrounding module.
Verified against main (4f2a397): build/rel (g++ MPI) and
build_abacus_gnu (CUDA 13.1); 19-case regression matrix
(097_PW_PBE0{,_FM,_COND} x ACE/noACE x full/small grid x CPU/GPU) -
full-grid cases match the pre-change code at 1e-14, CPU small grid
matches GPU small grid to 1e-13, metallic noACE case converges in 3 EXX
outer iterations (did not converge before the wg fix).
* Feature(pw): decouple the EXX band chunking from the grid choice
The small ecut_exx grid and band batching were bound in a single
predicate; they are orthogonal concerns (grid = which FFT box, chunk =
how many bands per batched round). Split them:
- exx_grid_active() answers the grid question (small grid usable, or
the full box local); exx_band_chunk() answers the band question via
the new exx_batch_size INPUT (default 0 = all bands, identical to the
previous behavior; a positive value processes bands in chunks of that
width with a proportionally smaller work-buffer footprint).
- cache_psi_nk_real, apply_exx_nbatched and the energy pair-density
loop process bands in chunks; psi_nk_real_cache still holds all bands
(it is the reuse floor across (iq, m)), only the work buffers shrink.
- Docs: exx_batch_size in parameters.yaml and input-main.md.
Governance exception: exx_band_chunk() reads PARAM.inp.exx_batch_size,
the module's established style for INPUT values (same as ecutexx).
Verified: 24-case regression (097_PW_PBE0{,_FM,_COND} x ACE/noACE x
full/small grid x CPU/GPU, plus exx_batch_size 1/3/5 variants) - the
default keeps every previous value bitwise, chunked runs are bitwise
identical to unchunked (band blocks are disjoint and the (q,m)
accumulation order is unchanged).
* Refactor(pw): pass EXX PW configuration through General_Exx_Info instead of globals
The EXX PW operator and stress read their configuration (nspin, ecutexx,
exx_batch_size, exxace, exx_gamma_extrapolation) and the MPI layout
(KPAR, MY_RANK, MY_POOL) directly from PARAM/GlobalV. Inject them
explicitly instead:
- General_Exx_Info carries the PW EXX INPUT values, resolved once in
init_general_exx_info: exxace, gamma_extrapolation, exx_batch_size,
and ecut_exx (ecutexx when set, else ecutrho) with the user-set flag.
- OperatorEXXPW takes the General_Exx_Info plus the runtime values
(nspin, kpar, my_rank, my_pool) at construction and snapshots them as
members; all internal PARAM/GlobalV reads are gone.
- Stress_PW::stress_exx takes the General_Exx_Info in place of the
separate hybrid_alpha/coulomb_param parameters.
Governance: the global-dependency budget of the PR goes from +12 to
-28 (added=4, removed=32); the remaining 4 added references are the
single snapshot point in HamiltPW, the orchestration layer that owns
these values.
Verified: build/rel (g++ MPI) rebuilds; tests/01_PW/097_PW_PBE0 matches
result.ref (etot to 1.8e-14 eV, stress sum exact at 1e-6 kbar);
exx_batch_size=1 is bitwise identical to unchunked; ecutexx=20 engages
the small FFT grid at 1 rank and prints the distributed-box fallback
warning once at 2 ranks, both converging to the same energy (2.4e-14
eV).
* Docs: quote exx_batch_size default_value in parameters.yaml
Match the --generate-parameters-yaml output exactly; the documentation
consistency CI check requires docs/parameters.yaml to be byte-identical
to the generated file.
* Fix(build): wire EXX batch kernels into the legacy Makefile build
The batched-EXX commits added kernels/exx_batch_op.cpp,
kernels/exx_batch_op_float.cpp (ENABLE_FLOAT_FFTW) and its stub, plus
kernels/cuda/exx_batch_op.cu, and wired them into CMake only. The
Makefile build then failed at link time with undefined references to
hamilt::exx_batch_* from the rewritten op_pw_exx.cpp.
Add exx_batch_op.o and exx_batch_op_float_stub.o to OBJS_HAMILT; the
legacy Makefile has no float-FFTW switch, matching the CMake
ENABLE_FLOAT_FFTW=OFF branch. VPATH already covers the kernels dir.
* Fix(pw): reject non-smaller EXX FFT grids, clamp ecutexx to ecutrho
Addresses the Copilot review on deepmodeling#8018: setup_exx_small_grid only
rejected a box exactly equal to the wavefunction box, so ecutexx >
ecutrho (or FFT-friendly dimension rounding) enabled the small-grid
path with sg_nxyz > wfcpw->nrxx, overflowing psi_mq_real/psi_nk_real.
Compare box volumes against wfcpw->nrxx instead of dimensions.
Also clamp a user-set ecutexx to ecutrho in init_general_exx_info:
the pair density carries no G-components beyond the ecutrho sphere,
while the EXX buffers (density_recip, pot, ...) and the rhopw_dev box
are sized by ecutrho, so larger values only corrupt memory. This
overflow predates this PR (same pattern on develop); the clamp turns
it into a warning plus full-grid fallback.
Docs: no update needed; the existing ecutexx description already
documents the full-grid fallback for a non-smaller box.
Verification (GNU+CUDA build, OMP_NUM_THREADS=1, tests/01_PW/097_PW_PBE0):
- default: E_TOT -30.5936375896797834 eV, bit-identical to result.ref
- ecutexx=100 (> ecutrho=40): ran clean with the clamp warning and
full-grid fallback, E_TOT bit-identical to default; valgrind shows
no invalid accesses in ABACUS code (previously heap corruption and
SIGSEGV/SIGABRT)
- ecutexx=15: small grid (9,9,9) vs (15,15,15) engaged as before,
dE = 2.7e-5 eV from the intended pair-density truncation1 parent 0e1a2cf commit be015ef
20 files changed
Lines changed: 1799 additions & 223 deletions
File tree
- docs
- advanced/input_files
- source
- source_hamilt/module_xc
- source_io/module_parameter
- source_pw/module_pwdft
- kernels
- cuda
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
345 | 345 | | |
346 | 346 | | |
347 | 347 | | |
| 348 | + | |
348 | 349 | | |
349 | 350 | | |
350 | 351 | | |
| |||
3450 | 3451 | | |
3451 | 3452 | | |
3452 | 3453 | | |
3453 | | - | |
| 3454 | + | |
3454 | 3455 | | |
3455 | 3456 | | |
3456 | 3457 | | |
| 3458 | + | |
| 3459 | + | |
| 3460 | + | |
| 3461 | + | |
| 3462 | + | |
| 3463 | + | |
3457 | 3464 | | |
3458 | 3465 | | |
3459 | 3466 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5279 | 5279 | | |
5280 | 5280 | | |
5281 | 5281 | | |
5282 | | - | |
| 5282 | + | |
5283 | 5283 | | |
5284 | 5284 | | |
5285 | 5285 | | |
| 5286 | + | |
| 5287 | + | |
| 5288 | + | |
| 5289 | + | |
| 5290 | + | |
| 5291 | + | |
| 5292 | + | |
| 5293 | + | |
5286 | 5294 | | |
5287 | 5295 | | |
5288 | 5296 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
639 | 639 | | |
640 | 640 | | |
641 | 641 | | |
642 | | - | |
| 642 | + | |
| 643 | + | |
643 | 644 | | |
644 | 645 | | |
645 | 646 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
380 | 380 | | |
381 | 381 | | |
382 | 382 | | |
| 383 | + | |
| 384 | + | |
383 | 385 | | |
384 | 386 | | |
385 | 387 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
| 6 | + | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
| |||
177 | 178 | | |
178 | 179 | | |
179 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
180 | 195 | | |
181 | 196 | | |
182 | 197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
34 | 41 | | |
35 | 42 | | |
36 | 43 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
751 | 751 | | |
752 | 752 | | |
753 | 753 | | |
| 754 | + | |
754 | 755 | | |
755 | 756 | | |
756 | 757 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
897 | 897 | | |
898 | 898 | | |
899 | 899 | | |
900 | | - | |
| 900 | + | |
901 | 901 | | |
902 | 902 | | |
903 | 903 | | |
| |||
910 | 910 | | |
911 | 911 | | |
912 | 912 | | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
| 924 | + | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
| 929 | + | |
913 | 930 | | |
914 | 931 | | |
915 | 932 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
15 | 25 | | |
16 | 26 | | |
17 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
140 | 146 | | |
141 | 147 | | |
142 | 148 | | |
| |||
0 commit comments