Skip to content

Commit be015ef

Browse files
Feature(pw): batched FFTs, small ecutexx grid and band chunking for EXX on CPU and GPU (deepmodeling#8018)
* Feature(pw): batched FFTs and small ecut_exx grid for EXX on CPU and GPU Unify all EXX PW entry points (act_op, act_op_kpar, cal_exx_energy_op) on one code path built on shared primitives; the batched kernels are a device specialization of the per-band operations, selected inside the primitives (batch_active). - exx_batch kernels templated on Device: host loops + FFTW plan_many on CPU (single-precision plans compiled only with ENABLE_FLOAT_FFTW, stubs otherwise), CUDA kernels + cuFFT as DEVICE_GPU specializations. - QE ecutfock-style small FFT grid from ecutexx when every |k+G|^2 fits, on CPU and GPU; falls back to the full grid with a warning otherwise. The full-grid batched path runs on both devices when the box is local. - Physics fix: the Fock operator weight now uses the source-state occupation f_{mq} and the source k-point weight (was the target-k wg in act_op and target-k wk in act_op_kpar). With k-dependent occupations (smearing) the old operator was inconsistent with the energy and the EXX outer loop never converged. - The per-(q,m) scalar MPI_Bcast of wg is replaced by one broadcast of the occupation row + wk per source k-point via Parallel_Common wrappers. - stress_exx G-sum truncated to the ecutexx sphere (CPU), consistent with the operator and energy. - Docs: ecutexx describes the small-grid behavior and fallback. Governance exception: the PARAM/GlobalV budget flags are migration- neutral moves - the refactor rewrites existing blocks (act_op_kpar, cal_exx_energy_op, setup) that already read PARAM.inp.nspin/ecutexx and GlobalV::MY_POOL in the same style as the surrounding module. Verified against main (4f2a397): build/rel (g++ MPI) and build_abacus_gnu (CUDA 13.1); 19-case regression matrix (097_PW_PBE0{,_FM,_COND} x ACE/noACE x full/small grid x CPU/GPU) - full-grid cases match the pre-change code at 1e-14, CPU small grid matches GPU small grid to 1e-13, metallic noACE case converges in 3 EXX outer iterations (did not converge before the wg fix). * Feature(pw): decouple the EXX band chunking from the grid choice The small ecut_exx grid and band batching were bound in a single predicate; they are orthogonal concerns (grid = which FFT box, chunk = how many bands per batched round). Split them: - exx_grid_active() answers the grid question (small grid usable, or the full box local); exx_band_chunk() answers the band question via the new exx_batch_size INPUT (default 0 = all bands, identical to the previous behavior; a positive value processes bands in chunks of that width with a proportionally smaller work-buffer footprint). - cache_psi_nk_real, apply_exx_nbatched and the energy pair-density loop process bands in chunks; psi_nk_real_cache still holds all bands (it is the reuse floor across (iq, m)), only the work buffers shrink. - Docs: exx_batch_size in parameters.yaml and input-main.md. Governance exception: exx_band_chunk() reads PARAM.inp.exx_batch_size, the module's established style for INPUT values (same as ecutexx). Verified: 24-case regression (097_PW_PBE0{,_FM,_COND} x ACE/noACE x full/small grid x CPU/GPU, plus exx_batch_size 1/3/5 variants) - the default keeps every previous value bitwise, chunked runs are bitwise identical to unchunked (band blocks are disjoint and the (q,m) accumulation order is unchanged). * Refactor(pw): pass EXX PW configuration through General_Exx_Info instead of globals The EXX PW operator and stress read their configuration (nspin, ecutexx, exx_batch_size, exxace, exx_gamma_extrapolation) and the MPI layout (KPAR, MY_RANK, MY_POOL) directly from PARAM/GlobalV. Inject them explicitly instead: - General_Exx_Info carries the PW EXX INPUT values, resolved once in init_general_exx_info: exxace, gamma_extrapolation, exx_batch_size, and ecut_exx (ecutexx when set, else ecutrho) with the user-set flag. - OperatorEXXPW takes the General_Exx_Info plus the runtime values (nspin, kpar, my_rank, my_pool) at construction and snapshots them as members; all internal PARAM/GlobalV reads are gone. - Stress_PW::stress_exx takes the General_Exx_Info in place of the separate hybrid_alpha/coulomb_param parameters. Governance: the global-dependency budget of the PR goes from +12 to -28 (added=4, removed=32); the remaining 4 added references are the single snapshot point in HamiltPW, the orchestration layer that owns these values. Verified: build/rel (g++ MPI) rebuilds; tests/01_PW/097_PW_PBE0 matches result.ref (etot to 1.8e-14 eV, stress sum exact at 1e-6 kbar); exx_batch_size=1 is bitwise identical to unchunked; ecutexx=20 engages the small FFT grid at 1 rank and prints the distributed-box fallback warning once at 2 ranks, both converging to the same energy (2.4e-14 eV). * Docs: quote exx_batch_size default_value in parameters.yaml Match the --generate-parameters-yaml output exactly; the documentation consistency CI check requires docs/parameters.yaml to be byte-identical to the generated file. * Fix(build): wire EXX batch kernels into the legacy Makefile build The batched-EXX commits added kernels/exx_batch_op.cpp, kernels/exx_batch_op_float.cpp (ENABLE_FLOAT_FFTW) and its stub, plus kernels/cuda/exx_batch_op.cu, and wired them into CMake only. The Makefile build then failed at link time with undefined references to hamilt::exx_batch_* from the rewritten op_pw_exx.cpp. Add exx_batch_op.o and exx_batch_op_float_stub.o to OBJS_HAMILT; the legacy Makefile has no float-FFTW switch, matching the CMake ENABLE_FLOAT_FFTW=OFF branch. VPATH already covers the kernels dir. * Fix(pw): reject non-smaller EXX FFT grids, clamp ecutexx to ecutrho Addresses the Copilot review on deepmodeling#8018: setup_exx_small_grid only rejected a box exactly equal to the wavefunction box, so ecutexx > ecutrho (or FFT-friendly dimension rounding) enabled the small-grid path with sg_nxyz > wfcpw->nrxx, overflowing psi_mq_real/psi_nk_real. Compare box volumes against wfcpw->nrxx instead of dimensions. Also clamp a user-set ecutexx to ecutrho in init_general_exx_info: the pair density carries no G-components beyond the ecutrho sphere, while the EXX buffers (density_recip, pot, ...) and the rhopw_dev box are sized by ecutrho, so larger values only corrupt memory. This overflow predates this PR (same pattern on develop); the clamp turns it into a warning plus full-grid fallback. Docs: no update needed; the existing ecutexx description already documents the full-grid fallback for a non-smaller box. Verification (GNU+CUDA build, OMP_NUM_THREADS=1, tests/01_PW/097_PW_PBE0): - default: E_TOT -30.5936375896797834 eV, bit-identical to result.ref - ecutexx=100 (> ecutrho=40): ran clean with the clamp warning and full-grid fallback, E_TOT bit-identical to default; valgrind shows no invalid accesses in ABACUS code (previously heap corruption and SIGSEGV/SIGABRT) - ecutexx=15: small grid (9,9,9) vs (15,15,15) engaged as before, dE = 2.7e-5 eV from the intended pair-density truncation
1 parent 0e1a2cf commit be015ef

20 files changed

Lines changed: 1799 additions & 223 deletions

‎docs/advanced/input_files/input-main.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -345,6 +345,7 @@
345345
- [exxace](#exxace)
346346
- [exx\_gamma\_extrapolation](#exx_gamma_extrapolation)
347347
- [ecutexx](#ecutexx)
348+
- [exx\_batch\_size](#exx_batch_size)
348349
- [exx\_thr\_type](#exx_thr_type)
349350
- [exx\_ene\_thr](#exx_ene_thr)
350351
- [Molecular dynamics](#molecular-dynamics)
@@ -3450,10 +3451,16 @@
34503451
### ecutexx
34513452

34523453
- **Type**: Real
3453-
- **Description**: The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. Reducing ecutexx below ecutrho may significantly accelerate EXX computations. This speed improvement comes with a reduced numerical accuracy in the exchange energy calculation.
3454+
- **Description**: The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. The pair-density G-sphere of the exchange operator, the EXX energy, and the EXX stress are all truncated at this value. If ecutexx yields a smaller FFT box and every |k+G|^2 of the wavefunctions fits inside it (i.e. ecutexx should not be smaller than ecutwfc), all EXX FFTs run on that smaller grid (QE ecutfock-style), which can significantly accelerate EXX computations. If the small grid is not usable (box not smaller, wavefunctions do not fit, or the FFT box is distributed over MPI), a warning is printed and the full grid is used. Reducing ecutexx below ecutrho reduces the numerical accuracy of the exchange contribution.
34543455
- **Default**: same as ecutrho
34553456
- **Unit**: Ry
34563457

3458+
### exx_batch_size
3459+
3460+
- **Type**: Integer
3461+
- **Description**: Number of bands processed per round of the EXX batched FFT path. 0 (the default) processes all bands in one round, which is fastest but needs nbands * nxyz work buffers; a positive value processes the bands in chunks of that width, trading some performance for a proportionally smaller memory footprint. The result is independent of the chunking.
3462+
- **Default**: 0
3463+
34573464
### exx_thr_type
34583465

34593466
- **Type**: String

‎docs/parameters.yaml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5279,10 +5279,18 @@ parameters:
52795279
category: Exact Exchange (PW)
52805280
type: Real
52815281
description: |
5282-
The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. Reducing ecutexx below ecutrho may significantly accelerate EXX computations. This speed improvement comes with a reduced numerical accuracy in the exchange energy calculation.
5282+
The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. The pair-density G-sphere of the exchange operator, the EXX energy, and the EXX stress are all truncated at this value. If ecutexx yields a smaller FFT box and every |k+G|^2 of the wavefunctions fits inside it (i.e. ecutexx should not be smaller than ecutwfc), all EXX FFTs run on that smaller grid (QE ecutfock-style), which can significantly accelerate EXX computations. If the small grid is not usable (box not smaller, wavefunctions do not fit, or the FFT box is distributed over MPI), a warning is printed and the full grid is used. Reducing ecutexx below ecutrho reduces the numerical accuracy of the exchange contribution.
52835283
default_value: same as ecutrho
52845284
unit: Ry
52855285
availability: ""
5286+
- name: exx_batch_size
5287+
category: Exact Exchange (PW)
5288+
type: Integer
5289+
description: |
5290+
Number of bands processed per round of the EXX batched FFT path. 0 (the default) processes all bands in one round, which is fastest but needs nbands * nxyz work buffers; a positive value processes the bands in chunks of that width, trading some performance for a proportionally smaller memory footprint. The result is independent of the chunking.
5291+
default_value: "0"
5292+
unit: ""
5293+
availability: ""
52865294
- name: exx_thr_type
52875295
category: Exact Exchange (PW)
52885296
type: String

‎source/CMakeLists.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -639,7 +639,8 @@ if(USE_CUDA)
639639
source_pw/module_pwdft/kernels/cuda/cal_density_real_op.cu
640640
source_pw/module_pwdft/kernels/cuda/mul_potential_op.cu
641641
source_pw/module_pwdft/kernels/cuda/vec_mul_vec_complex.cu
642-
source_pw/module_pwdft/kernels/cuda/exx_cal_energy_op.cu)
642+
source_pw/module_pwdft/kernels/cuda/exx_cal_energy_op.cu
643+
source_pw/module_pwdft/kernels/cuda/exx_batch_op.cu)
643644
endif()
644645

645646
if(USE_ROCM)

‎source/Makefile.Objects‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -380,6 +380,8 @@ OBJS_HAMILT=hamilt_pw.o\
380380
exx_cal_energy_op.o\
381381
cal_density_real_op.o\
382382
mul_potential_op.o\
383+
exx_batch_op.o\
384+
exx_batch_op_float_stub.o\
383385

384386
OBJS_HAMILT_OF=kedf_tf.o\
385387
kedf_extwt.o\

‎source/source_hamilt/module_xc/exx_info.cpp‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
#include "source_io/module_parameter/input_parameter.h"
55
#include "source_base/global_function.h"
6+
#include "source_base/tool_quit.h"
67

78
#include <algorithm>
89
#include <cassert>
@@ -177,6 +178,20 @@ bool init_general_exx_info(General_Exx_Info& info, const Input_para& inp)
177178
info.separate_loop = inp.exx_separate_loop;
178179
info.hybrid_step = inp.exx_hybrid_step;
179180
info.mixing_beta_for_loop1 = inp.exx_mixing_beta;
181+
182+
// PW EXX operator configuration
183+
info.exxace = inp.exxace;
184+
info.gamma_extrapolation = inp.exx_gamma_extrapolation;
185+
info.ecutexx_user_set = inp.ecutexx > 0.0;
186+
info.ecut_exx = info.ecutexx_user_set ? inp.ecutexx : inp.ecutrho;
187+
if (info.ecutexx_user_set && info.ecut_exx > inp.ecutrho)
188+
{
189+
// no pair-density G-components exist beyond the ecutrho sphere,
190+
// and buffers sized by ecutrho would overflow; clamp instead
191+
ModuleBase::WARNING("General_Exx_Info", "ecutexx is larger than ecutrho; ecutexx is set to ecutrho");
192+
info.ecut_exx = inp.ecutrho;
193+
}
194+
info.exx_batch_size = inp.exx_batch_size;
180195
}
181196

182197
return generate_opt_orb;

‎source/source_hamilt/module_xc/general_exx_info.h‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,13 @@ struct General_Exx_Info
3131

3232
bool separate_loop = true;
3333
size_t hybrid_step = 1;
34+
35+
// PW EXX operator configuration:
36+
bool exxace = false; // ACE acceleration of the EXX operator (INPUT exxace)
37+
bool gamma_extrapolation = true; // INPUT exx_gamma_extrapolation
38+
double ecut_exx = 0.0; // resolved EXX cutoff (Ry): ecutexx when set, else ecutrho
39+
bool ecutexx_user_set = false; // ecutexx explicitly set (controls small-grid fallback warnings)
40+
int exx_batch_size = 0; // band chunk width of the batched EXX path; 0 = all bands
3441
};
3542

3643
/// Forward declaration for Input_para (full definition in input_parameter.h)

‎source/source_io/module_parameter/input_parameter.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -751,6 +751,7 @@ struct Input_para
751751
std::string exx_thr_type = "density"; ///< threshold type for exx outer loop
752752
double exx_ene_thr = 1e-5; ///< threshold when exx_thr_type = energy
753753
double ecutexx = 0.0; ///< energy cutoff for exx calculation, Ry
754+
int exx_batch_size = 0; ///< band chunk width of the EXX batched path, 0 = all bands
754755

755756
// ==== #Parameters (23.XC external parameterization) ========
756757
/*

‎source/source_io/module_parameter/read_inp_other.cpp‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -897,7 +897,7 @@ When false (default), both the direction and magnitude of the magnetic moment ar
897897
item.annotation = "energy cutoff for exx calculation, Ry";
898898
item.category = "Exact Exchange (PW)";
899899
item.type = "Real";
900-
item.description = "The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. Reducing ecutexx below ecutrho may significantly accelerate EXX computations. This speed improvement comes with a reduced numerical accuracy in the exchange energy calculation.";
900+
item.description = "The energy cutoff for EXX (Fock) exchange operator in plane wave basis calculations. The pair-density G-sphere of the exchange operator, the EXX energy, and the EXX stress are all truncated at this value. If ecutexx yields a smaller FFT box and every |k+G|^2 of the wavefunctions fits inside it (i.e. ecutexx should not be smaller than ecutwfc), all EXX FFTs run on that smaller grid (QE ecutfock-style), which can significantly accelerate EXX computations. If the small grid is not usable (box not smaller, wavefunctions do not fit, or the FFT box is distributed over MPI), a warning is printed and the full grid is used. Reducing ecutexx below ecutrho reduces the numerical accuracy of the exchange contribution.";
901901
item.default_value = "same as ecutrho";
902902
item.unit = "Ry";
903903
read_sync_double(input.ecutexx);
@@ -910,6 +910,23 @@ When false (default), both the direction and magnitude of the magnetic moment ar
910910
this->add_item(item);
911911
}
912912

913+
{
914+
Input_Item item("exx_batch_size");
915+
item.annotation = "band chunk width of the EXX batched path, 0 = all bands";
916+
item.category = "Exact Exchange (PW)";
917+
item.type = "Integer";
918+
item.description = "Number of bands processed per round of the EXX batched FFT path. 0 (the default) processes all bands in one round, which is fastest but needs nbands * nxyz work buffers; a positive value processes the bands in chunks of that width, trading some performance for a proportionally smaller memory footprint. The result is independent of the chunking.";
919+
item.default_value = "0";
920+
read_sync_int(input.exx_batch_size);
921+
item.check_value = [](const Input_Item& item, const Parameter& para) {
922+
if (para.input.exx_batch_size < 0)
923+
{
924+
ModuleBase::WARNING_QUIT("ReadInput", "exx_batch_size must >= 0");
925+
}
926+
};
927+
this->add_item(item);
928+
}
929+
913930
{
914931
Input_Item item("exx_thr_type");
915932
item.annotation = "threshold type for exx outer loop, energy or density";

‎source/source_pw/module_pwdft/CMakeLists.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,16 @@ list(APPEND objects
1212
op_pw_exx.cpp
1313
op_pw_exx_ace.cpp
1414
op_pw_exx_pot.cpp
15+
kernels/exx_batch_op.cpp
16+
)
17+
18+
if(ENABLE_FLOAT_FFTW)
19+
list(APPEND objects kernels/exx_batch_op_float.cpp)
20+
else()
21+
list(APPEND objects kernels/exx_batch_op_float_stub.cpp)
22+
endif()
23+
24+
list(APPEND objects
1525
dftu_base.cpp
1626
dftu_base_io.cpp
1727
dftu_pw.cpp

‎source/source_pw/module_pwdft/hamilt_pw.cpp‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -133,10 +133,16 @@ HamiltPW<T, Device>::HamiltPW(elecstate::Potential* pot_in,
133133
}
134134
if (exx_info && exx_info->cal_exx)
135135
{
136-
bool separate_loop = exx_info->separate_loop;
137-
double hybrid_alpha = exx_info->hybrid_alpha;
138-
auto coulomb_param = exx_info->coulomb_param;
139-
auto exx = new OperatorEXXPW<T, Device>(isk, wfc_basis, pot_in->get_rho_basis(), pkv, ucell, separate_loop, hybrid_alpha, coulomb_param);
136+
auto exx = new OperatorEXXPW<T, Device>(isk,
137+
wfc_basis,
138+
pot_in->get_rho_basis(),
139+
pkv,
140+
ucell,
141+
*exx_info,
142+
PARAM.inp.nspin,
143+
GlobalV::KPAR,
144+
GlobalV::MY_RANK,
145+
GlobalV::MY_POOL);
140146
if (this->ops == nullptr)
141147
{
142148
this->ops = exx;

0 commit comments

Comments
 (0)