Skip to content

fix: resolve the #285 follow-ups (DOS labels, autobandsplot report, BS2D grid, overlay weights, Elk lmirep) - #304

Merged
lllangWV merged 40 commits into
devfrom
285-small-fixes
Oct 4, 2026
Merged

lllangWV merged 40 commits into
devfrom
285-small-fixes

Conversation

@lllangWV

@lllangWV lllangWV commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Why

Issue #285 collects small correctness and cleanup follow-ups. This PR fixes every item that is still open. Each fix has a test that fails on dev and passes here.

What changed

  • DOS labels. A vertical DOS plot, which is what bandsdosplot draws, labelled both spin channels "Total - up" and did not mirror the down channel. DOSPlotter.plot swapped energies and values before it split the channels (plotter/dos_plot.py:154). It now splits the channels first and then orients each one.
  • autobandsplot report. autobandsplot wrote report.txt into the current directory on every call (scriptAutoBandsplot.py:87). It now writes the report only to the path you pass as report=, and writes no file by default. A default inside the calculation directory was rejected, because fix(ebs): stop parsers writing caches into calculation directories #245 stopped pyprocar writing into input directories. The report also prints plain numbers again, so it matches the report.txt shipped with the bands/auto fixture.
  • Legacy PROCAR k-points. io/procarparser.py:153 split k-point headers on whitespace, so a header such as 0.50000000-0.25000000 raised. It now uses the per-coordinate regex that fix(io): resolve the #279 parser follow-ups for Abinit, QE, VASP, Elk and Siesta #280 added to io/vasp/procar.py.
  • Graphene BS2D. The surfaces were jagged for two pyprocar reasons:
    • The resampling grid covered the Cartesian bounding box of graphene's rhombic k-patch. About half of the grid was outside the data, so those points were NaN (core/bandstructure2D.py:148). This held in both modes: as_cartesian=False, which from_code and the legacy handler use, and as_cartesian=True, the from_ebs default. The grid is now regular in the two fractional coordinates that span the plane, in both modes. For a Cartesian plane the fractional normal is reciprocal_lattice @ normal. The grid edges are pulled in by 1e-12 of the span, because a grid corner that equals a data corner up to rounding can fall just outside the triangulation and come back NaN (BiSb lost its corner point on every sheet).
    • extract_surface() renumbered the surface points while every property stayed in grid order, so colours were drawn on other points. A non-square grid crashed the interpreter. The surface is now built from quads in grid order.
    • The ripples left on the upper sheet are real crossings with the sigma band and the 2.2 eV band, because bands are indexed by sorted energy.
  • scripts/procar.py. filter read args.infile, args.outfile and args.human_atoms, which its parser never defines, so every call raised. cat and fermi2D already worked.
  • SACATTER. The member is renamed to SCATTER in UnfoldPlotMode and in both BandStructureMode enums. SACATTER stays in each enum as a deprecated alias of the same value, because the rendered band_structure config page named it and unfold(mode=UnfoldPlotMode.SACATTER) worked on dev. Iterating an enum lists only SCATTER.
  • CHANGELOG. An Unreleased section records the two user-visible changes: autobandsplot writes no report.txt unless report= names a path, and the SCATTER spelling with its alias.
  • cfg docs. The dos, unfold and band_structure_2d pages were generated from yml files that no code reads, and their values had drifted (unfold showed color: black, not #eeeeee). These pages now use automodule, like the band-structure and Fermi-surface-3D pages. The yml files and their generator are removed, and the config docstrings now state the real defaults. The index lists the config modules.
  • Notebooks. The unfolding notebook kept plots drawn before fix(unfold): make pyprocar.unfold draw correct unfolded bands #282 fixed pyprocar.unfold. Getting Started and Comparing Bands kept option dumps that list weighted_color and weighted_width. Those outputs are cleared rather than re-run, so the PR adds no images.
  • Overlay weights (deferred until fix(core): check projection axes against bands and structure, and give every parser its orbital names #298). build_overlay_orbitals_weights and build_overlay_weights called .get on a list and raised AttributeError (core/ebs.py:1467,1527). They now resolve shells with the same present_shells and orbital_indices that bandsplot uses. Those two functions move into core/atomic_orbital_index.py, beside orbital_shells from fix(core): check projection axes against bands and structure, and give every parser its orbital names #298.
  • Elk lmirep (deferred until fix(core): check projection axes against bands and structure, and give every parser its orbital names #298). Elk 10.7.8 and later write task 22's band characters in each site's irreducible-representation basis unless lmirep is .false.. The l blocks stay, but the columns inside each l are no longer Y_lm. The parser now reads the version from INFO.OUT and lmirep from elk.in, and names those columns Y{l}_ir{i} in ELMIREP.OUT order. Shell sums still work. It reads both header forms: Elk 6 writes | Elk version 6.3.02 started | and Elk 11 writes Elk code version 11.2.3 (writeinfo.f90). Without a version the columns keep their Y{l}{m} names. ELMIREP.OUT alone does not decide it, because task 10 writes it too (with lmirep on, the default), and the usual workflow leaves a task-10 ELMIREP.OUT beside the bands. When ELMIREP.OUT is present and no version is found, the parser warns and says whether INFO.OUT is missing or has no version line. This rule never renames columns on a guess, so a version-less directory reads exactly as on dev.

Scope

Closes #285. Everything else stays out of scope. The notes below are defects I found but did not fix.

Blast Radius

The orbital resolvers move from scripts/_selection.py to core, and pyprocar.utils.info.orbital_names is removed (it had no other user). The BS2D change affects the grid of BandStructure2D in both modes and every surface it builds. Elk naming changes only for task-22 runs in the irreducible-representation basis.

Verification

Round 2 (fixes for the round-1 verdicts) is in "Round 2" below. The round-1 runs follow it.

Round 2

  • Elk 6.3 header (blocking finding). Repro from the verdict: a private copy of data/codes/elk/6.3/SrVO3/spin-polarized-colinear/bands plus ../dos/ELMIREP.OUT, read with ElectronicBandStructurePath.from_code("elk", dir).orbital_names[:9].

    • round-1 head 2fd6c01: ['Y0_ir1', 'Y1_ir1', 'Y1_ir2', ...].
    • this head: ['Y00', 'Y1-1', 'Y10', 'Y11', 'Y2-2', 'Y2-1', 'Y20', 'Y21', 'Y22'], with no warning. The same copy without INFO.OUT keeps those names and warns.
    • Tests, red at 2fd6c01 and green here: test_task_22_characters_in_the_ylm_basis_keep_their_l_m_names[6.3-with-task-10-elmirep] (the real 6.3 header, CRLF included), test_task_22_basis_that_cannot_be_told_apart_keeps_l_m_names_and_warns[no-info, info-without-version], and the data test test_real_elk_6_3_bands_beside_a_task_10_elmirep_keep_their_l_m_names (the repro above). The irrep tests now write the box header Elk 11.2.3 writes.
    • Mutants, both killed: the old Elk code version regex (2 tests fail), and the old "ELMIREP.OUT without task 10 means irrep" fallback (2 tests fail).
  • SCATTER alias. test_scatter_mode_is_spelled_scatter_and_the_old_spelling_still_resolves checks mode_enum["SACATTER"] is mode_enum.SCATTER and mode_enum(mode_enum.SACATTER) is mode_enum.SCATTER for all three enums. It failed at 2fd6c01 with KeyError: 'SACATTER' and fails on dev because SCATTER is missing.

  • BS2D Cartesian grid and BiSb corner. NaN points from BandStructure2D.from_ebs on private fixture copies (padding 10):

    Case 2fd6c01 this head
    graphene as_cartesian=True 30x30 960/1800 0/1800
    graphene as_cartesian=True 40x25 1056/2000 0/2000
    BiSb as_cartesian=True 30x30 2196/5400 0/5400
    BiSb as_cartesian=False 30x30 6/5400 (corner (0,0), every sheet) 0/5400
    graphene as_cartesian=True default 20x20, padding 15 440/800 0/800
    BiSb as_cartesian=True default 20x20, padding 15 1056/2400 0/2400
    • Independent reference: a periodic bilinear interpolation of the DFT bands on the 60x60 k-grid, evaluated at each surface point's own (x, y)/2 pi. On an 80x80 grid, whose vertices are DFT k-points, the surface heights match it to 1.1e-11 eV (graphene) and 3.3e-7 eV (BiSb, whose lattice has 1e-7 z components) in both modes. On 30x30 and 40x25 grids the difference is the gap between two linear interpolants (p99 at most 0.012 eV).
    • Projected area of each sheet's quads equals (79/60)^2 |b1 x b2| to 4e-12 relative for graphene.
    • Since no point is NaN any more, the NaN x and y that the 2 pi scale gave NaN-height points no longer occur on these fixtures. A plane oblique to every lattice axis can still leave NaN corners, and those points still get NaN x and y.
    • Tests, red at 2fd6c01 and green here: test_hexagonal_bs2d_surface_covers_the_k_patch_with_the_analytic_bands[cartesian], test_bs2d_on_a_lattice_slightly_off_square_leaves_no_grid_point_empty[cartesian, fractional] (BiSb's reciprocal lattice, padding 10), test_cartesian_bs2d_plane_across_the_kz_layers_covers_its_patch (a* tilted out of kz = 0, so a Cartesian kz plane crosses the fractional layers).
  • BS2D mutants that survived round 1, now killed. Each was run one at a time against tests/pyprocar/core/test_bandstructure2d_grid.py:

    • plane level dropped (the round-1 offset, now level = 0.0): killed by test_bs2d_plane_off_the_origin_on_a_skewed_lattice_has_the_bands_of_that_plane (kz = 0.25 on a lattice whose c* has x and y parts, heights against the analytic band to 1e-9) and one more test.
    • quads indexed in the wrong order (reshape(shape[::-1]).T): killed by test_bs2d_on_a_non_square_grid_builds_every_point, which now also checks 29x19 quads whose projected area equals the patch area.
    • surfaces with no faces: killed by 5 tests through the same area check.
    • Also killed: no edge inset (2 tests), the Cartesian normal used as fractional (1 test), the Cartesian bounding box restored (2 tests), kz not solved from the plane (1 test).
  • Before and after, graphene with the from_ebs default as_cartesian=True at 20x20, iso and top views. The holes and the stair-step edges are gone:

graphene cartesian 20x20
graphene cartesian 20x20 top

The same at 80x80, where the Dirac cones are resolved:

graphene cartesian 80x80

BiSb, as_cartesian=True, 20x20:

bisb cartesian 20x20

The 20x20 default still looks spiky near the cones. Every vertex is right; a 20-point grid over 79 k-steps samples the cone tips coarsely.

  • Gates on this head: pytest -m "not data" 2056 passed, 2 skipped. -m data 406 passed, 1 skipped (405 at 2fd6c01; the new one is the Elk 6.3 data test). basedpyright 0 errors, 0 warnings. Both ruff_new_violations checks and the F821 check pass. red_green.py origin/dev passes.

Round 1

  • python .github/scripts/red_green.py origin/dev: 19 added tests. 17 fail on dev for the reason they name. 2 guard existing behaviour (cat, Ylm-basis Elk names). The 2 autobandsplot data tests fail on dev, run by hand: report.txt lands in the cwd, and report= is a TypeError.
  • Gates on the head: pytest -m "not data" passes (2049 passed), the data tests of core, io, plotter, scripts and cfg pass (405 here, 403 on dev, no new failure), basedpyright passes (0 errors, 8 stale baseline entries removed in their own commit), both ruff_new_violations checks pass, and the F821 check passes.
  • Live runs through verify.sh run:
    • bandsdosplot on SrVO3 spin-polarized: legend up/up becomes up/down, with the down channel mirrored.
    • Graphene BandStructure2D.from_code: at 20x20, 440 of 800 points were NaN and are now 0. At 90x90, 8280 of 16200 were NaN and are now 0. The apparent Dirac gap at 90x90 goes from 0.267 eV to 2.511e-5 eV, which is the EIGENVAL value at K.
    • procar.py filter and cat on a fixture copy, and procar.py fermi2D on fermi2d/non-spin-polarized.
    • The legacy parser reads issues/issue-199/PROCAR (80 k-points, k41 = [0.50877193, -0.01754386, 0]).
    • Overlay weights on SrVO3 equal compute_projected_sum over the same orbitals.

bandsdosplot, before and after:

bandsdos before
bandsdos after

Graphene BS2D at 90x90, before and after:

bs2d before
bs2d after

Not verified

  • The Sphinx build of the three automodule pages was not run locally, because the dev environment has no Sphinx. They use the same directive as the band-structure page.
  • No Elk 10.7.8+ fixture exists. The irrep naming is tested on synthetic directories and checked against the Elk 11.2.3 source (bandstr.f90, genlmirep.f90) and its release notes.

DOSPlotter.plot swapped energies and values before it split the spin
channels, so in vertical orientation (bandsdosplot) every channel got the
first channel's label and the down channel was not mirrored.
autobandsplot wrote report.txt into the current working directory on
every call. The report now goes to the path the caller passes as
report=, and no file is written by default. Energies and band indexes
print as plain numbers again, so the text matches the reference report
shipped with the bands/auto fixture.
The legacy ProcarParser split k-point headers on whitespace, so a header
such as 0.50000000-0.25000000 0.00000000 gave two values and readFile
raised. It now uses the per-coordinate regex that #280 put in
io/vasp/procar.py.
UnfoldPlotMode and both BandStructureMode enums spelled the scatter
member SACATTER. Users select the mode by its value "scatter", which
does not change, and no public namespace or doc names the member, so no
alias is kept.
The cfg index listed yml files that do not exist, and the dos, unfold
and band-structure-2D pages were generated from yml files that no code
reads and that had drifted from the config defaults (unfold showed
color black instead of #eeeeee). Those pages now use automodule, like
the band-structure and Fermi-surface-3D pages. The yml files and their
generator are removed, and the config docstrings state the real
defaults.
procar.py filter read args.infile, args.outfile and args.human_atoms,
which its parser never defines (it defines inFile, outFile and --human),
so every filter call raised AttributeError.
The unfolding notebook kept plots drawn before #282 fixed pyprocar.unfold,
and the Getting Started and Comparing Bands notebooks kept option dumps
that list the removed weighted_color and weighted_width. Those outputs
are cleared rather than re-run, so no new images enter the repo.
…heir points

With as_cartesian=False the k-mesh slice is a parallelogram in Cartesian
space, and the uv grid spanned its bounding box. For graphene half of the
grid fell outside the data, so those points were NaN and the surface edge
became a sawtooth. The grid is now regular in fractional coordinates and
mapped to Cartesian.

The band surfaces went through cast_to_unstructured_grid().extract_surface(),
which renumbers points, while every property stayed in grid order, so
colours were drawn on other points. The surface is now built from quads
in grid order, which also fixes the dimensions of a non-square grid.
build_overlay_orbitals_weights and build_overlay_weights called .get on
pyprocar.utils.info.orbital_names, a list, so both raised AttributeError.
They now take the shells from orbital_shells(self.orbital_names), the
definition #298 added, so a PROCAR with only s and p orbitals gives s and p,
and a shell letter the orbitals do not hold raises a ValueError.
…asis

Elk 10.7.8 and later write task 22's (l,m) band characters in each site's
irreducible-representation basis unless lmirep is .false., so the Y{l}{m}
names were wrong for those runs. The parser reads the version from
INFO.OUT and lmirep from elk.in, and names those columns Y{l}_ir{i}, the
i-th basis function of l in ELMIREP.OUT order; orbital_shells still finds
each shell. Without INFO.OUT, ELMIREP.OUT from a task-22-only run marks the
basis, and a run that also lists task 10 warns.
…erlay weights

present_shells and orbital_indices move from scripts/_selection.py to
core/atomic_orbital_index.py beside orbital_shells. The overlay weight
builders call them, so they accept the same shell names as
bandsplot(orbitals=[...]), including a single column named by its letter.
The fixed orbital table in utils/info.py, whose last user was the broken
overlay code, is removed.
@lllangWV lllangWV changed the title fix: #285 small correctness and cleanup follow-ups fix: resolve the #285 follow-ups (DOS labels, autobandsplot report, BS2D grid, overlay weights, Elk lmirep) Oct 4, 2026
@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: FAIL

Reviewed SHA: 2fd6c01 (base origin/dev 4de5cc0, which is the merge base, so no trunk merge was needed). Lane: audit and regression. Independent verifier.

Blocking finding

The Elk version check misses the INFO.OUT header that older Elk writes, so an Elk 6.3 run gets irrep names it should not get. pyprocar/io/elk/parser.py:22 matches only Elk code version x.y.z. Elk 6.3, the version of the repo's own fixture, writes | Elk version 6.3.02 started |. That header does not match, so the parser takes the fallback that is meant for a directory with no INFO.OUT: an ELMIREP.OUT file plus an elk.in without task 10 counts as the irrep basis. The usual Elk workflow runs the DOS (task 10) and the bands (task 22) in the same directory. That directory then holds the ELMIREP.OUT from task 10, and elk.in lists only 0, 22.

  • Smallest repro: copy data/codes/elk/6.3/SrVO3/spin-polarized-colinear/bands into a scratch directory and add ../dos/ELMIREP.OUT. Then call ElectronicBandStructurePath.from_code("elk", dir).orbital_names[:9].
    • origin/dev: ['Y00','Y1-1','Y10','Y11','Y2-2',...], which is correct because Elk 6.3 always writes Ylm.
    • head: ['Y0_ir1','Y1_ir1','Y1_ir2','Y1_ir3','Y2_ir1',...], which is wrong.
  • The same copy without ELMIREP.OUT keeps the Ylm names at head.
  • The new tests write only the synthetic Elk code version ... line, so they cannot catch this.
  • Fix I checked: re.compile(r"Elk (?:code )?version (\d+)\.(\d+)\.(\d+)"). With it the repro gives the Ylm names, and the 8 task_22 tests in tests/pyprocar/io/elk/test_parser.py still pass. Add a test that uses the real 6.3 header.
  • The warning text "no INFO.OUT with the Elk version" is also wrong in this case, because the file exists.

What I ran

  • Full -m data suite with -n 4:
    • head: 405 passed, 1 skipped, exit 0.
    • origin/dev: 403 passed, 1 skipped, exit 0.
    • The 2 extra tests at head are the new autobandsplot data tests. No new failure.
  • gh pr checks 304: format, lint, red-green, test and typecheck all pass.
  • BS2D regression, origin/dev against head, through BandStructure2D.from_code on private copies of 2d-bands/graphene and 2d-bands/bisb_monolayer. I ran both as_cartesian modes at 30x30, 60x60 (graphene only) and 40x25, with padding 10. Independent reference: griddata of the padded EBS bands and projected_sum_spin_texture at each surface point's own (x, y)/2π.
    • Fractional mode, NaN points:

      Fixture, grid origin/dev head
      graphene 30x30 960 of 1800 0
      graphene 60x60 3720 of 7200 0
      bisb 30x30 2196 of 5400 6
    • Band values against z: on origin/dev, 16, 16 and 210 points have get_property('bands') != z. On head the count is 0 in every case. Surface z against the reference has a maximum error of about 4e-15 in fractional mode.

    • Spin texture (bisb, non-collinear): the maximum error against the reference falls from 0.578 to 2.3e-15. Shape is (5400, 3) in both.

    • Cartesian mode: after sorting, the point sets and NaN counts are identical between origin/dev and head (maximum difference 0.0). Only the order changed, and values now sit on their own points.

    • Non-square 40x25 grid: origin/dev segfaults (exit 139) for graphene frac, graphene cart and bisb frac. Head builds 2000 and 6000 points, with 0 values off their points.

  • Live public-API checks, origin/dev against head:
    • build_overlay_orbitals_weights(atoms=[2,3,4]) on SrVO3: origin/dev raises AttributeError. Head gives s 473.483, p 1153.606, d 0.0, which equals compute_projected_sum over the same orbitals.
    • build_overlay_weights({'V':['d']}) gives 875.557, equal to the reference.
    • Vertical spin-polarized dosplot on dos/spin-polarized: the labels change from ↑/↑ to ↑/↓, and the down channel now spans [-29.77, 0].
    • The legacy ProcarParser._readKpoints on fused headers such as 0.50000000-0.25000000: origin/dev raises ValueError. Head returns [[0,0,0],[0.5,-0.25,0],[-0.5,-0.25,-0.125]].
    • None of these runs wrote a file into the cwd.

Audit

  • Removed and renamed names: I grepped pyprocar/, scripts/, examples/ (including notebooks), docs/, tests/, .claude/, .github/ and the packaging files for each of these:

    • SACATTER
    • pyprocar.utils.info.{orbital_names, orbital_index_name_map, OrbitalOrdering}
    • the moved present_shells, orbital_indices and SHELL_LETTERS
    • generate_yml_docs and the three deleted yml files

    No caller is left. The remaining hits are the new test's not in __members__ assertion and an old CHANGELOG line. utils/config.py still reads package.yml, which is kept. setup.py keeps *.yml package data.

  • Baseline: 8 entries removed and none added. Two are the .get-on-list errors in ebs.py. Six are in scriptAutoBandsplot.py, from the fermi: int = None typing and its comparisons. All are tied to rewritten code.

  • Acceptance:

    • Overlay weights: done.
    • DOS labels: done.
    • autobandsplot report: done (data tests pass).
    • Legacy PROCAR regex: done.
    • Graphene BS2D: done.
    • procar.py filter: done, by the PR's CLI test.
    • SACATTER: done.
    • cfg docs: done. Not built with Sphinx.
    • Notebook outputs: done for the listed notebooks.
    • Elk lmirep: claimed but partly false, because older-version runs do not always keep their names (blocking finding above).

Notes (non-blocking)

  • SCATTER without an alias: the PR body says "no doc names the member". That is not quite true: docs/source/api/cfg/band_structure.rst was already an automodule with :undoc-members:, so BandStructureMode.SACATTER was rendered. pyprocar.unfold(mode=UnfoldPlotMode.SACATTER) also worked on dev, because UnfoldPlotMode(member) accepts a member. The risk is low because users pass strings. A one-line Enum alias (SACATTER = "scatter") would keep that code working for free.
  • report=None: a user who read report.txt from the cwd now gets no file and no message. This is documented in the docstring and the 04 notebook, but the next release notes should mention it. write_report's positional order also changed to (filename, verbosity). Only the module calls it.
  • BS2D fractional mode on bisb: grid corner (0,0) is NaN on all 6 sheets, probably a floating-point boundary miss in LinearNDInterpolator. The PR's "0 NaN" claim covers graphene only.
  • BS2D cartesian mode, before this PR and still now: every point with a NaN z also gets NaN x and y, because the 2π scale transform computes 0*NaN.
  • The unfolding notebook's unfold cells now have no outputs. nbsphinx runs in auto mode and the notebook still has outputs in cells 2 and 4, so the docs page will show those cells without figures until the notebook is re-run.
  • Elk DOS (task 10) columns are in the irrep basis whenever lmirep is on, but they keep their Ylm slot names. The feature map documents this, and it is out of scope here.
  • Small wording point: task 10 writes ELMIREP.OUT only when lmirep is on (dos.f90:80), not "in every version" unconditionally.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: PASS+NOTES

Reviewed SHA: 2fd6c01. The head contains origin/dev 4de5cc0, so no merge was needed.
Lane: live runs, red/green and gates (independent verifier).

Gates (run at the head, verify.sh exec, -n 4)

  • pytest tests -m "not data": exit 0, 2049 passed, 2 skipped.
  • pytest tests -m data: exit 0 at both commits. Head: 405 passed, 1 skipped. origin/dev: 403 passed, 1 skipped. The 2 extra tests are the new autobandsplot data tests. No new failure.
  • basedpyright --project pyrightconfig.json: exit 0, with 0 errors and 0 warnings.
  • ruff_new_violations.py check origin/dev and format origin/dev: exit 0 (31 changed files, ok). The F821/F822/F823/NPY201 check: exit 0.
  • red_green.py origin/dev: exit 0. 17 of the 19 added tests fail on dev for their stated reason. 2 guard existing behaviour. The 2 autobandsplot data tests are covered by the live runs below.
  • gh pr checks: format, lint, red-green, test and typecheck all pass on 2fd6c01.

Live runs (private fixture copies, origin/dev vs head, compared against independent expected values)

  • DOS labels and mirroring. Reference: total DOS read from dos/spin-polarized/DOSCAR columns 2 and 3 (6000 points).

    • Vertical dosplot, dev: legend reads up, up. The second line equals +down (max diff 0.0048, which is the vasprun vs DOSCAR rounding).
    • Vertical dosplot, head: legend reads up, down. The second line equals -down (max diff 0.0048).
    • The bandsdosplot DOS panel gives the same result: dev up/up, unmirrored; head up/down, mirrored.
    • Horizontal dosplot was already correct on dev and is unchanged.
  • autobandsplot. The calc dir was snapshotted, and a separate empty cwd was used.

    • dev: writes report.txt into the cwd. report= raises TypeError.
    • head default: writes nothing to the cwd or the calc dir.
    • head with report=path: writes only that path. With fermi=3.0667 (OUTCAR E-fermi) the text equals the fixture's shipped report.txt byte for byte. With 3.06668137 the energy window moves by about 2e-5.
  • Legacy ProcarParser on issues/issue-199/PROCAR. Reference: my own fixed-width (3x11-char) parse of all 80 k-point headers.

    • dev: raises "Badly formated Kpoints headers".
    • head: 80x3 k-points, max |parsed - reference| = 0. File k-point 41 is (0.5, 0, 0) and file k-point 42 is (0.50877193, -0.01754386, 0). The PR body's "k41" uses the 0-based index.
    • spd shape (80, 72, 4, 6, 11).
  • Graphene BandStructure2D.from_code. Reference: the 331 IBZ k-points of EIGENVAL and PROCAR, unfolded myself onto the 60x60 mesh with the 12 hexagonal operations. Every operation is integer in fractional coordinates, coverage is complete, and the 3600 points have no conflicting energies.

    config NaN points, dev NaN points, head
    frac 20x20 440/800 0/800
    frac 90x90 8280/16200 0/16200
    • Head grid steps in fractional coordinates are [1/60, 0] and [0, 1/60]: aligned with b1 and b2, with a range of [-44/60, 45/60], exactly the padded patch. The dev grid steps are diagonal, with a range of +-1.48.
    • Head at 90x90: all 2x8100 surface points sit on DFT k-points. |z - EIGENVAL| <= 5.2e-7 on bands 4 and 5.
    • Surface colours (get_property("projected")) at those points equal my PROCAR sums over both ions to <= 2e-14, for s and for pz. get_property("bands") equals points[:,2] exactly.
    • Dirac gap at K = (1/3, 1/3): the surface gives 2.511e-5 eV, EIGENVAL gives 2.50e-5 eV. dev gives 0.267 eV at 90x90 and 1.22 eV at 20x20.
    • as_cartesian=True is unchanged from dev: 440/800 and 8280/16200 NaN. Finite points agree with my bilinear reference to the interpolation error (p99 0.07 eV).
    • Skewed-lattice check, which reaches beyond the PR's tests: a synthetic 16x16x8 mesh with c* having x and y components, an analytic band, planes kz = 0 and 0.25, and grids 40x30 and 30x30. Head: 0 NaN, error within the linear-interpolation bound (max 0.055), face count (nu-1)(nv-1), and every quad on grid neighbours. dev on the 40x30 grid: 1521 faces instead of 1131, garbage indices, then a segfault (exit 139).
    • Rendered legacy handler at its default 120x120 grid, at both commits: both look smooth. The 20x20 driver image is still coarse because of the sampling.
  • scripts/procar.py. All subcommands ran for real inside bwrap, with only a fake root writable. Input modes and sizes were unchanged afterwards.

    • dev: all 4 filter variants raise AttributeError: 'Namespace' object has no attribute 'infile'.
    • head filter -a 0 1 -a 2 3 4 and the --human variant: every block equals my sums of the input ions, with tot kept.
    • head -o 0 -o 1 2 3 -o 4..8 --orbital_names s p d: exact.
    • head -b 10 20: 2200 blocks, bands 10-20.
    • cat and cat --gz: 400 k-points, both halves identical to the input.
    • fermi2D plain and parametric: exit 0, and the PNGs look correct.
    • bandsplot, repair, kpath and unfold: exit 0.
  • Overlay weights on SrVO3. Reference: my PROCAR parse, which equals ebs.projected exactly.

    • dev: both functions raise AttributeError: 'list' object has no attribute 'get'.
    • head build_overlay_orbitals_weights() s/p/d: equal to compute_projected_sum (diff 0) and to my sums (<= 2e-16).
    • head build_overlay_weights({"V":["d"],"O":["p"],"Sr":[0]}): equal to the per-species sums.
    • {"V":["f"]} and {"V":["dxy"]} raise a clear ValueError.
    • The bands.md numbers (O s 473.483, p 1153.606, d 0.0) reproduce.
  • Elk lmirep. I ran the code itself in place of a synthetic directory: Elk 11.2.3, Si with Td sites, tasks 0 and 22. One run used default lmirep, a second used lmirep=.false..

    • Task 22 writes ELMIREP.OUT only in the irrep run, in the documented format.
    • Head names the irrep run Y{l}_ir{i} and the Ylm run Y{l}{m}. dev names both Y{l}{m}.
    • Per-column characters differ between the two runs by up to 0.148, while per-l sums agree to <= 3e-6, the 6-decimal print. So shell sums are right and the Ylm names were wrong.
    • At Gamma, the valence triplet's d weight sits in Y2_ir1..3 (ELMIREP 288 x3, t2), with ir4..5 (528 x2, e) at 0, which confirms the ELMIREP order.
    • Without INFO.OUT the run still counts as irrep. With task 10 added it warns and keeps Ylm.
    • Head overlay shells s/p/d/f agree between the two runs to <= 6e-6. On dev, overlay raises.
    • The parser writes nothing into the calc dir.
  • SCATTER. bandsplot(mode="scatter") and unfold(mode="scatter") run at both commits. No SACATTER is left in the repo.

  • cfg docs and notebooks. No references to the removed yml files remain in docs, the code or make.bat. The index lists the config modules. No notebook output names an option missing from its config: I scanned every examples/**/*.ipynb against BandStructureConfig and DensityOfStatesConfig. The Sphinx build was not run here either.

Mutation tests (separate worktree, one mutant at a time, the PR's tests plus the neighbouring test modules)

  • Killed (16 of 19):

    • BS2D: grid reverted to the bounding box; surface reverted to extract_surface.
    • DOS: orient before the split; line drawn un-oriented.
    • Overlay: hard-coded shell table; empty result; resolver keeps only the first index; species ignored.
    • Elk: irrep off; >= changed to >; lmirep ignored; no warning.
    • Other fixes: PROCAR regex reverted; report always written to the cwd; --human dropped.
  • Survived (3), all test gaps, none blocking:

    1. bandstructure2D.py:162: dropping the normal offset (frac = 0*offset + ...). No test uses an off-origin plane on a skewed lattice. On my skewed mesh at kz=0.25, this mutant gives 98/1200 NaN where the head gives 0.
    2. bandstructure2D.py:204: quads indexed in the wrong order (reshape(shape[::-1]).T), which connects non-neighbours on non-square grids.
    3. bandstructure2D.py:209: pv.PolyData(points) with no faces at all.

    No BS2D test checks the surface cells, only points and values. My live adjacency check shows the head's faces are right.

Findings

None blocking. Every acceptance item checked is done: DOS labels, autobandsplot report, legacy k-point regex, graphene BS2D, procar.py fermi2D/filter/cat, SACATTER, cfg docs, notebook outputs, overlay weights, and Elk lmirep naming.

Notes (non-blocking)

  • Test gap: the 3 surviving BS2D mutants above. A face-adjacency assertion and an off-origin skewed-lattice case in test_bandstructure2d_grid.py would kill them.
  • Unchanged path: BandStructure2D.from_ebs defaults to as_cartesian=True. That path is unchanged and still leaves 51-55% of graphene grid points NaN. from_code and the legacy handler use as_cartesian=False and are fixed. The PR scopes the fix to that path.
  • Pre-existing: procar.py generate2dkmesh raises TypeError: 'str' object cannot be interpreted as an integer at both commits, because its subparser has no type=.
  • Pre-existing: VASP autobandsplot() with no fermi, as the fixture's own plot.py calls it, uses fermi=0. Its report then differs from the shipped report.txt, with a different window and no defect states, at both commits.
  • Pre-existing: for VASP, dosplot and bandsdosplot do not shift energies when fermi is None. bandsdosplot still draws its Fermi line at y=0, which is 4.96 eV below E_F on dos/spin-polarized.
  • Left open: Elk DOS (task 10) columns are in the irrep basis whenever lmirep is on, the Elk default in every version, but keep Y{l}{m} names. The PR documents this in parsers.md. It is outside the frozen task-22 item.
  • Wording: the PR body's "k41" is 0-based. It is k-point 42 in the file.
  • Environment: Elk 11.2.3 (local build) aborts with "double free or corruption" after finishing task 22 with lmirep=.true.. The outputs are complete, and the lmirep=.false. run exits cleanly.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: PASS+NOTES

Reviewed SHA: ced180c. The head contains origin/dev 4de5cc0, so no merge was needed.
Lane: live runs, red/green and gates (independent verifier, round 2).

Gates (run at the head, verify.sh exec, -n 4)

  • pytest -m "not data": exit 0, 2056 passed, 2 skipped.
  • pytest -m data: exit 0 at both commits. Head: 406 passed, 1 skipped. origin/dev: 403 passed, 1 skipped. The extra tests are 2 autobandsplot tests and the Elk 6.3 test. No new failure.
  • basedpyright: 0 errors and 0 warnings.
  • ruff_new_violations check and format: ok (31 files). The F821/F822/F823/NPY201 check passes.
  • red_green.py origin/dev: exit 0. 23 added tests: 18 fail on dev for their stated reason, 2 guard existing behaviour, and 3 are data tests.
  • gh pr checks: format, lint, red-green, test and typecheck all pass. The CI run's head SHA is ced180c.

BS2D live runs (private fixture copies, head vs origin/dev, compared against references I built myself)

References:

  • Graphene: EIGENVAL and PROCAR on the 60x60 mesh. The 331 irreducible k-points are unfolded with the 12 hexagonal operations, which are integer in fractional coordinates and give no conflicts.
  • BiSb: the full 60x60 mesh, because the run used ISYM=-1.
  • Projections and spin texture: sums of the ion rows, not VASP's rounded tot rows.
path graphene NaN, dev → head BiSb NaN, dev → head
from_code frac 20x20 440/800 → 0 1056/2400 → 0
from_code frac 90x90 8280/16200 → 0 17436/48600 → 0
from_code cart 20x20 440/800 → 0 1056/2400 → 0
from_code cart 90x90 8280/16200 → 0 17436/48600 → 0
legacy handler, default 120x120 (graphene parametric, BiSb spin_texture) 14640/28800 → 0 30456/86400 → 0
legacy handler 90x90 8280/16200 → 0 17436/48600 → 0
  • Heights. At 90x90 every vertex is a DFT k-point, in both modes and in the handler. Graphene: 2x8100 points. BiSb: 6x8100 points, 6 bands.
    • |z - EIGENVAL| <= 5e-7 for graphene and <= 1e-6 for BiSb.
    • The handler surfaces are E - E_F. They match once E_F is added back.
    • Off-mesh grids match my bilinear reference to the interpolation error (p99 <= 0.033 eV).
  • Surface colours and spin texture at those vertices. The colours are the projected property: the total over ions and orbitals, and pz.
    • Graphene colours: <= 1e-7.
    • BiSb colours: <= 2e-7.
    • BiSb spin texture (projected_sum_spin_texture in the handler's spin_texture mode, against Sx, Sy, Sz from the ion rows): <= 1.9e-6, with |S| up to 0.475.
  • Band values on their own points. get_property("bands") equals points[:,2] exactly at the head. On dev BiSb they differ by up to 0.68 eV, which is the old point renumbering.
  • Dirac gap at K = (1/3, 1/3).
    • 90x90, frac, cart and handler alike: 2.511e-5 eV, against 2.50e-5 eV in EIGENVAL. dev gives 0.267 eV.
    • Handler default 120x120: 0.072 eV, because K is not a grid point.
  • Oblique planes. Synthetic 24x24x16 meshes, one skewed lattice and one with c* tilted, an analytic band, 30x30 and 40x30 grids. The linear-interpolation bound is 0.055.
    • Exact-fit cases (all head; dev values in brackets):

      plane head NaN head max error dev NaN
      frac, n=(0,0,1), kz=0.25 0 0.000 (grid on mesh) 254/900
      frac, n=(1,0,0) 0 0.029 94/900
      frac, n=(1,1,0) 0 0.044 130/900
      cart, n=z, tilted lattice 0 0.028 318/900
    • Planes whose cut through the padded box is not a parallelogram leave NaN at the head too, from the geometry: frac n=(1,1,1) 435/900 (dev 530), cart n=(1,0,0) on the tilted lattice 372/900 (dev 488), and cart n=(0.3,0.2,1) 435/900 (dev 497).

    • Every finite point is within the interpolation bound. Every quad joins grid neighbours. dev segfaults on the 40x30 grid.

Elk lmirep

Inputs: my own Elk 11.2.3 runs of Si (task 22, once with lmirep on and once off), and the real Elk 6.3 fixture (data/codes/elk/6.3/SrVO3). The 6.3 bands directory was given the INFO.OUT that reads "Elk version 6.3.02 started" and the task-10 ELMIREP.OUT from the dos run.

case names warning
11.2.3, lmirep on Y{l}_ir{i} none
11.2.3, lmirep=.false. Y{l}{m} none
11.2.3 irrep run, INFO.OUT removed Y{l}{m} yes
11.2.3 irrep run, version line edited out of INFO.OUT Y{l}{m} yes
11.2.3 irrep run, INFO.OUT edited to 10.7.7 Y{l}{m} none
11.2.3 irrep run, INFO.OUT edited to 10.7.8 Y{l}_ir{i} none
Elk 6.3 with INFO.OUT and stale ELMIREP.OUT Y{l}{m} none
Elk 6.3 with stale ELMIREP.OUT, no INFO.OUT Y{l}{m} yes
Elk 6.3 plain Y{l}{m} none
  • Per-l sums agree between the irrep and Ylm runs to <= 3e-6. Individual columns differ by up to 0.169.
  • The parser writes nothing into any calc directory.

SACATTER alias

In all three enums, SACATTER is SCATTER, E["SACATTER"] resolves, and E("scatter") resolves. Iteration lists SCATTER once and never lists the alias. from_str("scatter") works.

Mutation tests (separate worktree, one mutant at a time)

  • Killed (16 of 17):

    • grid: plane level set to 0; quad order on non-square grids; no faces; Cartesian normal left untransformed; Cartesian normal transposed; inset removed; inset pointing outward; old bounding-box grid in Cartesian mode; level sign flipped.
    • Elk: old regex; version-less directory guessing irrep; warning removed; threshold moved to 10.7.9.
    • SACATTER alias removed from each of the 3 enums.

    This includes my three round-1 survivors.

  • Survived (1): across = 2 in place of the argmax at bandstructure2D.py:159. No test uses a plane whose normal is not along kz. On my oblique meshes this mutant leaves 900/900 points NaN for frac n=(1,0,0) and n=(1,1,0), where the head leaves 0.

Findings

None blocking. Changes A to E behave as described: Elk version handling, the deprecated alias, CHANGELOG, tests for the three round-1 mutants, and one fractional grid in both modes.

Notes (non-blocking)

  • Test gap: the across survivor above. A vertical-plane case, for example normal (1,0,0) on the skewed lattice, would kill it.
  • Expected NaN on oblique planes: planes whose cut through the padded mesh is not a parallelogram still leave about half the grid NaN, because the grid is a rectangle in the two in-plane fractional coordinates. This is fewer than on dev in every case I ran.
  • Environment: the local Elk 11.2.3 build again aborts with "double free or corruption" after finishing task 22 with lmirep=.true.. The outputs are complete.
  • Pre-existing: the round-1 notes still apply. They are the generate2dkmesh CLI TypeError, VASP autobandsplot() without fermi, the bandsdosplot Fermi line when fermi is None, and the Elk task-10 DOS irrep names.

@lllangWV

lllangWV commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

VERDICT: PASS+NOTES

Reviewed SHA: ced180c. Base origin/dev 4de5cc0 is the merge base, so no trunk merge was needed (git merge-tree exit 0). Lane: audit and regression, round 2. Independent verifier. Comparisons ran at three commits: origin/dev, the round-1 head 2fd6c01 and this head, each in its own detached worktree, on private fixture copies.

Gates (this head, verify.sh exec)

  • pytest tests -m "not data" -n 8: exit 0, 2056 passed, 2 skipped.
  • pytest tests -m data -n 4: exit 0 at both commits. Head: 406 passed, 1 skipped. origin/dev: 403 passed, 1 skipped. The 3 extra tests are the 2 autobandsplot data tests and the new Elk 6.3 data test. No new failure.
  • basedpyright --project pyrightconfig.json: exit 0, with 0 errors and 0 warnings.
  • ruff_new_violations.py check origin/dev and format origin/dev: exit 0 (31 files). The F821/F822/F823/NPY201 check: exit 0.
  • red_green.py origin/dev: exit 0. 23 tests: 18 fail on the base, 2 are guarded, 3 are data tests that the script does not judge.
  • gh pr checks 304: format, lint, red-green, test and typecheck all pass.

BS2D regression

I called BandStructure2D.from_ebs on graphene and BiSb. Each EBS was parsed in its own worktree and reduced to the bands near the Fermi level. I ran 17 configurations per fixture at each commit:

  • both modes at 30x30, 40x25 and 80x80
  • padding 5, 10 and 15
  • normals (1,0,0), (0,1,0) and (1,1,0) in both modes
  • origins off the plane
  • save/load
  • BS2DPlotter renders, plus the legacy handler at a non-square grid

NaN z points (origin/dev / 2fd6c01 / head):

case graphene BiSb
frac z 30x30 960 / 0 / 0 2196 / 6 / 0
cart z 30x30 960 / 960 / 0 2196 / 2196 / 0
frac z 40x25 segfault / 0 / 0 segfault / 6 / 0
cart z 40x25 segfault / 1056 / 0 segfault / 2346 / 0
cart z 20x20, padding 15 440 / 440 / 0 1056 / 1056 / 0
frac and cart, x and y normals 0 / 0 / 0 672-684 / 0-684 / 0
cart (1,1,0) 0 / 0 / 0 684 / 684 / 0
frac (1,1,0) 0 / 0 / 0 690 / 12 / 12
  • Values on their own points. get_property('bands') equals z on every head point, in every case. On origin/dev the number of points where they differ ranges from 16 to 594.
  • Independent check at DFT nodes. On an 80x80 grid with padding 10, every vertex is a 60x60 DFT k-point. Head's grid sits on those nodes to within 8e-11 (graphene) and 2.4e-6 (BiSb, whose lattice has components of order 1e-7). I compared against the parsed EBS, looked up by fractional k mod 1:
    • graphene: max |z - DFT| = 4.2e-11 in both modes, and projected_sum differs by at most 5.7e-11
    • BiSb: max |z - DFT| = 4.6e-7 in both modes, and the spin texture differs by at most 1.4e-6
  • Cases where origin/dev was already NaN-free (graphene with x, y and (1,1,0) normals): after sorting, head's point set equals origin/dev's. Only the order changes, plus the round-1 fix that keeps each value on its own point.
  • Fractional mode, head against 2fd6c01: points move by at most 6e-11. This is the inset.
  • Save/load gives back identical points in every case.
  • Renders: the BS2DPlotter renders and the handler's parametric and spin_texture modes at 60x40 and 50x50 produce correct images at head. At origin/dev the handler segfaults (exit 139) on the 60x40 grid.
  • Inset mutant: with inset = 0, test_bs2d_on_a_lattice_slightly_off_square_leaves_no_grid_point_empty[cartesian] fails. The inset is covered by a test.
  • Callers. compute_plane_info is called only by from_ebs. from_code and BandStructure2DHandler (which passes as_cartesian=False) go through from_ebs. PlaneInfo.u_limits and v_limits (the Cartesian bounding box) are read only by save and load. u_grid, v_grid and uv_grid_points are not read outside bandstructure2D.py. No code consumes them wrongly.

Elk regression

I ran all 4 Elk fixtures (SrVO3 6.3, bands and dos, non-spin-polarized and collinear spin-polarized) and 6 variants. The variants add the task-10 ELMIREP.OUT to the bands directory: with INFO.OUT, without it, and with the Elk 6.3 INFO.OUT copied in.

  • For each case I compared orbital names and SHA-1 digests of the bands, projected, total and DOS arrays between origin/dev and head. They are identical in all 10 cases: always Y00, Y1-1, ..., 16 names.
  • Head warns only when ELMIREP.OUT is present and no version is found ("there is no INFO.OUT"). The round-1 blocking repro now gives the Ylm names with no warning.
  • The local Elk 11.2.3 binary contains the format ("Elk code version ",I0,".",I0,".",I0," started"). Elk 6.3 writes Elk version 6.3.02. The new regex matches both.

Enum aliases, CHANGELOG and guards

  • Enum aliases, in all three enums:
    • Iteration order and len are unchanged. Iteration lists SCATTER in the old slot and does not list SACATTER.
    • __members__ contains SACATTER as an alias.
    • E("scatter"), E["SACATTER"] and E(E.SACATTER) all return SCATTER.
    • to_list() and the string values match origin/dev. from_str("SACATTER") raises at both commits.
    • The cfg BandStructureMode has no consumer.
  • CHANGELOG: the format matches the existing sections (# ..., ##### ..., ___).
  • guards_existing_behaviour reasons: all 3 are accurate.
    • Ylm-basis Elk: origin/dev never renames.
    • The Elk 6.3 data test: it guards against the round-1 regression.
    • cat: it already worked.

Claims

  • Elk 6.3 header fix: confirmed.
  • Version-less directory reads as on dev: confirmed (all fixtures and variants are identical).
  • SCATTER alias and iteration: confirmed.
  • CHANGELOG entries: confirmed.
  • The BS2D NaN table in the PR body: confirmed. Every row matches my numbers.
  • "Values on their own points" in both modes: confirmed, at DFT nodes against an independent lookup.
  • Gate numbers: confirmed.
  • Sphinx build: not checked, as the PR body also says.

Notes (none blocking)

  1. The 1e-12 inset does not always beat the rounding in the data. Smallest repro: BiSb EBS, from_ebs(normal=(1,1,0), grid_interpolation=(30,30), padding=10, as_cartesian=False).
    • Result: 12 NaN z points, which are grid points (28,0) and (29,0) on each of the 6 sheets. Those points also get NaN x and y. The result is the same at 2fd6c01, and origin/dev has 690.
    • Cause: the fractional slice points carry float32 rounding (for example -0.55000001 and -0.55000002, near-duplicates), so the edge of the hull is jagged by about 1e-8, against an inset of 1e-12 of the span.
    • The code comment at core/bandstructure2D.py:154-155 (the phrase is on line 155) ("by far more than rounding") and features/bs2d.md ("no BiSb grid corner is lost to rounding") overstate the fix. This is not a regression.
  2. The docstrings of from_ebs, from_code and the handler still say "Number of grid points in (u, v) directions". At head, grid_interpolation[0] follows the first in-plane fractional axis (across+1) % 3. For normal=(0,1,0) that axis is f2, which is the v direction. Axis 1 runs along -u. At origin/dev, axis 0 was u. The surface itself is correct.
  3. The # Unreleased section is hand-written. Both pixi run bump-changelog (git-cliff --prepend) and workflow_generate_changlog_on_recent_tag.py prepend their output above it. Whoever cuts the next release has to fold it into the release section by hand.
  4. Pre-existing, outside scope: plotter/bs_2d_plot.py:31 get_uv_bands_grid still builds a Cartesian bounding-box grid, but it has no callers. A NaN z point still gets NaN x and y from the 2π scale transform (0*NaN).

No .pixi/ directory appeared in my worktrees. I removed the worktrees after the run.

@lllangWV
lllangWV merged commit f93c8bd into dev Oct 4, 2026
5 checks passed
@lllangWV
lllangWV deleted the 285-small-fixes branch October 4, 2026 23:21
lllangWV added a commit that referenced this pull request Oct 4, 2026
…able_copy

The check reads tests/ and flags shutil.copy, copy2, copyfile, copytree,
copymode, copystat, os.link, Path.hardlink_to, and cp or rsync through
subprocess or os.system, when an argument holds a path built from DATA_DIR
or the repo root joined with data. It follows names, attributes, defaults,
call-site arguments, return values, pytest fixtures and parametrize values,
across modules. It flags #304's Elk test and the auto bandsplot fixture.

The module-name helpers move from test_user_warnings into tests/utils so
both checks share them.
lllangWV added a commit that referenced this pull request Oct 4, 2026
…ixtures stay usable

#304's Elk test copied a 0555 bands dir with copytree, then copied ELMIREP.OUT
into it, which raised PermissionError on locked fixtures. writable_copy now
also copies a single file.
lllangWV added a commit that referenced this pull request Oct 5, 2026
…y.sh inside data/ (#303)

* test: copy fixtures with writable_copy so read-only fixtures work

copytree keeps a read-only fixture's modes, so a test that writes into
its copy fails with PermissionError once fixtures are locked.
writable_copy dereferences symlinks and adds user write to the copy
only.

* test(verify-pyprocar): verify.sh locks only fixtures and unlocks only its own copies

Run a copy of the harness in scratch repos (data in the repo, data
linked, and a linked worktree) with a stub env. Cover the round-2
escapes: a symlinked fixture given to run, a symlinked work/ given to
clean and gc, glob characters given to fetch, and a fixture root that
is a symlink.

* feat(verify-pyprocar): make fetched fixtures read-only through one path resolver

fixture_path turns an argument into the real path of a fixture inside
data/ or refuses it: plain names only, so no glob reaches the Hugging
Face pattern, and the resolved path must be a fixture path. fetch, run
and doctor act only on that path. run copies with cp -RL, so its copy
holds no symlink. Every chmod runs through find -P ! -type l, which
never follows a symlink. fetch needs the env only to download.

* docs(verify-pyprocar): document read-only fixtures and the verify.sh path rule

* refactor(verify-pyprocar): name the path helpers for what they check and drop their comments

* test: state the writable_copy guard reason as one string

* test(verify-pyprocar): verify.sh changes nothing outside data/ under hostile data roots

Unenterable data, data at the top of the tree, data linked to a copy,
a leading // spelling, a verify-runs link out of data/, hard links to
outside files, and unlockable roots in doctor. A sweep runs every
command against a sacrificial tree and compares it before and after.
download_from_hf fetches only the archive named exactly.

Refs #288

* fix(verify-pyprocar): act only strictly below the main checkout's data/

real_path fails on any failed cd and drops a leading //. Every command
that touches data/ first requires data to resolve to <main>/data and
verify-runs to be a directory in it, compares paths by component, and
chmods only through chmod_below_data, which skips symlinks and files
with a second hard link. doctor lists roots fetch cannot lock.
download_from_hf fetches exactly <relpath>.zip and no longer deletes
<output>/.cache; fetch points its HF cache into data/verify-runs.

Refs #288

* docs(verify-pyprocar): state the data-root rule and name copies to unlock

Refs #288

* test: assert the prefix download first and drop guard marks on tests dev fails

Refs #288

* test: download_from_hf leaves the shared Hugging Face cache alone

Refs #288

* fix(download): fetch the one archive into a staging dir, not the shared HF cache

download_from_hf deleted the dataset's folder from the active Hugging
Face cache after every download. snapshot_download with local_dir set
to a staging dir beside the target touches no shared cache, and the
archive path is spelled once.

Refs #288

* refactor(verify-pyprocar): guard deletes like chmods and share the doctor list printer

scrub requires a resolved run dir below verify-runs before rm -rf, the
lint now covers rm and mv too, fetch no longer redirects HF_HUB_CACHE,
and the tests share one env-and-driver setup.

Refs #288

* test: take the hub arguments the fake ignores as keywords

Refs #288

* test(verify-pyprocar): run refuses a run dir that already exists; doctor exits 2 on a wrong data root

Plants a symlink, a real dir, and real dirs with symlinked work, .pid or
evidence at the next run names, and snapshots the whole fake tree.
The sweep test now gives the fake worktree a pyprocar/ dir, so
worktree-setup reaches its mkdir and ln lines.

* fix(verify-pyprocar): run writes only into a run dir it has just created

run named its dir <timestamp>-<name> and wrote into it with mkdir -p,
a .pid redirect and cp -RL without checking it was new, so an entry
planted at that name (a symlink, or a run dir whose work, .pid or
evidence links out of data/) made run overwrite and then delete files
outside data/ before chmod_below_data refused. A plain mkdir now
creates the dir and fails on any existing entry, the dir must resolve
strictly below verify-runs, its subdirs use plain mkdir, and noclobber
guards every redirect. This also stops two same-second runs with one
name from sharing a dir.

doctor exits 2, like the other data commands, when the data root is
wrong. The lint now lists every line of verify.sh that writes, links,
moves or deletes, not only chmod and rm.

* docs(verify-pyprocar): name the Hugging Face ref a download writes, the run-dir rule and how to refresh a fixture

* refactor(verify-pyprocar): scope noclobber to the .pid write and plant one stamped run in the test

A date stub fixes the run name, so the planted-run test plants one
entry instead of guessing the next 31 seconds, and the post-driver
writes into evidence keep their old behaviour.

* test(verify-pyprocar): decode verify.sh output as UTF-8 and keep census lines under 100 columns

CI runs in an ASCII locale, where mkdir's quoted error message failed
to decode. Two long verify.sh lines are split so the census lint pins
them without implicit string concatenation.

* test(filter): copy the fixture writable in test_filtered_procar so a locked PROCAR can be edited

* test(verify-pyprocar): gc keeps the work of a run whose pid is alive

* test(download): download_from_hf stages beside the fixture, not in the system tmp

* test(verify-pyprocar): fetch downloads with hf_xet log cleanup off, through pixi run --locked in the main checkout

* fix(verify-pyprocar): keep hf_xet's log cleanup off in fetch, run pixi --locked, and stop doctor's git status rewriting the index

* test(verify-pyprocar): census also flags fd and & redirects, in-place edits, install, tar, unzip and a command held in a variable

* docs(verify-pyprocar): list every write outside data/ and unlock a fixture by its directories only

* docs(verify-pyprocar): name the staging dir hf_xet creates beside its chunk cache

* docs: only worktree-setup refreshes a git index

* test(verify-pyprocar): assert gc's kept and removed work, not its message, and mark the census self-test as guarding

* test: fail when a test copies a fixture path without tests.utils.writable_copy

The check reads tests/ and flags shutil.copy, copy2, copyfile, copytree,
copymode, copystat, os.link, Path.hardlink_to, and cp or rsync through
subprocess or os.system, when an argument holds a path built from DATA_DIR
or the repo root joined with data. It follows names, attributes, defaults,
call-site arguments, return values, pytest fixtures and parametrize values,
across modules. It flags #304's Elk test and the auto bandsplot fixture.

The module-name helpers move from test_user_warnings into tests/utils so
both checks share them.

* test(elk,autobandsplot): copy fixtures with writable_copy so locked fixtures stay usable

#304's Elk test copied a 0555 bands dir with copytree, then copied ELMIREP.OUT
into it, which raised PermissionError on locked fixtures. writable_copy now
also copies a single file.

* test: make the fixture-copy self-test reach writable_copy with a fixture path, so its exemption is exercised

* fix(verify-pyprocar): start main-checkout python with PYTHONDONTWRITEBYTECODE=1

py ran pixi run without it, so doctor, a download and a run driver rewrote
stale __pycache__ files in the main checkout (measured in an overlay sandbox).
The env stub now logs the variable, and the harness clears it so the
caller's env cannot satisfy the test.

* test(verify-pyprocar): doctor leaves a stat-dirty git index alone

Dropping GIT_OPTIONAL_LOCKS=0 from doctor's git status survived the suite.
The test makes the index stale, runs doctor, and checks the index bytes; a
plain git status afterwards must rewrite it, so the setup is proven stale.

* test(verify-pyprocar): census flags "${cmd[@]}", "${@}" and if/while "$w", and ignores [[ ]] and arithmetic comparisons

A > inside [[ ]], (( )) or $(( )) is a comparison, not a redirect.

* docs: list matplotlib's config and font cache among verify.sh's writes, and add the fixture-copy rule

The rule table names tests/test_fixture_copies.py as the enforcer of
copying fixtures only through tests.utils.writable_copy.

* refactor(tests): inline one-caller helpers in the fixture-copy check, and map call arguments to positional parameters only

* fix(tests): type-narrow the fixture-copy scope, keep the Elk calc dir untyped for from_code, and wrap two long self-test lines

* refactor(verify-pyprocar): export PYTHONDONTWRITEBYTECODE once for every python verify.sh starts

* refactor(tests): simplify the fixture-copy check (derived shell set, hardlink first, fixture registration in the bind loop) and document writable_copy's file form

* style(tests): one condition for fixture registration, and keep the exports line under 100 columns
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant