This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
STDPipe (Simple Transient Detection Pipeline) is a Python library for astronomical image processing, focusing on astrometry, photometry, and transient detection. It operates on standard Python objects (NumPy arrays for images, Astropy Tables for catalogs) and wraps external astronomical tools with transparent Python interfaces.
Key design principle: All external tool execution happens via temporary files - nothing is kept after the run unless explicitly requested.
STDPipe uses modern Python packaging standards (PEP 517/621) with all configuration in pyproject.toml.
Install in editable/development mode:
# Basic editable install
python3 -m pip install -e .
# With development tools (pytest, coverage, etc.)
python3 -m pip install -e ".[dev]"
# With documentation tools
python3 -m pip install -e ".[docs]"
# Everything
python3 -m pip install -e ".[all]"This allows immediate reflection of local changes without reinstalling. The package is in constant development, so editable mode is recommended.
Build source and wheel distributions:
python3 -m pip install build
python3 -m buildOutput: dist/stdpipe-{version}.tar.gz and dist/stdpipe-{version}-py3-none-any.whl
STDPipe wraps these external astronomical tools (optional but commonly used):
- SExtractor - Object detection and photometry
- SCAMP - Astrometric calibration
- PSFEx - PSF model extraction
- SWarp - Image resampling
- HOTPANTS - Image subtraction
- Astrometry.Net - Blind WCS solving
Install via package manager:
# Debian/Ubuntu
sudo apt install sextractor scamp psfex swarp
# Conda
conda install -c conda-forge astromatic-source-extractor astromatic-scamp astromatic-psfex astromatic-swarpHOTPANTS must be compiled manually - see install_hotpants.sh for installation script.
Comprehensive pytest-based test suite with 783 tests covering core functionality (including 47 SFFT subtraction tests, 16 optimal extraction tests, 18 photutils PSF photometry tests, 17 DAOPHOT PSF photometry tests, and 32 photutils object detection tests).
# Run all tests
pytest
# Run only fast unit tests
pytest -m unit
# Run with coverage report
pytest --cov=stdpipe --cov-report=html
# Run in parallel (faster)
pytest -n auto
# Run specific test file
pytest tests/test_photometry.py
# Run specific test function
pytest tests/test_photometry.py::TestSEPPhotometry::test_get_objects_sep_simple_imageTests are marked by category:
unit- Fast tests without external dependencies (including 47 SFFT subtraction tests + 16 optimal extraction tests + 18 photutils PSF tests + 17 DAOPHOT PSF tests + 32 photutils detection tests)integration- Tests requiring external tools like SExtractor (6 tests)slow- Time-consuming testsrequires_sextractor,requires_hotpants,requires_pyraf, etc. - Auto-skip if tool not installedrequires_network- Tests needing internet access
All test configuration is in pyproject.toml under [tool.pytest.ini_options]. The test suite includes:
- 20+ reusable fixtures (images, WCS, catalogs, temp directories)
- Automatic skipping when external tools unavailable
- Property-based testing with hypothesis
- Coverage reporting integrated
See TESTING.md for comprehensive testing documentation.
The codebase is organized into focused modules in stdpipe/:
Core Processing Modules:
pipeline.py- Higher-level building blocks combining lower-level functionality. Contains utilities likemake_mask()for image masking (cosmic rays, saturation, bad pixels) and other multi-step operationsphotometry.py- Object detection and basic photometry. Main functions:get_objects_sep()- fast SEP-based detection (compiled Python). Supports position-dependent auto-FWHM viafwhm_spatial_order(applied per source for aperture/bkgann/optimal PSF width; only the scalar median summary is stored inobj.meta['fwhm_phot']so tables stay serializable). Also accepts a callablefwhm(e.g.FWHMMap) directly. Optionalpsf=(PSFEx /create_psf_modeldict orsep.PSF) switches detection to the PSF-matched significance image from sep-xpsf_snr()(spatially varying PSF supported); objects are then re-measured on the image inside those footprints. Defaultr0=0(no detection smoothing, likeget_objects_sextractor); withr0=Noneand a numeric/callablefwhmthe detection kernel sigma is0.8*FWHM/2.355(_DETECT_KERNEL_FWHM_FRACTION), else 0.5.minnthresh(default 0) counts unsmoothed footprint pixels abovethresh*err*f,f = sqrt(sum k^2)/sum k(_detection_noise_factor, from the kernel or PSF), via the segmentation map; good values grow with kernel size (~2 for r0=0.5, ~5 for the FWHM kernel);minnthresh='auto'(_minnthresh_auto) requires 1 + (npix-1)p + 2 sigma of the binomial noise count per object (best with the FWHM kernel orpsf=).r0='auto'(opt-in) without a known FWHM detects twice:r0=0.5, then the FWHM-following kernel from the robust (unflagged-object) estimate; apertures keep the first-pass FWHM.get_objects_sextractor()- external SExtractor tool wrapperget_objects_photutils()- pure Python photutils-based detection with 3 methods (segmentation with deblending, DAOStarFinder, IRAFStarFinder). Key features: optional deblending for crowded fields, multiple detection algorithms, extended API exposing photutils parameters. Drop-in replacement forget_objects_sep().estimate_fwhm_from_objects()- robust mode-based FWHM from a detection table. Scalar by default;spatial_order >= 1returns aFWHMMapcallable (2-D polynomialFWHM(x, y)) with transparent scalar fallback on low candidate counts. The spatial fit reuses the scalar path's anti-contamination guard (candidates windowed to 0.5-2x the guarded scalar estimate before the sigma-clipped fit).FWHMMap- callable polynomial FWHM(x, y) (scalar viafloat(fmap)), usable anywhere afwhmargument is accepted.
photometry_measure.py- Aperture and optimal extraction photometry. Main function:measure_objects()for photometry at detected object positions. Supports both standard aperture photometry and optimal extraction (Naylor 1998) that provides ~10% S/N improvement for point sources. Advanced features: grouped optimal extraction for crowded fields (simultaneously fits overlapping sources via weighted least squares). Handles background estimation, error propagation, centroiding, and local background annuli. Bothmeasure_objectsandmeasure_objects_sepaccept a callablefwhm(e.g.FWHMMap): the SEP backend broadcasts per-source aperture/bkgann/fwhm arrays to SEP; the photutils backend uses per-source width in ungrouped optimal extraction and falls back to the scalar median elsewhere. Noise semantics (both backends):erris the full per-pixel noise, used as is;gainonly adds source Poisson noise whenerr=None(never passed to SEP);bg_err(background-only map or scalar) definesbg_fluxerr(background noise in the aperture, used for local limits) and is the base of the automaticerr. Withbkgann, the noise of the local background (bg_local_err, predicted from err map, estimator factors in_LOCAL_BG_ESTIMATOR_NOISE,_local_bg_order_factorfor bkg_order>0) is added tofluxerr(times d flux/d bg: npix_aper, ordflux_dbg6th element of_optimal_extraction/grouped results, 0 for groups fitting bg) and tobg_fluxerr(times npix_aper). SEP optimal with bg_estimator='mean' relies on sep-x's own bkgann error (fixed in sep-x 1.5.3, the minimal version; before it the annulus error was 0 whenever clip_iters>0).photometry_background.py- Background estimation (get_background()with sep/photutils/percentile/morphology/gp/polynomial methods) andestimate_noise_excess(): ratio of actual to predicted noise of aperture (or PSF-weighted,optimal=True) sums at random unmasked positions, i.e. the correlated-noise correction for resampled/stacked/difference images. Aperture-dependent, so pass the sameaper/fwhm/optimal/psfas the photometry; scalebg_err(anderrif given) by it. Predicts the variance of the sum as sum(w^2*err^2) (pixel overlap w), like photutils >= 3.1 and sep-x >= 1.5.4, so ~1 on white noise;bg_fluxerrin both backends is propagated by photutils/SEP themselves. Kernel path uses global bg only; withbkgann(mode estimator adds ~13% noise) or PSF fitting passmeasure=callable(obj)->tableto run the real photometry on a grid of empty positions (spacing > grouping radius, no centroiding).photometry_model.py- Photometric modeling and calibration. Main functions:match()for photometric matching with spatial zero-point models,make_sn_model()for S/N modeling,get_detection_limit_sn()for detection limit estimation. Core calibration routines used by higher-level workflows.photometry_psf.py- PSF photometry using photutils. Main functions:measure_objects_psf()for PSF fitting photometry,create_psf_model()for building empirical ePSF. Provides more accurate flux measurements than aperture photometry, especially in crowded fields. Supports Gaussian, PSFEx, and empirical PSF models. Advanced features: position-dependent PSF (evaluates PSFEx polynomials at each position), grouped PSF fitting (simultaneously fits overlapping sources), and crowdsource-style quality metrics (qf,fracflux,spread_model,dspread_model;compute_quality=Trueby default, NaN in position-dependent mode). Stamp-based metric routines live inphotometry_quality.py(backend-agnostic).photometry_iraf.py- IRAF DAOPHOT backend for aperture and PSF photometry. Main functions:measure_objects()for aperture photometry using DAOPHOT phot task,measure_objects_psf()for PSF photometry using complete DAOPHOT workflow (phot → psf → allstar). Requires PyRAF/IRAF. Provides classic, battle-tested algorithms as alternative to photutils.astrometry.py- Astrometric calibration and coordinate transformations. Wraps Astrometry.Net for blind solving, SCAMP for refinement. Utilities for WCS, pixel scales, spherical distancessubtraction.py- Image subtraction wrappers. Main functions:run_sfft()(built-in SFFT),run_hotpants()(external HOTPANTS),run_zogy()(built-in ZOGY). All share the same interface pattern (masks, error maps, gain,get_noise/get_scaled/get_convolved)sfft.py- SFFT (Hu et al. 2022) low-level solver. Main function:solve()for spatially varying kernel fitting in a single global least-squares problem. Helper functions:evaluate_kernel_at(),evaluate_flux_scale(),evaluate_background()templates.py- Template image acquisition from HiPS surveys (Pan-STARRS, Legacy Survey). Main function:get_hips_image()using CDS hips2fits servicecatalogs.py- Online catalog queries via Vizier. Supports PS1, Gaia DR2/EDR3/DR3, SDSS, ATLAS-REFCAT2, SkyMapper, etc. with automatic photometric augmentation (magnitude conversions). Main function:get_cat_vizier()psf.py- PSF modeling via PSFEx wrapper. Main function:run_psfex()
Supporting Modules:
cutouts.py- Cutout extraction and manipulationutils.py- FITS I/O helpers, downloading, coordinate formatting, header parsingplots.py- Visualization utilities for images and catalogslcs.py- Light curve handlingresolve.py- Object name resolutiondb.py- Database interactions (likely for PostgreSQL/psycopg2)
Typical processing pipeline:
- Load image (NumPy array) + optional mask/header
- Preprocessing (
pipeline.make_mask()) - mask bad pixels, cosmic rays, saturation - Object detection - Choose one approach:
- SEP:
photometry.get_objects_sep()- fast, compiled, SExtractor-like - photutils:
photometry.get_objects_photutils()- pure Python, 3 methods (segmentation/DAOStarFinder/IRAFStarFinder), optional deblending - SExtractor:
photometry.get_objects_sextractor()- external tool, most features All return Astropy Table with identical column structure
- SEP:
- Astrometry (
astrometry.pyfunctions) - solve/refine WCS - Photometry - Choose one approach:
- Aperture:
photometry.measure_objects()orphotometry_iraf.measure_objects()- fast, robust, good for isolated sources - PSF:
photometry_psf.measure_objects_psf()(photutils) orphotometry_iraf.measure_objects_psf()(DAOPHOT) - more accurate, especially in crowded fields
- Aperture:
- Photometric calibration - match to catalogs (
catalogs.get_cat_vizier()), fit zero-points - Image subtraction (optional) - get template (
templates.get_hips_image()), subtract (subtraction.run_sfft(),subtraction.run_hotpants(), orsubtraction.run_zogy()) - Transient detection - analyze difference image
Modules wrapping external tools (SExtractor, SCAMP, PSFEx, HOTPANTS) follow a consistent pattern:
- Accept NumPy arrays and Python parameters
- Create temporary directory (cleaned up after, unless
_workdirspecified for debugging) - Write FITS files and configuration files to temp directory
- Execute external binary with
subprocess - Parse output files back into Python objects (tables, arrays)
- Return results + optionally write to user-specified files
Key parameters for debugging wrappers:
_workdir- Keep temporary files for inspection_tmpdir- Custom temporary directory location_exe- Override executable pathverbose- Enable detailed logging
image- NumPy array (2D, float)mask- Boolean array whereTrue= masked/excluded pixelsheader- FITS header (astropy.io.fits.Header)wcs- Astrometric solution (astropy.wcs.WCS)err- Error/noise map (NumPy array). Can beTrueto auto-generate from image statisticsgain- Detector gain in e-/ADUverbose- Boolean or callable (print-like function) for loggingextra- Dictionary of additional parameters for external tools
SEP-Based Photometry (photometry_measure.measure_objects_sep()):
- Alternative backend using new SEP 1.4+ features
- Sigma-clipped local background via
sep.stats_circann()(more robust in crowded fields) - Grouped optimal extraction via
sep.sum_circle_optimal() - PSF fitting photometry via
sep.psf_fit()with PSFEx or Gaussian models - Grouped flux-only fitting (
group_sources=True, fit_positions=False): NNLS deblending at fixed positions — fast, accurate for crowded fields with known positions - Fitting radius control (
fit_radius): limits pixels used in PSF fitting to a circular region (e.g., 2-3× FWHM), reducing contamination from distant neighbors - C-based implementation (potentially faster)
- No polynomial background gradient fitting
- Excellent agreement with
measure_objects()(<0.1% bias) - Best for: crowded fields, performance-critical applications, consistency with SEP detection
Aperture Photometry (photometry_measure.measure_objects()):
- Sums flux within circular aperture
- Fast and robust
- Best for: isolated sources, quick measurements
- Truncates PSF wings at aperture edge
Optimal Extraction (photometry_measure.measure_objects(optimal=True)):
- Weighted sum using PSF profile (Naylor 1998 algorithm)
- ~10% S/N improvement over aperture for point sources
- Requires PSF model (
psfparameter) or FWHM (fwhmparameter) - Uses pixel-integrated Gaussian PSF - eliminates FWHM-dependent systematic bias:
- Old: +11.6% at FWHM=1.5, +2.7% at FWHM=3.0
- New: < 0.1% bias for all FWHM values
- Grouped extraction (enabled by default,
group_sources=True):- Simultaneously fits overlapping sources via weighted least squares
- Properly accounts for flux sharing between nearby sources
- Dramatic accuracy improvement: 51× at 0.5 FWHM, 14× at 1.0 FWHM, 3× at 1.5 FWHM
- No downside for isolated fields (identical results at >3 FWHM)
- Returns
group_idandgroup_sizecolumns - Set
group_sources=Falseonly for sparse fields with performance constraints
- Best for: point sources, all crowding levels, when PSF is known
PSF Photometry (two backends available):
Photutils Backend (photometry_psf.measure_objects_psf()):
- Modern Python-based PSF fitting using photutils
- PSF models: Gaussian (auto), PSFEx, empirical ePSF
- Advanced features: position-dependent PSF, grouped fitting
- Fast, flexible, pure Python
DAOPHOT Backend (photometry_iraf.measure_objects_psf()):
- Classic IRAF DAOPHOT workflow (phot → psf → allstar)
- PSF models: Analytical functions (auto, gauss, moffat, lorentz, penny)
- Battle-tested, conservative fitting, widely cited
- Requires PyRAF/IRAF
Both approaches:
- More accurate than aperture photometry, especially in crowded fields
- Account for full PSF including wings
- Best for: accurate fluxes, crowded fields, faint sources
- Fitted positions replace
x,y(input ones kept inx_orig,y_orig, as with centroiding), plus quality metrics
PSF Output Columns (all with _psf suffix):
qfit_psf- Fit quality (0 = good)cfit_psf- Central pixel fit quality (0 = good)flags_psf- Photutils fit flagsnpix_psf- Number of unmasked pixels used in fitreduced_chi2_psf- Reduced chi-squared (photutils ≥ 2.3.0)
PSF Quality Metrics (photutils backend, compute_quality=True by default, no suffix):
qf- PSF quality factor: fraction of PSF footprint on unmasked pixels (1 = clean)fracflux- Fraction of PSF-weighted stamp flux from this source after neighbour subtraction (crowding indicator, 1 = isolated)spread_model- Star/galaxy discriminant (~0 for stars, > 0 for extended); matches SExtractor'sSPREAD_MODELdspread_model- Uncertainty ofspread_modelCrowdsource-style (Schlafly et al.), computed inphotometry_quality.pyfrom neighbour-subtracted residual stamps. NaN for failed fits and in position-dependent PSF mode.
Advanced PSF Features:
- Position-dependent PSF (
use_position_dependent_psf=True) - For wide-field imaging where PSF varies across the field. Evaluates PSFEx polynomial at each source position (constant, linear, quadratic terms). Slower but more accurate for fields with PSF variation. - Grouped PSF fitting (enabled by default,
group_sources=True) - Fits nearby sources simultaneously to reduce neighbor contamination. Dramatically more accurate in crowded fields (51× improvement at 0.5 FWHM, 14× at 1.0 FWHM, 3× at 1.5 FWHM). No downside for isolated fields (identical results at >3 FWHM). Setgroup_sources=Falseonly for sparse fields with performance constraints. Configure withgrouper_radius(default: max(2.5×FWHM, fit_size/2)). - Combined mode - Both features can be used together for maximum accuracy in crowded, wide-field images.
Optimal Extraction Output Columns:
flux,fluxerr- Measured flux and errornpix_optimal- Number of pixels used in extractionchi2_optimal- Reduced chi-squared of the fitnorm_optimal- PSF normalization factorgroup_id- Group identifier (whengroup_sources=True)group_size- Number of sources in group (whengroup_sources=True)
Optimal Extraction Flags (flags column):
0x800- Optimal extraction failed (NaN result)
PSF Photometry Flags (flags column, in addition to standard flags):
0x1000- PSF fit failed (NaN result)0x2000- Large centroid shift during fit (>1 pixel)
Real-Bogus Flags (flags column, set by artefacts/realbogus/realbogus_features):
0x4000- Object classified as bogus by real-bogus filtering (FLAG_BOGUSinrealbogus_features)
Examples:
from stdpipe import photometry, photometry_psf, psf
from stdpipe.photometry_measure import measure_objects_sep, _HAS_SEP_OPTIMAL
# Check if SEP optimal extraction is available
if _HAS_SEP_OPTIMAL:
print("SEP 1.4+ features available")
# Object detection - SEP backend (fast, compiled)
obj = photometry.get_objects_sep(image, thresh=3.0, aper=5.0)
# Object detection - photutils backend (pure Python, 3 methods)
obj = photometry.get_objects_photutils(
image,
thresh=3.0,
method='segmentation', # or 'dao' or 'iraf'
deblend=True, # Deblend crowded sources
aper=5.0
)
# photutils: Segmentation with deblending (crowded fields)
obj = photometry.get_objects_photutils(
image,
thresh=2.0,
method='segmentation',
deblend=True,
nlevels=32,
contrast=0.001,
aper=3.0,
sn=5.0
)
# photutils: DAOStarFinder (stellar fields)
obj = photometry.get_objects_photutils(
image,
thresh=5.0,
method='dao',
fwhm=3.0,
aper=5.0
)
# Aperture photometry (fast)
from stdpipe import photometry_measure
result_aper = photometry_measure.measure_objects(obj, image, aper=3.0, fwhm=3.0)
# Optimal extraction (~10% S/N improvement for point sources)
# Grouped fitting is now enabled by default for safety
result_opt = photometry_measure.measure_objects(
obj, image,
aper=5.0,
fwhm=3.0,
optimal=True # Uses Naylor 1998 algorithm, group_sources=True by default
)
# Check grouping results (for crowded fields)
print(result_opt['group_id'], result_opt['group_size'])
# For sparse fields with performance constraints (optional)
result_sparse = photometry_measure.measure_objects(
obj, image,
aper=5.0,
fwhm=3.0,
optimal=True,
group_sources=False # Disable grouped fitting (not recommended unless field is sparse)
)
# PSF photometry - Photutils backend (modern, flexible)
# Grouped fitting is now enabled by default for safety
result_psf = photometry_psf.measure_objects_psf(obj, image, fwhm=3.0)
# PSF photometry - DAOPHOT backend (classic, battle-tested)
from stdpipe import photometry_iraf
result_psf = photometry_iraf.measure_objects_psf(obj, image, fwhm=3.0)
# Photutils: Position-dependent PSF (wide-field images)
psfex_model = psf.run_psfex(image)
result = photometry_psf.measure_objects_psf(
obj, image,
psf=psfex_model,
use_position_dependent_psf=True
)
# Photutils: Custom grouper radius (crowded fields)
result = photometry_psf.measure_objects_psf(
obj, image,
fwhm=3.0,
grouper_radius=10.0 # Custom radius (group_sources=True by default)
)
# DAOPHOT: Custom PSF function and radii
result = photometry_iraf.measure_objects_psf(
obj, image,
fwhm=3.0,
psf_function='moffat25', # Force Moffat β=2.5
psfrad=12.0, # PSF radius
fitrad=4.0, # Fitting radius
sn=5.0 # S/N filter
)
# Photutils: Combined position-dependent + grouped (maximum accuracy)
result = photometry_psf.measure_objects_psf(
obj, image,
psf=psfex_model,
use_position_dependent_psf=True,
group_sources=True,
grouper_radius=15.0
)
# Access quality metrics
good_fits = result[result['qfit_psf'] < 0.5] # Low qfit = good quality
converged = result[result['cfit_psf'] < 0.01] # Close to 0 = good quality in central pixel
if 'reduced_chi2_psf' in result.colnames:
excellent = result[result['reduced_chi2_psf'] < 1.5]
# SEP-based photometry (requires SEP 1.4+ with new features)
if _HAS_SEP_OPTIMAL:
# Aperture photometry with sigma-clipped local background
result = measure_objects_sep(
obj, image,
aper=1.5, fwhm=3.0,
bkgann=(3.0, 5.0), # Sigma-clipped annulus background (robust)
sn=5, verbose=True
)
# Grouped optimal extraction (crowded fields)
result = measure_objects_sep(
obj, image,
aper=1.5, fwhm=3.0,
optimal=True,
group_sources=True, # Automatic grouping in SEP
sn=5, verbose=True
)
# PSF fitting with grouped deblending at fixed positions
result = measure_objects_sep(
obj, image,
psf=psfex_model, fwhm=3.0,
group_sources=True,
fit_positions=False, # Fix positions, only fit fluxes (NNLS)
fit_radius=2.0 * 3.0, # Limit fitting to 2× FWHM radius
sn=5, verbose=True
)Important: SEP and SExtractor use different flag values for object detection quality. When using get_objects_sep(), consult https://sep.readthedocs.io/en/v1.1.x/reference.html for flag meanings (they differ from SExtractor).
Documentation is built with Sphinx and hosted on ReadTheDocs: https://stdpipe.readthedocs.io/
Build locally:
cd doc
make htmlTutorial notebook: notebooks/stdpipe_tutorial.ipynb demonstrates a complete processing workflow.
- Uses
from __future__ importfor Python 2/3 compatibility (legacy) - Extensive use of temporary file operations via
tempfilemodule - Functions include detailed docstrings with parameter descriptions
- Verbose logging pattern:
log = (verbose if callable(verbose) else print) if verbose else lambda *args, **kwargs: None
- When debugging a regression or bug, always investigate the root cause first. Never 'fix' by reverting defaults or papering over symptoms without understanding why the issue occurs.
- After modifying any function signature or return type, immediately check and update all callers and all tests that reference that function. Run the full test suite before considering the task complete.
- When the user asks for an investigation or analysis, do the analysis first — do NOT jump to implementing code changes. Ask clarifying questions if the scope is unclear. Especially distinguish between 'evaluate/compare approaches' vs 'implement a fix'.
- Keep changes minimal and focused. Do not create summary documents, elaborate writeups, or refactor code unless explicitly asked. When reviewing or editing docstrings, edit them in-place — don't create separate files.
- When implementing features, prefer existing library conventions (e.g., use existing flags like
_HAS_SEP_OPTIMAL, existing parameter naming patterns). Check existing code patterns before introducing new ones.