Skip to content

Fix pin-staleness job: stop update_toolchains.py crashing, bump stale pypa pins #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
o-murphy merged 1 commit into from
Sep 7, 2026
Merged

Fix pin-staleness job: stop update_toolchains.py crashing, bump stale pypa pins #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 64 additions & 30 deletions bin/update_toolchains.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,15 @@
Three upstream shapes, which is why [0046] asked for one script per shape
rather than one generic checker:

- **GitHub releases** (`arm-none-eabi`, `riscv-none-elf` -- both now
`embedded_base`, record 0096 --, `xtensa_esp`, `windows`) -- the pinned
tag is in the URL; compare against the repo's
own latest release.
- **GitHub releases.** For `xtensa_esp`/`windows` the pinned tag is in the
Dockerfile's own URL; compare against the repo's own latest release.
`arm-none-eabi`/`riscv-none-elf` (both `embedded_base` since record 0096)
are the same upstream shape but no longer a Dockerfile fact at all --
[0087]/[0089] moved their real pins into `resources/pinned_toolchains.toml`'s
own `[cross]` tables, keyed by `(cross, version)` because more than one
verified version is pinned at once (per-row floor/ceiling windows, e.g.
`mimxrt`'s below-13 ceiling). This checker reads every version pinned for
that cross and reports whether the newest has fallen behind upstream.
- **emsdk** (`webassembly`) -- pinned by *build hash*, which looks
uncomparable and is not: `emscripten-core/emsdk` publishes
`emscripten-releases-tags.json` mapping every release to its hash, so
Expand All @@ -39,13 +44,16 @@
import json
import re
import sys
import tomllib
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path

REPO = Path(__file__).resolve().parent.parent
DOCKER = REPO / "docker"
RESOURCES = REPO / "src" / "cibuildmp" / "resources"
PINNED_TOOLCHAINS = RESOURCES / "pinned_toolchains.toml"

GITHUB_LATEST = "https://api.github.com/repos/{repo}/releases/latest"
EMSDK_TAGS = (
Expand All @@ -60,37 +68,41 @@ class Pin:
dockerfile: str
# How to find the pinned value in that file, group(1) being the value.
pattern: str
kind: str # "github" | "emsdk" | "unversioned"
upstream: str = "" # owner/repo for "github"


# `arm-none-eabi`/`riscv-none-elf` below point at `embedded_base.Dockerfile`
# (record 0096 merged what used to be `arm_embedded.Dockerfile`/
# `riscv_embedded.Dockerfile`) purely so `_pinned()` reads a file that
# exists -- **neither actually matches any more.** [0087]/[0089] already
# deleted the `ARG TOOLCHAIN_URL=` line this regex needs from both former
# files (the tarball is fetched at container run time now, per-row, not
# baked at image-build time), so `_pinned()` has raised
# `SystemExit(f"{pin.name}: no pin found...")` for both entries since
# [0087] landed -- a real, pre-existing gap this record does not close,
# only avoids widening into a harder `FileNotFoundError` by keeping the
# filename real. The actual fix (reading `resources/pinned_toolchains.toml`'s
# own per-cross pin instead of grepping a Dockerfile `ARG`) is [0090]'s own
# scope, not this Dockerfile-merge's.
kind: str # "github" | "cross-toml" | "emsdk" | "unversioned"
upstream: str = "" # owner/repo for "github"/"cross-toml"
cross: str = "" # `pinned_toolchains.toml` table key, for "cross-toml"


# `arm-none-eabi`/`riscv-none-elf` used to be grepped straight out of
# `embedded_base.Dockerfile`'s own `ARG TOOLCHAIN_URL=`. [0087]/[0089]
# deleted that line (the tarball is fetched at container run time now, per
# row, not baked at image-build time) and moved the real pins into
# `resources/pinned_toolchains.toml`'s own `[cross]` tables -- and unlike
# the Dockerfile's single shared `ARG`, that table genuinely holds more
# than one verified version per cross at once (e.g. `mimxrt`'s own
# below-13 ceiling, record 0088), because different `(tag, scope)` windows
# resolve to different versions. There is no longer one "the" pin to
# compare against upstream; `kind="cross-toml"` instead reads every
# version pinned for that cross and reports whether the *newest* of them
# has fallen behind upstream's own latest release -- the question this
# checker can still answer ("is it time to add a newer entry"), without
# claiming the older, intentionally-kept versions are drift.
PINS = (
Pin(
"arm-none-eabi",
"embedded_base.Dockerfile",
r"xpack-dev-tools/arm-none-eabi-gcc-xpack/releases/download/v([^/]+)/",
"github",
"",
"",
"cross-toml",
"xpack-dev-tools/arm-none-eabi-gcc-xpack",
"arm-none-eabi-",
),
Pin(
"riscv-none-elf",
"embedded_base.Dockerfile",
r"xpack-dev-tools/riscv-none-elf-gcc-xpack/releases/download/v([^/]+)/",
"github",
"",
"",
"cross-toml",
"xpack-dev-tools/riscv-none-elf-gcc-xpack",
"riscv64-unknown-elf-",
),
Pin(
"xtensa-esp",
Expand Down Expand Up @@ -135,6 +147,24 @@ def _pinned(pin: Pin) -> str:
return match.group(1)


def _version_key(version: str) -> tuple[int, ...]:
return tuple(int(part) for part in re.findall(r"\d+", version))


def _newest_cross_pin(pin: Pin) -> tuple[str, int]:
"""(newest version string, how many versions are pinned) for `pin.cross`
in `pinned_toolchains.toml`."""
with PINNED_TOOLCHAINS.open("rb") as handle:
data = tomllib.load(handle)
versions = list(data.get(pin.cross, {}))
if not versions:
raise SystemExit(
f"{pin.name}: no versions pinned for {pin.cross!r} in "
f"{PINNED_TOOLCHAINS.relative_to(REPO)}"
)
return max(versions, key=_version_key), len(versions)


def _latest_github(repo: str) -> str:
data = json.loads(_get(GITHUB_LATEST.format(repo=repo)))
return str(data["tag_name"]).lstrip("v")
Expand Down Expand Up @@ -166,12 +196,16 @@ def _tarball_url(pin: Pin) -> str:

def check(pin: Pin, *, slow: bool) -> int:
"""0 when current, 1 when behind. Prints one line either way."""
pinned = _pinned(pin)
if pin.kind == "cross-toml":
pinned, count = _newest_cross_pin(pin)
else:
pinned = _pinned(pin)
try:
if pin.kind == "github":
if pin.kind in ("github", "cross-toml"):
latest = _latest_github(pin.upstream)
stale = pinned.lstrip("v") != latest
arrow = f"{pinned} -> {latest}" if stale else pinned
suffix = f" (newest of {count} pinned)" if pin.kind == "cross-toml" else ""
arrow = f"{pinned} -> {latest}{suffix}" if stale else f"{pinned}{suffix}"
elif pin.kind == "emsdk":
version, latest_hash = _latest_emsdk()
stale = pinned != latest_hash
Expand Down
37 changes: 37 additions & 0 deletions docs/records/0046-pin-staleness-checker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,47 @@ Still unbuilt from this record: nothing writes results anywhere but the job log
([0029]'s `stepsummary.py` is still unreused here), and the "are consumers' own
`micropython` pins in scope" question stays open.

## Addendum, 2026-09-07 — `update_toolchains.py`'s own `arm-none-eabi`/`riscv-none-elf`
## rows fixed; the weekly job had been crashing, not merely reporting drift

[0096]'s own text ("`bin/update_toolchains.py`'s `PINS`... a pre-existing gap... the real
fix... is [0090]'s own scope") turned out to point at the wrong record: [0090]'s item 1
fixed `bin/refresh_toolchain_pins.py`'s per-row `gcc` check, a different script for a
different question (is a row's pinned compiler inside its own floor/ceiling window). It
never touched `update_toolchains.py`, so the gap [0096] described — `_pinned()` regexing
an `ARG TOOLCHAIN_URL=` line that [0087]/[0089] had already deleted from
`embedded_base.Dockerfile` for both crosses — was still live and unclosed. Found live
while investigating a failing `pin-staleness.yml` run (2026-09-07): the "toolchain
tarballs" step raised `SystemExit("arm-none-eabi: no pin found...")` and crashed the whole
checker before it ever reached `riscv-none-elf`/`xtensa-esp`/`llvm-mingw`/`emsdk`/
`xtensa-lx106` — every scheduled run since [0087] landed had been reporting a hard
failure, not real staleness, with the other five pins never actually checked.

Fixed by making `PINS` read those two crosses from `resources/pinned_toolchains.toml`'s
own `[cross]` tables (the same file [0087]/[0089] actually moved the real pins into)
instead of a Dockerfile `ARG` that no longer exists. That table is a genuinely different
shape from the rest of `PINS`: it holds more than one verified version per cross at once
(`arm-none-eabi-` alone carries three, kept for different rows' own floor/ceiling
windows — [0088]'s `mimxrt` ceiling among them), so there is no single "the" pin left to
compare against upstream the way a Dockerfile `ARG` was. `check()` now reads every version
pinned for a cross and reports whether the *newest* has fallen behind upstream's latest
release, which is the question this checker can still honestly answer without mislabelling
an intentionally-kept older version as drift. Verified live: the script now runs to
completion and reports each of the six pins (three came back `UNKNOWN` on an unauthenticated
GitHub API rate limit in this sandbox, exactly like the two `github`-kind pins that were
never in question — not a regression this fix introduced), instead of crashing on the first.

Still open: `--slow`'s `xtensa-lx106` path, and everything this record's own "Still unbuilt"
paragraph above already named.

[0002]: 0002-delegate-compile-own-environment.md
[0010]: 0010-pinned-data-in-resources.md
[0013]: 0013-micropython-list-dedup-by-abi.md
[0029]: 0029-github-actions-job-summary.md
[0033]: 0033-cibuildmp-never-builds-docker-image-itself.md
[0044]: 0044-unix-native-images-landed.md
[0068]: 0068-docker-dependabot-grouping-and-mipsel-ubuntu-26-04.md
[0087]: 0087-arm-riscv-embedded-thin-out-toolchain-version-lands.md
[0088]: 0088-mimxrt-own-floor.md
[0090]: 0090-toolchain-pins-checker-and-0058-text-followup.md
[0096]: 0096-arm-riscv-embedded-collapse-into-embedded-base.md
10 changes: 10 additions & 0 deletions docs/records/0096-arm-riscv-embedded-collapse-into-embedded-base.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,3 +154,13 @@ digest that run's own "Record the pinned digest" step printed, replacing the int
(`refresh_toolchain_pins.py`/`update_toolchains.py`'s pre-existing gap, and
`plan_test_matrix.py`'s arithmetic-estimate weight) are unaffected by this addendum and stay
open, as [0090]'s own scope and a future re-measurement respectively.

## Addendum, 2026-09-07 — the "[0090]'s own scope" pointer above was wrong for half of it

[0090] closed `refresh_toolchain_pins.py`'s half of the gap this record names above (its own
item 1), but never touched `bin/update_toolchains.py`'s `PINS` — a different script, checking
a different question, that this record's own text conflated with it. That half sat unfixed
and, worse, silently crashing (`SystemExit` on the very first pin) every week `pin-staleness.yml`
ran since [0087] landed, rather than merely reporting stale drift the way [0046] intends. Fixed
in [0046]'s own 2026-09-07 addendum, not here — noted in this record only because this record's
own forward pointer is what sent a later reader looking in the wrong place.
28 changes: 14 additions & 14 deletions src/cibuildmp/resources/pinned_docker_images.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,33 +133,33 @@
[image_group]

# ── unix -- one native image per (arch, libc floor), identity-keyed ────
manylinux_2_28_x86_64 = "quay.io/pypa/manylinux_2_28_x86_64@sha256:94c816d38ad56d2dc1df2f28007d18d306b0f5d05096c6f6b9141345df504d3e"
musllinux_1_2_x86_64 = "quay.io/pypa/musllinux_1_2_x86_64@sha256:8900a53ed236d85edd6868387b3e3327c45774156519ad492fe8b1c2a434d1dc"
manylinux_2_28_x86_64 = "quay.io/pypa/manylinux_2_28_x86_64@sha256:53390351aeb4688114b02c36a23b3e6ce1166ee9b7afc5df1a4f776354fc764c"
musllinux_1_2_x86_64 = "quay.io/pypa/musllinux_1_2_x86_64@sha256:621f8004ed526a5a6bf6a866fb415ad8da54d59a991e50b3b69167c3a768a616"

manylinux_2_28_i686 = "quay.io/pypa/manylinux_2_28_i686@sha256:bce1c1f15a59f9b4aa3e8a82aab777f0dc987213dcae66609d25651c277e86c5"
musllinux_1_2_i686 = "quay.io/pypa/musllinux_1_2_i686@sha256:e9029eacb01a207fc991eed8abccf55f653900b4b83068e37fb0ea201176a64a"
manylinux_2_28_i686 = "quay.io/pypa/manylinux_2_28_i686@sha256:48790e05c01457c8a8daa0185e2c5e09d9684c017f930b3f2aa6d8afebe6784b"
musllinux_1_2_i686 = "quay.io/pypa/musllinux_1_2_i686@sha256:7ff9262769ecadb9b889d2977f2e1fd47400a161664502ecb0eda9291d9abdfb"

manylinux_2_28_aarch64 = "quay.io/pypa/manylinux_2_28_aarch64@sha256:45ea412cfbd2cc5e51323cecab614ea8e4e5522b4d4f668946c54056b40152f6"
musllinux_1_2_aarch64 = "quay.io/pypa/musllinux_1_2_aarch64@sha256:ab0391c77648e2b15d37e3e5b8c3d43a0c4a1ca0fb3ab2f60d252be3474d2975"
manylinux_2_28_aarch64 = "quay.io/pypa/manylinux_2_28_aarch64@sha256:ad74e53b713f3b07d8c889c526dc0c6500da9827b45e38739570875fef52e28f"
musllinux_1_2_aarch64 = "quay.io/pypa/musllinux_1_2_aarch64@sha256:4dffcd49f0b6fc6928a49915f3cd939f973bbecbdfe96e1e7926b6049bc0bad5"

manylinux_2_28_ppc64le = "quay.io/pypa/manylinux_2_28_ppc64le@sha256:c30b4161aaab2ad55d56e12d52e2cae1e5f1d981194e7db4d9d57ac2516807b4"
musllinux_1_2_ppc64le = "quay.io/pypa/musllinux_1_2_ppc64le@sha256:e745f8e8e8c7c8e02e6379502d0b73f7cc4fd95283e011a78063292740f3cf42"
manylinux_2_28_ppc64le = "quay.io/pypa/manylinux_2_28_ppc64le@sha256:f1efe96ed791399d23ad3d63ba2c9b20d06139203d1f3affad4534f44d53ee0b"
musllinux_1_2_ppc64le = "quay.io/pypa/musllinux_1_2_ppc64le@sha256:fbc1983eb04e9bf311bd743863472b3ac9840edc3f2ca4e9dd4085d26f24fc84"

manylinux_2_28_s390x = "quay.io/pypa/manylinux_2_28_s390x@sha256:2ff3207ba86c8fef999eb2e1139a612501501493f3fd47bbc8a71199a29e9de4"
musllinux_1_2_s390x = "quay.io/pypa/musllinux_1_2_s390x@sha256:411d3d433dde4b3a9fce85122f7080110e2ec3543ca6c22578acb4a61c38cc14"
manylinux_2_28_s390x = "quay.io/pypa/manylinux_2_28_s390x@sha256:d93d3fe69923df58ec8ca45a6e0bb1af915d62888f95baef6d7596e5d002646f"
musllinux_1_2_s390x = "quay.io/pypa/musllinux_1_2_s390x@sha256:a7254c085d6c28ae74acb229af697133a8827535ac26b542b7be014272ba81d2"

# armv7l's manylinux line starts at `_2_31` upstream (Ubuntu 20.04/apt,
# unlike the AlmaLinux 8/dnf `_2_28` images) -- `_2_31` is the lower,
# more compatible of the two floors pypa offers, and cibuildwheel makes
# the same default choice.
manylinux_2_31_armv7l = "quay.io/pypa/manylinux_2_31_armv7l@sha256:bb0c6355c62cd0971f17a66969b1bce15222e67e4064667e25030923bb216bfe"
musllinux_1_2_armv7l = "quay.io/pypa/musllinux_1_2_armv7l@sha256:bb213c77861583faa326b24de1c413dab0e7c51af76f8a30f5abcaf749da5265"
manylinux_2_31_armv7l = "quay.io/pypa/manylinux_2_31_armv7l@sha256:3503ad3cef1b2a35ee05048eaebdb93ad1b4a6e9426af7f43dbe7658243ab0db"
musllinux_1_2_armv7l = "quay.io/pypa/musllinux_1_2_armv7l@sha256:7fa1e5ac26e79b9aa6b64aac4ca2479eadd8d00229f178eb5495edd538e9ff4b"

# riscv64 is the newest arch upstream and carries the highest floor:
# `manylinux_2_39` (Rocky Linux 10/dnf) is the only manylinux image pypa
# publishes for it.
manylinux_2_39_riscv64 = "quay.io/pypa/manylinux_2_39_riscv64@sha256:4e74e5408c307666116196383a3bbf741b171fda049c776dac731e5b6087a1d5"
musllinux_1_2_riscv64 = "quay.io/pypa/musllinux_1_2_riscv64@sha256:3c2e2c26c10c2788046e54e816d8880011e4af9c4292cb735705a89870248082"
manylinux_2_39_riscv64 = "quay.io/pypa/manylinux_2_39_riscv64@sha256:000de3e1037325a6cc13615c1218061a56e1ece60ecdbe1a4572933a90654d18"
musllinux_1_2_riscv64 = "quay.io/pypa/musllinux_1_2_riscv64@sha256:f8bf1ede4138d95937a98362ce482453d717c62d22207f038619262b180d4ab8"

# mipsel is cibuildmp's own architecture, not one of cibuildwheel's
# seven, and the one documented exception to record 0043's model: pypa
Expand Down
Loading
Loading