Skip to content

CI: add shared fix-ubuntu-ca-roots action to distrust compromised CA roots - #124

Merged
taoeffect merged 10 commits into
mainfrom
122-cas
Nov 4, 2026
Merged

taoeffect merged 10 commits into
mainfrom
122-cas

Conversation

@taoeffect

Copy link
Copy Markdown
Member

Closes #122.

Adds a shared composite action, .github/actions/fix-ubuntu-ca-roots/, taken from okTurtles/forkana#502 and made generic. It does four things:

  • Upgrades ca-certificates and libnss3.

  • Removes emSign ECC G3, emSign ECC C3, emSign C1 and Comodo AAA from the system CA bundle. If any of them is still there, the step fails.

  • Exports DENO_TLS_CA_STORE=system and SSL_CERT_FILE so that Deno and uv use the fixed bundle.

  • With sync-nss-db: true, it can also update the NSS database for jobs that run Chromium or Playwright.

  • ci.yml and pull-review-crush.yml run the action as the first step, through okTurtles/libcheloniajs/...@main, and set NODE_OPTIONS: --use-openssl-ca at workflow level.

  • pull-review-crush.yml:

    • Passes NODE_OPTIONS through env -i and bwrap.
    • Posts the failure comment for every failure, including a failure in the CA step.
    • Widens the timeout-detection slack from 60s to 300s, because the apt upgrade now runs before JOB_STARTED_AT is set.
  • AGENTS.md: new CI section listing the workflows and the rules for callers.

AI Notes

Opus 5.5:xhigh used to write and review.

Warnings

  1. CI on this PR will fail in "Set up job". ci.yml loads the action from @main, and the action is not on main yet: gh api .../contents/.github/actions?ref=main returns 404. CI should pass after the merge. If you want green CI before merging, land the .github/actions/ folder in a separate PR first.
  2. Running /review on this PR will not test these changes. issue_comment workflows run from the default branch, so /review uses the old pull-review-crush.yml on main.
  3. Other repos change as soon as this merges. Every repo that calls pull-review-crush.yml@main will start running the action, including its apt-get update/upgrade, which makes jobs a little slower.
  4. The upstream source is not merged yet. forkana#502 is still open. This copy matches its latest head, 8925fb1e.
  5. Not tested on real GitHub runners. It was tested with actionlint, shellcheck and an ubuntu:24.04 container on arm64, with curl, Node, Deno and NSS checks. npm run build, npm run lint and npm test all pass (476/476). It has not run on a real runner (that needs the action on main), on amd64, or with Chromium.

Adopted from okTurtles/forkana PR #502 at 8925fb1e4298, with a generic header
for all okTurtles repos (caller contract, @main reference) and a new step that
sets DENO_TLS_CA_STORE=system through GITHUB_ENV. The pem is unchanged.
Widen the timeout-report slack to 300 s. JOB_STARTED_AT is now set after
fix-ubuntu-ca-roots, so a real timeout could be skipped as a cancellation.
Export SSL_CERT_FILE so uv (before 0.13.4) and httpx use the fixed
system bundle, and name the remaining gap (requests with an old certifi)
in the caller contract.
Limit the CI rules to jobs that have steps. A job that only calls a

reusable workflow (pr-review.yml) needs neither the action nor NODE_OPTIONS.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failure reporting can expose GH_TOKEN through an authenticated request before CA hardening succeeds, so it must be gated on successful hardening while CA-setup failures remain in Actions logs.

1 open issue
What changed in this PR

Adds centralized CA hardening for Ubuntu CI runners, addressing #122 and providing a shared action for other okTurtles repositories.

Changes:

  • Upgrades certificate packages, removes four distrusted roots, and optionally synchronizes Chromium’s NSS database.
  • Integrates the action into CI and review workflows, including Node trust-store configuration.
  • Updates failure reporting, timeout detection, and caller documentation.
File Description
AGENTS.md Documents CI workflows and shared-action usage rules.
.github/​workflows/​pull-review-crush.yml Adds CA hardening, propagates Node settings, and adjusts failure handling.
.github/​workflows/​ci.yml Runs CA hardening before checkout and configures Node’s trust store.
.github/​actions/​fix-ubuntu-ca-roots/​distrusted-ca-roots.pem Supplies four root certificates for explicit NSS distrust.
.github/​actions/​fix-ubuntu-ca-roots/​action.yml Implements system CA cleanup, runtime configuration, and optional NSS synchronization.

Review effort: Balanced

Comment thread .github/workflows/pull-review-crush.yml Resolved
corrideat
corrideat previously approved these changes Nov 4, 2026

@corrideat corrideat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Address Copilot review on PR #124. When fix-ubuntu-ca-roots fails, the
CA bundle can still trust a removed root, so "Post failure comment" and
"Report timeout on trigger comment" must not send GH_TOKEN through it.
Add the rule to the caller contract in the action header.

@corrideat corrideat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@taoeffect
taoeffect merged commit 287765e into main Nov 4, 2026
5 of 6 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

This security-sensitive TLS trust change affects all @main consumers, but PR CI cannot exercise it, requiring staged human validation of GitHub-hosted runner and Chromium/NSS behavior before rollout.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

taoeffect added a commit to okTurtles/forkana that referenced this pull request Nov 4, 2026
okTurtles/libcheloniajs#124 made this action shared by all okTurtles
repos. Its copy does everything ours did, and also makes Deno and uv use
the fixed system bundle (uv runs in pull-compliance). So delete the local
copy and call okTurtles/libcheloniajs/.github/actions/fix-ubuntu-ca-roots@main
in all 36 jobs. The action is on libcheloniajs main already, so the CI of
this branch can load it.

Follow the caller rules in the header of the shared action:
- Skip post-failure steps that send a token when the CA step did not
  succeed: the e2e artifact upload, and the failure comment and timeout
  report in pull-review-crush.
- pull-review-crush: post the failure comment for every failure after the
  CA step, as libcheloniajs does, and widen the timeout-detection slack
  from 60s to 300s, because JOB_STARTED_AT is set after the apt upgrade.

AGENTS.md lists the rules for workflows.
taoeffect added a commit to taoeffect/oh-my-pi that referenced this pull request Nov 4, 2026
…r CA bundle

Every Linux job of release-taoeffect.yml and natives-taoeffect.yml now starts with okTurtles/libcheloniajs/.github/actions/fix-ubuntu-ca-roots@main instead of the inline step that upgraded ca-certificates and libnss3 and removed the eMudhra and Comodo AAA roots. The action does the same work and also makes Deno and uv use the fixed bundle.

Both workflows set NODE_OPTIONS=--use-openssl-ca, as the action requires, so Node (JavaScript actions, npm) uses the fixed system bundle instead of its built-in roots. The macOS smoke step still cleans /etc/ssl/cert.pem, which the action does not cover, and now exports SSL_CERT_FILE so Node on macOS reads that bundle.

okTurtles/libcheloniajs#124
https://emudhra.we-have-a-problem.com/#linux
taoeffect added a commit to okTurtles/forkana that referenced this pull request Nov 6, 2026
…502)

* CI: remove compromised eMudhra and Comodo AAA roots on every runner

Add the composite action .github/actions/fix-ubuntu-ca-roots. It upgrades
ca-certificates and libnss3, removes emSign ECC Root CA - G3, emSign ECC
Root CA - C3, emSign Root CA - C1 and Comodo AAA from the system bundle,
and fails the job if one of them is still trusted. With sync-nss-db, it
also updates the NSS database that Playwright's Chromium reads.

Every job calls it first, as okTurtles/forkana/...@main. The e2e job sets
NODE_OPTIONS=--use-openssl-ca so that Node uses the fixed system bundle.

https://emudhra.we-have-a-problem.com/

* CI: address review of the CA root fix

- Call the action at @master. origin has no main branch, so @main does
  not resolve.
- Set NODE_OPTIONS=--use-openssl-ca in every workflow, so Node (the
  runner's own Node for JavaScript actions too) uses the fixed system
  bundle and not its built-in CA list. Pass it through the cleared
  environments in pull-review-crush (env -i and bwrap --clearenv). The
  action cannot set it, because GitHub blocks NODE_OPTIONS in GITHUB_ENV.
- Parse the bundle before the check, so the step fails when openssl
  cannot read the bundle.
- Do not call emSign ECC Root CA - C3 and emSign Root CA - C1
  compromised, and give the reason for removing them.

* CI: use the shared fix-ubuntu-ca-roots action from libcheloniajs

okTurtles/libcheloniajs#124 made this action shared by all okTurtles
repos. Its copy does everything ours did, and also makes Deno and uv use
the fixed system bundle (uv runs in pull-compliance). So delete the local
copy and call okTurtles/libcheloniajs/.github/actions/fix-ubuntu-ca-roots@main
in all 36 jobs. The action is on libcheloniajs main already, so the CI of
this branch can load it.

Follow the caller rules in the header of the shared action:
- Skip post-failure steps that send a token when the CA step did not
  succeed: the e2e artifact upload, and the failure comment and timeout
  report in pull-review-crush.
- pull-review-crush: post the failure comment for every failure after the
  CA step, as libcheloniajs does, and widen the timeout-detection slack
  from 60s to 300s, because JOB_STARTED_AT is set after the apt upgrade.

AGENTS.md lists the rules for workflows.

* CI: skip the Codecov upload when the CA fix fails

The upload runs with always(). codecov-action downloads the Codecov CLI,
its checksum, its signature and the PGP key that verifies them with
curl, which uses the system CA bundle. Then it runs the CLI. If
fix-ubuntu-ca-roots fails, that bundle can still trust a removed root,
so a MITM could supply a CLI that passes the check.

When the CA step fails, the tests do not run, so there is no coverage to
upload. Coverage still uploads after a test failure.

* remove 1 sentence from agents

This branch had an error being deployed

1 failed deployment
CI — 2e7aaeb1 Deployed Nov 4, 2026 by taoeffect via build #312
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt compromised CA fixes for github workflow

3 participants