Skip to content

Static typing in main.nf, nf-schema validation, mock-form CI, and a new-event script - #48

Merged
robsyme merged 10 commits into
mainfrom
cursor/types-schema-ci-improvements-8119
Oct 16, 2026
Merged

robsyme merged 10 commits into
mainfrom
cursor/types-schema-ci-improvements-8119

Conversation

@FloWuenne

Copy link
Copy Markdown
Collaborator

Targets Nextflow 26.10 (manifest.nextflowVersion = '!>=26.10.0'). This PR doesn't change where the form endpoint lives: event configs stay in event_configs/*.json, so #13 is untouched. Nothing in this PR or its CI submits to a real event form.

1. Static typing in main.nf

  • Added nextflow.enable.types = true. nextflow lint . is clean on 26.10.2 across all 8 scripts. The 11 errors it reported before are fixed.
  • Event config parsing: the new loadEventConfig() in utils.nf parses the JSON and validates it against the new assets/schema_event_config.json with nf-schema validate(). It then casts it with as EventConfig, which fails if a field is missing. Bad configs now fail with specific messages, e.g. /form_fields/uuid ("1234") does not match ^entry\.[0-9]+$, an unknown form_fields key, or an invalid required_fields value.
  • platform_enabled: the typed workflow namespace in 26.10 has no accessor for the session or the tower config. I checked the docs and the WorkflowNamespace source, and there's no public replacement. A config-file workaround wouldn't see -with-tower, because the CLI applies it after config parsing. So access is isolated in one helper, towerConfig() in utils.nf. It reads nextflow.Global.session.config['tower']. It passes type checking and gives the same value as before (tower.enabled ?: false), both for -with-tower and for nextflow auth login. nextflow.Global is the only internal API used. Verified by running against a mock Platform endpoint: the form receives platform_enabled=true and the "extra tickets" message prints.
  • PUBLISH_REPORT.out is replaced by an assigned output, which the typed workflow requires.

2. nf-schema params validation

  • nf-schema@3.0.0: validateParameters() runs first, and --help now uses paramsHelp().
  • An invalid --event stops the run before anything is submitted:
    * --event (nope): Expected any of [ashg_2026, biotechx_basel_2026, eccb_2026, fog_2026, ismb_bosc_2026]
  • Email check uses a pattern with a clear errorMessage. The old pattern rejected TLDs longer than 5 characters, so it's loosened.
  • Enum can't drift: the enum is generated from event_configs/*.json by scripts/new_event.py sync, and CI runs sync --check. To make that work, configs for past events (2024/2025, plus general) moved to event_configs/archive/. They were already not selectable from the schema.
  • Schema $id and title fixed. first_name, last_name and affiliation stay in required so the Seqera Platform launch form still marks them. Their empty defaults still pass on the CLI, and required_fields in each event config decides what's actually needed (checked).

3. Default event: ashg_2026

Before this PR, main.nf said eccb_2026, the schema said biotechx_basel_2026, and the README said ismb_bosc_2026 (README text) and eccb_2026. docs/INSTRUCTIONS.md said fog_2026. ASHG 2026 is the next upcoming event (October 2026) and was the most recently added config (#47). ECCB (2 Sept) and ISMB/BOSC (July) are over, and FOG was in January. ashg_2026 is now the default in main.nf, the schema, README, docs and CLAUDE.md. The examples include --first_name/--last_name/--affiliation, since ASHG requires them.

4. CI never hits a real form

  • The new test profile sets a hidden --event_config tests/test_event.json, which points at http://localhost:8765/formResponse. tests/mock_form_server.py (standard library only) records submissions and also stubs the Platform API.
  • CI steps:
    • nextflow lint .
    • sync --check
    • validate every event config against the schema
    • test the new-event script against a synthetic fixture form
    • run the pipeline and check the exact set of submitted fields and values, and that raffle_ticket.html has the event name and the run name used as ticket number
    • run with -with-tower http://localhost:8765, expecting platform_enabled=true, platform_workflow_id sent, and the unmapped workspace_id skipped
    • an invalid --event and a missing required field must both fail with zero new submissions
  • Removed the workflow_dispatch email/event inputs that posted to the live form. Nextflow is pinned to 26.10.2.

5. Staff-demo fields

  • ENTER_RAFFLE now builds a single field→value map and sends only fields that are mapped in form_fields and non-empty. Unmapped fields are skipped. Values are shell-quoted, so O'Brien $HOME arrives intact. uuid falls back to /proc/sys/kernel/random/uuid when uuidgen isn't installed.
  • Documented in README ("What gets submitted"), CLAUDE.md, and docs/ADDING_AN_EVENT.md, which lists the questions a form should include and which must never be Required.
  • Correction to the premise: the live ASHG 2026 form does have hostname, user_name, workspace_id and platform_workflow_id questions. Only ashg_2026.json was missing them. The entry IDs were read from the public viewform page with a GET; nothing was submitted. They're added in a separate commit (168ee08), so it's easy to drop if you'd rather not change ASHG's payload days before the event.

6. Cleanup

  • Removed the unused PRINT_ASCII_LOGO and CONGRATULATIONS modules and their assets (congratulations.txt, ismb_bosc2024/). Nothing referenced them.
  • README, docs/INSTRUCTIONS.md, docs/index.html and CLAUDE.md now match actual behaviour:
    • Nextflow 26.10 and Java 17
    • the default event
    • required fields
    • --help
    • the test workflow

7. Adding a new event

  • scripts/new_event.py add --event ashg_2027 --name "ASHG 2027" --url <link> [--winner-announcement ...] [--set-default] (standard library only).
    • A pre-filled link (answers typed as field names) maps entry IDs exactly.
    • A public viewform link is fetched read-only, and questions are matched by title.
    • Either way, the form's Required questions become required_fields. The script refuses forms where a question the pipeline can't always fill is Required, since Google Forms silently drops those entries.
    • It writes the config, then sync updates the schema enum and default, the main.nf default, and the README event table.
  • Run against the live ASHG 2026 viewform page, it reproduces ashg_2026.json exactly, plus the four demo mappings.
  • docs/ADDING_AN_EVENT.md covers:
    • the question table
    • form settings: Collect emails off, no sign-in or 1-response limit, responder access Anyone with the link, accepting responses
    • script usage and a finish-up checklist
    • a config reference

Validation

  • nextflow lint .: ✅ 8 files, no errors, with types on (26.10.2).
  • Local run with -profile test against the mock server produced results/raffle_ticket.html. Submitted fields were as expected.
  • An invalid --event, an invalid --email, an unknown param, and a missing required field each fail early with no submission.
  • I replayed all the CI workflow steps locally, and they pass.

Trade-offs / not done

  • towerConfig() relies on nextflow.Global (internal), because 26.10 has no typed accessor. It's isolated to one function.
  • -preview hangs on typed map operators in 26.10.2, so CI validates configs with tests/validate_event_configs.nf instead of previewing the pipeline.
  • The nf-schema --help footer suggests --showHidden/--helpFull. The typed params block rejects those flags. This is cosmetic and comes from nf-schema.
  • The script can't detect the "Collect email addresses" setting from the public page; the checklist covers it.
  • With --event_config set, the required-field error still names --event's value. This only affects the test-only param.
Open in Web Open in Cursor 

cursoragent and others added 10 commits October 16, 2026 14:42
…assets

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…s current events

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…th nf-schema

- nextflow.enable.types in main.nf; event configs are validated against
  assets/schema_event_config.json and cast to the EventConfig record
- tower config read via a single towerConfig() helper (the typed workflow
  namespace has no session accessor), preserving platform_enabled
- nf-schema 3.0.0 validateParameters() and --help
- default event unified to ashg_2026; require Nextflow 26.10
- hidden --event_config param and test profile for local mock-form runs

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…lues

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
… link

- 'add' builds event_configs/<event>.json from a pre-filled link or a public
  viewform page (read-only), deriving required_fields from Required questions
- 'sync' regenerates the --event enum/default in nextflow_schema.json and
  main.nf plus the README event table from event_configs/; --check for CI

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
The form already has hostname, user_name, workspace_id and
platform_workflow_id questions; entry IDs read from the public viewform page.

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…d nextflow lint

- test profile posts to tests/mock_form_server.py (also stubs the Platform API
  so -with-tower can be exercised with platform_enabled=true)
- checks submitted fields, the generated ticket, invalid --event and missing
  required fields, event config schema validation and event list drift

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…vent checklist

- default event ashg_2026, Nextflow 26.10, Java 17
- example commands include name/affiliation required by the default event
- docs/ADDING_AN_EVENT.md: form questions, form settings checklist,
  new_event.py usage and config reference

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
…launch form

Empty defaults still pass on the CLI; per-event requiredness is enforced by
required_fields in main.nf.

Co-authored-by: Florian Wuennemann <flowuenne@gmail.com>
- Restore workflow name 'Run nf-raffle Pipeline' and job id 'run-pipeline'
- Drop 'message' from the mock Platform reply so runs no longer print 'WARN: ok'
- Remove shebang from tests/validate_event_configs.nf
- Restore blank line in the onComplete warning and the docker example in CLAUDE.md
- Ignore requests.jsonl written by the local mock server

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@robsyme robsyme left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is mostly just a re-org. lgtm

@robsyme
robsyme merged commit 856fb1c into main Oct 16, 2026
@FloWuenne
FloWuenne deleted the cursor/types-schema-ci-improvements-8119 branch October 16, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants