Request for technical feedback on reproducible ML-DSA cross-implementation verification

[mokkunsuzuki-code]

Hello Open Quantum Safe community,

I am looking for technical feedback on a narrowly scoped ML-DSA independent-reproduction workflow.

I maintain an open verification project called QSP.

Current Stage391 repository:

https://github.com/mokkunsuzuki-code/stage391

Public verification page:

https://mokkunsuzuki-code.github.io/stage391/

This is not a vulnerability report, certification request, or request for endorsement.

The part I would most value OQS feedback on is the ML-DSA verification and cross-implementation reproduction path developed in the preceding QSP stages and carried into Stage391.

The current work includes:

• ML-DSA signature verification evidence,
• independent verifier / multiple-implementation checks,
• deterministic reproduction,
• exact SHA-256 evidence binding,
• explicit environment metadata,
• Fail-Closed rejection of contradictory assessment claims,
• separation between developer self-tests and genuine independent third-party reproduction.

Exact Stage391 assessment commit:

a74b7f47f8caabf37a910ea9f4da2b87c0cd5f15

Canonical Stage391 result SHA-256:

ed644d11bd49f67f89cfda50364d619066b4da3a36bf1fb26b38e111b6092b23

The authoritative Stage391 state is intentionally:

third_party_submission_pending

verification_status:

waiting_for_external_submission

No independent external assessment has been accepted yet.

My question is deliberately limited:

From the perspective of people familiar with liboqs / ML-DSA implementations, is this type of cross-implementation and reproducible verification methodology technically meaningful?

I would especially appreciate any of the following:

1. identification of an invalid ML-DSA assumption,
2. identification of a missing interoperability test,
3. identification of a reproducibility weakness,
4. an independent reproduction of the ML-DSA-related path,
5. advice on a better way to compare implementations or verification outputs,
6. or direction to a more appropriate OQS venue or maintainer.

A full review of QSP is not required.

Negative results are fully acceptable. A disagreement, mismatch, or concrete methodological criticism is as useful as agreement.

A self-contained assessor handoff package is prepared, with:

• exact commit binding,
• SHA-256 manifest,
• environment schema,
• reproduction result schema,
• agreement / disagreement / incomplete outcome format.

I am not attaching the archive unsolicited, but I can provide it if someone wants to reproduce the evidence.

One upstream state also remains intentionally unresolved:

Stage389 = dual_timestamp_pending

Stage389 records RFC3161 verification as verified, while OpenTimestamps / Bitcoin verification remains pending.

QSP does not promote that pending state to success.

Related external requests already opened:

mldsa-native:
pq-code-package/mldsa-native#1393

PQCA Readiness Tracking WG:
PQCA/wg-readiness-tracking#41

Thank you for any technical criticism, reproduction result, or direction.

Best regards,
Motohiro Suzuki

[Kxrma47]

The methodology is meaningful as an evidence-preservation and reproducibility workflow, but it should be careful not to equate “two implementations produced the same result” with an independent validation of ML-DSA correctness.

For ML-DSA, I would separate four claims and test each explicitly:

1. Known-answer conformance: run the official ACVP/NIST vectors for key generation, signing, and verification, including deterministic and hedged signing where supported.
2. Cross-implementation interoperability: generate keys and signatures in implementation A and verify in B, then reverse the direction for every supported parameter set. Include ML-DSA-44, 65, and 87 rather than only one profile.
3. Negative behavior: mutate the public key, message, context, signature hint, challenge, and length fields; verification must reject. Include truncated/extended encodings and non-canonical values. Agreement on valid fixtures alone misses a large class of parser and validation errors.
4. Reproduction integrity: pin source commits, compiler/toolchain, flags, CPU architecture, dependencies, test-vector revision, and container/base-image digest. Hashes prove which bytes were assessed, but the manifest must also describe how those bytes were produced.

Two additional assumptions deserve explicit treatment:

• ML-DSA contexts are part of the signed input. Test empty, maximum-length, and mismatched contexts where the APIs expose them.
• Randomized/hedged signing should not be compared by expecting identical signatures. Compare successful cross-verification and record the randomness mode; byte equality is appropriate only for a deterministic, identically specified path.

For independence, avoid counting two wrappers around the same underlying implementation as two implementations. Record the implementation lineage (for example, whether both ultimately call the same native code). A stronger matrix uses separately maintained codebases and includes at least one implementation not derived from the same upstream source.

The fail-closed state model and explicit third_party_submission_pending status are good practices. I would add a machine-readable claim table in which every claim points to its exact command, inputs, output, exit status, and artifact hashes. That makes partial reproduction possible without forcing an assessor to endorse the entire project.

Finally, RFC 3161/OpenTimestamps evidence establishes ordering or existence of artifacts, not cryptographic correctness. Keeping timestamp status separate from ML-DSA conformance status, as you describe, is the correct boundary.

Relevant upstream starting points:

• https://csrc.nist.gov/pubs/fips/204/final
• https://github.com/open-quantum-safe/liboqs
• https://github.com/pq-code-package/mldsa-native

The most useful next deliverable would be a small public interoperability corpus: versioned keys/messages/contexts/signatures, expected accept/reject outcomes, provenance for each producer, and a runner that executes the same corpus against every implementation.

[mokkunsuzuki-code]

Thank you — this is extremely helpful. The distinction between agreement and correctness is especially important. I’ll make sure not to treat two implementations producing the same result as independent validation of ML-DSA correctness. I’m going to separate the work into the four claims you outlined: known-answer conformance, cross-implementation interoperability, negative behavior, reproduction integrity. I also want to incorporate the other boundaries you pointed out: implementation lineage, context handling, deterministic vs. hedged signing, and keeping timestamp evidence separate from ML-DSA conformance. The machine-readable claim table is particularly useful. I like the idea that an assessor should be able to reproduce and agree or disagree with one specific claim without having to endorse the whole project. I’ll also work toward the small public interoperability corpus you suggested, with versioned inputs, expected accept/reject outcomes, producer provenance, and a common runner. One question as I start designing the corpus: do you think it would be better to keep the format implementation-neutral, and reference ACVP/NIST vectors as one conformance source, rather than structuring the corpus itself around the ACVP format? Thanks again for taking the time to give such detailed feedback.

…

2026/08/31 8:42、Mahidul Haque ***@***.***>のメール: The methodology is meaningful as an evidence-preservation and reproducibility workflow, but it should be careful not to equate “two implementations produced the same result” with an independent validation of ML-DSA correctness. For ML-DSA, I would separate four claims and test each explicitly: Known-answer conformance: run the official ACVP/NIST vectors for key generation, signing, and verification, including deterministic and hedged signing where supported. Cross-implementation interoperability: generate keys and signatures in implementation A and verify in B, then reverse the direction for every supported parameter set. Include ML-DSA-44, 65, and 87 rather than only one profile. Negative behavior: mutate the public key, message, context, signature hint, challenge, and length fields; verification must reject. Include truncated/extended encodings and non-canonical values. Agreement on valid fixtures alone misses a large class of parser and validation errors. Reproduction integrity: pin source commits, compiler/toolchain, flags, CPU architecture, dependencies, test-vector revision, and container/base-image digest. Hashes prove which bytes were assessed, but the manifest must also describe how those bytes were produced. Two additional assumptions deserve explicit treatment: ML-DSA contexts are part of the signed input. Test empty, maximum-length, and mismatched contexts where the APIs expose them. Randomized/hedged signing should not be compared by expecting identical signatures. Compare successful cross-verification and record the randomness mode; byte equality is appropriate only for a deterministic, identically specified path. For independence, avoid counting two wrappers around the same underlying implementation as two implementations. Record the implementation lineage (for example, whether both ultimately call the same native code). A stronger matrix uses separately maintained codebases and includes at least one implementation not derived from the same upstream source. The fail-closed state model and explicit third_party_submission_pending status are good practices. I would add a machine-readable claim table in which every claim points to its exact command, inputs, output, exit status, and artifact hashes. That makes partial reproduction possible without forcing an assessor to endorse the entire project. Finally, RFC 3161/OpenTimestamps evidence establishes ordering or existence of artifacts, not cryptographic correctness. Keeping timestamp status separate from ML-DSA conformance status, as you describe, is the correct boundary. Relevant upstream starting points: https://csrc.nist.gov/pubs/fips/204/final https://github.com/open-quantum-safe/liboqs https://github.com/pq-code-package/mldsa-native The most useful next deliverable would be a small public interoperability corpus: versioned keys/messages/contexts/signatures, expected accept/reject outcomes, provenance for each producer, and a runner that executes the same corpus against every implementation. — Reply to this email directly, view it on GitHub <#2533?email_source=notifications&email_token=BYN6ZAWUP2CTLXGILDJBBU35MS3VJA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBSGEYDSNBVUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-18210945>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BYN6ZAS373SUDCJKJLPA2AD5MS3VJAVCNFSNUABHKJSXA33TNF2G64TZHM3DKNJRGQZDANJ3IRUXGY3VONZWS33OHMYTANRWGAZDQNNBOYBA>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BYN6ZATKGQQEQNUUCQ7QF6D5MS3VJA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBSGEYDSNBVUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVJTG633UMVZF62LPOM> and Android <https://github.com/notifications/mobile/android/BYN6ZASBTTYKONBP6ULTD2L5MS3VJA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBSGEYDSNBVUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today! You are receiving this because you authored the thread.

[Kxrma47]

Yes—keep the corpus format implementation-neutral, and treat ACVP/NIST vectors as one provenance-tagged conformance source rather than making the whole corpus an ACVP-shaped wrapper.

I would give each case a small stable envelope containing: schema version; algorithm and parameter set; operation; message, key, signature, and context bytes in an explicit encoding; deterministic or hedged mode; expected accept/reject result; producer implementation and pinned commit; source/provenance; and hashes of every artifact. Preserve imported ACVP vectors unchanged and reference their source identifiers, with a separate adapter that maps them into this envelope. That makes it possible to verify the mapping and avoids silently changing the authoritative vector.

Then keep runner adapters implementation-specific. Each adapter consumes the same neutral case and emits a normalized result containing the implementation version, command, exit status, verification outcome, and output hashes. Negative cases and cross-produced signatures fit the same schema, while ACVP-specific metadata can live in an optional provenance object rather than becoming mandatory for every case.

This separation matters because ACVP answers “does this implementation conform to these standardized vectors?”, while the shared corpus also needs to express interoperability, malformed-input rejection, implementation lineage, and reproduction metadata. It also lets you add Wycheproof or project-generated regression cases later without revising the core format. liboqs itself follows a similar separation by consuming NIST ACVP vectors in its test harness rather than defining its public API around ACVP: https://github.com/open-quantum-safe/liboqs and https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-files.

I would version the neutral schema independently, publish a JSON Schema plus a few canonical examples, and require that adapters preserve the original fixture hash. That gives assessors a compact common contract without confusing corpus interoperability with NIST conformance.

[mokkunsuzuki-code]

Thank you again for the detailed feedback. I went back through your points and tried to implement the separation you described rather than turning the corpus into an ACVP-shaped wrapper. The substantive changes are now public here: mokkunsuzuki-code/qsv-mldsa-interoperability-corpus@5d6c0ca The corpus now has an append-only neutral fixture v0.3 layer with: - an implementation-neutral JSON Schema; - deterministic offline mapping from pinned NIST ACVP prompt.json / expectedResults.json inputs; - positive and negative ACVP canonical examples; - a negative Wycheproof canonical example; - explicit source-case and neutral-fixture canonicalization rules; - pinned upstream source identifiers and SHA-256 bindings; and - fail-closed checks for malformed or tampered metadata. I kept ACVP-specific information in provenance rather than making it mandatory in the neutral format. I also kept the fixture model separate from the execution/result layer. The existing runner interface and normalized-result machinery remain separate artifacts; I am not claiming that the new v0.3 neutral fixture is already consumed directly by implementation-specific runners. The existing F7 runtime and normalized-result evidence were not rewritten, and no new ML-DSA execution was performed for this change. Current main is: mokkunsuzuki-code/qsv-mldsa-interoperability-corpus@d49dee4 That follow-up commit is documentation-only and just corrects the README from “locally defined” to “publicly available in this repository.” I have deliberately kept the claims narrow: this is not NIST validation, FIPS 204 certification, complete ML-DSA conformance, or third-party independent reproduction. The part I would most value your view on is the boundary itself: Does the separation now look consistent with what you had in mind — ACVP and Wycheproof as provenance-tagged sources, a stable implementation-neutral fixture, and a separate implementation-specific execution/result layer? Before I implement the runner adapters against the v0.3 neutral fixture, is there anything in the current envelope or source-binding model that you would change? Even one concrete example would be very helpful.

…

2026/09/07 2:33、Mahidul Haque ***@***.***>のメール: Yes—keep the corpus format implementation-neutral, and treat ACVP/NIST vectors as one provenance-tagged conformance source rather than making the whole corpus an ACVP-shaped wrapper. I would give each case a small stable envelope containing: schema version; algorithm and parameter set; operation; message, key, signature, and context bytes in an explicit encoding; deterministic or hedged mode; expected accept/reject result; producer implementation and pinned commit; source/provenance; and hashes of every artifact. Preserve imported ACVP vectors unchanged and reference their source identifiers, with a separate adapter that maps them into this envelope. That makes it possible to verify the mapping and avoids silently changing the authoritative vector. Then keep runner adapters implementation-specific. Each adapter consumes the same neutral case and emits a normalized result containing the implementation version, command, exit status, verification outcome, and output hashes. Negative cases and cross-produced signatures fit the same schema, while ACVP-specific metadata can live in an optional provenance object rather than becoming mandatory for every case. This separation matters because ACVP answers “does this implementation conform to these standardized vectors?”, while the shared corpus also needs to express interoperability, malformed-input rejection, implementation lineage, and reproduction metadata. It also lets you add Wycheproof or project-generated regression cases later without revising the core format. liboqs itself follows a similar separation by consuming NIST ACVP vectors in its test harness rather than defining its public API around ACVP: https://github.com/open-quantum-safe/liboqs and https://github.com/usnistgov/ACVP-Server/tree/master/gen-val/json-files. I would version the neutral schema independently, publish a JSON Schema plus a few canonical examples, and require that adapters preserve the original fixture hash. That gives assessors a compact common contract without confusing corpus interoperability with NIST conformance. — Reply to this email directly, view it on GitHub <#2533?email_source=notifications&email_token=BYN6ZAS3FEEFXGEWZEYJWMT5NWNVNA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBTGE4TGNJXUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-18319357>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BYN6ZARUNJIRHWZ6TROJAWD5NWNVNAVCNFSNUABHKJSXA33TNF2G64TZHM3DKNJRGQZDANJ3IRUXGY3VONZWS33OHMYTANRWGAZDQNNBOYBA>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/BYN6ZAUMTQN4NX6GS4UH4GL5NWNVNA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBTGE4TGNJXUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVJTG633UMVZF62LPOM> and Android <https://github.com/notifications/mobile/android/BYN6ZAUX5VFF4RLHZDG63G35NWNVNA5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCOBTGE4TGNJXUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today! You are receiving this because you authored the thread.

[Kxrma47]

I checked the v0.3 schema and verifier. Yes—the separation is consistent with what I meant: ACVP/Wycheproof are provenance-tagged sources, the fixture is implementation-neutral, and runner results remain a separate layer.

One gate to make explicit before wiring adapters: verify_fixture() checks fixture hashes and declared source metadata, but actual regeneration from the pinned ACVP/Wycheproof files occurs only when all three source paths are supplied. The default run reports SOURCE_REGENERATION_PERFORMED=NO. Therefore a default PASS is metadata/fixture validation, not source-byte verification. Require source regeneration for any result called source-bound, and carry that gate's status and source-file digests into the normalized result. For example, changing a pinned source vector while leaving the published fixture unchanged must fail that gate before an adapter result is accepted.

When adapters are added, I would also constrain the currently free-form consumer field to identify the implementation, commit, and lineage. This is a format/source-binding review, not a new ML-DSA execution or cryptographic certification.

[mokkunsuzuki-code]

Thank you — that distinction is very helpful. I agree that the current default PASS should remain a metadata / fixture-integrity validation when SOURCE_REGENERATION_PERFORMED=NO, and should not be treated as source-byte verification. Before wiring any adapters, I’ll make source regeneration an explicit fail-closed requirement for any result marked source-bound: require the exact pinned source files, verify their digests, regenerate the relevant source cases and neutral fixtures, and confirm the regenerated fixture against the published one. I’ll also carry the regeneration status and verified source-file digests into the normalized result. I’ll use the source-mutation case you suggested as a required negative test, so a changed pinned source file with an unchanged published fixture must fail the source-bound gate before an adapter result can contribute to any result classified as source-bound. I’ll also constrain consumer to identify the implementation, exact version/commit, adapter identity, and implementation lineage. I’ll keep these changes limited to format/source binding, with no new ML-DSA execution implied. Thank you again — this gives me a clear acceptance boundary before adapter work. *Edit note: Removed automatically quoted GitHub notification email content. Technical response unchanged.*