Skip to content

feat: add catalog and identity linking conformance tests - #125

Open
evoleinik wants to merge 1 commit into
Universal-Commerce-Protocol:mainfrom
evoleinik:feat/catalog-identity-linking-tests
Open

evoleinik wants to merge 1 commit into
Universal-Commerce-Protocol:mainfrom
evoleinik:feat/catalog-identity-linking-tests

Conversation

@evoleinik

@evoleinik evoleinik commented Nov 9, 2026 •

Copy link
Copy Markdown

Description

The suite has no tests for catalog or identity linking. This PR adds 12. It follows up on my comment on #120.

catalog_test.py has 8 tests for search, lookup and get product over the REST binding. identity_linking_test.py has 4. They read the authorization metadata and send one request with a made-up token. They never run an authorization flow or need an account.

Each assertion rests on one of three things: a MUST sentence in the specification or in an RFC it cites, an item of the REST binding's Conformance list, or a constraint in the schemas. The table below names the rule behind each test. The rules are the same in the 2026-04-08 and 2026-08-25 releases. The one exception is the oauth2 provider rule, which is new in 2026-08-25 and applies only to a profile that lists such a provider.

Test Rule
test_search_returns_products_in_ucp_envelope REST Conformance 1. Overview, Business Requirements 4: every response MUST include ucp with version and the active capabilities. The search response schema requires ucp and products
test_search_cursor_returns_next_page REST Conformance 3. Pagination schema: cursor MUST be present when has_next_page is true
test_search_variants_carry_valid_prices REST Conformance 2. Schemas: a product has at least one variant, a variant requires price, and a Price has an integer amount of at least 0 and a currency matching ^[A-Z]{3}$
test_lookup_unknown_id_returns_no_product REST Conformance 4. Lookup: variants without an inputs entry MUST NOT appear
test_lookup_dedupes_ids_and_correlates_inputs Lookup: MUST support lookup by product ID and variant ID. Duplicate identifiers MUST be deduplicated. A product MUST be returned once. Variants without an inputs entry MUST NOT appear
test_lookup_over_batch_limit_returns_request_too_large REST Conformance 5
test_get_product_endpoint_is_available REST Conformance 1: both /catalog/lookup and /catalog/product MUST be available. The get product response schema requires product
test_get_product_by_variant_id_features_that_variant Get Product: for a variant ID, the requested variant MUST be the first element
test_protected_resource_metadata_matches_its_resource RFC 9728 §3.3: resource MUST be identical to the identifier the metadata URL was built from
test_authorization_server_metadata_issuer_matches RFC 8414 §3.3, repeated in Identity linking, Discovery: issuer MUST match byte for byte
test_authorization_server_metadata_declares_scopes_and_auth For Businesses: MUST populate scopes_supported. MUST declare token_endpoint_auth_methods_supported. With an oauth2 provider, MUST include the JWT bearer grant in grant_types_supported
test_rejected_token_on_gated_operation_gets_bearer_challenge Error Handling, identity_required: a gated operation MUST answer an invalid token with 401, a Bearer challenge and an identity_required message. The header MUST include realm set to the issuer, and MUST include error="invalid_token"

Each test skips when the server does not advertise its capability. The catalog tests send no user token, so they also skip a capability gated by a scope in config.scopes. Three lookup tests find their product by search, so they also need the search capability. A test that needs a product skips until the merchant sets catalog.search_query. The batch test skips until the merchant sets catalog.max_lookup_ids. The token test needs a gated operation: identity_linking.request, or a catalog search when the profile gates it. The README documents these keys.

Other changes:

  • shopping-agent-test.json declares dev.ucp.shopping.catalog.search and dev.ucp.shopping.catalog.lookup. A business computes the intersection of its capabilities and the platform's (Overview, Business Requirements 2), so without them the suite's agent gets no catalog capability. fix: declare exercised capabilities in agent profile #109 added the shopping capabilities for the same reason.
  • runner.py adds shopping.catalog.catalog_test to the shopping suite and common.identity_linking.identity_linking_test to the common suite. The packages shopping/catalog/ and common/identity_linking/ are new.
  • framework/services.py reads three things from the profile: a service's REST endpoint, a capability's config, and the gated scopes. It takes the REST entry even when that entry is not listed first.
  • ietf joins the cspell word list, for the JWT bearer grant URN.

ruff check, ruff format, prettier and cspell pass on the changed files.

Results

Server Result
Python sample server, samples 52319df, --suite=all Main 0642adf: 81 passed and 2 skipped. With this PR, the same 81 pass, and the 12 new tests skip because the sample has no catalog or identity linking
Our UCP server, a catalog with identity linking 10 pass. The price test fails. The token test skips because no operation is gated
Our UCP server, a catalog only 6 pass and the price test fails. 5 skip: the 4 identity linking tests, and the batch test, which had no limit set

The price test fails because the catalogs our server carries publish no prices. The spec requires a price on every variant today. Universal-Commerce-Protocol/ucp#969 and Universal-Commerce-Protocol/ucp#1042 propose a way to mark a price that is not published. So the failure is the expected result for these catalogs.

Our server fetches the agent profile named in UCP-Agent and refuses a localhost URL. So I ran it on #123's branch with --agent_profile_url, using the same test files as this PR. Nothing in this PR needs #123 to run against a local server.

Each test was also run against defects planted for it. A local proxy sat in front of the first configuration and changed one thing at a time. Each defect failed the tests listed. The price test kept failing in every row, because of the missing prices above.

Planted defect Tests that failed
a lookup returns the same product twice test_lookup_dedupes_ids_and_correlates_inputs
a lookup variant loses inputs, repeats an entry, loses the product-id entry, or carries the variant-id entry under another variant id test_lookup_dedupes_ids_and_correlates_inputs (each one separately)
a lookup for an unknown id returns a product test_lookup_unknown_id_returns_no_product
has_next_page true with no cursor test_search_cursor_returns_next_page
search drops ucp.capabilities test_search_returns_products_in_ucp_envelope, test_search_cursor_returns_next_page
every variant gets a price with currency usd, where the same price with USD passes test_search_variants_carry_valid_prices
get product returns no product test_get_product_endpoint_is_available, test_get_product_by_variant_id_features_that_variant
get product by variant id puts another variant first test_get_product_by_variant_id_features_that_variant
the batch error code is invalid_request test_lookup_over_batch_limit_returns_request_too_large
protected-resource resource differs from its URL test_protected_resource_metadata_matches_its_resource
metadata issuer gains a trailing slash test_authorization_server_metadata_issuer_matches
metadata drops scopes_supported, or drops token_endpoint_auth_methods_supported test_authorization_server_metadata_declares_scopes_and_auth (each one separately)
the profile lists an oauth2 provider, and no metadata has the JWT bearer grant test_authorization_server_metadata_declares_scopes_and_auth
authorization server metadata answers 500 test_authorization_server_metadata_issuer_matches, test_authorization_server_metadata_declares_scopes_and_auth
the profile gates catalog search, and search accepts the made-up token test_rejected_token_on_gated_operation_gets_bearer_challenge
the profile gates catalog search, and the 401 challenge drops error="invalid_token", has another realm, or has no realm test_rejected_token_on_gated_operation_gets_bearer_challenge (each one separately)

These changes should fail nothing new, and none did:

Change Result
every variant gets a valid price, as an integer or as 100.0 the price test passes, and so does everything else that ran
the batch error is a flat {"code": "request_too_large", ...} body the batch test passes
the profile gates catalog search, and search answers with the server's real 401 the token test passes, and the 6 tests that need an anonymous search skip
the protected-resource metadata lists no authorization_servers the protected-resource test passes, and the 2 tests that need an issuer skip

In the gated rows the server's 401 is real. The proxy only added the gate to the profile.

Choices a reviewer may question

  • The batch test accepts request_too_large at the top level or in messages. The spec does not define the body of that 400.
  • Without protected-resource metadata, the issuer is "the business domain". The spec does not say whether it ends in /, so both spellings are accepted there. Issuers listed in protected-resource metadata are compared byte for byte.
  • A metadata URL that answers 5xx or does not answer fails the test. A 404, or a 2xx that is not JSON, counts as not published.
  • When several authorization servers are listed, the JWT bearer grant must appear in at least one of them.

Left out

The new files follow the layout from #121: shopping/catalog/ and common/identity_linking/. If #123 lands first, the endpoint lookup can use its rest_entry helper instead.

Type of change

  • New feature (non-breaking change which adds functionality)

Is this a Breaking Change or Removal?

No. The change removes no schema file and no field.


Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

🤖 Generated with Claude Code

@evoleinik
evoleinik force-pushed the feat/catalog-identity-linking-tests branch from a5c2ca5 to 13c68a9 Compare November 13, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:needs-triage Signal that the PR is ready for human triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants