test_format_schema_source fails on arm on stable-26.3: ZooKeeper never starts because the arm image override is missing

[CarlosFelipeOR]

✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report — CI configuration, not product code

Describe the situation

All 17 tests in test_format_schema_source error at module setup on Integration tests (arm_binary, distributed plan, 1/4). The Kafka cluster the module needs never comes up on arm.

This issue:

• Is deterministic, not flaky — 3 of 3 module starts failed, on three different 16-CPU Hetzner runners
• Is not a product bug, and not an Altinity divergence — upstream's 26.3 branch is byte-identical on every relevant file
• Only reaches us because we run arm integration tests on 26.3 and upstream does not

Actual behavior

Attempt 1 hit the pytest timeout; attempt 2 exhausted the retry loop and gave the clean message:

helpers/cluster.py:3078: in wait_kafka_is_available
    raise Exception("Kafka is not available")

The Kafka broker cannot reach ZooKeeper:

kafka1-1 | ClientCnxn - Socket error occurred: kafka_zookeeper/172.16.3.2:2181: Connection refused

And ZooKeeper logged two lines, then nothing — it never bound 2181:

kafka_zookeeper-1 | ZooKeeper JMX enabled by default
kafka_zookeeper-1 | Using config: /conf/zoo.cfg

zookeeper:3.4.9 is an amd64-only image. Docker confirms it runs under emulation:

kafka_zookeeper The requested image's platform (linux/amd64) does not match
the detected host platform (linux/arm64/v8) and no specific platform was requested

Root cause

From 26.6 onward, tests/integration/helpers/cluster.py swaps in arm64-native images in setup_kafka_cmd:

if is_arm():
    env_variables["KAFKA_IMAGE_TAG"] = "7.9.0"
    env_variables["KAFKA_SCHEMA_REGISTRY_IMAGE_TAG"] = "7.9.0"
    env_variables["KAFKA_ZOOKEEPER_IMAGE_TAG"] = "3.8"
    env_variables["KAFKA_ZOOKEEPER_SERVERS"] = "server.1=kafka_zookeeper:2888:3888;2181"

That comes from upstream 156f1ef9347 (2026-04-21, Enable Kafka integration tests on arm64), which is not an ancestor of cde98ffcf06. The 26.3 line stays pinned to zookeeper:3.4.9 and cp-kafka:5.2.0.

The test started needing Kafka via the backport of upstream #117737 (commit 897a240e903), which added with_kafka=True. So the half that needs Kafka reached 26.3; the half that makes Kafka work on arm did not.

Every other Kafka test in the same job is SKIPPED on arm, because helpers/kafka/common_direct.py sets pytestmark = pytest.mark.skip under is_arm(). test_format_schema_source imports helpers.kafka.common directly, so it inherits no guard.

Comparing docker_compose_kafka.yml between branches is misleading — the 26.3 and upstream-26.3 files are identical, and master's only differs by ${VAR:-default} placeholders. The deciding file is cluster.py. Grep KAFKA_ZOOKEEPER_IMAGE_TAG there.

Which branches have what

ref	arm image override	test needs Kafka	arm result
upstream 26.3	no	yes	would break — job not run there
Altinity/ClickHouse:stable-26.3	no	yes	broken
antalya-26.3	no	no	ok
upstream 26.6 / 26.8, antalya-26.6 / -26.8	yes	yes	ok

Upstream runs 496 Integration tests (amd_tsan, N/6) on branch 26.3 in the last 120 days and zero arm integration jobs. On master they run 27,568. That is why only we see it.

Suggested fix

Add the guard the sibling tests already carry, in tests/integration/test_format_schema_source/test.py:

from helpers.cluster import ClickHouseCluster, is_arm

# Skip on ARM due to Confluent/Kafka
if is_arm():
    pytestmark = pytest.mark.skip

The larger alternative is to port 156f1ef9347 to stable-26.3, which enables Kafka on arm properly. No other Kafka test runs on arm in 26.3 today, so it buys nothing for this release.

CI failure

• Job: Integration tests (arm_binary, distributed plan, 1/4)
• Ref / commit: v26.3.33.10001.altinitystable · cde98ffcf061cd23b49f1f65da841522b05c2e94
• Run: https://github.com/Altinity/ClickHouse/actions/runs/35342360645 — attempt 1 · attempt 2
• Logs: json.html
• Runners: attempt 1 github-hetzner-runner-standby-1789736637213980, attempt 2 github-hetzner-runner-standby-1789769777521319 — both 16 CPUs