Skip to content

[improve][fs] Reduce bundled S3 and Hadoop dependencies - #9786

Closed
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/reduce-filesystem-package-size
Closed

JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/reduce-filesystem-package-size

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

The S3 plugin bundles AWS service clients that Hadoop S3A does not use. Remove the unused SDK v2 IAM, Glue, DynamoDB and Lake Formation clients, and move the SDK v1 S3 client and its SAX parser workaround to test scope for the MinIO fixture. Retain SDK v1 core and STS for legacy credential providers, as well as the SDK v2 clients required for S3 access, credentials and encryption.

Exclude the unshaded Guava dependency from the shared Hadoop 3.4 bundle: Hadoop uses hadoop-shaded-guava, while Curator, which needs the ordinary Guava copy, is already excluded. Update the bundled dependency notices.

Local JDK 8 builds against master reduce the packaged artifacts as follows (decimal MB):

Artifact Before After
paimon-s3 96.70 MB 72.18 MB
paimon-s3-impl 93.42 MB 69.40 MB

Tests

Passed on JDK 8, with the normal build checks enabled:

mvn -o -pl paimon-filesystems/paimon-s3,paimon-filesystems/paimon-azure -am \
  -DwildcardSuites=none -DfailIfNoTests=false \
  -Dtest=S3MultiPartUploadTest,S3LoaderTest clean verify

Additional local smoke checks loaded the final shaded plugins in isolation and verified S3A initialization with ordinary, SSE-C and SSE-KMS settings; SDK v2 S3/STS/KMS/SSO client construction; SDK v1 credential adaptation and STS client construction; and Azure WASB/ABFS loading with Hadoop UGI and shaded Guava. Retained class and native-library entries were unchanged in the S3 JAR comparison.

MinIO integration tests were not run because the Docker daemon was unavailable.

@JingsongLi JingsongLi closed this Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant