[improve][fs] Reduce bundled S3 and Hadoop dependencies - #9786
Closed
JingsongLi wants to merge 1 commit into
Closed
JingsongLi wants to merge 1 commit into
JingsongLi wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The S3 plugin bundles AWS service clients that Hadoop S3A does not use. Remove the unused SDK v2 IAM, Glue, DynamoDB and Lake Formation clients, and move the SDK v1 S3 client and its SAX parser workaround to test scope for the MinIO fixture. Retain SDK v1 core and STS for legacy credential providers, as well as the SDK v2 clients required for S3 access, credentials and encryption.
Exclude the unshaded Guava dependency from the shared Hadoop 3.4 bundle: Hadoop uses hadoop-shaded-guava, while Curator, which needs the ordinary Guava copy, is already excluded. Update the bundled dependency notices.
Local JDK 8 builds against master reduce the packaged artifacts as follows (decimal MB):
Tests
Passed on JDK 8, with the normal build checks enabled:
Additional local smoke checks loaded the final shaded plugins in isolation and verified S3A initialization with ordinary, SSE-C and SSE-KMS settings; SDK v2 S3/STS/KMS/SSO client construction; SDK v1 credential adaptation and STS client construction; and Azure WASB/ABFS loading with Hadoop UGI and shaded Guava. Retained class and native-library entries were unchanged in the S3 JAR comparison.
MinIO integration tests were not run because the Docker daemon was unavailable.