Skip to content

[improve][fs] Reduce bundled S3 and Hadoop dependencies #9786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JingsongLi wants to merge 1 commit into from
Closed

[improve][fs] Reduce bundled S3 and Hadoop dependencies #9786

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions paimon-filesystems/paimon-hadoop-shaded-3.4/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,12 @@
<groupId>org.apache.kerby</groupId>
<artifactId>kerb-simplekdc</artifactId>
</exclusion>
<exclusion>
<!-- Hadoop uses hadoop-shaded-guava. The unshaded copy is only needed by Curator,
which is excluded below. -->
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>curator-client</artifactId>
Expand Down
7 changes: 0 additions & 7 deletions paimon-filesystems/paimon-hadoop-shaded-3.4/src/main/resources/META-INF/NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,6 @@ This project bundles the following dependencies under the Apache Software Licens
- com.fasterxml.jackson.core:jackson-core:2.12.7
- com.fasterxml.jackson.core:jackson-databind:2.12.7
- com.fasterxml.woodstox:woodstox-core:5.3.0
- com.google.guava:failureaccess:1.0
- com.google.guava:guava:27.0-jre
- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
- com.google.j2objc:j2objc-annotations:1.1
- commons-beanutils:commons-beanutils:1.9.4
- commons-collections:commons-collections:3.2.2
- commons-io:commons-io:2.8.0
Expand All @@ -35,10 +31,7 @@ This project bundles the following dependencies under the Apache Software Licens
- com.google.code.findbugs:jsr305:1.3.9

This project bundles the following dependencies under the MIT (https://opensource.org/licenses/MIT).
You find them under licenses/LICENSE.checker-framework-qualifiers and licenses/LICENSE.animal-sniffer.

- org.checkerframework:checker-qual:2.5.2
- org.codehaus.mojo:animal-sniffer-annotations:1.17
- org.slf4j:slf4j-api:1.7.32

This project bundles the following dependencies under BSD-2 License (https://opensource.org/licenses/BSD-2-Clause).
Expand Down
58 changes: 12 additions & 46 deletions paimon-filesystems/paimon-s3-impl/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,27 +49,12 @@
<scope>provided</scope>
</dependency>

<!-- AWS dependencies (bundled) -->
<!-- AWS SDK v1 support for legacy credential providers. S3A uses SDK v2 clients. -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-core</artifactId>
<version>${fs.s3.aws.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>${fs.s3.aws.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-kms</artifactId>
<version>${fs.s3.aws.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-dynamodb</artifactId>
<version>${fs.s3.aws.version}</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
Expand Down Expand Up @@ -111,7 +96,9 @@
</exclusions>
</dependency>

<!-- AWS SDK v2 dependencies - only include necessary modules -->
<!-- AWS SDK v2 dependencies used by S3A and its credential providers.
IAM, Glue, DynamoDB and Lake Formation clients are not needed for S3 access.
IAM role credentials are handled by auth and STS, not the IAM service client. -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
Expand All @@ -132,11 +119,6 @@
<artifactId>http-auth-aws-crt</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>iam</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sso</artifactId>
Expand All @@ -152,26 +134,11 @@
<artifactId>kms</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>glue</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>dynamodb</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>lakeformation</artifactId>
<version>${fs.s3.awssdk.version}</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
Expand All @@ -187,6 +154,14 @@
<scope>provided</scope>
</dependency>

<!-- The MinIO fixture uses the SDK v1 S3 client. -->
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
<version>${fs.s3.aws.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-common</artifactId>
Expand Down Expand Up @@ -309,15 +284,6 @@
<exclude>META-INF/LICENSE.txt</exclude>
</excludes>
</filter>
<filter>
<artifact>com.amazonaws:aws-java-sdk-s3</artifact>
<!-- Make sure we are using the overridden XmlResponsesSaxParser.
Filter must be removed as soon as XmlResponsesSaxParser of this module is
dropped, for example when discontinuing support for Java 8. -->
<excludes>
<exclude>com/amazonaws/services/s3/model/transform/XmlResponsesSaxParser**</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
Expand Down
3 changes: 0 additions & 3 deletions paimon-filesystems/paimon-s3-impl/src/main/resources/META-INF/NOTICE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,6 @@ The Apache Software Foundation (http://www.apache.org/).
This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)

- com.amazonaws:aws-java-sdk-core:1.12.319
- com.amazonaws:aws-java-sdk-dynamodb:1.12.319
- com.amazonaws:aws-java-sdk-kms:1.12.319
- com.amazonaws:aws-java-sdk-s3:1.12.319
- com.amazonaws:aws-java-sdk-sts:1.12.319
- com.amazonaws:jmespath-java:1.12.319
- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.12.6
Expand Down
2 changes: 1 addition & 1 deletion ...odel/transform/XmlResponsesSaxParser.java → ...odel/transform/XmlResponsesSaxParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@
// This class is copied from aws-java-sdk-s3 (Apache License 2.0)
// in order to replaced {@code XMLReaderFactory.createXMLReader} by
// {@code SAX_PARSER_FACTORY.newSAXParser().getXMLReader()} to
// avoid JDK-8015099.
// avoid JDK-8015099 when the MinIO test fixture uses the SDK v1 client.
//
// ----------------------------------------------------------------------------
// ----------------------------------------------------------------------------
Expand Down
Loading