Zstd-jni

Overview

JNI bindings for Zstd native library that provides fast and high compression lossless algorithm for Android, Java and all JVM languages:

• static compress/decompress methods

• implementation of InputStream and OutputStream for transparent compression of data streams fully compatible with the "zstd" program.

• minimal performance overhead

Zstd

Zstd, short for Zstandard, is a new lossless compression algorithm, which provides both good compression ratio and speed for your standard compression needs. "Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).

Zstd is developed by Yann Collet and the source is available at: https://github.com/facebook/zstd

The motivation for development, the algorithm used and its properties are explained in the blog post that introduces the library: http://fastcompression.blogspot.com/2015/01/zstd-stronger-compression-algorithm.html

Status and availability

Zstd is production ready with a stable format.

Zstd-jni is tracking the release branch of Zstd (master) without compatibility support for the legacy formats. The last version that was released with support for the legacy formats (0.4 - 0.7) is 1.5.7-12.

Zstd-jni version uses the base Zstd version with Zstd-jni release appended with a dash, e.g. "1.2.0-2" is the second Zstd-jni release based on Zstd version 1.2.0.

Limitations

The Java classes cannot be renamed/minimized/relocated. JVM linking the native library depends on the class name that is trying to link the native part, so changing the class names will lead to failed linking at runtime.

Foreign Function & Memory API

The library's native operations have been ported from JNI to the Foreign Function & Memory API (JEP 454). Every class that calls into libzstd has an FFM version. These are compiled for JDK 22 and shipped inside the same Jar under META-INF/versions/22, which makes it a Multi-Release Jar: a JDK 22+ runtime loads them, anything older keeps loading the JNI implementation from the Jar root. The public API and the behaviour are identical either way, so there is nothing to configure and nothing to change in your code.

If you need to force the JNI implementation on a JDK 22+ runtime, start the JVM with:

-Djdk.util.jar.enableMultiRelease=false

Note that this disables Multi-Release dispatch for every Jar on the classpath, not just this one, so it is an escape hatch rather than a supported configuration. Repackaging into a fat Jar has the same effect whenever the Multi-Release: true manifest attribute is not carried over.

Building and dependencies

Zstd-jni uses SBT for building the library and running the tests.

The build system depends on Scala and the tests depend on ScalaTest and ScalaCheck but the produced JAR does not have any dependencies. It also embeds the native library.

Note: For the moment the project depends on a local build of sbt-java-module-info v0.5.2, as that version is not published to Maven. Before compiling, you need to publish it locally:

$ cd sbt-java-module-info && ./sbt publishLocal && cd -

Compile and test:

 $ ./sbt compile test package

Note: packaging does not wait for the native library to be built, so run compile as a separate, earlier command (as above) before anything that packages the jar.

Note: ./sbt test covers the JNI implementation only. The JDK 22+ one built on the Foreign Function & Memory API ships in META-INF/versions/22, and Multi-Release dispatch only happens from a jar, so testing it means packaging first and then choosing the runtime:

 $ ./sbt compile testFromJarSetup testFromJar

If you want to publish it to you local ivy2 repository:

 $ ./sbt compile publishLocal

Binary releases

The binary releases are architecture dependent because we are embedding the native library in the provided Jar file. Currently they are built for linux-amd64, linux-i386, linux-aarch64, linux-armhf, linux-ppc64, linux-ppc64le, linux-s390x, linux-riscv64, linux-loongarch64, win-amd64, win-x86, win-aarch64, darwin-x86_64 (MacOS X), darwin-aarch64, aix-ppc64, freebsd-amd64, and freebsd-i386.

More builds will be available if I get access to more platforms.

You can find published releases on Maven Central.

<dependency>
    <groupId>com.github.luben</groupId>
    <artifactId>zstd-jni</artifactId>
    <version>VERSION</version>
</dependency>

sbt dependency:

libraryDependencies += "com.github.luben" % "zstd-jni" % "VERSION"

Single architecture classified jars are also published. The classifier name is following the "$OS_$ARCH" convention. There is also a "cloud" classifier that packs binaries just for linux-amd64, linux-aarch64 (for deployment), and darwin-aarch64 (to help local development).

They can be used like:

<dependency>
    <groupId>com.github.luben</groupId>
    <artifactId>zstd-jni</artifactId>
    <version>VERSION</version>
    <classifier>linux_amd64</classifier>
</dependency>

or for sbt:

libraryDependencies += "com.github.luben" % "zstd-jni" % "VERSION" classifier "linux_amd64"

Link for direct download if you don't use a dependency manager:

• https://repo1.maven.org/maven2/com/github/luben/zstd-jni/

If there is not yet a binary release compatible with your platform look how to build it locally under the Building section.

Android support

Zstd-jni is usable in Android applications by importing the sources in Android Studio. I guess using git submodules will also work. Import src/main/java rather than src/main as a whole: src/main/java22 holds an alternative implementation of a few classes built on the Java 22 java.lang.foreign API, which the published jar ships under META-INF/versions/22 for JDK 22+ runtimes only. It duplicates class names from src/main/java and does not compile on Android.

Android archive (zstd-jni.aar) is also published on maven central that support Android 5.0 and above. You will need to add the repository in your build.gradle, e.g.:

allprojects {
    repositories {
        jcenter()
        mavenCentral()
    }
}

as it is not added by default by Android Studio. And then add dependency on the prebuilt android archive (aar):

dependencies {
    implementation "com.github.luben:zstd-jni:VERSION@aar"
    testImplementation "com.github.luben:zstd-jni:VERSION"
}

For example Android app and how to declare dependencies and use zstd-jni, consult the 2nd and 3rd commit of: https://github.com/luben/ZstdAndroidExample

License

The code for these JNI bindings is licenced under 2-clause BSD license. The native Zstd library is licensed under 3-clause BSD license or GPL2. See the LICENSE file and LICENSE and COPYRIGHT in src/main/native for full copyright and conditions.