Skip to content

[Bug] Spark 4.2.0 History Server fails to replay Uniffle events with newer Jackson versions #2792

Description

@wForget

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

Opening an application's history page can fail with HTTP 500 when its event log contains Uniffle custom listener events:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
Cannot construct instance of
org.apache.spark.shuffle.events.ShuffleAssignmentInfoEvent
(no Creators, like default constructor, exist)

The exception occurs during event log replay:

FsHistoryProvider → ReplayListenerBus → JsonProtocol → ObjectMapper.readValue

Uniffle's Java event classes have parameterized constructors without explicit deserialization mappings or no-argument constructors. Older Jackson Scala modules inferred their constructor parameter names. Jackson 2.16 changed this behavior to limit interference with non-Scala classes (jackson-module-scala#644), exposing this compatibility issue.

Local reproduction using the same event class shows:

Spark Jackson Result
3.5.1 2.15.2 Successful deserialization
4.1.1 2.20.0 InvalidDefinitionException
4.2.0 2.21.2 InvalidDefinitionException

Expected behavior: History Server should replay Uniffle events and display the application's history, including Uniffle metrics.

Proposed fix: Add no-argument constructors and JavaBean setters to the affected event classes and nested metric/timing objects. Preserve existing JSON property names and parameterized constructors. Cover non-empty nested metrics, boolean failure flags, and shaded JAR replay in compatibility tests.

This failure affects history-page loading; it does not by itself indicate that the Spark application failed.

Spark 4.2.0 History Server error:

com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.apache.spark.shuffle.events.ShuffleAssignmentInfoEvent` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 71]
	at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
	at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1935)
	at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:415)
	at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1422)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1514)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:340)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:212)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:179)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:169)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:135)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1384)
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:5052)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3954)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3922)
	at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:968)
	at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:938)
	at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:89)
	at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:60)
	at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3(FsHistoryProvider.scala:1437)
	at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3$adapted(FsHistoryProvider.scala:1435)
	at org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:51)
	at org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:48)
	at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:96)
	at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$1(FsHistoryProvider.scala:1435)
	at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$1$adapted(FsHistoryProvider.scala:1432)
	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
	at org.apache.spark.deploy.history.FsHistoryProvider.parseAppEventLogs(FsHistoryProvider.scala:1432)
	at org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:1413)
	at org.apache.spark.deploy.history.FsHistoryProvider.createInMemoryStore(FsHistoryProvider.scala:1668)
	at org.apache.spark.deploy.history.FsHistoryProvider.getAppUI(FsHistoryProvider.scala:444)
	at org.apache.spark.deploy.history.HistoryServer.getAppUI(HistoryServer.scala:203)
	at org.apache.spark.deploy.history.ApplicationCache.$anonfun$loadApplicationEntry$2(ApplicationCache.scala:179)
	at org.apache.spark.deploy.history.ApplicationCache.time(ApplicationCache.scala:150)
	at org.apache.spark.deploy.history.ApplicationCache.org$apache$spark$deploy$history$ApplicationCache$$loadApplicationEntry(ApplicationCache.scala:177)
	at org.apache.spark.deploy.history.ApplicationCache$$anon$1.load(ApplicationCache.scala:67)
	at org.apache.spark.deploy.history.ApplicationCache$$anon$1.load(ApplicationCache.scala:58)
	at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3556)
	at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2307)
	at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2180)
	at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2071)
	at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:3985)
	at org.sparkproject.guava.cache.LocalCache.getOrLoad(LocalCache.java:4006)
	at org.sparkproject.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
	at org.apache.spark.deploy.history.ApplicationCache.get(ApplicationCache.scala:104)
	at org.apache.spark.deploy.history.ApplicationCache.withSparkUI(ApplicationCache.scala:116)
	at org.apache.spark.deploy.history.HistoryServer.org$apache$spark$deploy$history$HistoryServer$$loadAppUi(HistoryServer.scala:265)
	at org.apache.spark.deploy.history.HistoryServer$$anon$1.doGet(HistoryServer.scala:106)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
	at org.sparkproject.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:754)
	at org.sparkproject.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)
	at org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:100)
	at org.sparkproject.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
	at org.sparkproject.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)
	at org.sparkproject.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1556)
	at org.sparkproject.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:871)
	at org.sparkproject.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:449)
	at org.sparkproject.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:469)
	at org.sparkproject.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1226)
	at org.sparkproject.jetty.compression.server.CompressionHandler.handle(CompressionHandler.java:343)
	at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:178)
	at org.sparkproject.jetty.server

Affects Version(s)

master

Uniffle Server Log Output

Uniffle Engine Log Output

Uniffle Server Configurations

Uniffle Engine Configurations

Additional context

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions