Code of Conduct
Search before asking
Describe the bug
Opening an application's history page can fail with HTTP 500 when its event log contains Uniffle custom listener events:
com.fasterxml.jackson.databind.exc.InvalidDefinitionException:
Cannot construct instance of
org.apache.spark.shuffle.events.ShuffleAssignmentInfoEvent
(no Creators, like default constructor, exist)
The exception occurs during event log replay:
FsHistoryProvider → ReplayListenerBus → JsonProtocol → ObjectMapper.readValue
Uniffle's Java event classes have parameterized constructors without explicit deserialization mappings or no-argument constructors. Older Jackson Scala modules inferred their constructor parameter names. Jackson 2.16 changed this behavior to limit interference with non-Scala classes (jackson-module-scala#644), exposing this compatibility issue.
Local reproduction using the same event class shows:
| Spark |
Jackson |
Result |
| 3.5.1 |
2.15.2 |
Successful deserialization |
| 4.1.1 |
2.20.0 |
InvalidDefinitionException |
| 4.2.0 |
2.21.2 |
InvalidDefinitionException |
Expected behavior: History Server should replay Uniffle events and display the application's history, including Uniffle metrics.
Proposed fix: Add no-argument constructors and JavaBean setters to the affected event classes and nested metric/timing objects. Preserve existing JSON property names and parameterized constructors. Cover non-empty nested metrics, boolean failure flags, and shaded JAR replay in compatibility tests.
This failure affects history-page loading; it does not by itself indicate that the Spark application failed.
Spark 4.2.0 History Server error:
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `org.apache.spark.shuffle.events.ShuffleAssignmentInfoEvent` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 71]
at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1935)
at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:415)
at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1422)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1514)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:340)
at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:212)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:179)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:169)
at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:135)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1384)
at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:5052)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3954)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3922)
at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:968)
at org.apache.spark.util.JsonProtocol$.sparkEventFromJson(JsonProtocol.scala:938)
at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:89)
at org.apache.spark.scheduler.ReplayListenerBus.replay(ReplayListenerBus.scala:60)
at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3(FsHistoryProvider.scala:1437)
at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$3$adapted(FsHistoryProvider.scala:1435)
at org.apache.spark.util.SparkErrorUtils.tryWithResource(SparkErrorUtils.scala:51)
at org.apache.spark.util.SparkErrorUtils.tryWithResource$(SparkErrorUtils.scala:48)
at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:96)
at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$1(FsHistoryProvider.scala:1435)
at org.apache.spark.deploy.history.FsHistoryProvider.$anonfun$parseAppEventLogs$1$adapted(FsHistoryProvider.scala:1432)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:630)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:628)
at scala.collection.AbstractIterable.foreach(Iterable.scala:936)
at org.apache.spark.deploy.history.FsHistoryProvider.parseAppEventLogs(FsHistoryProvider.scala:1432)
at org.apache.spark.deploy.history.FsHistoryProvider.rebuildAppStore(FsHistoryProvider.scala:1413)
at org.apache.spark.deploy.history.FsHistoryProvider.createInMemoryStore(FsHistoryProvider.scala:1668)
at org.apache.spark.deploy.history.FsHistoryProvider.getAppUI(FsHistoryProvider.scala:444)
at org.apache.spark.deploy.history.HistoryServer.getAppUI(HistoryServer.scala:203)
at org.apache.spark.deploy.history.ApplicationCache.$anonfun$loadApplicationEntry$2(ApplicationCache.scala:179)
at org.apache.spark.deploy.history.ApplicationCache.time(ApplicationCache.scala:150)
at org.apache.spark.deploy.history.ApplicationCache.org$apache$spark$deploy$history$ApplicationCache$$loadApplicationEntry(ApplicationCache.scala:177)
at org.apache.spark.deploy.history.ApplicationCache$$anon$1.load(ApplicationCache.scala:67)
at org.apache.spark.deploy.history.ApplicationCache$$anon$1.load(ApplicationCache.scala:58)
at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3556)
at org.sparkproject.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2307)
at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2180)
at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2071)
at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:3985)
at org.sparkproject.guava.cache.LocalCache.getOrLoad(LocalCache.java:4006)
at org.sparkproject.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4946)
at org.apache.spark.deploy.history.ApplicationCache.get(ApplicationCache.scala:104)
at org.apache.spark.deploy.history.ApplicationCache.withSparkUI(ApplicationCache.scala:116)
at org.apache.spark.deploy.history.HistoryServer.org$apache$spark$deploy$history$HistoryServer$$loadAppUi(HistoryServer.scala:265)
at org.apache.spark.deploy.history.HistoryServer$$anon$1.doGet(HistoryServer.scala:106)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
at org.sparkproject.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:754)
at org.sparkproject.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1622)
at org.apache.spark.ui.HttpSecurityFilter.doFilter(HttpSecurityFilter.scala:100)
at org.sparkproject.jetty.ee10.servlet.FilterHolder.doFilter(FilterHolder.java:205)
at org.sparkproject.jetty.ee10.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1594)
at org.sparkproject.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1556)
at org.sparkproject.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:871)
at org.sparkproject.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:449)
at org.sparkproject.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:469)
at org.sparkproject.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1226)
at org.sparkproject.jetty.compression.server.CompressionHandler.handle(CompressionHandler.java:343)
at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:178)
at org.sparkproject.jetty.server
Affects Version(s)
master
Uniffle Server Log Output
Uniffle Engine Log Output
Uniffle Server Configurations
Uniffle Engine Configurations
Additional context
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Describe the bug
Opening an application's history page can fail with HTTP 500 when its event log contains Uniffle custom listener events:
The exception occurs during event log replay:
Uniffle's Java event classes have parameterized constructors without explicit deserialization mappings or no-argument constructors. Older Jackson Scala modules inferred their constructor parameter names. Jackson 2.16 changed this behavior to limit interference with non-Scala classes (jackson-module-scala#644), exposing this compatibility issue.
Local reproduction using the same event class shows:
InvalidDefinitionExceptionInvalidDefinitionExceptionExpected behavior: History Server should replay Uniffle events and display the application's history, including Uniffle metrics.
Proposed fix: Add no-argument constructors and JavaBean setters to the affected event classes and nested metric/timing objects. Preserve existing JSON property names and parameterized constructors. Cover non-empty nested metrics, boolean failure flags, and shaded JAR replay in compatibility tests.
This failure affects history-page loading; it does not by itself indicate that the Spark application failed.
Spark 4.2.0 History Server error:
Affects Version(s)
master
Uniffle Server Log Output
Uniffle Engine Log Output
Uniffle Server Configurations
Uniffle Engine Configurations
Additional context
No response
Are you willing to submit PR?