[core] Use Locale.ROOT for machine-facing case conversions - #9771
Draft
LuciferYang wants to merge 3 commits into
Draft
[core] Use Locale.ROOT for machine-facing case conversions#9771LuciferYang wants to merge 3 commits into
LuciferYang wants to merge 3 commits into
Conversation
toLowerCaseIfNeed, toLowerCase, and toUpperCase converted with the JVM default locale. Under a Turkish or Azeri default locale, 'I' lowercases to a dotless glyph and 'i' uppercases to a dotted capital, so the case-sensitive=false identifier matching used by CDC table mapping, computed columns, and the Arrow readers silently broke for columns containing 'I'/'i'. Convert with Locale.ROOT, and align the record side of the same CDC flow: CdcRecord.fieldNameLowerCase also lowercased with the default locale, so fixing only the schema side would newly diverge the two halves of the record-schema join under tr/az (previously both sides mangled identically and data still flowed). Assisted-by: GLM-5.3
LuciferYang
marked this pull request as draft
September 13, 2026 03:06
…common/core
Fixing only StringUtils left the same bug in sites that actually throw. Under a
Turkish default locale 'i' uppercases to a dotted capital, so
PartitionMarkDoneAction.valueOf("SUCCESS_FILE") gets "SUCCESS_FİLE" and
IllegalArgumentException, RowKind.fromShortString("+i") stops matching "+I",
JdbcProtocol.valueOf loses SQLITE and MARIADB, and Solaris detection in
OperatingSystem stops recognising its own name. The DLF request signers, option
key lookups, format identifiers and system table names have the same exposure.
Every converted site here is machine-facing: enum names, protocol tokens,
option keys, header names, OS names, format identifiers, hex digits. None
should follow the JVM default locale. BinaryString.toLowerCase/toUpperCase are
left alone: their ASCII path uses Character.toLowerCase and their fallback
already pins Locale.ROOT, so the SQL upper()/lower() transforms over user data
were never locale-dependent.
Co-Authored-By: Claude Code <noreply@anthropic.com>
getIdentifierPrefixOptions lowercased the key to test the prefix and then
sliced the original by the prefix length, which assumes lowercasing preserves
length. It does not: ROOT maps 'İ' to 'i' plus a combining dot. Match
case-insensitively on the original key instead.
RowKind.fromShortString("-d") passes whichever conversion the code uses, since
Turkish differs from ROOT on 'i' and 'I' alone. Only the "+i" case can catch
the bug, so keep that one and say why.
Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9770
String.toLowerCase()andString.toUpperCase()follow the JVM default locale. Under a Turkish or Azeri default,iuppercases to the dottedİandIlowercases to the dotlessı, so any token that is case-folded before being matched, parsed or signed stops matching what it is compared against. This is a real failure, not a theoretical one:CoreOptions.partitionMarkDoneActions()didPartitionMarkDoneAction.valueOf(x.replace('-','_').toUpperCase()), and bothsuccess-fileanddone-partitioncontain ani, so the default configuration threwIllegalArgumentException: No enum constant ...PartitionMarkDoneAction.SUCCESS_FİLE.RowKind.fromShortString("+i")uppercases to+İ, matches no case arm, and threwUnsupportedOperationException.DistributedLockDialectFactorylostSQLITEandMARIADBthe same way, so JDBC catalog locking failed to resolve its dialect.OperatingSystemlowercasesos.nameand then looks forsolaris, which becomessolarıs, so the OS came backUNKNOWN.HttpClientlooks for therequest-idheader by lowercased name, so error messages silently lost the request id.StartupModeandFormatenum parsing, and vector index type names are all exposed the same way, sincemanifests,INCREMENTAL,FROM_TIMESTAMPandIVFall contain ani.The issue started from identifier matching:
StringUtils.toLowerCaseIfNeeddrives case-insensitive column and table matching, andCdcRecord.fieldNameLowerCaseis the record side of that same join, so the two had to agree or a column silently nulled out.Fixing three call sites and leaving the ones above would have been arbitrary, so this pins
Locale.ROOTon every locale-sensitive case conversion inpaimon-api,paimon-common,paimon-coreandpaimon-flink-cdc. Afterwards those modules have no bareString.toLowerCase()ortoUpperCase()left.BinaryString.toLowerCaseandtoUpperCaseare untouched: their ASCII path usesCharacter.toLowerCaseand the non-ASCII fallback already pinsLocale.ROOT, so Paimon's data-level case conversion was already locale-independent.Two changes that are not machine tokens, called out deliberately
StringUtils.toUpperCase/toLowerCaseare what the CDCupper()andlower()computed columns run on, so this changes data written into the table for atr,azorltdefault locale:lower("ISTANBUL")now persistsistanbulwhere it previously persistedıstanbul. I think locale-independence is the behaviour you want there, since otherwise the value depends on which TaskManager ran the job, and it matches whatBinaryStringalready does everywhere else. But it is a data change, not a token fix.There is also an upgrade caveat for the narrow case of a table whose schema was inferred under such a locale. The persisted column name is
ıd; after this change both sides of the join produceid, so the old column stops matching and schema evolution can append a second column beside it.TableNameConverterlikewise resolves a different physical name. Being bug-compatible with a locale-dependent schema is not possible while also being correct, so this is a disclosure rather than something the patch works around.Tests
TurkishLocaleParsingTestsets a Turkish default locale and covers the two failures reachable through a public entry point: parsingsuccess-file,done-partitioninto the enum set, andRowKind.fromShortString("+i"). Only+ican discriminate, because Turkish differs from ROOT oniandIalone.StringUtilsTestandCdcRecordTestcover the identifier path:toLowerCaseIfNeed("INDEX", false),toUpperCase("ıindex"), and the record-side key conversion.FileFormat.getIdentifierPrefixOptionsalso stopped assuming that lowercasing preserves a string's length, which ROOT does not forİ; it now matches case-insensitively on the original key before slicing it.Verified fail-on-base on JDK 11: with
CoreOptionsandRowKindreverted, the new test errors withNo enum constant org.apache.paimon.CoreOptions.PartitionMarkDoneAction.SUCCESS_FİLEandUnsupported short string '+i' for row kind.. The four modules build clean with checkstyle and spotless enabled, and the ORC, Parquet and Avro format tests that consume the prefix options still pass.