Skip to content

[core] Use Locale.ROOT for machine-facing case conversions - #9771

Draft
LuciferYang wants to merge 3 commits into
apache:masterfrom
LuciferYang:fix/stringutils-tolowercase-locale
Draft

[core] Use Locale.ROOT for machine-facing case conversions#9771
LuciferYang wants to merge 3 commits into
apache:masterfrom
LuciferYang:fix/stringutils-tolowercase-locale

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

close #9770

String.toLowerCase() and String.toUpperCase() follow the JVM default locale. Under a Turkish or Azeri default, i uppercases to the dotted İ and I lowercases to the dotless ı, so any token that is case-folded before being matched, parsed or signed stops matching what it is compared against. This is a real failure, not a theoretical one:

  • CoreOptions.partitionMarkDoneActions() did PartitionMarkDoneAction.valueOf(x.replace('-','_').toUpperCase()), and both success-file and done-partition contain an i, so the default configuration threw IllegalArgumentException: No enum constant ...PartitionMarkDoneAction.SUCCESS_FİLE.
  • RowKind.fromShortString("+i") uppercases to , matches no case arm, and threw UnsupportedOperationException.
  • DistributedLockDialectFactory lost SQLITE and MARIADB the same way, so JDBC catalog locking failed to resolve its dialect.
  • OperatingSystem lowercases os.name and then looks for solaris, which becomes solarıs, so the OS came back UNKNOWN.
  • HttpClient looks for the request-id header by lowercased name, so error messages silently lost the request id.
  • System table lookup, StartupMode and Format enum parsing, and vector index type names are all exposed the same way, since manifests, INCREMENTAL, FROM_TIMESTAMP and IVF all contain an i.

The issue started from identifier matching: StringUtils.toLowerCaseIfNeed drives case-insensitive column and table matching, and CdcRecord.fieldNameLowerCase is the record side of that same join, so the two had to agree or a column silently nulled out.

Fixing three call sites and leaving the ones above would have been arbitrary, so this pins Locale.ROOT on every locale-sensitive case conversion in paimon-api, paimon-common, paimon-core and paimon-flink-cdc. Afterwards those modules have no bare String.toLowerCase() or toUpperCase() left.

BinaryString.toLowerCase and toUpperCase are untouched: their ASCII path uses Character.toLowerCase and the non-ASCII fallback already pins Locale.ROOT, so Paimon's data-level case conversion was already locale-independent.

Two changes that are not machine tokens, called out deliberately

StringUtils.toUpperCase / toLowerCase are what the CDC upper() and lower() computed columns run on, so this changes data written into the table for a tr, az or lt default locale: lower("ISTANBUL") now persists istanbul where it previously persisted ıstanbul. I think locale-independence is the behaviour you want there, since otherwise the value depends on which TaskManager ran the job, and it matches what BinaryString already does everywhere else. But it is a data change, not a token fix.

There is also an upgrade caveat for the narrow case of a table whose schema was inferred under such a locale. The persisted column name is ıd; after this change both sides of the join produce id, so the old column stops matching and schema evolution can append a second column beside it. TableNameConverter likewise resolves a different physical name. Being bug-compatible with a locale-dependent schema is not possible while also being correct, so this is a disclosure rather than something the patch works around.

Tests

TurkishLocaleParsingTest sets a Turkish default locale and covers the two failures reachable through a public entry point: parsing success-file,done-partition into the enum set, and RowKind.fromShortString("+i"). Only +i can discriminate, because Turkish differs from ROOT on i and I alone.

StringUtilsTest and CdcRecordTest cover the identifier path: toLowerCaseIfNeed("INDEX", false), toUpperCase("ıindex"), and the record-side key conversion.

FileFormat.getIdentifierPrefixOptions also stopped assuming that lowercasing preserves a string's length, which ROOT does not for İ; it now matches case-insensitively on the original key before slicing it.

Verified fail-on-base on JDK 11: with CoreOptions and RowKind reverted, the new test errors with No enum constant org.apache.paimon.CoreOptions.PartitionMarkDoneAction.SUCCESS_FİLE and Unsupported short string '+i' for row kind.. The four modules build clean with checkstyle and spotless enabled, and the ORC, Parquet and Avro format tests that consume the prefix options still pass.

toLowerCaseIfNeed, toLowerCase, and toUpperCase converted with the
JVM default locale. Under a Turkish or Azeri default locale, 'I'
lowercases to a dotless glyph and 'i' uppercases to a dotted capital,
so the case-sensitive=false identifier matching used by CDC table
mapping, computed columns, and the Arrow readers silently broke for
columns containing 'I'/'i'.

Convert with Locale.ROOT, and align the record side of the same CDC
flow: CdcRecord.fieldNameLowerCase also lowercased with the default
locale, so fixing only the schema side would newly diverge the two
halves of the record-schema join under tr/az (previously both sides
mangled identically and data still flowed).

Assisted-by: GLM-5.3
@LuciferYang
LuciferYang marked this pull request as draft September 13, 2026 03:06
…common/core

Fixing only StringUtils left the same bug in sites that actually throw. Under a
Turkish default locale 'i' uppercases to a dotted capital, so
PartitionMarkDoneAction.valueOf("SUCCESS_FILE") gets "SUCCESS_FİLE" and
IllegalArgumentException, RowKind.fromShortString("+i") stops matching "+I",
JdbcProtocol.valueOf loses SQLITE and MARIADB, and Solaris detection in
OperatingSystem stops recognising its own name. The DLF request signers, option
key lookups, format identifiers and system table names have the same exposure.

Every converted site here is machine-facing: enum names, protocol tokens,
option keys, header names, OS names, format identifiers, hex digits. None
should follow the JVM default locale. BinaryString.toLowerCase/toUpperCase are
left alone: their ASCII path uses Character.toLowerCase and their fallback
already pins Locale.ROOT, so the SQL upper()/lower() transforms over user data
were never locale-dependent.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@LuciferYang LuciferYang changed the title [api] Use Locale.ROOT in StringUtils case conversions [core] Use Locale.ROOT for machine-facing case conversions Sep 13, 2026
getIdentifierPrefixOptions lowercased the key to test the prefix and then
sliced the original by the prefix length, which assumes lowercasing preserves
length. It does not: ROOT maps 'İ' to 'i' plus a combining dot. Match
case-insensitively on the original key instead.

RowKind.fromShortString("-d") passes whichever conversion the code uses, since
Turkish differs from ROOT on 'i' and 'I' alone. Only the "+i" case can catch
the bug, so keep that one and say why.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Locale-sensitive case conversions break identifier matching, enum parsing and request signing

1 participant