Summary
contrib/datalake_fdw's Parquet format layer (#1951) refuses numeric columns at CREATE TABLE ... USING iceberg. DECIMAL is the most common column type in real lake tables, so this is the first type to add.
What has to be decided
- Storage form. Parquet stores DECIMAL four ways (INT32, INT64, FIXED_LEN_BYTE_ARRAY, BYTE_ARRAY). Iceberg's spec fixes the writer to
decimal(P,S) with P <= 38 backed by fixed-length bytes; the reader has to accept all four.
- Unconstrained
numeric. atttypmod = -1 has no precision or scale. Computing ((typmod - 4) >> 16) & 65535 without checking gives precision 65535 / scale 65531, so bare numeric columns silently match nothing (the trap in lithium-tech/tea validate.cpp:59-61). Iceberg needs P and S, so the likely answer is to refuse bare numeric at CREATE TABLE and require numeric(P,S).
- Precision above 38. PostgreSQL allows up to 1000;
decimal128 caps at 38. Refuse at CREATE TABLE.
- Conversion.
NumericVar <-> two's-complement int128, the way tea's bridge.cpp:269-277 and numeric_var.cpp do it, with NaN/Infinity refused on write.
Where
format/format_types.h / arrow_support.cpp: dl_format_type_refusal() decides what CREATE TABLE accepts; the mapping to arrow::decimal128(P, S) goes next to the other types.arrow_builder.cpp / arrow_decode.c: append and decode. The decoder reads the Arrow C data interface directly; the format string is d:P,S.- Tests in
test/automation/sqlrepo/smoke/format_parquet/, plus CREATE TABLE refusals in iceberg_am_reject.sql.
Deferred from #1951 on purpose: the framework there is settled, the type work is separate.
Summary
contrib/datalake_fdw's Parquet format layer (#1951) refusesnumericcolumns atCREATE TABLE ... USING iceberg. DECIMAL is the most common column type in real lake tables, so this is the first type to add.What has to be decided
decimal(P,S)with P <= 38 backed by fixed-length bytes; the reader has to accept all four.numeric.atttypmod = -1has no precision or scale. Computing((typmod - 4) >> 16) & 65535without checking gives precision 65535 / scale 65531, so barenumericcolumns silently match nothing (the trap in lithium-tech/teavalidate.cpp:59-61). Iceberg needs P and S, so the likely answer is to refuse barenumericatCREATE TABLEand requirenumeric(P,S).decimal128caps at 38. Refuse atCREATE TABLE.NumericVar<-> two's-complement int128, the way tea'sbridge.cpp:269-277andnumeric_var.cppdo it, withNaN/Infinityrefused on write.Where
format/format_types.h/arrow_support.cpp:dl_format_type_refusal()decides whatCREATE TABLEaccepts; the mapping toarrow::decimal128(P, S)goes next to the other types.arrow_builder.cpp/arrow_decode.c: append and decode. The decoder reads the Arrow C data interface directly; the format string isd:P,S.test/automation/sqlrepo/smoke/format_parquet/, plusCREATE TABLErefusals iniceberg_am_reject.sql.Deferred from #1951 on purpose: the framework there is settled, the type work is separate.