Skip to main content

Release 0.4.0-incubating

Release 0.4.0-incubating (docs)​

This is the fourth official Apache release for Apache XTable (Incubating), an incubating project under the Apache Software Foundation. Apache XTable™ (Incubating) facilitates omni-directional interoperability across data processing systems and query engines by allowing users to convert between open table formats without the need to rewrite any data files. Apache XTable™ (Incubating) converts between Apache Hudi, Apache Iceberg and Delta Lake, and with this release it can also read Apache Paimon and Apache Parquet tables as sources.

✨ What's Changed​

📦 Run an XTable sync on Apache Spark​

This release introduces xtable-spark-runtime, a runtime jar that runs a sync on a Spark cluster you already have. Add it to an existing job with --jars and call the sync right after your write, or submit it on its own with spark-submit. Every engine dependency is provided, so the jar reuses the Hudi, Iceberg and Delta libraries already on your cluster.

shell
$SPARK_HOME/bin/spark-submit \
  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
  --class com.example.OrdersJob \
  orders-job.jar

It is published to Maven Central as org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating and supports Spark 3.4 and 3.5. See Run an XTable sync on Apache Spark.

🔄 New source formats​

  • Apache Paimon source, with incremental sync and column statistics.
  • Apache Parquet source, with schema conversion, column statistics, partition extraction and incremental sync.

🆕 Delta Kernel​

  • Added a Delta Kernel conversion source and target, which no longer depend on Spark for the conversion itself.

🔧 Conversion improvements​

  • Delta source: extract statistics for nested fields.
  • Hudi source: support nested partition columns when hive-style partitioning is enabled.
  • Fall back to Parquet footers when the metadata table has no column statistics.
  • Fail fast on a corrupt Delta checkpoint CRC, instead of continuing with bad statistics.

🐞 Bug fixes​

  • Delta: null partition value for composite generated-column partitions, map-key path handling, a null pointer on binary fields inside maps and arrays, and conversion from a source with column mapping enabled.
  • Hudi: batch INSERT_OVERWRITE replace commits that dropped adds, and an empty EarliestCommitToRetain.
  • Iceberg: nested comments with a qualified name, schema sync, and an empty source table.
  • Parquet: snapshot sync across several commits on a partitioned table, and schema conversion.

🧰 Build and infrastructure​

  • Upgraded Apache Iceberg to 1.9.2 and Apache Avro to 1.12.0.
  • Added a Maven wrapper so every build uses the same Maven version.
  • Bundled license files are now generated and verified as part of the build.
  • Added a source build check that builds the source release with no XTable artifact installed.
  • Restored the delombok step, so release builds produce javadoc jars again.

🧪 Testing​

  • Added a first integration test for the XTable REST service.
  • Added an integration test that runs the Spark runtime jar through a real spark-submit.
  • Parameterised TestHudiInstantUtils and added more cases.
  • Removed wildcard imports and enforced the rule with Spotless.

Known issues​

Three license documentation issues remain open and are targeted for the next release:

  • Incomplete license information for the bundled jars (#700).
  • More than one LICENSE file in some jars (#701).
  • Orphaned license files in the bundled jars (#865).

📥 Download​

See the Downloads page for the source release, its signature and its checksum.