-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support MERGE on cloned table in Delta Lake #24756
base: master
Are you sure you want to change the base?
Conversation
37ff93a
to
60e9e73
Compare
60e9e73
to
6432fab
Compare
9bc68fc
to
0cb664f
Compare
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCloneTableCompatibility.java
Show resolved
Hide resolved
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCloneTableCompatibility.java
Show resolved
Hide resolved
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCloneTableCompatibility.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Show resolved
Hide resolved
...sts/src/main/java/io/trino/tests/product/deltalake/TestDeltaLakeCloneTableCompatibility.java
Show resolved
Hide resolved
plugin/trino-delta-lake/src/main/java/io/trino/plugin/deltalake/DeltaLakeMergeSink.java
Outdated
Show resolved
Hide resolved
throw new RuntimeException(e); | ||
} | ||
|
||
checkArgument(sourceTableName != null && sourceTableName.contains(".") && sourceTableName.split("\\.").length == 3, "Unexpected source table in operation_parameters: %s", sourceTableName); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add test with resources like: https://github.com/trinodb/trino/blob/6ae6646eacf50b440b83544fcdaf4ecda82a2127/plugin/trino-delta-lake/src/test/resources/deltalake/allow_column_defaults
to showcase how _delta_log
entries look like?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A question about the path
in the remove entry in the _/delta_log/xxx.json
file:
Since it is possible that the cloned table reference a source table file(absolute path in s3), so here means I have to modify the path
that to make it work befor upload the files?
i, e: the path in removeEntry is s3://aaa/bbb/ccc.parquet
, then for the test, I may create two directory(source, cloned), so here the path
may change to ../source/aaa/bbb/ccc.parquet
in cloned table
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the test testMergeOnClonedTable
in TestDeltaLakeBasic
, PTAL
The absolute path of file in the source table that referenced by cloned table is changed to relative path with logic mentioned above.
0cb664f
to
7ef5f5d
Compare
7ef5f5d
to
be52ef7
Compare
Description
Fix problem that fail update on cloned table, reproduce steps:
testing/bin/ptl env up --environment singlenode-delta-lake-oss
In Trino:
create schema delta.tiny with (location='s3://test-bucket/tiny/');
In Spark-sql:
CREATE TABLE tiny.t1 (id int, v string, part date) USING DELTA PARTITIONED BY (part);
In Trino:
insert into delta.tiny.t1 values (1, 'A', TIMESTAMP '2024-01-01'), (2, 'B', TIMESTAMP '2024-01-01'), (3, 'C', TIMESTAMP '2024-02-02'), (4, 'D', TIMESTAMP '2024-02-02');
In Spark-sql:
CREATE TABLE tiny.t1clone SHALLOW CLONE tiny.t1;
In Trino:
update delta.tiny.t1clone set v = 'update1' where id in (1,3);
It fails with:Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text: