Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support MERGE on cloned table in Delta Lake #24756

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

chenjian2664
Copy link
Contributor

@chenjian2664 chenjian2664 commented Jan 21, 2025

Description

Fix problem that fail update on cloned table, reproduce steps:

testing/bin/ptl env up --environment singlenode-delta-lake-oss

In Trino: create schema delta.tiny with (location='s3://test-bucket/tiny/');

In Spark-sql: CREATE TABLE tiny.t1 (id int, v string, part date) USING DELTA PARTITIONED BY (part);

In Trino: insert into delta.tiny.t1 values (1, 'A', TIMESTAMP '2024-01-01'), (2, 'B', TIMESTAMP '2024-01-01'), (3, 'C', TIMESTAMP '2024-02-02'), (4, 'D', TIMESTAMP '2024-02-02');

In Spark-sql: CREATE TABLE tiny.t1clone SHALLOW CLONE tiny.t1;

In Trino: update delta.tiny.t1clone set v = 'update1' where id in (1,3); It fails with:

Query 20240904_130833_00010_r4mc9 failed: path [s3://test-bucket/tiny/t1/part=2024-02-02/20240904_125421_00003_r4mc9_8e16bc1c-e33d-45da-b085-427d66e55960] must be a subdirectory of basePath [s3://test-bucket/tiny/t1clone]

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
(x) Release notes are required, with the following suggested text:

## Delta Lake
* Support MERGE on cloned table. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Jan 21, 2025
@github-actions github-actions bot added the delta-lake Delta Lake connector label Jan 21, 2025
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 6 times, most recently from 37ff93a to 60e9e73 Compare January 21, 2025 11:47
@ebyhr ebyhr requested review from ebyhr, pajaks and vinay-kl January 22, 2025 04:20
@chenjian2664 chenjian2664 force-pushed the delta_clone branch 2 times, most recently from 9bc68fc to 0cb664f Compare January 23, 2025 08:52
throw new RuntimeException(e);
}

checkArgument(sourceTableName != null && sourceTableName.contains(".") && sourceTableName.split("\\.").length == 3, "Unexpected source table in operation_parameters: %s", sourceTableName);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@chenjian2664 chenjian2664 Jan 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question about the path in the remove entry in the _/delta_log/xxx.json file:
Since it is possible that the cloned table reference a source table file(absolute path in s3), so here means I have to modify the path that to make it work befor upload the files?
i, e: the path in removeEntry is s3://aaa/bbb/ccc.parquet, then for the test, I may create two directory(source, cloned), so here the path may change to ../source/aaa/bbb/ccc.parquet in cloned table

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the test testMergeOnClonedTable in TestDeltaLakeBasic, PTAL
The absolute path of file in the source table that referenced by cloned table is changed to relative path with logic mentioned above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed delta-lake Delta Lake connector
Development

Successfully merging this pull request may close these issues.

2 participants