Arrow, Parquet, Spark 3.5, Flink 1.20: Avoid deprecated method #11874
Conversation
| @@ -251,7 +251,8 @@ private String getCompressionType(InputFile inputFile) throws Exception { | |||
| return orcReader.getCompressionKind().name(); | |||
| case PARQUET: | |||
| ParquetMetadata footer = | |||
| ParquetFileReader.readFooter(CONF, new Path(inputFile.location()), NO_FILTER); | |||
There was a problem hiding this comment.
readFooter method is deprecated: https://javadoc.io/doc/org.apache.parquet/parquet-hadoop/latest/index.html
| @@ -107,8 +107,8 @@ public void testAlterTable() throws NoSuchTableException { | |||
|
|
|||
| assertThat(table).as("Should return updated table").isNotNull(); | |||
|
|
|||
| StructField expectedField = DataTypes.createStructField(fieldName, DataTypes.StringType, true); | |||
| assertThat(table.schema().fields()[2]) | |||
There was a problem hiding this comment.
|
|
||
| } finally { | ||
| visitor.fieldNames.pop(); | ||
| if (annotation.equals(LogicalTypeAnnotation.listType())) { |
There was a problem hiding this comment.
I would recommend reviewing this changer with "Hide whitespace" option.
I changed switch to if because LogicalTypeAnnotation isn't enum.
| @@ -59,106 +59,101 @@ public static <T> T visit(DataType sType, Type type, ParquetWithSparkSchemaVisit | |||
| } else { | |||
| // if not a primitive, the typeId must be a group | |||
| GroupType group = type.asGroupType(); | |||
| OriginalType annotation = group.getOriginalType(); | |||
There was a problem hiding this comment.
OriginalType enum is deprecated: https://www.javadoc.io/doc/org.apache.parquet/parquet-column/latest/org/apache/parquet/schema/OriginalType.html
There was a problem hiding this comment.
there are a few more places in the codebase that use this, so maybe those places should be updated as well as part of this PR?
There was a problem hiding this comment.
Updated except for Spark 3.4 & 3.3 and Flink 1.19 & 1.18.
|
CI hit #11651 |
|
|
||
| } finally { | ||
| visitor.fieldNames.pop(); | ||
| if (annotation.equals(LogicalTypeAnnotation.listType())) { |
There was a problem hiding this comment.
it would be better if the check is the other way around, because the annotation could be null: LogicalTypeAnnotation.listType().equals(annotation).
That way we could save ourselves one nesting and remove the if (annotation != null) block
There was a problem hiding this comment.
This block is inside of annotation != null. Do you want to remove the outer if?
There was a problem hiding this comment.
yes, let's remove that if block
ebyhr
changed the title
No description provided.