FAQ-Caused by: org.apache.spark.SparkUpgradeExcep

问题描述/异常栈
Caused by: org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: reading dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z from Parquet INT96 files can be ambiguous, as the files may be written by Spark 2.x or legacy versions of Hive, which uses a legacy hybrid calendar that is different from Spark 3.0+'s Proleptic Gregorian calendar. See more details in SPARK-31404. You can set spark.sql.legacy.parquet.int96RebaseModeInRead to 'LEGACY' to rebase the datetime values w.r.t. the calendar difference during reading. Or set spark.sql.legacy.parquet.int96RebaseModeInRead to 'CORRECTED' to read the datetime values as it is.
发现版本
spark3
解决方案
spark.sql.legacy.parquet.int96RebaseModeInRead=LEGACY或CORRECTED
问题原因
参照https://kms.netease.com/topics/topic/257/item/15794

作者：林帅