问题描述/异常栈
离线传输,源端字段设置自定义表达式: FAQ-离线传输,源端报错“自定义表达式”not found in schema - 图1
任务报错如下:
2025-12-30 17:59:57 CST SparkExecutor ERROR - Diagnostics: User class threw exception: java.lang.Exception: Column "substring(name, 2, 3)" not found in schema Some(StructType(StructField(id,IntegerType,true),StructField(name,StringType,true)))
    at com.netease.music.da.transfer.mysql.writer.MySQLWriter.$anonfun$getInsertStatement$4(MySQLWriter.scala:55)
    at scala.Option.getOrElse(Option.scala:189)
    at com.netease.music.da.transfer.mysql.writer.MySQLWriter.$anonfun$getInsertStatement$2(MySQLWriter.scala:55)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike.map(TraversableLike.scala:286)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
    at com.netease.music.da.transfer.mysql.writer.MySQLWriter.getStatement$1(MySQLWriter.scala:53)
    at com.netease.music.da.transfer.mysql.writer.MySQLWriter.getInsertStatement(MySQLWriter.scala:76)
    at com.netease.music.da.transfer.jdbc.writer.JDBCWriter.doWrite(JDBCWriter.scala:176)
    at com.netease.music.da.transfer.common.writer.AbstractDataWriter.write(AbstractDataWriter.scala:32)
    at com.netease.music.da.transfer.common.Worker$.main(Worker.scala:98)
    at com.netease.music.da.transfer.common.Worker.main(Worker.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:723)
发现版本
EasyTransfer 3.37.0
问题原因
源端导入字段设置的自定义表达式没有指定别名导致
解决方案
将源端导入字段as一个别名后解决
FAQ-离线传输,源端报错“自定义表达式”not found in schema - 图2

作者:曹俊