离线开发 - INFO-Spark Jar开发提交

INFO-Spark Jar开发提交

更新时间: 2024-03-11 02:50:51 | 阅读 2763

扫码

复制

导出

Spark Jar应用开发

Spark Jar应用开发

maven工程中导入依赖，版本供参考

<properties>
    <scala.version>2.11.8</scala.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <hadoop.version>2.9.2</hadoop.version>
    <spark.version>2.3.2</spark.version>
    <hive.version>2.1.1</hive.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>${scala.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>${spark.version}</version>
        <!--<scope>provided</scope>-->
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <version>${spark.version}</version>
        <!--<scope>provided</scope>-->
    </dependency>
</dependencies>

新建Spark作业：demo.SparkDemo

package demo

import org.apache.spark.sql.SparkSession

object SparkDemo {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("SparkDemo")
      .enableHiveSupport()
      .getOrCreate()

    val table = "demo.ods_acct_acc_transaction"
    val df = spark.sql(s"select count(*) from $table")

    df.show()

    spark.stop()
  }
}