spark怎么结合es使用

 时间:2026-02-12 08:06:29

1、启动es后,spark shell导入es-hadoop jar包:cp elasticsearch-hadoop-2.1.2/dist/elasticsearch-spark* spark-1.6.0-bin-hadoop2.6/lib/

cd spark-1.6.0-bin-hadoop2.6/bin

./spark-shell --jars ../lib/elasticsearch-spark-1.2_2.10-2.1.2.jar

2、互交结果如下:import org.apache.spark.SparkConf

import org.elasticsearch.spark._

val conf = new SparkConf()

conf.set("es.index.auto.create", "true")

conf.set("es.nodes", "127.0.0.1")

val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)

val airports = Map("OTP" -> "Otopeni", "SFO" -> "San Fran")

sc.makeRDD(Seq(numbers, airports)).saveToEs("spark/docs")

3、然后查看ES中的数据:http://127.0.0.1:9200/spark/docs/_search?q=*

4、结果就是这样啦:{"took":71,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":2,"max_score":1.0,"hits":[{"_index":"spark","_type":"docs","_id":"AVfhVqPBv9dlWdV2DcbH","_score":1.0,"_source":{"OTP":"Otopeni","SFO":"San Fran"}},{"_index":"spark","_type":"docs","_id":"AVfhVqPOv9dlWdV2DcbI","_score":1.0,"_source":{"one":1,"two":2,"three":3}}]}}

  • 流水线生产看板触摸屏云终端解决方案
  • CentOS下Apache多站点配置
  • [QtCreator(3)] 新建Qt静态库项目[TZZ]
  • win10如何开启远程桌面
  • win10如何查看启动程序信息?
  • 热门搜索
    什么是科学发展观 赣州有什么好玩的地方 多汗是什么原因 今天是什么节日查询 脸颊两边长痘痘是什么原因引起的 restaurant什么意思 什么是命题 今年是什么生肖年 省略号的作用是什么 茶喝多了有什么坏处