elasticsearch学习笔记

1.Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot determine write shards for

在config/analysis下创建 synonym.txt文件即可

The index operation automatically creates an index if it has not been created before (check out thecreate
index API for manually creating an index), and also automatically creates a dynamic type mapping for the specific type if one has
not yet been created (check out the put
mapping API for manually creating a type mapping)

es 会自动创建index和type如果不存在的话

2.

//利用隐式转换，native的支持

  implicit class ESConfig(conf: SparkConf) {

    def build(index: String, typeTable: String, mapId: Option[String] = None): SparkConf = {

      val sc = conf

        //        .set("spark.buffer.pageSize", "8m")

        .set("es.nodes", config.nodes)

        .set("es.port", config.port)

        .set("es.scroll.size", "2000")

        .set("es.resource", s"$index/$typeTable")

        .set("es.index.auto.create", "true")

        .set("es.write.operation", "upsert")

        .set("es.batch.size.entries", config.batchSize)

      mapId match {

        case Some(id) => sc.set("es.mapping.id", id)

        case None => sc

      }

    }

  }