ElasticSearch 2 (2) - Setup

ElasticSearch 2.1.1 (2) - Setup

Installation

Elasticsearch can be started using:

$ bin/elasticsearch

Running as a daemon

$ bin/elasticsearch -d

PID

The PID is written to a file called pid.

$ bin/elasticsearch -d -p pid

The kill command sends a TERM signal to the PID stored in the pid file.

$ kill `cat pid`

Another feature is the ability to pass -D or getopt long style configuration parameters directly to the script. When set, all override anything set using either JAVA_OPTS or ES_JAVA_OPTS. For example:

$ bin/elasticsearch -Des.index.refresh_interval=5s --node.name=my-node

Configuration

Environment Variables

JAVA_OPTS

ES_JAVA_OPTS > JAVA_OPTS

ES_HEAP_SIZE

ES_MIN_MEM (defaults to 256m)
ES_MAX_MEM (defaults to 1g)

It is recommended to set the min and max memory to the SAME value, and enable mlockall.

System Configuration

File Descriptors

Setting it to 32k or even 64k is recommended.

-Des.max-open-files=true

max_file_descriptors

curl localhost:9200/_nodes/stats/process?pretty

Virtual memory

Elasticsearch uses a hybrid mmapfs / niofs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions. On Linux, you can increase the limits by running the following command as root:

sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in /etc/sysctl.conf.

Memory Settings

Most operating systems try to use as much memory as possible for file system caches and eagerly swap out unused application memory, possibly resulting in the elasticsearch process being swapped. Swapping is very bad for performance and for node stability, so it should be avoided at all costs.

Disable swap

Box -

ES_HEAP_SIZE

Linux -

Temporarily

sudo swapoff -a

Permanently (comment out swap)

/etc/fstab

Windows -

System Properties → Advanced → Performance → Advanced → Virtual memory

Configure swappiness

This reduces the kernel’s tendency to swap and should not lead to swapping under normal circumstances, while still allowing the whole system to swap in emergency conditions.

vm.swappiness

mlockall

Linux - mlockall

bootstrap.mlockall: true

Windows - VirtualLock

Check:

curl http://localhost:9200/_nodes/process?pretty

Grant(as root):

$ ulimit -l unlimited

Directory(/tmp):

./bin/elasticsearch -Djna.tmpdir=/path/to/new/dir

Elasticsearch Settings

Directory

ES_HOME/config

module - elasticsearch.yml

logging - logging.yml

network

The address all network based modules will use to bind and publish to:

network :
host : 10.0.0.4

Path

path:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch

Cluster name

Don’t forget to give your production cluster a name, which is used to discover and auto-join other nodes:

cluster:
name: <NAME OF YOUR CLUSTER>

Don’t reuse the same cluster names in different environment

development: logging-dev

staging: logging-stage

production: logging-prod

Node name

Default (Marvel Character Name)

Provided

node:
name: <NAME OF YOUR NODE>

Single Node on Machine

node:
name: ${HOSTNAME}

Configuration style

JSON

elasticsearch.json:

{
"network" : {
"host" : "10.0.0.4"
}
}

Command

$ elasticsearch -Des.network.host=10.0.0.4

Default

es.default.

Environment

{
"network" : {
"host" : "${ES_NET_HOST}"
}
}

Non-store

${prompt.text} or ${prompt.secret}

node:
name: ${prompt.text}

On execution:

Enter value for [node.name]:

Elasticsearch will not start if ${prompt.text} or ${prompt.secret} is used in the settings and the process is run as a service or in the background.

Index Settings

Index Level (YAML or JSON)

$ curl -XPUT http://localhost:9200/kimchy/ -d \
'
index:
refresh_interval: 5s
'

Node Level (elasticsearch.yml)

index :
refresh_interval: 5s

Collapsed

$ elasticsearch -Des.index.refresh_interval=5s

Logging

Log4j (log4j-extras)

Format

.yml

.yaml

.json

.properties

Deprecation

config/logging.yml

deprecation: DEBUG, deprecation_log_file

Running as a Service on Linux

Linux

ES_USER

The user to run as, defaults to elasticsearch

ES_GROUP

The group to run as, defaults to elasticsearch

ES_HEAP_SIZE

The heap size to start with

ES_HEAP_NEWSIZE

The size of the new generation heap

ES_DIRECT_SIZE

The maximum size of the direct memory

MAX_OPEN_FILES

Maximum number of open files, defaults to 65535

MAX_LOCKED_MEMORY

Maximum locked memory size. Set to "unlimited" if you use the bootstrap.mlockall option in elasticsearch.yml. You must also set ES_HEAP_SIZE.

MAX_MAP_COUNT

Maximum number of memory map areas a process may have. If you use mmapfs as index store type, make sure this is set to a high value. For more information, check the linux kernel documentation about max_map_count. This is set via sysctl before starting elasticsearch. Defaults to 65535

LOG_DIR

Log directory, defaults to /var/log/elasticsearch

DATA_DIR

Data directory, defaults to /var/lib/elasticsearch

CONF_DIR

Configuration file directory (which needs to include elasticsearch.yml and logging.yml files), defaults to /etc/elasticsearch

ES_JAVA_OPTS

Any additional java options you may want to apply. This may be useful, if you need to set the node.name property, but do not want to change the elasticsearch.yml configuration file, because it is distributed via a provisioning system like puppet or chef. Example: ES_JAVA_OPTS="-Des.node.name=search-01"

RESTART_ON_UPGRADE

Configure restart on package upgrade, defaults to false. This means you will have to restart your elasticsearch instance after installing a package manually. The reason for this is to ensure, that upgrades in a cluster do not result in a continuous shard reallocation resulting in high network traffic and reducing the response times of your cluster.

ES_GC_LOG_FILE

The absolute log file path for creating a garbage collection logfile, which is done by the JVM. Note that this logfile can grow pretty quick and thus is disabled by default.

Debian/Ubuntu

runlevels

update-rc.d

init script

/etc/init.d/elasticsearch

configuration

/etc/default/elasticsearch

after install

dpkg -i

start on boot and start up

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

Oracle JDK

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version

RPM based distribution

Using chkconfig

init script

/etc/init.d/elasticsearch

configuration

/etc/sysconfig/elasticsearch

manual start

sudo /sbin/chkconfig --add elasticsearch
sudo service elasticsearch start

Using systemd

system rpm based

/etc/default/elasticsearch

configuration

/etc/sysconfig/elasticsearch

up

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
sudo /bin/systemctl start elasticsearch.service

MAX_MAP_COUNT setting

/etc/sysconfig/elasticsearch (no effect)

/usr/lib/sysctl.d/elasticsearch.conf

Running as a Service on Windows

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-service-win.html

Directory Layout

Type(Setting) - Default Location

Description

home(path.home) - N/A

Home of elasticsearch installation.

bin(N/A) - {path.home}/bin

Binary scripts including elasticsearch to start a node.

conf(path.conf) - {path.home}/config

Configuration files including elasticsearch.yml

data(path.data) - {path.home}/data

The location of the data files of each index / shard allocated on the node. Can hold multiple locations.

logs(path.logs) - {path.home}/logs

Log files location.

plugins(path.plugins) - {path.home}/plugins

Plugin files location. Each plugin will be contained in a subdirectory.

repo(path.repo) - Not configured

Shared file system repository locations. Can hold multiple locations. A file system repository can be placed in to any subdirectory of any directory specified here.

script(path.script) - {path.conf}/scripts

Location of script files.

Multiple data paths

---------------------------------
path.data: /mnt/first,/mnt/second
---------------------------------

Or array format

----------------------------------------
path.data: ["/mnt/first", "/mnt/second"]
----------------------------------------

Default Path

deb & rpm

-----------------------------------------------------------------------------------
Type    | Location Debian/Ubuntu            | Location RHEL/CentOS
-----------------------------------------------------------------------------------
home    | /usr/share/elasticsearch          | /usr/share/elasticsearch
-----------------------------------------------------------------------------------
bin     | /usr/share/elasticsearch/bin      | /usr/share/elasticsearch/bin
-----------------------------------------------------------------------------------
conf    | /etc/elasticsearch                | /etc/elasticsearch
-----------------------------------------------------------------------------------
conf    | /etc/default/elasticsearch        | /etc/sysconfig/elasticsearch
-----------------------------------------------------------------------------------
data    | /var/lib/elasticsearch/data       | /var/lib/elasticsearch
-----------------------------------------------------------------------------------
logs    | /var/log/elasticsearch            | /var/log/elasticsearch
-----------------------------------------------------------------------------------
plugins | /usr/share/elasticsearch/plugins  | /usr/share/elasticsearch/plugins
-----------------------------------------------------------------------------------
repo    | Not configured                    | Not configured
-----------------------------------------------------------------------------------
script  | /etc/elasticsearch/scripts        | /etc/elasticsearch/scripts
-----------------------------------------------------------------------------------

zip

-----------------------------------------------------------------------------------
Type    | Description                                   | Location
-----------------------------------------------------------------------------------
home    | Home of elasticsearch installation            | {extract.path}
-----------------------------------------------------------------------------------
bin     | scripts to start a node                       | {extract.path}/bin
-----------------------------------------------------------------------------------
conf    | files elasticsearch.yml and logging.yml       | {extract.path}/config
-----------------------------------------------------------------------------------
data    | location of files of each index / shard       | {extract.path}/data
-----------------------------------------------------------------------------------
logs    | Log files location                            | {extract.path}/logs
-----------------------------------------------------------------------------------
plugins | Plugin files location.                        | {extract.path}/plugins
-----------------------------------------------------------------------------------
repo    | Shared file system repository locations.      | Not configured
-----------------------------------------------------------------------------------
script  | Location of script files.                     | {extract.path}/config/scripts
-----------------------------------------------------------------------------------

Repositories

PGP Key

We use the PGP key D88E42B4, Elasticsearch Signing Key, with fingerprint

4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4

APT

Download and install the Public Signing Key:

wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Save the repository definition to /etc/apt/sources.list.d/elasticsearch-2.x.list:

echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

Run apt-get update and the repository is ready for use. You can install it with:

sudo apt-get update && sudo apt-get install elasticsearch

Configure Elasticsearch to automatically start during bootup. If your distribution is using SysV init, then you will need to run:

sudo update-rc.d elasticsearch defaults 95 10

Otherwise if your distribution is using systemd:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

YUM

Download and install the public signing key:

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch[/code] Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example elasticsearch.repo

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

And your repository is ready for use. You can install it with:

yum install elasticsearch


SysV init

chkconfig --add elasticsearch


Systemd

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service


Reference
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html