[[elasticsearch-integration]]
== Integration with Elasticsearch

// vim: set colorcolumn=100:

=== Status

[CAUTION]
====
This feature is a work in progress.
Make sure to read the <<elasticsearch-limitations,Limitations>> section!
====

The integration with Elasticsearch is in development and should be considered experimental.
We do think we have the basics covered and we are looking for feedback.

Patches can be sent as pull requests to the https://github.com/hibernate/hibernate-search[Github repository],
but also general feedback, suggestions and questions are very welcome.
To get in touch or find other interesting links for contributors, see the http://hibernate.org/community/[Community page]
of the Hibernate website.

=== Goal of the Elasticsearch integration

The goal of integrating with Elasticsearch is to allow Hibernate Search users to benefit
from the full-text capabilities integrated with Hibernate ORM
but replacing the local Lucene based index with a remote Elasticsearch service.

There could be various reasons to prefer this over an "embedded Lucene" approach:

* wish to separate the service running your application from the Search service (Microservices)
* benefit from Elasticsearch's out of the box horizontal scalability and high availability features
* integrate with an existing Elasticsearch cluster
* explore the data updated by an Hibernate powered application using the Elasticsearch dashboard integrations such as Kibana

There are a couple of drawbacks compared to the embedded Lucene approach though:

* incur a performance penalty of remote RPCs both for index updates and to run queries
* need to manage an additional service
* possibly need to buy and manage additional servers

Which solution is best will depend on the specific needs of your system and your organization.

[NOTE]
.Why not use Elasticsearch directly?
--
The #1 reason is that Hibernate Search integrates perfectly with Hibernate ORM.
All changes done to your objects will trigger the necessary index changes transparently.

* it will honor the transaction boundary - i.e. not do the indexing work if the transaction ends up in rollback
* changes to cascaded objects are handled
* changes to nested object embedded in a root indexed entity are handled
* changes will be sent in bulk - i.e. optimized systematically for you
* it generates JSON encoded requests from your entities, parses reponses
* manages the connection pools
* etc.

There is no more paradigm shift in your code.
You are working on Hibernate ORM managed objects,
doing your queries on object properties with a nice Java DSL.
--

=== Getting started and configuration

To experiment with the Elasticsearch integration you will have to download Elasticsearch and run it:
Hibernate Search connects to an Elasticsearch node but does not provide one.

One option is to use the https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html[Elasticsearch Docker image].

==== Elasticsearch version

Hibernate Search expects an Elasticsearch cluster running version 2.x or 5.x.
The version running on your cluster will be automatically detected on startup,
and Hibernate Search will adapt based on the detected version.

The targeted version is mostly transparent to Hibernate Search users,
but there are a few differences in how Hibernate Search behaves depending
on the Elasticsearch version that may affect you.
The following table details those differences.

[cols="h,3*",options="header"]
|===============
||2.x|5.0/5.1|5.2+
|Configuration required for purges|<<elasticsearch-integration-server-configuration,Enable the delete-by-query plugin>> 2+|None
|Datatype used for String fields in Elasticsearch
 |`string`
 |`text` (if analyzed) or `keyword` (if not).
 The `string` datatype https://www.elastic.co/guide/en/elasticsearch/reference/5.0/string.html[has been deprecated in Elasticsearch 5.0].
 |`text` (if analyzed with an <<analyzer,analyzer>>) or `keyword` (if not analyzed or analyzed with a <<section-normalizers,normalizer>>).
|<<field-annotation-norms,Norms>>|Not implemented 2+|Implemented
|Implementation of <<field-annotation-indexNullAs,`@Field.indexNullAs`>> for analyzed text fields
 |https://www.elastic.co/guide/en/elasticsearch/reference/2.4/null-value.html[`null_value`] is added to the mapping,
  `null` values are indexed as such.
 2+|https://www.elastic.co/guide/en/elasticsearch/reference/5.0/null-value.html[`null_value`] is not available for `text` fields,
  null values are replaced with the `indexNullAs` value explicitly when indexing.
  +
  This does not apply to `keyword` fields.
|Configuration required for spatial projections without a spatial sort
 |<<elasticsearch-integration-server-configuration,Enable and configure the groovy plugin>>
 2+|None
|===============

[NOTE]
--
The testsuite of Hibernate Search runs against Elasticsearch {testElasticsearchVersion} by default.
--

===== Upgrading Elasticsearch

When upgrading your Elasticsearch cluster, some
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html[administrative tasks]
are still required on your cluster: Hibernate Search will not take care of those.

On top of that, there are some fundamental differences between some versions of Elasticsearch,
and for that reason, some upgrades will require you to delete your indexes manually and <<search-batchindex,reindex your data>>.

The upgrades that require such changes are the following:

* upgrading from 2.x to 5.x, because Hibernate Search does not support the `string` datatype on Elasticsearch 5.x;
* upgrading from 5.0/5.1 to 5.2+ if you used <<section-normalizer,normalizers>>,
because Hibernate Search normalizers are mapped to native Elasticsearch normalizers starting from Elasticsearch 5.2,
thereby changing the datatype of fields using normalizers from `text` to `keyword`.

[[elasticsearch-integration-dependencies]]
==== Dependencies in your Java application

In addition to the usual dependencies like Hibernate ORM and Hibernate Search,
you will need the new `hibernate-search-elasticsearch` jar.

.Maven dependencies for Hibernate Search with Elasticsearch
====
[source, XML]
[subs="verbatim,attributes"]
----
<dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-search-elasticsearch</artifactId>
   <version>{hibernateSearchVersion}</version>
</dependency>
----
====

If you use Amazon's proprietary IAM authentication through request signing to access Elasticsearch,
you will need an additional dependency to handle authentication:
`hibernate-search-elasticsearch-aws`.

.Maven dependencies for Hibernate Search with AWS-hosted Elasticsearch
====
[source, XML]
[subs="verbatim,attributes"]
----
<dependency>
   <groupId>org.hibernate</groupId>
   <artifactId>hibernate-search-elasticsearch-aws</artifactId>
   <version>{hibernateSearchVersion}</version>
</dependency>
----
====

[[elasticsearch-integration-server-configuration]]
==== Elasticsearch configuration

Hibernate Search can work with an Elasticsearch server without altering its configuration.

However some features offered by Hibernate Search require specific configuration:

* on Elasticsearch 2.x only (not necessary on 5.x): if you want to be able to use the Hibernate Search <<search-batchindex-massindexer,MassIndexer>>
with `purgeAllOnStart` enabled - it is enabled by default -, or to use `FullTextSession.purge()` or `FullTextSession.purgeAll()`,
install the link:https://www.elastic.co/guide/en/elasticsearch/plugins/2.2/plugins-delete-by-query.html[`delete-by-query` plugin]
* on Elasticsearch 2.x only (not necessary on 5.x): if you want to retrieve the distance in a geolocation query, enable the `lang-groovy` plugin,
see <<elasticsearch-query-spatial, Elasticsearch Spatial queries>>
* if you want to use paging (as opposed to <<elasticsearch-scrolling,scrolling>>) on result sets larger than 10000 elements
(for instance access the 10001st result),
you may increase the value of the `index.max_result_window` property (default is 10000).

[[elasticsearch-integration-configuration]]
==== Hibernate Search configuration

Configuration is minimal.
Add the configuration properties to your `persistence.xml` or where you put the rest of your Hibernate Search configuration.

Select Elasticsearch as the backend:: `hibernate.search.default.indexmanager elasticsearch`
Hostname and port for Elasticsearch:: `hibernate.search.default.elasticsearch.host \http://127.0.0.1:9200` (default)
+
You may also select multiple hosts (separated by whitespace characters), so that they are assigned requests in turns (load balancing):
+
`hibernate.search.default.elasticsearch.host \http://es1.mycompany.com:9200 \http://es2.mycompany.com:9200`
+
In the example above, the first request will go to `es1`, the second to `es2`, the third to `es1`, and so on.
+
Also note having multiple hosts will enable failover:
if one node happens to fail to serve a request (timeout, server error, invalid HTTP response, ...),
the same request will be sent to the next one; if the second request is served without error,
the failure will be blamed on the node and no error will be reported to the application.
+
The failover feature will also be enabled when you only have one configured host
but other hosts have been added thanks to automatic discovery (see below).
Username for Elasticsearch connection:: `hibernate.search.default.elasticsearch.username ironman` (default is empty, meaning anonymous access)
Password for Elasticsearch connection:: `hibernate.search.default.elasticsearch.password j@rV1s` (default is empty)
+
[CAUTION]
====
If you use HTTP instead of HTTPS in any of the Elasticsearch host URLs (see above),
your password will be transmitted in clear text over the network.
====
[[elasticsearch-schema-management-strategy]]Select the index creation strategy::
`hibernate.search.default.elasticsearch.index_schema_management_strategy CREATE` (default)
+
Let's see the options for the `index_schema_management_strategy` property:
+
[options="header"]
|===============
|Value|Definition
|`none`|The index, its mappings and the analyzer definitions will not be created, deleted nor altered.
Hibernate Search will **not even check** that the index already exists.
|`validate`|The index, its existing mappings and analyzer definitions will be checked to be compatible with the metamodel of your application.
The index, its mappings and analyzer definitions will not be created, deleted nor altered.
|`update`|The index, its mappings and analyzer definitions will be created, existing mappings will be updated if there are no conflicts.
Caution: if analyzer definitions have to be updated, the index will be closed automatically during the update.
|`create`|**The default**: an existing index will not be altered, a missing index will be created along with their mappings and analyzer definitions.
|`drop-and-create`|Indexes will be deleted if existing and then created along with their mappings and analyzer definitions to match the requirements of your appllication.
This will delete all content from the indexes! Useful during development.
|`drop-and-create-and-drop`|Similar to `drop-and-create` but will also delete the index at shutdown. Commonly used for tests.
|===============
+
[WARNING]
--
Since Elasticsearch on Amazon Web Services (AWS)
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-supported-es-operations.html[does not support the `_close`/`_open` operations],
the `update` strategy will fail when trying to update analyzer definitions
on an AWS Elasticsearch cluster.

The only workaround is to avoid the `update` strategy on AWS.
--
+
[WARNING]
.Strategies in production environments
====
It is strongly recommended to use either `none` or `validate` in a production environment.

The alternatives `drop-and-create` and `drop-and-create-and-drop` are obviously unsuitable in this context
unless you want to reindex everything upon every startup,
and `update` may leave your mapping half-updated in case of conflict.

To be precise, if your mapping changed in an incompatible way, such as a field having its type changed,
updating the mapping may be impossible without manual intervention.
In this case, the `update` strategy will prevent Hibernate Search from starting,
but it may already have successfully updated the mappings for another index, making a rollback difficult.

When updating analyzer definitions Hibernate Search will temporarily stop the affected indexes during the update.
This means the `update` strategy should be used with caution when multiple clients use Elasticsearch indexes managed by Hibernate Search:
those clients should be synchronized in such a way that while Hibernate Search is starting, no other client needs to access the index.

For these reasons, migrating your mapping on a live cluster should be properly planned within the deployment process.
====
+
[NOTE]
--
Mapping validation is as permissive as possible. Fields or mappings that are unknown to Hibernate Search will be ignored, and settings that are more powerful than required (e.g. a field annotated with `@Field(index = Index.NO)` in Search but marked as `"index": analyzed` in Elasticsearch) will be deemed valid.

One exception: date formats must match exactly the formats specified by Hibernate Search, due to implementation constraints.
--
Maximum time to wait for the successful execution of a request to the Elasticsearch server before failing (in ms):: `hibernate.search.default.elasticsearch.request_timeout 60000` (default)
+
The execution time of a request includes the time needed to establish a connection,
to send the request, and to receive the whole response,
optionally retrying in case of node failures.
Maximum time to wait for a connection to the Elasticsearch server before failing (in ms):: `hibernate.search.default.elasticsearch.connection_timeout 3000` (default)
Maximum time to wait for a response from the Elasticsearch server before failing (in ms):: `hibernate.search.default.elasticsearch.read_timeout 60000` (default)
Maximum number of simultaneous connections to the Elasticsearch cluster:: `hibernate.search.default.elasticsearch.max_total_connection 20` (default)
Maximum number of simultaneous connections to a single Elasticsearch server:: `hibernate.search.default.elasticsearch.max_total_connection_per_route 2` (default)
Whether to enable automatic discovery of servers in the Elasticsearch cluster (`true` or `false`):: `hibernate.search.default.elasticsearch.discovery.enabled false` (default)
+
When using automatic discovery, the Elasticsearch client will periodically probe for new nodes in the cluster,
and will add those to the server list (see `host` above).
Similarly, the client will periodically check whether registered servers still respond,
and will remove them from the server list if they don't.
Time interval between two executions of the automatic discovery (in seconds):: `hibernate.search.default.elasticsearch.discovery.refresh_interval 10` (default)
+
This setting will only be taken into account if automatic discovery is enabled (see above).
Scheme to use when connecting to automatically discovered nodes (`http` or `https`):: `hibernate.search.default.elasticsearch.discovery.default_scheme http` (default)
+
This setting will only be taken into account if automatic discovery is enabled (see above).
Maximum time to wait for the indexes to become available before failing (in ms):: `hibernate.search.default.elasticsearch.index_management_wait_timeout 10000` (default)
+
This setting is ignored when the `NONE` strategy is selected, since the index will not be checked on startup (see above).
+
This value must be lower than the read timeout (see above).
Status an index must at least have in order for Hibernate Search to work with it (one of "green", "yellow" or "red")::
`hibernate.search.default.elasticsearch.required_index_status green` (default)
+
Only operate if the index is at this level or safer.
In development, set this value to `yellow` if the number of nodes started is below the number of expected replicas.
Whether to perform an explicit refresh after a set of operations has been executed against a specific index (`true` or `false`)::
`hibernate.search.default.elasticsearch.refresh_after_write false` (default)
+
This is useful in unit tests to ensure that a write is visible by a query immediately without delay.
This keeps unit tests simpler.
You should not rely on the synchronous behaviour for your production code except in rare cases as Elasticsearch is optimised for asynchronous writes:
leave at `false` for optimal performance.
When <<elasticsearch-scrolling,scrolling>>, the minimum number of previous results kept in memory at any time:: `hibernate.search.elasticsearch.scroll_backtracking_window_size 10000` (default)
When <<elasticsearch-scrolling,scrolling>>, the number of results fetched by each Elasticsearch call:: `hibernate.search.elasticsearch.scroll_fetch_size 1000` (default)
When <<elasticsearch-scrolling,scrolling>>, the maximum duration `ScrollableResults` will be usable if no other results are fetched from Elasticsearch, in seconds::
`hibernate.search.elasticsearch.scroll_timeout 60` (default)
The behavior with respect to dynamically added fields in the Elasticsearch mapping::
`hibernate.search.default.elasticsearch.dynamic_mapping strict` (default)
+
Possible values are:
+
* `true`: Add unknown fields to the schema dynamically
* `false`: Ignore unknown fields
* `strict`: Throw an exception on unknown fields
+
This setting may be overridden in custom field bridges on a per-field basis:
see <<elasticsearch-custom-field-bridge-dynamic-mapping>>.
[[elasticsearch-log-json-pretty-printing]] Whether JSON included in logs should be pretty-printed (indented, with line breaks) (`true` or `false`)::
`hibernate.search.elasticsearch.log.json_pretty_printing false` (default)

[NOTE]
--
Properties prefixed with `hibernate.search.default` can be given globally as shown above and/or be given for specific indexes:

`hibernate.search.someindex.elasticsearch.index_schema_management_strategy update`

This excludes properties related to the internal Elasticsearch client, which at the moment is common to every index manager (but this will change in a future version).
Excluded properties are `host`, `username`, `password`, `read_timeout`, `connection_timeout`, `max_total_connection`, `max_total_connection_per_route`, `http_client_configurer`,
`discovery.enabled`, `discovery.refresh_interval` and `discovery.scheme`.
--

[[elasticsearch-integration-configuration-aws]]
===== Authentication on Amazon Web Services

Amazon's proprietary IAM authentication through request signing <<elasticsearch-integration-dependencies,requires additional dependencies>>.
With these dependencies, you will be able to use the following configuration options.

Enable or disable AWS request signing:: `hibernate.search.default.elasticsearch.aws.signing.enabled false` (default)
+
The value must be either `true` or `false`.
+
AWS access key:: `hibernate.search.default.elasticsearch.aws.access_key AKIDEXAMPLE`
+
This property has no default and must be provided for the AWS authentication to work.
+
AWS secret key:: `hibernate.search.default.elasticsearch.aws.secret_key wJalrXUtnFEMI/K7MDENG+bPxRfiCYEXAMPLEKEY`
+
This property has no default and must be provided for the AWS authentication to work.
+
AWS region:: `hibernate.search.default.elasticsearch.aws.region us-east-1`
+
This property has no default and must be provided for the AWS authentication to work.

Should you need help with finding the correct values to use,
please refer to the AWS documentation on http://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html[security credentials]
and http://docs.aws.amazon.com/general/latest/gr/rande.html[regions].

[NOTE]
--
Setting these AWS-specific configuration properties is not enough to enable the Elasticsearch integration:
you should also have a look at the <<elasticsearch-integration-configuration,main configuration properties>>,
most notably `hibernate.search.default.indexmanager`
and `hibernate.search.default.elasticsearch.host`.
--

=== Mapping and indexing

Like in Lucene embedded mode, indexes are transparently updated when you create or update
entities mapped to Hibernate Search.
Simply use familiar annotations from <<search-mapping>>.

The name of the index will be the lowercased name provided to `@Indexed` (non qualified class name by default).
Hibernate Search will map the fully qualified class name to the Elasticsearch type.

==== Annotation specificities

===== Field.indexNullAs

The `org.hibernate.search.annotations.Field` annotation allows you to provide a replacement value for null properties through the `indexNullAs` attribute (see <<field-annotation>>), but this value must be provided as a string.

In order for your value to be understood by Hibernate Search (and Elasticsearch), the provided string must follow one of those formats:

 * For string values, no particular format is required.
 * For numeric values, use formats accepted by `Double.parseDouble`, `Integer.parseInteger`, etc., depending on the actual type of your field.
 * For booleans, use either `true` or `false`.
 * For dates (`java.util.Calendar`, `java.util.Date`, `java.time.*`), use the ISO-8601 format.
+
The full format is `yyyy-MM-dd'T'HH:mm:ss.nZ[ZZZ]` (for instance `2016-11-26T16:41:00.006+01:00[CET]`).
Please keep in mind that part of this format must be left out depending on the type of your field, though.
For a `java.time.LocalDateTime` field, for instance,
the provided string must not include the zone offset (`+01:00`) or the zone ID (`[UTC]`), because those don't make sense.
+
Even when they make sense for the type of your field, the time and time zone may be omitted
 (if omitted, the time zone will be interpreted as the default JVM time zone).

===== Index-time boosting

The `org.hibernate.search.annotations.DynamicBoost` annotation is not (and cannot be)
supported with Elasticsearch, because the platform lacks per-document, index-time boosting capabilities.

The `@Boost` annotation will work, but since it's deprecated
we suggest to use query-time boosting instead: see <<search-query>>.

[[elasticsearch-mapping-analyzer]]
==== Analyzers

NOTE: Analyzers are treated differently than in Lucene embedded mode.

===== Built-in or server-defined analyzers

Using the `definition` attribute in the `@Analyzer` annotation, you can refer to the name of the
built-in Elasticsearch analyzer, or custom analyzers already registered on your Elasticsearch instances.

More information on analyzers, in particular those already built in Elasticsearch, can be found
in link:https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html[the Elasticsearch documentation].

[source, yml]
.Example of custom analyzers defined in the elasticsearch.yml
--
# Custom analyzer
index.analysis:
  analyzer.custom-analyzer:
    type: custom
    tokenizer: standard
    filter: [custom-filter, lowercase]
  filter.custom-filter:
    type : stop
    stopwords : [test1, close]
--

From there, you can use the custom analyzers by name in your entity mappings.

[source,java]
.Example of mapping that refers to custom and built-in analyzers on Elasticsearch
--
@Entity
@Indexed(index = "tweet")
public class Tweet {

    @Id
    @GeneratedValue
    private Integer id;

    @Field
    @Analyzer(definition = "english") // Elasticsearch built-in analyzer
    private String englishTweet;

    @Field
    @Analyzer(definition = "whitespace") // Elasticsearch built-in analyzer
    private String whitespaceTweet;

    @Field(name = "tweetNotAnalyzed", analyzer = Analyze.NO, store = Store.YES)
    // Custom analyzer:
    @Field(
        name = "tweetWithCustom",
        analyzer = @Analyzer(definition = "custom-analyzer")
    )
    private String multipleTweets;
}
--

You may also reference a built-in Lucene analyzer implementation using the `@Analyzer.impl` attribute:
Hibernate Search will translate the implementation to an equivalent Elasticsearch built-in type, if possible.

[NOTE]
====
Using the `@Analyzer.impl` attribute is not recommended with Elasticsearch because it will
never allow you to take full advantage of Elasticsearch analysis capabilities.
You cannot, for instance, use custom analyzer implementations: only built-in Lucene implementations are supported.

This feature is designed to help migrating an application that already used Hibernate Search,
moving from an embedded Lucene instance to an Elasticsearch cluster.
====

[source,java]
.Example of mapping that refers to a built-in analyzer on Elasticsearch using a Lucene implementation class
--
@Entity
@Indexed(index = "tweet")
public class Tweet {

    @Id
    @GeneratedValue
    private Integer id;

    @Field
    @Analyzer(impl = EnglishAnalyzer.class) // Elasticsearch built-in "english" analyzer
    private String englishTweet;

    @Field
    @Analyzer(impl = WhitespaceAnalyzer.class) // Elasticsearch built-in "whitespace" analyzer
    private String whitespaceTweet;

}
--

===== Custom analyzers using the `@AnalyzerDef` annotation

You can also define analyzers within your Hibernate Search mapping using the `@AnalyzerDef` annotation,
like you would <<section-named-analyzers,do with an embedded Lucene instance>>.
When Hibernate Search creates the Elasticsearch indexes, the relevant definitions will then be automatically added as a
https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html[custom analyzer]
in  the index settings.

Two different approaches allow you to define your analyzers with Elasticsearch.

The first, recommended approach is to use the factories provided by the `hibernate-search-elasticsearch` module:

 * `org.hibernate.search.elasticsearch.analyzer.ElasticsearchCharFilterFactory`
 * `org.hibernate.search.elasticsearch.analyzer.ElasticsearchTokenFilterFactory`
 * `org.hibernate.search.elasticsearch.analyzer.ElasticsearchTokenizerFactory`

Those classes can be passed to the `factory` attribute of
the `@CharFilterDef`, `@TokenFilterDef` and `@TokenizerDef` annotations.

The `params` attribute may be used to define the `type` parameter and any other parameter
accepted by Elasticsearch for this type.

The parameter values will be interpreted as JSON. The parser is not strict, though:

 * quotes around strings may be left out in some cases, as when a string only contains letters.
 * when quotes are required (e.g. your string may be interpreted as a number, and you don't want that),
   you may use single quotes instead of double quotes (which are painful to write in Java).

[NOTE]
====
You may use the `name` attribute of the `@CharFilterDef`, `@TokenFilterDef` and `@TokenizerDef` annotations
to define the exact name to give to that definition in the Elasticsearch settings.
====

[source,java]
.Example of mapping that defines analyzers on Elasticsearch using the `Elasticsearch*Factory` types
--
@Entity
@Indexed(index = "tweet")
@AnalyzerDef(
	name = "tweet_analyzer",
	charFilters = {
		@CharFilterDef(
			name = "custom_html_strip",
			factory = ElasticsearchCharFilterFactory.class,
			params = {
				@Parameter(name = "type", value = "'html_strip'"),
				// One can use Json arrays
				@Parameter(name = "escaped_tags", value = "['br', 'p']")
			}
		),
		@CharFilterDef(
			name = "p_br_as_space",
			factory = ElasticsearchCharFilterFactory.class,
			params = {
				@Parameter(name = "type", value = "'pattern_replace'"),
				@Parameter(name = "pattern", value = "'<p/?>|<br/?>'"),
				@Parameter(name = "replacement", value = "' '"),
				@Parameter(name = "tags", value = "'CASE_INSENSITIVE'")
			}
		)
	},
	tokenizer = @TokenizerDef(
		factory = ElasticsearchTokenizerFactory.class,
		params = {
			@Parameter(name = "type", value = "'whitespace'"),
		}
	)
)
public class Tweet {

    @Id
    @GeneratedValue
    private Integer id;

    @Field
    @Analyzer(definition = "tweet_analyzer")
    private String content;
}
--


The second approach is to configure everything as if you were using Lucene:
use the Lucene factories, their parameter names, and format the parameter values as required in Lucene.
Hibernate Search will automatically convert these definitions to the Elasticsearch equivalent.

[NOTE]
====
Referencing Lucene factories is not recommended with Elasticsearch because it will
never allow you to take full advantage of Elasticsearch analysis capabilities.

Here are the known limitations of the automatic translation:

 * a few factories have unsupported parameters, because those have no equivalent in Elasticsearch.
   An exception will be raised on startup if a parameter is not supported.
 * the `hyphenator` parameter for `HyphenatedWordsFilterFactory` must refer to
   a file on the Elasticsearch servers, on the contrary to other factories
   where the files are accessed by Hibernate Search directly.
   This is due to an Elasticsearch limitation
   (there is no way to forward the content of a local hyphenation pattern file).
 * some built-in Lucene factories are not (and cannot) be translated, because of incompatible
   parameters between the Lucene factory and the Elasticsearch equivalent.
   This is in particular the case for `HunspellStemFilterFactory`.

Therefore, Lucene factories should only be referenced within analyzer definitions
when migrating an application that already used Hibernate Search,
moving from an embedded Lucene instance to an Elasticsearch cluster.
====


[source,java]
.Example of mapping that defines analyzers on Elasticsearch using Lucene factories
--
@Entity
@Indexed(index = "tweet")
@AnalyzerDef(
	name = "tweet_analyzer",
	charFilters = {
		@CharFilterDef(
			name = "custom_html_strip",
			factory = HTMLStripCharFilterFactory.class,
			params = {
				@Parameter(name = "escapedTags", value = "br,p")
			}
		),
		@CharFilterDef(
			name = "p_br_as_space",
			factory = PatternReplaceCharFilterFactory.class,
			params = {
				@Parameter(name = "pattern", value = "<p/?>|<br/?>"),
				@Parameter(name = "replacement", value = " ")
			}
		)
	},
	tokenizer = @TokenizerDef(
		factory = WhitespaceTokenizerFactory.class
	)
)
public class Tweet {

    @Id
    @GeneratedValue
    private Integer id;

    @Field
    @Analyzer(definition = "tweet_analyzer")
    private String content;
}
--

===== Custom analyzers using a definition provider

As <<section-programmatic-analyzer-definition,with Lucene>>, you can define analyzers programmatically.

In the case of Elasticsearch, this requires to set the
`hibernate.search.elasticsearch.analysis_definition_provider` configuration property.

The property can be set to the fully-qualified name of a class with a public, no-arg constructor in your application.
This class must either implement `org.hibernate.search.elasticsearch.analyzer.definition.ElasticsearchAnalysisDefinitionProvider` directly
or expose a `@Factory`-annotated method that returns such implementation. 

[source,java]
.Example of mapping that defines analyzers on Elasticsearch using the `Elasticsearch*Factory` types
--
public static class CustomAnalyzerProvider implements ElasticsearchAnalysisDefinitionProvider {
    @Override
    public void register(ElasticsearchAnalysisDefinitionRegistryBuilder builder) {
        builder.analyzer( "tweet_analyzer" )
                .withTokenizer( "whitespace" )
                .withCharFilters( "custom_html_strip" )
                .withCharFilters( "p_br_as_space" );
        
        builder.charFilter( "custom_html_strip" )
                .type( "html_strip" )
                .param( "escaped_tags", "br", "p" );

        builder.charFilter( "p_br_as_space" )
                .type( "pattern_replace" )
                .param( "pattern", "<p/?>|<br/?>" )
                .param( "replacement", " " )
                .param( "tags", "CASE_INSENSITIVE" );
    }
}
--

===== Normalizers

Depending on the Elasticsearch version you're using,
<<section-normalizers,normalizers>> will be handled differently:

* On version 5.2 and above, Elasticsearch supports normalizers natively and
thus Hibernate Search normalizers will be translated directly to Elasticsearch normalizers.
* On version 5.1 and below, Elasticsearch does not support normalizers natively,
thus Hibernate Search normalizers will be translated to Elasticsearch analyzers
with a keyword tokenizer.

[WARNING]
====
On Elasticsearch 5.1 and below,
the fact that normalizers are translated to Elasticsearch analyzers
means in particular that no normalizer definition should have the
same name as an analyzer definition.
====

All the features mentioned above for analyzers are also available when using normalizers:

* They can be defined using Lucene factories with `@NormalizerDef`,
which will be translated to the equivalent types in Elasticsearch.
* They can be defined using the special factories `ElasticsearchCharFilterFactory`
and `ElasticsearchTokenFilterFactory` with `@NormalizerDef`.
* They can be defined using an `ElasticsearchAnalysisDefinitionProvider`
(use `builder.normalizer(name)`).
* The definitions can be referenced using
`@Field(normalizer = @Normalizer(definition = "myDefinition"))`


==== Field bridges

===== Built-in field bridges

The default field bridges are the same as those mentioned in the <<section-built-in-bridges,main documentation>>,
with the following exceptions.

[NOTE]
====
You can refer to the Elasticsearch documentation for a description of
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html[date formats].
====

`boolean`, `Boolean`::
are converted to native JSON booleans.

`Date`, `Calendar`::
are converted to the Elasticsearch `date` type
with the default format (`strict_date_optional_time||epoch_millis`).

`Instant`::
are converted to the Elasticsearch `date` type
with the default format (`strict_date_optional_time||epoch_millis`).

`LocalDate`::
are converted to the Elasticsearch `date` type
with format `strict_date||yyyyyyyyy-MM-dd`.

`LocalDateTime`::
are converted to the Elasticsearch `date` type
with format `strict_date_hour_minute_second_fraction||yyyyyyyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS`.

`LocalTime`::
are converted to the Elasticsearch `date` type
with format `strict_hour_minute_second_fraction`.

`OffsetDateTime`::
are converted to the Elasticsearch `date` type
with format `strict_date_time||yyyyyyyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ`.

`OffsetTime`::
are converted to the Elasticsearch `date` type
with format `strict_time`.

`ZonedDateTime`::
are converted to the Elasticsearch `date` type
with format `yyyy-MM-dd'T'HH:mm:ss.SSSZZ'['ZZZ']'||yyyyyyyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZZ'['ZZZ']'`.

`Year`::
are converted to the Elasticsearch `date` type
with format `strict_year||yyyyyyyyy`.

`YearMonth`::
are converted to the Elasticsearch `date` type
with format `strict_year_month||yyyyyyyyy-MM`.

`MonthDay`::
are converted to the Elasticsearch `date` type
with format `--MM-dd`.


===== Custom field bridges

You can write custom field bridges and class bridges.
For class bridges and field bridges creating multiple fields,
make sure to make your bridge implementation also implement the `MetadataProvidingFieldBridge` contract.

[source,java]
--
public static class FirstAndMiddleNamesFieldBridge implements MetadataProvidingFieldBridge {

    @Override
    public void set(String name, Object value, Document document, LuceneOptions luceneOptions) {
        Explorer explorer = (Explorer) value;

        String firstName = explorer.getFirstName();
        luceneOptions.addFieldToDocument( name + "_firstName", firstName, document );
        document.add( new SortedDocValuesField( name + "_firstName", new BytesRef( firstName ) ) );

        String middleName = explorer.getMiddleName();
        luceneOptions.addFieldToDocument( name + "_middleName", middleName, document );
        document.add( new SortedDocValuesField( name + "_middleName", new BytesRef( middleName ) ) );
    }

    @Override
    public void configureFieldMetadata(String name, FieldMetadataBuilder builder) {
        builder
            .field( name + "_firstName", FieldType.STRING )
                .sortable( true )
            .field( name + "_middleName", FieldType.STRING )
                .sortable( true );
    }
}
--

[NOTE]
--
This interface and `FieldBridge` in general are likely going to evolve in the next major version of Hibernate Search
to remove its adherence to Lucene specific classes like `Document`.
--

====== Sub-fields

When your `MetadataProvidingFieldBridge` registers a field whose name is the name of an existing field,
with a dot and another string appended, like `name + ".mySubField"`,
Hibernate Search will translate it as an object with a property `mySubField`
in the JSON document sent to Elasticsearch.

As a result, sub-fields can only have a parent field with the `OBJECT` type:
obviously, Elasticsearch would reject a String or an Integer with `mySubField` property.
So every time a field named `foo.bar` is registered,
its parent field `foo` must be registered with the `OBJECT` type, as in the following example.
Not doing so will result in errors when Hibernate Search generates the Elasticsearch schema.

[source,java]
--
@Override
public void configureFieldMetadata(String name, FieldMetadataBuilder builder) {
    builder
        .field( name, FieldType.OBJECT )
        .field( name + ".firstName", FieldType.STRING )
            .sortable( true )
        .field( name + ".middleName", FieldType.STRING )
            .sortable( true );
}
--

[[elasticsearch-custom-field-bridge-dynamic-mapping]]
====== Dynamic mapping

By default Hibernate Search (or, more accurately, the Elasticsearch schema generated by Hibernate Search)
will not allow custom bridges to create fields that weren't
previously registered through the `configureFieldMetadata` method.

If you don't know in advance the name of the fields created by your custom bridge,
you can enable "dynamic mapping" globally or per-index through <<elasticsearch-integration-configuration,a configuration option>>,
or you can do it on a per-field basis using the `FieldMetadataCreationContext.mappedOn` method
and the `org.hibernate.search.elasticsearch.bridge.spi.Elasticsearch` extension:

[source,java]
--
@Override
public void configureFieldMetadata(String name, FieldMetadataBuilder builder) {
    builder
        .field( name, FieldType.OBJECT )
            .mappedOn( Elasticsearch.class )
                .dynamic( DynamicType.TRUE ); // Will allow to index fields named <name>.foo, <name>.bar, etc.
}
--

[NOTE]
--
Dynamic mapping involves Elasticsearch guessing the type and attributes of fields automatically,
and has its own limitations.
See https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html[the Elasticsearch documentation]
for more information. 
--

==== Tika bridges

If your metadata processors create fields with a different name from the one passed as a parameter, make sure to make your processor also implement the `MetadataProvidingTikaMetadataProcessor` contract.

=== Queries

You can write queries like you usually do in Hibernate Search: native Lucene queries and DSL queries (see <<search-query>>).
We do automatically translate the most common types of Apache Lucene queries
and all queries generated by the Hibernate Search DSL except more like this (see below).

[NOTE]
.Unsupported Query DSL features
--
Queries written via the DSL work.
Open a JIRA otherwise.

The notable exception is more like this queries.
Hibernate Search has a more advanced algorithm than Lucene (or Elasticsearch/Solr)
which is not easily portable with what Elasticsearch exposes.

If you need this feature, contact us.
--

On top of translating Lucene queries,
you can directly create Elasticsearch queries by using either its String format or a JSON format:

.Creating an Elasticsearch native query from a string
====
[source,java]
----
FullTextEntityManager fullTextEm = Search.getFullTextEntityManager(entityManager);
QueryDescriptor query = ElasticsearchQueries.fromQueryString("title:tales");
List<?> result = fullTextEm.createFullTextQuery(query, ComicBook.class).getResultList();
----
====

.Creating an Elasticsearch native query from JSON
====
[source,java]
----
FullTextEntityManager fullTextEm = Search.getFullTextEntityManager(entityManager);
QueryDescriptor query = ElasticsearchQueries.fromJson(
      "{ 'query': { 'match' : { 'lastName' : 'Brand' } } }");
List<?> result = fullTextEm.createFullTextQuery(query, GolfPlayer.class).getResultList();
----
====

[NOTE]
.Date/time in native Elasticsearch queries
====
By default Elasticsearch interprets the date/time strings lacking the time zone as if they were represented using the UTC time zone. If overlooked, this can cause your native Elasticsearch queries to be completely off.

The simplest way to avoid issues is to always explicitly provide time zone IDs or offsets when building native Elasticsearch queries. This may be achieved either by directly adding the time zone ID or offset in date strings, or by using the `time_zone` parameter (range queries only). See the Elasticsearch documentation for more information.
====

[NOTE]
.Simple query string queries and analyzers
====
The simple query string DSL is supported by the Elasticsearch integration.

In the general case, your query will automatically use the analyzers defined in the schema but be aware that, in the advanced case when you want to override the analyzer used, Elasticsearch only supports overriding the analyzer globally.

Thus, when overriding the analyzer, it must be consistent across all the queried fields.
====

[[elasticsearch-query-spatial]]
==== Spatial queries

The Elasticsearch integration supports spatial queries by using either the DSL or native Elasticsearch queries.

For regular usage, there are no particular requirements for spatial support.

[NOTE]
--
On Elasticsearch 2.x only (not necessary on 5.x),
if you want to calculate the distance from your entities to a point *without sorting* by the distance to this point,
you need to enable the Groovy plugin by adding the following snippet to your Elasticsearch configuration:

.Enabling Groovy support in your elasticsearch.yml
----
script.engine.groovy.inline.search: true
----
--

[[elasticsearch-scrolling]]
==== Paging and scrolling

You may handle large result sets in two different ways, with different limitations.

For (relatively) smaller result sets, you may use the traditional offset/limit querying provided by the `FullTextQuery` interfaces: `setFirstResult(int)` and `setMaxResults(int)`.
Limitations:

* This will only get you as far as the 10000 first documents, i.e. when requesting a window that includes documents beyond the 10000th result, Elasticsearch will return an error. If you want to raise this limit, see the `index.max_result_window` property in https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings[Elasticsearch's settings].

If your result set is bigger, you may take advantage of scrolling by using the `scroll` method on `org.hibernate.search.FullTextQuery`.
Limitations:

* This method is not available in `org.hibernate.search.jpa.FullTextQuery`.
* The Elasticsearch implementation has poor performance when an offset has been defined (i.e. `setFirstResult(int)` has been called on the query before calling `scroll()`).
  This is because Elasticsearch does not provide such feature, thus Hibernate Search has to scroll through every previous result under the hood.
* The Elasticsearch implementation allows only limited backtracking. Calling `scrollableResults.setRowNumber(4)` when currently positioned at index `1006`,
  for example, may result in a `SearchException` being thrown, because only 1000 previous elements had been kept in memory.
  You may work this around by tweaking the property: `hibernate.search.elasticsearch.scroll_backtracking_window_size` (see <<elasticsearch-integration-configuration, Elasticsearch integration configuration>>).
* The `ScrollableResults` will become stale and unusable after a given period of time spent without fetching results from Elasticsearch.
  You may work this around by tweaking two properties: `hibernate.search.elasticsearch.scroll_timeout` and `hibernate.search.elasticsearch.scroll_fetch_size` (see <<elasticsearch-integration-configuration, Elasticsearch integration configuration>>).
  Typically, you will solve timeout issues by reducing the fetch size and/or increasing the timeout limit, but this will also increase the performance hit on Elasticsearch.

[[elasticsearch-query-sorting]]
==== Sorting

Sorting is performed the same way as <<query-sorting,with the Lucene backend>>.

If you happen to need an advanced Elasticsearch sorting feature that is not natively supported in `SortField` or in Hibernate Search sort DSL, you may still create a sort from JSON, and even mix it with DSL-defined sorts:

.Mixing DSL-defined sorts with native Elasticsearch JSON sorts
====
[source, JAVA]
----
FullTextEntityManager fullTextEm = Search.getFullTextEntityManager(entityManager);
org.hibernate.search.query.dsl.QueryBuilder queryBuilder =
        fullTextEm.getSearchFactory()
        .buildQueryBuilder().forEntity(Book.class).get();
Query luceneQuery = /* ... use the query builder as explained in tje Querying section ... */;
FullTextQuery query = fullTextEm.createFullTextQuery( luceneQuery, Book.class );
Sort sort = queryBuilder.sort()
        .byNative("authors.name", "{'order':'asc', 'mode': 'min'}")
        .andByField("title")
        .createSort();
query.setSort(sort);
List results = query.getResultList();
----
====

==== Projections

All fields are stored by Elasticsearch in the JSON document it indexes,
there is no specific need to mark fields as stored when you want to project them.
The downside is that to project a field, Elasticsearch needs to read the whole JSON document.
If you want to avoid that, use the `Store.YES` marker.

You can also retrieve the full JSON document by using `org.hibernate.search.elasticsearch.ElasticsearchProjectionConstants.SOURCE`.

[source,java]
--
FullTextEntityManager fullTextEm = Search.getFullTextEntityManager(entityManager);
FullTextQuery query = fullTextEm.createFullTextQuery(
                    qb.keyword()
                    .onField( "tags" )
                    .matching( "round-based" )
                    .createQuery(),
                    VideoGame.class
            )
            .setProjection( ElasticsearchProjectionConstants.SCORE, ElasticsearchProjectionConstants.SOURCE );

Object[] projection = (Object[]) query.getSingleResult();
--

If you're looking for information about execution time, you may also use `org.hibernate.search.elasticsearch.ElasticsearchProjectionConstants.TOOK` and `org.hibernate.search.elasticsearch.ElasticsearchProjectionConstants.TIMED_OUT`: 

[source,java]
--
FullTextEntityManager fullTextEm = Search.getFullTextEntityManager(entityManager);
FullTextQuery query = fullTextEm.createFullTextQuery(
                    qb.keyword()
                    .onField( "tags" )
                    .matching( "round-based" )
                    .createQuery(),
                    VideoGame.class
            )
            .setProjection(
                    ElasticsearchProjectionConstants.SOURCE,
                    ElasticsearchProjectionConstants.TOOK,
                    ElasticsearchProjectionConstants.TIMED_OUT 
            );

Object[] projection = (Object[]) query.getSingleResult();
Integer took = (Integer) projection[1]; // Execution time (milliseconds)
Boolean timedOut = (Boolean) projection[2]; // Whether the query timed out
--

==== Filters

The Elasticsearch integration supports the definition of full text filters.

Your filters need to implement the `ElasticsearchFilter` interface.

[source,java]
--
public class DriversMatchingNameElasticsearchFilter implements ElasticsearchFilter {

    private String name;

    public DriversMatchingNameElasticsearchFilter() {
    }

    public void setName(String name) {
        this.name = name;
    }

    @Override
    public String getJsonFilter() {
        return "{ 'term': { 'name': '" + name + "' } }";
    }

}
--

You can then declare the filter in your entity.

[source,java]
--
@Entity
@Indexed
@FullTextFilterDef(name = "namedDriver",
  impl = DriversMatchingNameElasticsearchFilter.class)
public class Driver {
    @Id
    @DocumentId
    private int id;

    @Field(analyze = Analyze.YES)
    private String name;

    // ...
}
--

From then you can use it as usual.

[source,java]
--
FullTextQuery ftQuery = /* ... */;
ftQuery.enableFullTextFilter( "namedDriver" ).setParameter( "name", "liz" );
--

For static filters, you can simply extend the `SimpleElasticsearchFilter` and provide an Elasticsearch filter in JSON form.

=== Index optimization

The optimization features documented in <<search-optimize>> are only partially implemented. That kind of optimization is rarely needed with recent versions of Lucene (on which Elasticsearch is based), but some of it is still provided for the very specific case of indexes meant to stay read-only for a long period of time:

 * The automatic optimization is not implemented and most probably never will be.
 * The manual optimization (`searchFactory.optimize()`) is implemented.

[[elasticsearch-logging]]
==== Logging executed requests

Search queries are logged to the `org.hibernate.search.fulltext_query` category at `DEBUG` level,
as when using an embedded Lucene instance (the query format is Elasticsearch's, though).

In addition, you can enable the logging of every single request sent to the Elasticsearch cluster
by enabling `DEBUG` logging for the log category `org.hibernate.search.elasticsearch.request`.
The logs will include, in particular, the HTTP method, path, query parameters, execution time and response status.
At `DEBUG` level, the request body and response body will be omitted;
use the `TRACE` level to also print out the request body and response body.

[NOTE]
--
By default, JSON in logs will be printed inline, without indent.
If you wish the JSON to be pretty-printed, use the
<<elasticsearch-log-json-pretty-printing,`hibernate.search.elasticsearch.log.json_pretty_printing` configuration property>>.
--

[[elasticsearch-limitations]]
=== Limitations

Not everything is implemented yet.

At the time of writing this, the following features are known to not work yet; feel free to check the JIRA links to get updates:

* Query timeouts: https://hibernate.atlassian.net/browse/HSEARCH-2399[HSEARCH-2399]
* MoreLikeThis queries: https://hibernate.atlassian.net/browse/HSEARCH-2395[HSEARCH-2395]
* `@IndexedEmbedded.indexNullAs`: https://hibernate.atlassian.net/browse/HSEARCH-2389[HSEARCH-2389]
* <<search-monitoring,Statistics>>: https://hibernate.atlassian.net/browse/HSEARCH-2421[HSEARCH-2421]
* `@AnalyzerDiscriminator`: https://hibernate.atlassian.net/browse/HSEARCH-2428[HSEARCH-2428]
* Dynamic sharding: https://hibernate.atlassian.net/browse/HSEARCH-2725[HSEARCH-2725]
* Mixing Lucene based indexes and Elasticsearch based indexes (partial support is implemented though)
* Hibernate Search does not make use of nested objects nor parent child relationship mapping https://hibernate.atlassian.net/browse/HSEARCH-2263[HSEARCH-2263].
  This is largely mitigated by the fact that Hibernate Search does the denormalization itself and maintain data consistency when nested objects are updated.
* There is room for improvements in the performances of the MassIndexer implementation
* Our new Elasticsearch integration module does not work in OSGi environments. If you need this, please vote for: https://hibernate.atlassian.net/browse/HSEARCH-2524[HSEARCH-2524].

=== Specific versions of Elasticsearch to avoid

We might occasionally hit issues in specific versions of Elasticsearch.
This is not the right place to track Elasticsearch defects but we'll do our best to track specific issues which affect compatibility with Hibernate Search.

* Mapping `java.time.ZonedDateTime` won't work with Elasticsearch 2.4.1 because of https://github.com/elastic/elasticsearch/issues/20911[a JodaTime bug affecting Elasticsearch]: https://hibernate.atlassian.net/browse/HSEARCH-2414[HSEARCH-2414].
+
*Solution:* Upgrade to Elasticsearch 2.4.2 or later.

=== Acknowledgment

More information about Elasticsearch can be found on the https://www.elastic.co/products/elasticsearch[Elasticsearch website]
and its https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html[reference documentation].
