Hibernate Search: Diagnose Missing Catalog Results

See how an analyzer mismatch hides an indexed book from a name search, and how a focused change restores the match.

I’ve had a soft spot for Jo Nesbø for a long time, which is how Søren J. Thorsen ended up in my Hibernate Search tutorial last year. When Quarkus updated its Hibernate integration, I went back to the catalog with a small search problem to investigate.

Anyway, here’s the case. The Snowman is in PostgreSQL, with Jo Nesbø as its author. I search for Nesbo and get an empty result. So I look at the document in Elasticsearch and then ask the analyzer what it does with the name. You can follow the same steps when a search misses something you know is there.

I’m using Quarkus 3.39.4 for this example.

What you need

You’ll need Java 25 to compile and run the app. The Quarkus CLI creates the project and includes a Maven wrapper, so you can run the commands below from its directory. Start Podman before launching the app; Dev Services uses it to start PostgreSQL and Elasticsearch.

I assume you’re familiar with Panache entities and Jakarta REST resources. We’ll add a small inspector to the app so you can see what Elasticsearch has stored and which tokens it produces.

Create the application

With the Quarkus CLI or start from my Github repository.

quarkus create app com.themainthread:hibernate-search-noir \
  --platform-bom=io.quarkus:quarkus-bom:3.39.4 \
  --java=25 \
  --extension=rest-jackson,hibernate-orm-panache,jdbc-postgresql,hibernate-search-orm-elasticsearch
cd hibernate-search-noir

The catalog uses Panache and PostgreSQL. Hibernate Search indexes the entities, and the REST extension returns JSON. The CLI also includes REST Assured, which we’ll use to call the endpoints from a test later. Keep the generated dependencies and remove the sample classes:

rm src/main/java/com/themainthread/GreetingResource.java \
   src/main/java/com/themainthread/MyEntity.java \
   src/test/java/com/themainthread/GreetingResourceTest.java \
   src/test/java/com/themainthread/GreetingResourceIT.java

I let Dev Services choose the Elasticsearch image. The earlier example needed a pinned image because of a bug on my M4; this time the default image started on the same machine. Hibernate Search still needs an Elasticsearch version in the configuration to generate its mapping at build time. Put this in src/main/resources/application.properties:

quarkus.datasource.db-kind=postgresql

quarkus.hibernate-orm.sql-load-script=import.sql

quarkus.hibernate-search-orm.elasticsearch.version=9
quarkus.hibernate-search-orm.indexing.plan.synchronization.strategy=sync

# Disposable demo databases and indexes only.
%dev,test.quarkus.hibernate-orm.schema-management.strategy=drop-and-create
%dev,test.quarkus.hibernate-search-orm.schema-management.strategy=drop-and-create

%prod.quarkus.datasource.jdbc.url=jdbc:postgresql://localhost/quarkus_test
%prod.quarkus.datasource.username=quarkus_test
%prod.quarkus.datasource.password=quarkus_test
%prod.quarkus.hibernate-orm.schema-management.strategy=create
%prod.quarkus.hibernate-search-orm.elasticsearch.hosts=localhost:9200

sync makes ORM-triggered index changes searchable before the transaction commit returns. The seed data in this example goes through a mass indexer, which refreshes the indexes before startAndWait() returns. That’s what lets us search the imported books as soon as the app starts.

Use disposable services for this demo, including a packaged run with the %prod settings. SQL seeding and startup mass indexing run in that profile too. The mass indexer clears the existing index contents before filling them again.

Map the catalog

I kept the catalog small, with Jo Nesbø and three of his books. John Irving gets a book too, giving us another author to search for.

Create src/main/java/com/themainthread/noir/model/Author.java:

package com.themainthread.noir.model;

import java.util.ArrayList;
import java.util.List;

import org.hibernate.search.engine.backend.types.Sortable;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.FullTextField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.IndexedEmbedded;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.KeywordField;

import io.quarkus.hibernate.orm.panache.PanacheEntity;
import jakarta.persistence.CascadeType;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.OneToMany;

@Entity
@Indexed
public class Author extends PanacheEntity {

    @FullTextField(analyzer = "name")
    @KeywordField(name = "firstName_sort", sortable = Sortable.YES, normalizer = "sort")
    public String firstName;

    @FullTextField(analyzer = "name")
    @KeywordField(name = "lastName_sort", sortable = Sortable.YES, normalizer = "sort")
    public String lastName;

    @OneToMany(mappedBy = "author", cascade = CascadeType.ALL, orphanRemoval = true, fetch = FetchType.EAGER)
    @IndexedEmbedded
    public List<Book> books = new ArrayList<>();
}

Create src/main/java/com/themainthread/noir/model/Book.java:

package com.themainthread.noir.model;

import org.hibernate.search.engine.backend.types.Sortable;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.FullTextField;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.Indexed;
import org.hibernate.search.mapper.pojo.mapping.definition.annotation.KeywordField;

import com.fasterxml.jackson.annotation.JsonIgnore;

import io.quarkus.hibernate.orm.panache.PanacheEntity;
import jakarta.persistence.Entity;
import jakarta.persistence.ManyToOne;

@Entity
@Indexed
public class Book extends PanacheEntity {

    @FullTextField(analyzer = "english")
    @KeywordField(name = "title_sort", sortable = Sortable.YES, normalizer = "sort")
    public String title;

    @ManyToOne
    @JsonIgnore
    public Author author;
}

@Indexed marks each entity for indexing. The name and title fields use named analyzers through @FullTextField, while @KeywordField supplies the single-token fields used for sorting. Have a look at @IndexedEmbedded on the books collection: it includes the book fields in the author document, so an author search can also match books.title.

Replace the generated src/main/resources/import.sql with the catalog rows:

INSERT INTO author(id, firstname, lastname) VALUES (1, 'Jo', 'Nesbø');
INSERT INTO author(id, firstname, lastname) VALUES (2, 'John', 'Irving');
ALTER SEQUENCE author_seq RESTART WITH 3;

INSERT INTO book(id, title, author_id) VALUES (1, 'The Snowman', 1);
INSERT INTO book(id, title, author_id) VALUES (2, 'The Bat', 1);
INSERT INTO book(id, title, author_id) VALUES (3, 'The Redeemer', 1);
INSERT INTO book(id, title, author_id) VALUES (4, 'A Prayer for Owen Meany', 2);
ALTER SEQUENCE book_seq RESTART WITH 5;

import.sql writes straight into PostgreSQL, bypassing the ORM events that normally trigger indexing. So I added a startup mass indexer to read the imported entities and send them to Elasticsearch. Once it finishes, we can inspect an author document and try the search.

Create src/main/java/com/themainthread/noir/CatalogIndexer.java:

package com.themainthread.noir;

import org.hibernate.search.mapper.orm.mapping.SearchMapping;
import org.jboss.logging.Logger;

import com.themainthread.noir.model.Book;

import io.quarkus.runtime.StartupEvent;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;
import jakarta.transaction.Transactional;

@ApplicationScoped
public class CatalogIndexer {

    private static final Logger log = Logger.getLogger(CatalogIndexer.class);

    @Inject
    SearchMapping searchMapping;

    @Transactional
    void onStart(@Observes StartupEvent ev) throws InterruptedException {
        if (Book.count() == 0) {
            return;
        }
        log.info("SQL import is invisible to Hibernate Search; mass-indexing the catalog");
        searchMapping.scope(Object.class)
                .massIndexer()
                .startAndWait();
    }
}

Define the name analyzer

Create src/main/java/com/themainthread/noir/config/AnalysisConfigurer.java. My first name analyzer only converted letters to lowercase:

package com.themainthread.noir.config;

import org.hibernate.search.backend.elasticsearch.analysis.ElasticsearchAnalysisConfigurationContext;
import org.hibernate.search.backend.elasticsearch.analysis.ElasticsearchAnalysisConfigurer;

import io.quarkus.hibernate.search.orm.elasticsearch.SearchExtension;

@SearchExtension
public class AnalysisConfigurer implements ElasticsearchAnalysisConfigurer {

    @Override
    public void configure(ElasticsearchAnalysisConfigurationContext context) {
        context.analyzer("name").custom()
                .tokenizer("standard")
                .tokenFilters("lowercase");

        context.analyzer("english").custom()
                .tokenizer("standard")
                .tokenFilters("asciifolding", "lowercase", "porter_stem");

        context.normalizer("sort").custom()
                .tokenFilters("asciifolding", "lowercase");
    }
}

@SearchExtension registers the configurer for the default persistence unit. The name analyzer uses the standard tokenizer and converts letters to lowercase, keeping ø as ø. For the English book titles I picked porter_stem, following the current Quarkus guide. That filter reduces words to stems so related forms can match.

Search for Nesbo

I typed Nesbo, using a plain o. That’s the spelling I want this search to accept alongside Nesbø.

I added a small inspector so I can read a document and call _analyze through the app. This Quarkus release selects the REST5 client for Hibernate Search, so I unwrap it as Rest5Client. I first tried org.elasticsearch.client.RestClient and got HSEARCH400518. The type has to match the client Quarkus selected.

Create src/main/java/com/themainthread/noir/ElasticsearchInspector.java:

package com.themainthread.noir;

import java.io.IOException;
import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import org.hibernate.search.backend.elasticsearch.ElasticsearchBackend;
import org.hibernate.search.mapper.orm.mapping.SearchMapping;

import co.elastic.clients.transport.rest5_client.low_level.Request;
import co.elastic.clients.transport.rest5_client.low_level.Response;
import co.elastic.clients.transport.rest5_client.low_level.ResponseException;
import co.elastic.clients.transport.rest5_client.low_level.Rest5Client;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.ws.rs.NotFoundException;
import jakarta.ws.rs.WebApplicationException;
import jakarta.ws.rs.core.Response.Status;

@ApplicationScoped
public class ElasticsearchInspector {

    private final SearchMapping searchMapping;
    private final ObjectMapper mapper;

    public ElasticsearchInspector(SearchMapping searchMapping, ObjectMapper mapper) {
        this.searchMapping = searchMapping;
        this.mapper = mapper;
    }

    public JsonNode indexedAuthor(long id) {
        return get("/author-read/_doc/" + id);
    }

    public List<String> analyze(String analyzer, String text) {
        Map<String, String> body = new LinkedHashMap<>();
        body.put("analyzer", analyzer);
        body.put("text", text);
        JsonNode response = post("/author-read/_analyze", body);
        List<String> tokens = new ArrayList<>();
        for (JsonNode token : response.path("tokens")) {
            tokens.add(token.path("token").asText());
        }
        return tokens;
    }

    private JsonNode get(String endpoint) {
        Request request = new Request("GET", endpoint);
        return perform(request);
    }

    private JsonNode post(String endpoint, Object jsonBody) {
        Request request = new Request("POST", endpoint);
        try {
            request.setJsonEntity(mapper.writeValueAsString(jsonBody));
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
        return perform(request);
    }

    private JsonNode perform(Request request) {
        try {
            Response response = client().performRequest(request);
            return mapper.readTree(response.getEntity().getContent());
        } catch (ResponseException e) {
            int status = e.getResponse().getStatusCode();
            if (status == 404) {
                throw new NotFoundException("Indexed document not found");
            }
            throw new WebApplicationException("Elasticsearch request failed: " + status, Status.BAD_GATEWAY);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }

    private Rest5Client client() {
        return searchMapping.backend()
                .unwrap(ElasticsearchBackend.class)
                .client(Rest5Client.class);
    }
}

author-read is the Hibernate Search read alias for the Author index.

Create src/main/java/com/themainthread/noir/LibraryResource.java:

package com.themainthread.noir;

import java.util.List;

import org.hibernate.search.mapper.orm.session.SearchSession;

import com.fasterxml.jackson.databind.JsonNode;
import com.themainthread.noir.model.Author;
import com.themainthread.noir.model.Book;

import io.smallrye.common.annotation.Blocking;
import jakarta.transaction.Transactional;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.NotFoundException;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.PathParam;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.MediaType;

@Path("/library")
@Produces(MediaType.APPLICATION_JSON)
public class LibraryResource {

    private final SearchSession searchSession;
    private final ElasticsearchInspector inspector;

    public LibraryResource(SearchSession searchSession, ElasticsearchInspector inspector) {
        this.searchSession = searchSession;
        this.inspector = inspector;
    }

    public record CatalogEntry(
            Long bookId,
            String title,
            Long authorId,
            String authorFirstName,
            String authorLastName) {
    }

    public record AnalyzeResponse(String analyzer, String text, List<String> tokens) {
    }

    @GET
    @Path("/books")
    @Transactional
    public CatalogEntry bookFromDatabase(@QueryParam("title") String title) {
        if (title == null || title.isBlank()) {
            throw new NotFoundException("title is required");
        }
        Book book = Book.find("title", title).firstResult();
        if (book == null) {
            throw new NotFoundException("No catalog row for title " + title);
        }
        return new CatalogEntry(book.id, book.title, book.author.id, book.author.firstName, book.author.lastName);
    }

    @GET
    @Path("/search")
    @Transactional
    public List<Author> search(@QueryParam("q") String q) {
        if (q == null || q.isBlank()) {
            return List.of();
        }
        return searchSession.search(Author.class)
                .where(f -> f.simpleQueryString()
                        .fields("firstName", "lastName", "books.title")
                        .matching(q))
                .sort(f -> f.field("lastName_sort").then().field("firstName_sort"))
                .fetchHits(20);
    }

    @GET
    @Path("/index/author/{id}")
    @Blocking
    public JsonNode indexedAuthor(@PathParam("id") long id) {
        return inspector.indexedAuthor(id);
    }

    @GET
    @Path("/analyze")
    @Blocking
    public AnalyzeResponse analyze(@QueryParam("analyzer") String analyzer, @QueryParam("text") String text) {
        if (analyzer == null || analyzer.isBlank() || text == null) {
            throw new NotFoundException("analyzer and text are required");
        }
        return new AnalyzeResponse(analyzer, text, inspector.analyze(analyzer, text));
    }
}

With this mapping, simpleQueryString() analyzes q using each field’s analyzer. Hibernate Search then loads the matching authors through the ORM session, which is why I put @Transactional on the search method.

Now start the app:

./mvnw quarkus:dev

In a second terminal, try the name:

curl 'http://localhost:8080/library/search?q=Nesbo'

That came back as [].

Find the book in the database

Okay, let’s ask the database for The Snowman. This endpoint reads the book through Panache:

curl 'http://localhost:8080/library/books?title=The%20Snowman'

{
  "bookId": 1,
  "title": "The Snowman",
  "authorId": 1,
  "authorFirstName": "Jo",
  "authorLastName": "Nesbø"
}

There’s The Snowman, with Nesbø as the author’s last name. Now I can follow that author ID into Elasticsearch.

Inspect the indexed document

curl 'http://localhost:8080/library/index/author/1'

The endpoint returns Elasticsearch’s GET response. Here’s an excerpt showing the author and his books:

{
  "_index": "author-000001",
  "_id": "1",
  "found": true,
  "_source": {
    "lastName": "Nesbø",
    "firstName": "Jo",
    "books": [
      { "title": "The Snowman" },
      { "title": "The Bat" },
      { "title": "The Redeemer" }
    ]
  }
}

There’s the author document, with Nesbø in _source.lastName. Elasticsearch keeps the original spelling there. Let’s ask the analyzer which tokens it produces from that name.

Inspect the analyzer output

curl --get --data-urlencode 'analyzer=name' --data-urlencode 'text=Nesbø' \
  'http://localhost:8080/library/analyze'

{"analyzer":"name","text":"Nesbø","tokens":["nesbø"]}

curl --get --data-urlencode 'analyzer=name' --data-urlencode 'text=Nesbo' \
  'http://localhost:8080/library/analyze'

{"analyzer":"name","text":"Nesbo","tokens":["nesbo"]}

There it is. The analyzer turns the stored name into nesbø and my query into nesbo. Those are two distinct tokens, which explains the empty result. Lowercasing preserves the ø.

I also searched with the exact spelling:

curl --get --data-urlencode 'q=Nesbø' 'http://localhost:8080/library/search'

That returns Jo Nesbø and his books, including The Snowman. So the next change is to let the plain o spelling reach the same result.

Add ASCII folding

I added asciifolding to the name analyzer:

context.analyzer("name").custom()
        .tokenizer("standard")
        .tokenFilters("asciifolding", "lowercase");

The index needs to be recreated with the changed analyzer and filled again. Stop the app with Ctrl+C and run ./mvnw quarkus:dev again. The demo’s %dev settings recreate the database and indexes, and the startup mass indexer fills them from the imported catalog. A production rebuild needs its own plan for keeping searches available.

After the restart:

curl --get --data-urlencode 'analyzer=name' --data-urlencode 'text=Nesbø' \
  'http://localhost:8080/library/analyze'

{"analyzer":"name","text":"Nesbø","tokens":["nesbo"]}

Nesbo produces nesbo too. Let’s try the search again:

curl 'http://localhost:8080/library/search?q=Nesbo'

[
  {
    "id": 1,
    "firstName": "Jo",
    "lastName": "Nesbø",
    "books": [
      { "id": 1, "title": "The Snowman" },
      { "id": 2, "title": "The Bat" },
      { "id": 3, "title": "The Redeemer" }
    ]
  }
]

The catalog still contains the original spelling, and now the analyzer gives both spellings the token nesbo. That’s enough for this query to find Jo’s books.

Write a regression test

I want Nesbo to keep finding Jo’s books when I change the mapping later. So I added a regression test that follows the same path through the catalog and the analyzer. It also checks that the plain o spelling finds The Snowman.

Create src/test/java/com/themainthread/noir/LibrarySearchTest.java:

package com.themainthread.noir;

import static io.restassured.RestAssured.given;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.hasItem;
import static org.hamcrest.Matchers.hasItems;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.restassured.http.ContentType;

@QuarkusTest
class LibrarySearchTest {

    @Test
    void snowmanExistsInTheDatabase() {
        given()
                .when().get("/library/books?title=The Snowman")
                .then()
                .statusCode(200)
                .contentType(ContentType.JSON)
                .body("bookId", equalTo(1))
                .body("title", equalTo("The Snowman"))
                .body("authorLastName", equalTo("Nesbø"));
    }

    @Test
    void nesboAuthorDocumentIsIndexed() {
        given()
                .when().get("/library/index/author/1")
                .then()
                .statusCode(200)
                .contentType(ContentType.JSON)
                .body("found", equalTo(true))
                .body("_source.lastName", equalTo("Nesbø"))
                .body("_source.books.title", hasItem("The Snowman"));
    }

    @Test
    void nameAnalyzerFoldsNesboToAscii() {
        given()
                .when().get("/library/analyze?analyzer=name&text=Nesbø")
                .then()
                .statusCode(200)
                .body("tokens", hasItem("nesbo"));

        given()
                .when().get("/library/analyze?analyzer=name&text=Nesbo")
                .then()
                .statusCode(200)
                .body("tokens", hasItem("nesbo"));
    }

    @Test
    void asciiSearchFindsJoNesbo() {
        given()
                .when().get("/library/search?q=Nesbo")
                .then()
                .statusCode(200)
                .contentType(ContentType.JSON)
                .body("lastName", hasItems("Nesbø"))
                .body("find { it.lastName == 'Nesbø' }.books.title", hasItem("The Snowman"));
    }
}

./mvnw test

The tests pass with folding enabled, as it is in the companion project. You can try removing asciifolding to see the regression test catch the empty search again.

For more on the mapping and analyzers, have a look at the Quarkus Hibernate Search guide. Hibernate Search also explains how to retrieve the configured Elasticsearch client.

I started with a book I knew was in the catalog and an empty search result. Once I could see what the analyzer did with each spelling, the change was pretty small: fold ø to o, rebuild the index, and search again. Next time a name disappears from search, I’d ask the analyzer what happened to it.