Category: elasticsearch

java.lang.NoClassDefFoundError: org/elasticsearch/common/transport/InetSocketTransportAddress

Scenario

In a spring-boot application that uses spring-boot-dependencies instead of spring-boot-starter-parent and a third party library that depends on elasticsearch version that is not comparable with version 6.x, elasticsearch dependency version get mixed up creating class loading issue.

Steps to reproduce

Create a spring-boot project and include spring-boot-dependencies in the dependency management section of the maven pom.xml

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Later, in the dependencies section include a third party library that depends on elastic search version 5.6.8

<dependnecies>
  <dependency>
    <groupId>com.netflix.conductor</groupId>
    <artifactId>conductor-es5-persistence</artifactId>
    <version>2.8.1</version>
  </dependency>
</dependnecies>

Note that conductor-es5-persistence depends on elasticsearch version 5.6.8

The try to run mvn dependency:tree on your project and observe the following: 

[INFO] +- com.netflix.conductor:conductor-es5-persistence:jar:2.8.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:6.4.3:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:6.4.3:compile

Where 6.4.3 is coming from?

Diagnostic

Turns out mvn dependency:tree shows the issue, but it does not say where org.elasticsearch:elasticsearch:jar:6.4.3:compile is comming from. The best command for the job is:

mvn -Dverbose=true help:effective-pom

This command outputs effective POM that can be redirected to a file mvn -Dverbose=true help:effective-pom > ~/pom.xml and analysed. The effective pom will look something like this

...
<dependency>
  <groupId>org.elasticsearch</groupId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1965 -->
  <artifactId>elasticsearch</artifactId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1966 -->
  <version>6.4.3</version>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->
</dependency>
...

Magic! now we can see exactly where the problem is:

<!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->

Solution

Lets look at this import dependency:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-dependencies</artifactId>
    <version>${spring.boot.version}</version>
    <type>pom</type>
    <scope>import</scope>
  </dependency>

spring-boot-dependencies is imported, so there is no easy way to exclude elasticsearch dependency that it pulls into our project. A simple maven <exclude> will not work here. The only way to solve the issue is to bring elasticsearch dependencies into your project dependency management block like this:

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
  
  <elasticsearch.version>5.6.8</elasticsearch.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    
    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.distribution.integ-test-zip</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
      <type>zip</type>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>transport-netty4-client</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

  </dependencies>
</dependencyManagement>

Now run mvn -Dverbose=true help:effective-pom and confirm that the problem is fixed.

itzap-elasticsearch client library

Vision Statement

Build 7.3.x elasticsearch embedded server into itzap–elasticsearch library to enable easy prototyping and unit/integration testing

How It Works

Elasticsearch embedded server starts a single elasticsearch Node. Elasticsearch server node will be listening for incoming requests on port 9200 Elasticsearch client can be configured to access embedded server using the following url: http://localhost:9200 Elasticsearch embedded server will create data and home temp folders that will be deleted opon application close.

Code

Here is the main class that implements elasticsearch embedded server node

public class EmbeddedElasticSearchServer implements IOData {
  private static final Logger LOGGER = LoggerFactory.getLogger(EmbeddedElasticSearchServer.class);

  private Node instance;
  private int port;
  private final AnyConfig config;

  public EmbeddedElasticSearchServer(AnyConfig config) {
    this.config = config;
  }

  private static class PluginConfigurableNode extends Node {
    PluginConfigurableNode(Settings input,
                           Map<String, String> properties,
                           Path configPath,
                           Supplier<String> defaultNodeName,
                           Collection<Class<? extends Plugin>> classpathPlugins) {
      super(InternalSettingsPreparer.prepareEnvironment(input, properties, configPath, defaultNodeName), classpathPlugins, false);
    }
  }

  @Override
  public synchronized Completable start() {
    return new RunnableCommand<Void>("cmd-start") {
      @Override
      protected Void run() {
        Settings settings = getSettings();

        instance = new PluginConfigurableNode(settings, ImmutableMap.of(),
                                              null, () -> config.getString(EsConfig.CLUSTER_NAME),
                                              singletonList(Netty4Plugin.class));
        try {
          instance.start();
          Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            try {
              if (instance != null) {
                instance.close();
              }
            } catch (IOException e) {
              LOGGER.error("Error closing ElasticSearch");
            }
          }));
          LOGGER.info("ElasticSearch cluster {} started in local mode on port {}", instance.settings().get("cluster.name"),
                      port);
          return null;
        } catch (NodeValidationException e) {
          throw new IZapException("Failed to start embedded elastic search server", e);
        }
      }
    }.toCompletable();
  }

  @Override
  public synchronized Completable stop() {
    return new RunnableCommand<Void>("cmd-stop") {
      @Override
      protected Void run() {
        if (instance != null && !instance.isClosed()) {
          LOGGER.info("Stopping Elastic Search");
          try {
            instance.close();
            instance = null;
            LOGGER.info("Elastic Search on port {} stopped", port);
          } catch (IOException e) {
            throw new IZapException("Failed to close elastic search embedded server", e);
          }
        }

        return null;
      }
    }.toCompletable();
  }

  private Settings getSettings() {
    String clusterName = config.getString(EsConfig.CLUSTER_NAME);
    String host = config.getString(EsConfig.HOST);
    port = config.getInt(EsConfig.PORT);

    try {
      File dataDir = Files.createTempDirectory(clusterName + "_" + System.currentTimeMillis() + "data").toFile();
      FileUtils.forceDeleteOnExit(dataDir);
      cleanDataDir(dataDir.getAbsolutePath());

      File homeDir = Files.createTempDirectory(clusterName + "_" + System.currentTimeMillis() + "-home").toFile();
      cleanDataDir(homeDir.getAbsolutePath());
      FileUtils.forceDeleteOnExit(homeDir);

      Settings.Builder settingsBuilder = Settings.builder()
        .put("cluster.name", clusterName)
        .put("http.host", host)
        .put("http.port", port)
        .put("transport.tcp.port", port + 100)
        .put(EsConfig.DATA_PATH.getName(), dataDir.getAbsolutePath())
        .put(EsConfig.HOME_PATH.getName(), homeDir.getAbsolutePath())
        .put("http.cors.enabled", true)
        .put("node.data", true)
        .put("http.type", "netty4")
        .put("transport.type", "netty4");

      return settingsBuilder.build();
    } catch (IOException e) {
      throw new IZapException("Failed to create temp data/home dir.", e);
    }
  }
}

Elasticsearch embedded server implementation is using RunnableCommand command pattern to implement start and stopmethods. Please note that RunnableCommand returns Completable and not Observable. Sice Observable<Void> is no longer permitted in RxJava2 and start/stop methods have no return values. Here is the unit test for the elasticsearch embedded server

public class EmbeddedElasticSearchServerTest {
    private EmbeddedElasticSearchServer server;

    @Before
    public void setup() {
        AnyConfig config = ConfigBuilder.builder(ConfigType.TYPE_SAFE)
                .setFileName(this.getClass()
                        .getResource("/es-config.properties").getFile())
                .build();

        server = new EmbeddedElasticSearchServer(config);
    }

    @Test
    public void startTest() {
        server.start().blockingGet();
        server.stop().blockingGet();
    }
}

Elasticsearch dependencies

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-high-level-client</artifactId>
  <version>7.3.1</version>
</dependency>

<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch</artifactId>
  <version>7.3.1</version>
</dependency>

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>transport</artifactId>
  <version>7.3.1</version>
</dependency>