Scenario

Spring framework provides mechanism for binding controller method parameters to custom model. Here is a very good article on how to implement custom data binding https://www.baeldung.com/spring-mvc-custom-data-binder During data binding there might be a need to access HTTP request body.

Solution

HTTP request body is hidden in the ModelAndViewContainer object that spring provides to the argument resolver. Here is a code example:

public class MyContextArgumentResolver implements HandlerMethodArgumentResolver {
  @Override
  public boolean supportsParameter(@Nullable MethodParameter methodParameter) {
    return methodParameter != null &&
        methodParameter.getParameterAnnotation(MyContext.class) != null;
  }

  @Override
  public Object resolveArgument(@Nullable MethodParameter methodParameter,
                                ModelAndViewContainer modelAndViewContainer,
                                @Nullable NativeWebRequest nativeWebRequest,

    return toContext(modelAndViewContainer);
  }
                                
  private MyContext toContext(ModelAndViewContainer modelAndViewContainer) {
    if (modelAndViewContainer == null) {
      return MyContext.builder().build();
    }

    MyContext.Builder builder = MyContext.builder();

    modelAndViewContainer.getModel().forEach((k, v) -> {
      if (v instanceof BindingResult) {
        BindingResult result = (BindingResult) v;
        if (result.getTarget() instanceof Map) {
          // plain map
          builder
            .setPropertyOne(((Map<?, ?>) result.getTarget()).get("propertyOne"))
            .setPropertyTwo(((Map<?, ?>) result.getTarget()).get("propertyTwo"));         
        } else if (result.getTarget() instanceof MyRequest) {
          // handle MyRequest request
		  builder
            .setPropertyOne(((MyRequest) result.getTarget()).getpropertyOne())
            .setPropertyTwo(((MyRequest) result.getTarget()).getpropertyTwo());
        }
      }
    });

    return builder.builder();
  }                                
}

All details about setting up method argument resolver are omitted and the focus is on the toContext function that demonstrates how to get request body from ModelAndViewContainer.

Scenario

spring-cloud/spring-boot application is using zuul API gateway to route client requests. Here is zuul config snippet

zuul.ribbonIsolationStrategy=THREAD
zuul.threadPool.useSeparateThreadPools=true

ribbon.eureka.enabled = false

# sevices
...

Invoking service APIs through zuul gateway resulted in HTTP 500 response that looked something like this:

{
  "timestamp": "2019-12-18T21:37:10.168+0000",
  "status": 500,
  "error": "Internal Server Error",
  "message": "REJECTED_THREAD_EXECUTION"
}

Interesting part of this response is REJECTED_THREAD_EXECUTION. None of the services were reporting this message. After looking deeper, the Zuul log showed this error

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b17817e rejected from java.util.concurrent.ThreadPoolExecutor@67f89c35[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 2535368]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_212]

Solution

After realizing that exception is in the zuul log it was a matter of finding the right properties to configure services thread pools. By default, zuul was using 10 threads with the max thread count set to 10. To change this default across all services set these properties:

# defualt threadpool config
hystrix.threadpool.default.coreSize = 150
hystrix.threadpool.default.maximumSize = 150

It is also possible to configure thread pools for each service that zuul is routing to. For example:

zuul.routes.first.url=firstServiceId
zuul.routes.second.url=secondServiceId

hystrix.threadpool.firstServiceId.coreSize = 150
hystrix.threadpool.firstServiceId.maximumSize = 150

hystrix.threadpool.secondServiceId.coreSize = 50
hystrix.threadpool.secondServiceId.maximumSize = 150

Scenario

Suppose you are working on multiple Spring Boot/Spring Cloud projects that are using the spring cloud config server. You setup property encryption for the spring-cloud config. The encryption key is sored in your local profile as an environment variable ENCRYPT_KEY. Then, you switch to work on another spring-boot/spring-cloud project that does not use spring config server and have spring-cloud-context as a dependency. When you run this project you get the following exception stack trace:

Caused by: java.lang.NullPointerException: null
	at org.springframework.cloud.context.encrypt.EncryptorFactory.create(EncryptorFactory.java:54)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration.textEncryptor(EncryptionBootstrapConfiguration.java:83)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf.CGLIB$textEncryptor$0(<generated>)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf$$FastClassBySpringCGLIB$$4d33eb7b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf.textEncryptor(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 31 common frames omitted

Diagnostic

This is actually a spring bug! Another way to provide the encryption key to the config server is to use -Dencrypt.key=123 system property. If you do it this way everything is working fine. The problem only arises when the environment variable ENCRYPT_KEY is used. The root cause is in:

org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration.KeyCondition

This condition bean determines whether spring will initialize TextEncryptor class or not. When spring detects ENCRYPT_KEY environment variable the TextEncryptor class gets created. It uses KeyProperties to initialize encryptor. If -Dencrypt.key=123 is provided then KeyProperties is initialized with the key provided as a system property. If ENCRYPT_KEY environment variable is found KeyProperties are not initialized and the key member is null.

Solution

Remove ENCRYPT_KEY property from the environment prior to running spring boot application. The problem is fixed.

Vision Statement

Provide a library that can serve swagger UI in Jersey application. Inspiration for this library was the SpringFox Swagger UI project that works with spring boot application.

Use Case

Suppose you have a runnable Jar Jersey REST API application with Open API annotations and would like to provide a swagger UI for API consumers to learn and play with your APIs. Swagger UI comes as a set of static Javascript files that need to be included in your application. In addition, you need to configure a web context that will serve swagger UI pages. By including itzap-jerseyswagger.jar in your project you will get an endpoint that will load functional Swagger UI with your API definitions.

Implementation Details

Here is the code that starts embedded Tomcat in the project that I described in my previous post itzap-message. I modified it to use Open API annotations and include swagger UI.

private static void start() throws Exception {

        String contextPath = "";
        String appBase = ".";
        String port = System.getenv("PORT");
        if (port == null || port.isEmpty()) {
            port = "8025";
        }

        Tomcat tomcat = new Tomcat();
        tomcat.setPort(Integer.parseInt(port));
        tomcat.getHost().setAppBase(appBase);

        Context context = tomcat.addWebapp(contextPath, appBase);

        Tomcat.addServlet(context, "jersey-container-servlet",
                new ServletContainer(resourceConfig()));
        context.addServletMappingDecoded(UDecoder.URLDecode("/v1/itzap/message/*", Charset.defaultCharset()),
                "jersey-container-servlet");

        SwaggerContext.addSwaggerServlet(tomcat, context,
                ConfigBuilder.builder(ConfigType.TYPE_SAFE)
                        .build()
                        .getConfig("swagger"),
                EmailApplication.class);

        tomcat.getConnector();
        tomcat.start();
        tomcat.getServer().await();
    }

The code to notice here is:

SwaggerContext.addSwaggerServlet(tomcat, context,
                ConfigBuilder.builder(ConfigType.TYPE_SAFE)
                        .build()
                        .getConfig("swagger"),
                EmailApplication.class);

That all you need to do to add swagger UI to the REST API application.

There are a few properties that need to be added to the project configuration.

swagger.package="com.itzap"
swagger.apiBaseUrl="http://${email.host}/v1/itzap/message/"

Now you can start the project java -jar message-impl/target/message-impl-0.0.1-SNAPSHOT.jar and navigate your browser to

http://localhost:8025/api/v1/swagger

Readme

itzap-jerseyswagger

itzap-jerseyswagger library is designed to provide swagger UI endpoint to the Jersey REST API application.

Usage

Include itzap-jerseyswagger dependency to your project.

<dependency>
    <groupId>com.itzap</groupId>
    <artifactId>itzap-jerseyswagger</artifactId>
    <version>0.0.1-SNAPSHOT</version>
</dependency>

add the following properties to the application config file

swagger.package="com.itzap"
swagger.apiBaseUrl="http://${email.host}/v1/itzap/message/"

The following code will add swaggerUI artifacts and context

SwaggerContext.addSwaggerServlet(tomcat, context,
        ConfigBuilder.builder(ConfigType.TYPE_SAFE)
                .build()
                .getConfig("swagger"),
        EmailApplication.class);

How To Build

  • Clone the following projects:
    • git clone git@github.com:avinokurov/itzap-parent.git
    • git clone git@github.com:avinokurov/itzap-common.git
  • Build all projects
    • cd itzap-parent && mvn clean install
    • cd ../itzap-common && mvn clean install

Visit my ITZap blog to read more about this project.

Code

Complete implementation can be found here:

Vision Statement

Immutable POJOs are key to bug-free programs. It is not straight forward to write immutable POJOs in Java. Builder pattern comes handy to solve this problem by separating getters and setters. The problem with the Builder pattern in java is the need to duplicate POJO properties. The purpose of this solution is to minimize the downside of the Builder pattern in java by separating the state into a separate class.

Use Case

Suppose we have a User POJO like this:

public class User {
    private final String lastName;
  	private final String firstName;

    private User(Builder builder) {
        this.lastName = builder.lastName;
      	this.firstName = builder.firstName;
    }

    public String getLastName() {
        return this.lastName;
    }

    public String getFirstName() {
        return this.firstName;
    }

    public static class Builder {
		private String lastName;
  		private String firstName;
      
        public Builder setLastName(String lastName) {
            this.lastName = lastName;
            return this;
        }

        public Builder setFirstName(String firstName) {
            this.firstName = firstName;
            return this;
        }

        public User build() {
            return new User(this);
        }
    }

    public static Builder builder() {
        return new Builder();
    }
}

The problem with the above solution is that lastName and firstName attributes need to be repeated for both builder and POJO itself.

Solution

To address this issue we can try to separate attributes into a separate State class.

public class User {
  	private static class State {
    	private String lastName;
  		private String firstName;
    }
	
  	private final State state;
  
    private User(Builder builder) {
        this.state = builder.state;
    }

    public String getLastName() {
        return this.state.lastName;
    }

    public String getFirstName() {
        return this.state.firstName;
    }

    public static class Builder {
		private final State state = new State();
      
        public Builder setLastName(String lastName) {
            this.state.lastName = lastName;
            return this;
        }

        public Builder setFirstName(String firstName) {
            this.state.firstName = firstName;
            return this;
        }
      
        public User build() {
            return new User(this);
        }
      
      	public static Builder from(User user) {
          return User.builder()
            .setLastName(user.getLastName())
            .setFirstName(user.getFirstName());
        }
    }

    public static Builder builder() {
        return new Builder();
    }
}

Bonus for IntelliJ Users

IntelliJ’s generate Getter/Setter feature cannot be used effectively when attributes of the POJO are factored out into a separate State. Luckily, IntelliJ provides the ability to customize generators. Here is an example of a possible custom Getter/Setter generator scripts.

Getter

#if($field.modifierStatic)
static ##
#end
$field.type ##
#set($name = $StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project))))
#if ($field.boolean && $field.primitive)
  is##
#else
  get##
#end
${name}() {
  return this.state.$field.name;
}

Setter

#set($paramName = $helper.getParamName($field, $project))
public ##
#if($field.modifierStatic)
static void ##
#else
  Builder ##
#end
set$StringUtil.capitalizeWithJavaBeanConvention($StringUtil.sanitizeJavaIdentifier($helper.getPropertyName($field, $project)))($field.type $paramName) {
#if ($field.name == $paramName)
  #if (!$field.modifierStatic)
  this.state.##
  #else
    $this.state.##
  #end
#end
$field.name = $paramName;
#if(!$field.modifierStatic)
return this;
#end
}

Navigate to the State class and generate Getters/Setters. Then copy getters to the POJO and setters to the builder. A small extra step but save a lot of boilerplate code.

Vision Statement

NoSQL databases are dominating enterprise systems. One reason is that they do not require precise data schema and can deal with new data without triggering massive changes. The idea is to achieve schema flexibility with the typical SQL database like PostgreSQL or MySql. In general, NoSQL databases require some key/index/primary fields to enable fast lookups. If we preserve the concept of necessary fields and store the rest of the document in the JSON text field we can accomplish similar flexibility with the plain old SQL database.

Use Case

Imagine some users table where fields like username and email are required to identify a user, but user attributes like phone, lastName, firstName can be just stored as part of the user record.

Implementation Details

Here is the database schema for the above use case using H2 Java in-memory database:

drop table if exists users;
create table users
(
    id       bigint       NOT NULL auto_increment,
    username varchar(50)  not null,
    password varchar(255) not null,
    enabled  boolean      not null,
    email    varchar(255) not null,
    object   varchar(4048)         NOT NULL DEFAULT '{}',
    CONSTRAINT pk_users PRIMARY KEY (id)
);

In this example, application key fields defined as typical SQL schema and the rest of the document is stored in the object field. Now, all we need to do is merge key fields into a final object. I used ebean ORM library to implement database connectivity and statement execution and command/reactive design pattern explained in my RxJava post to implement DAO layer.

Model

itzap-ebeans project implements this idea and contains an example UserDao implementation. Simple user model looks like this:

@JsonDeserialize(builder = User.Builder.class)
public class User extends Auditable {
    private final String lastName;
    private final String firstName;
    private final String email;
    private final String phone;

    @JsonProperty("enabled")
    private final Boolean enabled;
    private final String username;

    @JsonIgnore
    private final String password;

    public User(Builder builder) {
        super(builder);

        this.lastName = builder.lastName;
        this.firstName = builder.firstName;
        this.email = builder.email;
        this.phone = builder.phone;
        this.enabled = builder.enabled;
        this.username = builder.username;
        this.password = builder.password;
    }

    public String getLastName() {
        return lastName;
    }

    public String getFirstName() {
        return firstName;
    }

    public String getEmail() {
        return email;
    }

    public String getPhone() {
        return phone;
    }

    public Boolean getEnabled() {
        return enabled;
    }

    public String getUsername() {
        return username;
    }
  
    public String getPassword() {
        return password;
    }

    @Override
    public String toString() {
        return MoreObjects.toStringHelper(this.getClass())
                .add("lastName", lastName)
                .add("email", email)
                .add("username", username)
                .toString();
    }

    @JsonPOJOBuilder(withPrefix = "set")
    @JsonIgnoreProperties(ignoreUnknown = true)
    public static class Builder extends Auditable.Builder<User, User.Builder> {
        private String lastName;
        private String firstName;
        private String email;
        private String phone;
        private Boolean enabled;
        private String username;
        private String password;

        public Builder() {
        }

        @Override
        protected Builder getThis() {
            return this;
        }

        public Builder setLastName(String lastName) {
            this.lastName = lastName;
            return this;
        }

        public Builder setFirstName(String firstName) {
            this.firstName = firstName;
            return this;
        }

        public Builder setEmail(String email) {
            this.email = email;
            return this;
        }

        public Builder setPhone(String phone) {
            this.phone = phone;
            return this;
        }

        public Builder setEnabled(Boolean enabled) {
            this.enabled = enabled;
            return this;
        }

        public Builder setUsername(String username) {
            this.username = username;
            return this;
        }

        public Builder setPassword(String password) {
            this.password = password;
            return this;
        }

        @Override
        public Builder merge(User org) {
            Builder bld = super.merge(org)
                    .setEmail(StringUtils.defaultIfBlank(this.email, org.getEmail()))
                    .setEnabled(ObjectUtils.defaultIfNull(this.enabled, org.getEnabled()))
                    .setLastName(StringUtils.defaultIfBlank(this.lastName, org.getLastName()))
                    .setUsername(StringUtils.defaultIfBlank(this.username, org.getUsername()))
                    .setFirstName(StringUtils.defaultIfBlank(this.firstName, org.getFirstName()))
                    .setPassword(StringUtils.defaultIfBlank(this.password, org.getPassword()))
                    .setPhone(StringUtils.defaultIfBlank(this.phone, org.getPhone()));

            return bld;
        }

        @Override
        public User build() {
            return new User(this);
        }
    }

    public static Builder from(User user) {
        Builder builder = new Builder();
        BeanUtils.copyProperties(user, builder);
        return builder;
    }

    public static Builder builder() {
        return new Builder();
    }
}

Note all the Json annotations that enable serialization/deserialization of the model and merge method that provides rules for merging database and Json objects.

Dao Implementation

Dao layer implemented using two base commands: ListBaseEbeanCommand and BaseBulkEbeanCommand. ListBaseEbeanCommand is the base command for all select operations and BaseBulkEbeanCommand is the base command for all Insert/Update/Delete operations. Base commands operate using EbeanHandler and report exceptions using EbeanErrorHandler. Complete DAO implementation looks like this:

public class UserDao extends AbstractServiceDao {
    private static final String SELECT_ALL_USERS = "select * from users";
    private static final String WHERE_USER = " where username=:username";
    private static final String WHERE_USER_ID = " where id=:id";

    private static final String ADD_USER = "INSERT INTO public.users(" +
            "            username, password, enabled, email, object)" +
            "    VALUES (:username, :password, :enabled, :email, :object)";
    private static final String UPDATE_USER = "UPDATE public.users " +
            "   SET username=:username, password=:password, enabled=:enabled, email=:email, object=:object" +
            " WHERE id=:id";
    private static final String DELETE_USER = "DELETE FROM users WHERE id=:id";

    public UserDao(Servers.Server server) {
        super(server);
    }

    public Observable<User> getUsers() {
        return new SelectUsersCommand()
                .toObservable()
                .flatMap(Observable::fromIterable);
    }

    public Observable<User> getUserByName(String name) {
        return new SelectUsersCommand()
                .setName(name)
                .toObservable()
                .map(users -> users.get(0));
    }

    public Observable<User> getUser(Long userId) {
        return new SelectUsersCommand()
                .setUserId(userId)
                .toObservable()
                .map(users -> users.get(0));
    }

    public Observable<Integer> addUser(User user) {
        return new BaseBulkEbeanCommand<BaseBulkEbeanCommand>("cmd-addUser-cmd",
                new UserHandler(user)) {
            @Override
            protected BaseBulkEbeanCommand getThis() {
                return this;
            }

            @Override
            protected SqlUpdate createSql() {
                return server.getServer().createSqlUpdate(ADD_USER);
            }
        }.toObservable();
    }

    public Observable<Integer> updateUser(Long userId, User user) {
        return new SelectUsersCommand()
                .setName(user.getUsername())
                .toObservable()
                .flatMap(orgUser -> new BaseBulkEbeanCommand<BaseBulkEbeanCommand>("cmd-updateUser-cmd",
                        new UserHandler(userId, user, orgUser.get(0))) {
                    @Override
                    protected BaseBulkEbeanCommand getThis() {
                        return this;
                    }

                    @Override
                    protected SqlUpdate createSql() {
                        return server.getServer().createSqlUpdate(UPDATE_USER);
                    }
                }.toObservable());
    }

    public Observable<Integer> deleteUser(Long id) {
        return new BaseBulkEbeanCommand<BaseBulkEbeanCommand>("cmd-deleteUser-cmd",
                new DeleteEbeanHandler<>(QueryParameter.id(id))) {
            @Override
            protected BaseBulkEbeanCommand getThis() {
                return this;
            }

            @Override
            protected SqlUpdate createSql() {
                return server.getServer().createSqlUpdate(DELETE_USER);
            }
        }.toObservable();
    }

    class UserHandler extends AbstractAuditableHandler<User, User.Builder> {
        UserHandler(User user) {
            this(null, user, User.builder().build());
        }

        UserHandler(Long userId, User user, User dbUser) {
            super(userId, user, dbUser);
        }

        @Override
        protected User.Builder rowProperties(SqlUpdate updateQuery, User user) {
            updateQuery.setParameter("username",
                    StringUtils.defaultIfBlank(user.getUsername(), orgObject.getUsername()));

            String pass = getPassword(user);
            if (StringUtils.isBlank(pass)) {
                updateQuery.setParameter("password", orgObject.getPassword());
            } else {
                updateQuery.setParameter("password", pass);
            }
            updateQuery.setParameter("enabled", ObjectUtils.defaultIfNull(user.getEnabled(),
                    orgObject.getEnabled()));
            updateQuery.setParameter("email",
                    StringUtils.defaultIfBlank(user.getEmail(), orgObject.getEmail()));

            return audibalBuilder(user,  User.from(user));
        }
    }

    private static String getPassword(User user) {
        return user.getPassword();
    }

    private static  User.Builder fromRow(SqlRow row, User user) {
        return User.from(user)
                .setUsername(row.getString("username"))
                .setEmail(row.getString("email"))
                .setEnabled(row.getBoolean("enabled"))
                .setPassword(row.getString("password"));
    }

    class SelectUsersCommand extends ListBaseEbeanCommand<User, SelectUsersCommand> {
        private String name;
        private Long userId;

        public SelectUsersCommand setName(String name) {
            this.name = name;
            return getThis();
        }

        public SelectUsersCommand setUserId(Long userId) {
            this.userId = userId;
            return this;
        }

        SelectUsersCommand() {
            super("cmd-get-users",
                    new AbstractSelectListEbeanHandler<User, User.Builder>(User.class) {
                        @Override
                        protected User.Builder from(SqlRow row, User user) {
                            return fromRow(row, user);
                        }
                    });
        }

        @Override
        protected SelectUsersCommand getThis() {
            return this;
        }

        @Override
        protected SqlQuery createSql() {
            String sql = SELECT_ALL_USERS;
            if (StringUtils.isNotBlank(this.name)) {
                sql = sql + WHERE_USER;
            } else if (this.userId != null) {
                sql = sql + WHERE_USER_ID;
            }

            SqlQuery qry = server.getServer().createSqlQuery(sql);
            if (StringUtils.isNotBlank(this.name)) {
                qry.setParameter("username", this.name);
            } else if (this.userId != null) {
                qry.setParameter("id", this.userId);
            }
            return qry;
        }
    }
}

Readme

itzap-ebeans

itzap-ebeans provides an easy to use library for building reactive DAO with flexible schema models. Visit my ITZap blog to read more about this project.

How To Build

  • Clone the following projects:
    • git clone git@github.com:avinokurov/itzap-parent.git
    • git clone git@github.com:avinokurov/itzap-common.git
    • git clone git@github.com:avinokurov/itzap-rxjava.git
    • git clone git@github.com:avinokurov/itzap-ebeans.git
  • Build all projects
    • cd itzap-parent && mvn clean install
    • cd ../itzap-common && mvn clean install
    • cd ../itzap-rxjava && mvn clean install
    • cd ../itzap-ebeans && mvn clean install
  • Example
    • itzap-beans project contains sample UserDao implementation

Code

Complete implementation can be found here:

Scenario

Let’s say you built spring-boot executable jar and uploaded it into nexus repository. To deploy your spring-boot application you create a Dockerfile that downloads your jar using nexus rest search and download API. For example:

FROM openjdk:8-jre-alpine

MAINTAINER itzap <mailer@itzap.com>

WORKDIR /

RUN wget -O app-service.jar  \
	'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=des'

EXPOSE 8080

ENTRYPOINT [ "java", "-jar","app-service.jar"]

When you run docker container you see the following error message:

no main manifest attribute, in app-service.jar

Diagnostic

When you build and run jar locally everything is working fine. app-service.jar inside docker container appears to be corrupted. To diagnose the issue run the wget command from the terminal:

wget -O app-service.jar  \
	'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=des'

Notice the following results on the command line:

$ wget -O app-service.jar 'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc'

--2019-10-21 21:48:27--  https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc

Распознаётся nexus.compnyUrl.com (nexus.companyUrl.com)… 00.00.00.00

Подключение к nexus.compnyUrl.com (nexus.compnyUrl.com)|00.00.00.00|:443... соединение установлено.

HTTP-запрос отправлен. Ожидание ответа… 302 Found

Адрес: https://nexus.compnyUrl.com/nexus/repository/company-snapshots/com/mycompany/service/app-service/0.0.1-SNAPSHOT/app-service-0.0.1-20191021.220127-1-javadoc.jar [переход]

--2019-10-21 21:48:28--  https://nexus.compnyUrl.com/nexus/repository/app-snapshots/com/mycompany/service/app-service/0.0.1-SNAPSHOT/app-service-0.0.1-20191021.220127-1-javadoc.jar

Повторное использование соединения с nexus.compnyUrl.com:443.

HTTP-запрос отправлен. Ожидание ответа… 200 OK

Длина: 247463 (242K) [application/java-archive]

Сохранение в: «app-service.jar»

Do not get all these messages in Russian fool you. This is not a Russian hacker attack. Pay attention to what is actually being downloaded. app-service-0.0.1-20191021.220127-1-javadoc.jar javadoc!!!

Solution

Solution to this issue can be found in sonatype Search API. Looking through documentation I came across this query parameter maven.classifier This issue happens when your repository contains javadoc jar alnog side your artifact. &sort=version&direction=desc parameters are also critical to insure you are downloading the latest version, but not enough to uniquely identify artifact. Final URL that works looks like this:

'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc&maven.classifier'

Scenario

In a spring-boot application that uses spring-boot-dependencies instead of spring-boot-starter-parent and a third party library that depends on elasticsearch version that is not comparable with version 6.x, elasticsearch dependency version get mixed up creating class loading issue.

Steps to reproduce

Create a spring-boot project and include spring-boot-dependencies in the dependency management section of the maven pom.xml

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Later, in the dependencies section include a third party library that depends on elastic search version 5.6.8

<dependnecies>
  <dependency>
    <groupId>com.netflix.conductor</groupId>
    <artifactId>conductor-es5-persistence</artifactId>
    <version>2.8.1</version>
  </dependency>
</dependnecies>

Note that conductor-es5-persistence depends on elasticsearch version 5.6.8

The try to run mvn dependency:tree on your project and observe the following:

[INFO] +- com.netflix.conductor:conductor-es5-persistence:jar:2.8.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:6.4.3:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:6.4.3:compile

Where 6.4.3 is coming from?

Diagnostic

Turns out mvn dependency:tree shows the issue, but it does not say where org.elasticsearch:elasticsearch:jar:6.4.3:compile is comming from. The best command for the job is:

mvn -Dverbose=true help:effective-pom

This command outputs effective POM that can be redirected to a file mvn -Dverbose=true help:effective-pom > ~/pom.xml and analysed. The effective pom will look something like this

...
<dependency>
  <groupId>org.elasticsearch</groupId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1965 -->
  <artifactId>elasticsearch</artifactId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1966 -->
  <version>6.4.3</version>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->
</dependency>
...

Magic! now we can see exactly where the problem is:

<!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->

Solution

Lets look at this import dependency:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-dependencies</artifactId>
    <version>${spring.boot.version}</version>
    <type>pom</type>
    <scope>import</scope>
  </dependency>

spring-boot-dependencies is imported, so there is no easy way to exclude elasticsearch dependency that it pulls into our project. A simple maven <exclude> will not work here. The only way to solve the issue is to bring elasticsearch dependencies into your project dependency management block like this:

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
  
  <elasticsearch.version>5.6.8</elasticsearch.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    
    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.distribution.integ-test-zip</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
      <type>zip</type>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>transport-netty4-client</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

  </dependencies>
</dependencyManagement>

Now run mvn -Dverbose=true help:effective-pom and confirm that the problem is fixed.

Vision Statement

Build 7.3.x elasticsearch embedded server into itzapelasticsearch library to enable easy prototyping and unit/integration testing

How It Works

Elasticsearch embedded server starts a single elasticsearch Node. Elasticsearch server node will be listening for incoming requests on port 9200 Elasticsearch client can be configured to access embedded server using the following url: http://localhost:9200 Elasticsearch embedded server will create data and home temp folders that will be deleted opon application close.

Code

Here is the main class that implements elasticsearch embedded server node

public class EmbeddedElasticSearchServer implements IOData {
  private static final Logger LOGGER = LoggerFactory.getLogger(EmbeddedElasticSearchServer.class);

  private Node instance;
  private int port;
  private final AnyConfig config;

  public EmbeddedElasticSearchServer(AnyConfig config) {
    this.config = config;
  }

  private static class PluginConfigurableNode extends Node {
    PluginConfigurableNode(Settings input,
                           Map<String, String> properties,
                           Path configPath,
                           Supplier<String> defaultNodeName,
                           Collection<Class<? extends Plugin>> classpathPlugins) {
      super(InternalSettingsPreparer.prepareEnvironment(input, properties, configPath, defaultNodeName), classpathPlugins, false);
    }
  }

  @Override
  public synchronized Completable start() {
    return new RunnableCommand<Void>("cmd-start") {
      @Override
      protected Void run() {
        Settings settings = getSettings();

        instance = new PluginConfigurableNode(settings, ImmutableMap.of(),
                                              null, () -> config.getString(EsConfig.CLUSTER_NAME),
                                              singletonList(Netty4Plugin.class));
        try {
          instance.start();
          Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            try {
              if (instance != null) {
                instance.close();
              }
            } catch (IOException e) {
              LOGGER.error("Error closing ElasticSearch");
            }
          }));
          LOGGER.info("ElasticSearch cluster {} started in local mode on port {}", instance.settings().get("cluster.name"),
                      port);
          return null;
        } catch (NodeValidationException e) {
          throw new IZapException("Failed to start embedded elastic search server", e);
        }
      }
    }.toCompletable();
  }

  @Override
  public synchronized Completable stop() {
    return new RunnableCommand<Void>("cmd-stop") {
      @Override
      protected Void run() {
        if (instance != null && !instance.isClosed()) {
          LOGGER.info("Stopping Elastic Search");
          try {
            instance.close();
            instance = null;
            LOGGER.info("Elastic Search on port {} stopped", port);
          } catch (IOException e) {
            throw new IZapException("Failed to close elastic search embedded server", e);
          }
        }

        return null;
      }
    }.toCompletable();
  }

  private Settings getSettings() {
    String clusterName = config.getString(EsConfig.CLUSTER_NAME);
    String host = config.getString(EsConfig.HOST);
    port = config.getInt(EsConfig.PORT);

    try {
      File dataDir = Files.createTempDirectory(clusterName + "_" + System.currentTimeMillis() + "data").toFile();
      FileUtils.forceDeleteOnExit(dataDir);
      cleanDataDir(dataDir.getAbsolutePath());

      File homeDir = Files.createTempDirectory(clusterName + "_" + System.currentTimeMillis() + "-home").toFile();
      cleanDataDir(homeDir.getAbsolutePath());
      FileUtils.forceDeleteOnExit(homeDir);

      Settings.Builder settingsBuilder = Settings.builder()
        .put("cluster.name", clusterName)
        .put("http.host", host)
        .put("http.port", port)
        .put("transport.tcp.port", port + 100)
        .put(EsConfig.DATA_PATH.getName(), dataDir.getAbsolutePath())
        .put(EsConfig.HOME_PATH.getName(), homeDir.getAbsolutePath())
        .put("http.cors.enabled", true)
        .put("node.data", true)
        .put("http.type", "netty4")
        .put("transport.type", "netty4");

      return settingsBuilder.build();
    } catch (IOException e) {
      throw new IZapException("Failed to create temp data/home dir.", e);
    }
  }
}

Elasticsearch embedded server implementation is using RunnableCommand command pattern to implement start and stopmethods. Please note that RunnableCommand returns Completable and not Observable. Sice Observable<Void> is no longer permitted in RxJava2 and start/stop methods have no return values. Here is the unit test for the elasticsearch embedded server

public class EmbeddedElasticSearchServerTest {
    private EmbeddedElasticSearchServer server;

    @Before
    public void setup() {
        AnyConfig config = ConfigBuilder.builder(ConfigType.TYPE_SAFE)
                .setFileName(this.getClass()
                        .getResource("/es-config.properties").getFile())
                .build();

        server = new EmbeddedElasticSearchServer(config);
    }

    @Test
    public void startTest() {
        server.start().blockingGet();
        server.stop().blockingGet();
    }
}

Elasticsearch dependencies

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-high-level-client</artifactId>
  <version>7.3.1</version>
</dependency>

<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch</artifactId>
  <version>7.3.1</version>
</dependency>

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>transport</artifactId>
  <version>7.3.1</version>
</dependency>