jar package conflict resolution in Hbase and elasticsearch integration

2020-12-05 17:13:58
OfStack

The problem background

Then data platform, the project need to use es and HBASE structures, data query interface, the integrated jar appeared in the process of packet conflict bug: com. google. common. base. Stopwatch. () V from class org. apache. hadoop. hbase. zookeeper. MetaTableLocator


org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)
    ...
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil
    ...
    at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
    ... 45 more

The solution

Upon investigation, it was confirmed that the conflict was caused by the package com.google.guava. es relies on version 18 and above, while HBASE only supports version 16 and above. guava, on the other hand, has changed internally since 17 and the methods have changed, so 18 is not compatible with version 16 downwards. During the course of the project, if both versions 16 and 18 are introduced, the calling procedures for es and hbase will be confused. The next step is to repackage, put guava18 inside es, and display the package that references guava16 in the pom file. Thus, es calls guava18 inside the package, and hbase calls guava16 outside.

repack

Create a new maven project and configure it in the pom file as follows:


<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <!--
    this maven project is aim to fix the jar comflic in hbase & elasticsearch.
    in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.

    cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.
    then, adjust your project pom.xml whth:
    <dependency>
      <groupId>douguo.shaded.elasticsearch</groupId>
      <artifactId>douguo_shaded_elasticsearch</artifactId>
      <version>1.0-SNAPSHOT</version>
      <exclusions>
        <exclusion>
          <groupId>org.elasticsearch</groupId>
          <artifactId>elasticsearch</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    finally, your jar comflic will be fixed.

    @date:2017-11-30
    @author:zhangjianfei
    @since:douguo.shaded.elasticsearch-1.0.0
  -->
  <groupId>douguo.shaded.elasticsearch</groupId>
  <artifactId>douguo_shaded_elasticsearch</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <elasticsearch.version>2.4.1</elasticsearch.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>shield</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <configuration>
          <createDependencyReducedPom>false</createDependencyReducedPom>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <relocations>
                <relocation>
                  <pattern>com.google.guava</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>org.joda</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>com.google.common</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>com.google.thirdparty</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>
                </relocation>
              </relocations>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <repositories>
    <repository>
      <id>elasticsearch-releases</id>
      <url>http://maven.elasticsearch.org/releases</url>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>daily</updatePolicy>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>
  </repositories>

</project>

Four potentially conflicting jar packages, such as ES51en.joda, were migrated through the ES54en-ES55en-ES56en plug-in and repackaged so that when the jar package was introduced, the jar package could use its own dependencies instead of external dependencies. . Here it is important to note that need to be com google. common etc. All 4 bags to move, can appear otherwise java. lang. IllegalAccessError: tried to access method com. google. common. base error

Project package


mvn clean install

The new dependency package will be in the.es80EN2 maven warehouse, and if the company builds the warehouse, the jar package will need to be uploaded. If you run the jar package directly, remember to recompile the project and replace the lib directory

The project loads the new package

You only need to configure it in the pom file:


<!-- douguo.shaded.elasticsearch -->
<dependency>
  <groupId>douguo.shaded.elasticsearch</groupId>
  <artifactId>douguo_shaded_elasticsearch</artifactId>
  <version>1.0-SNAPSHOT</version>
  <exclusions>
    <exclusion>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
    </exclusion>
  </exclusions>
</dependency>


<!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
<!-- this guava is only used in habse
   in es, 18.0+ is required, but hbase only supported 16.0 or blow.
   clean as install douguo.shaded.elasticsearch -->
<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>16.0</version>
</dependency>

Thus, the guava18 package is under douguo.shaded.elasticsearch, and es is called first. The externally configured guava16 is called by HBASE. Both versions of jar packages exist independently of each other!

Here, the problem is solved!