Of Stack

Android Web3j OOM Solution

  • 2021-09-20 21:24:41
  • OfStack

OOM may be generated when Web3j is used to create wallets and import wallets in Android clients. The related issue has been mentioned on Github: https://github.com/web3j/web3j/issues/299. This problem did not exist before Web3j 3.0. Because the new version of Web3j replaces lambdaworks library with spongycastle library, although the efficiency is improved, there is compatibility problem on Android side.

Code address of this project: https://github.com/uncleleonfan/WalletOOM.git

Create wallet OOM solution

When creating a wallet, if you create 1 Full Wallet, it will cause OOM: 


public void onCreateFullWallet(View view) {
  String filePath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/full";
  File file = new File(filePath);
  file.mkdirs();
  try {
    WalletUtils.generateFullNewWalletFile("a12345678", file);
  } catch (NoSuchAlgorithmException e) {
    e.printStackTrace();
  } catch (NoSuchProviderException e) {
    e.printStackTrace();
  } catch (InvalidAlgorithmParameterException e) {
    e.printStackTrace();
  } catch (CipherException e) {
    e.printStackTrace();
  } catch (IOException e) {
    e.printStackTrace();
  }
}

Log is as follows:

"Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1036 byte allocation with 16777216 free bytes and 48MB until OOM; failed due to fragmentation (required continguous free 16384 bytes for a new buffer where largest contiguous free 8192 bytes)",
"\tat org.spongycastle.util.Arrays.clone(Arrays.java:602)",
"\tat org.spongycastle.crypto.generators.SCrypt.SMix(SCrypt.java:126)",
"\tat org.spongycastle.crypto.generators.SCrypt.MFcrypt(SCrypt.java:87)",
"\tat org.spongycastle.crypto.generators.SCrypt.generate(SCrypt.java:66)",
"\tat org.web3j.crypto.Wallet.generateDerivedScryptKey(Wallet.java:136)",
"\tat org.web3j.crypto.Wallet.create(Wallet.java:74)",
"\tat org.web3j.crypto.Wallet.createStandard(Wallet.java:93)",
"\tat org.web3j.crypto.WalletUtils.generateWalletFile(WalletUtils.java:61)"

In generateFullNewWalletFile, createStandard will be called to create wallet, and N_STANDARD and P_STANDARD will be used to configure encryption strength, which will directly affect the memory size to be used and eventually lead to OOM. 


public static WalletFile createStandard(String password, ECKeyPair ecKeyPair)
    throws CipherException {
  return create(password, ecKeyPair, N_STANDARD, P_STANDARD);
}

The solution is as simple as creating an Light Wallet: 


public void onCreateLightWallet(View view) {
  String filePath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/light";
  File file = new File(filePath);
  file.mkdirs();
  try {
    WalletUtils.generateLightNewWalletFile("a12345678", file);
  } catch (NoSuchAlgorithmException e) {
    e.printStackTrace();
  } catch (NoSuchProviderException e) {
    e.printStackTrace();
  } catch (InvalidAlgorithmParameterException e) {
    e.printStackTrace();
  } catch (CipherException e) {
    e.printStackTrace();
  } catch (IOException e) {
    e.printStackTrace();
  }
}

generateLightNewWalletFile will call createLight to create a light wallet, using N_LIGHT, P_LIGHT, which are relatively small in value, so OOM will not be used. 


public static WalletFile createLight(String password, ECKeyPair ecKeyPair)
    throws CipherException {
  return create(password, ecKeyPair, N_LIGHT, P_LIGHT);
}

We can compare the sizes of N_STANDARD and P_STANDARD, N_LIGHT and P_LIGHT under 1: 


private static final int N_LIGHT = 1 << 12;
private static final int P_LIGHT = 6;

private static final int N_STANDARD = 1 << 18;
private static final int P_STANDARD = 1;

Import wallet OOM solution

OOM will not be generated when we import a light wallet, but OOM may be generated when we import not a light wallet. For example, we use Imtoken to create a wallet and export Keystore, Keystore as follows:

{"address":"9a2e2419f3af050d4730f80e7a65b9f8deb5e16f","crypto":{"cipher":"aes-128-ctr","cipherparams":{"iv":"eaccea79c27a91e307f24988186ef21a"},"ciphertext":"a163e532edf2d76beaee5c26fd2c2fab071a9cb37627aa185ac89e223e41ab97","kdf":"scrypt","kdfparams":{"dklen":32,"n":65536,"p":1,"r":8,"salt":"6a847392a029553f4152dea7bb0b6fb0ac9eec29f55e572fe94603182f5ed7f1"},"mac":"3fad2a31e18c611b10df84db9ae368ce2e189b5c35e9f111e40ca4b4bfb02491"},"id":"032c47c2-c7b7-46f8-a3f7-f526580f6f09","version":3}

It can be seen that n is 65536, p is 1, and n of light wallet is 1 < < 12, that is, 2 to the 12th power, 4096, so this is not a light wallet.

We put the Keystore into the SD card as an json file, push, and import it using Web3j: 


public void onImportWallet(View view) {
  try {
    // Need to be sent in advance assets Directory keystore.json File pushed to mobile phone SD Inside 
    String filePath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/keystore.json";
    File file = new File(filePath);
    WalletUtils.loadCredentials("a12345678", file);
  } catch (IOException e) {
    e.printStackTrace();
  } catch (CipherException e) {
    e.printStackTrace();
  }
}

The discovery will also be OOM:

Caused by: java.lang.OutOfMemoryError: Failed to allocate a 1036 byte allocation with 13588800 free bytes and 12MB until OOM; failed due to fragmentation (required continguous free 16384 bytes for a new buffer where largest contiguous free 12288 bytes)
at org.spongycastle.util.Arrays.clone(Arrays.java:602)
at org.spongycastle.crypto.generators.SCrypt.SMix(SCrypt.java:126)
at org.spongycastle.crypto.generators.SCrypt.MFcrypt(SCrypt.java:87)
at org.spongycastle.crypto.generators.SCrypt.generate(SCrypt.java:66)
at org.web3j.crypto.Wallet.generateDerivedScryptKey(Wallet.java:136)
at org.web3j.crypto.Wallet.decrypt(Wallet.java:214)
at org.web3j.crypto.WalletUtils.loadCredentials(WalletUtils.java:112)

As can be seen from log, it is the same as OOM that created the wallet, which is caused by the last call to generateDerivedScryptKey: 


private static byte[] generateDerivedScryptKey(
    byte[] password, byte[] salt, int n, int r, int p, int dkLen) throws CipherException {
  return SCrypt.generate(password, salt, n, r, p, dkLen);
}

You can create a light wallet by creating a wallet, but you can't let users change a light wallet to import it when importing it. Here, we can only replace the lambda library to complete the coding and decoding of keystore, that is, we can write an generateDerivedScryptKey method by ourselves, and replace SCrypt of spongycastle with SCrypt of lambda. We use MyWalletUtils and MyWallet to accomplish this task together. 


public void onImportWallet(View view) {
  try {
    // Need to be sent in advance assets Directory keystore.json File pushed to mobile phone SD Inside 
    String filePath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/keystore.json";
    File file = new File(filePath);
    Credentials credentials = MyWalletUtils.loadCredentials("a12345678", file);
    Log.d(TAG, "address:" + credentials.getAddress());
  } catch (IOException e) {
    e.printStackTrace();
  } catch (CipherException e) {
    e.printStackTrace();
  }
}

public class MyWalletUtils {
  public static Credentials loadCredentials(String password, File source)
      throws IOException, CipherException {
    WalletFile walletFile = objectMapper.readValue(source, WalletFile.class);
    return Credentials.create(MyWallet.decrypt(password, walletFile));
  }
}

public class MyWallet {

  private static final int CURRENT_VERSION = 3;
  private static final String CIPHER = "aes-128-ctr";
  static final String AES_128_CTR = "pbkdf2";
  static final String SCRYPT = "scrypt";

  private static byte[] generateDerivedScryptKey(
      byte[] password, byte[] salt, int n, int r, int p, int dkLen) {
    try {
      return SCrypt.scrypt(password, salt, n, r, p, dkLen);
    } catch (GeneralSecurityException e) {
      e.printStackTrace();
    }
    return null;
  }
}

After processing according to the above method, OOM can be solved, but the waiting time of users will be slightly longer by 1 point. In addition, it is better to add libscrpt. so library of Android platform under 1, which can be found in jniLibs of this project.

Related articles: