android Using okhttp may raise a point of OOM

Encountered a problem: it is necessary to add signature verification to all requests to prevent the interface from being brushed; The incoming requests url and body generate a text string as an header and transmit it to the server; There are already ready-made signature verification methods String doSignature (String url, byte [] body); The current network library is based on com. squareup. okhttp3: okhttp: 3.14. 2.
This is very simple, of course, is to write an interceptor and then pass in url and body of the request object. So there are:

public class SignInterceptor implements Interceptor {
  @NonNull
  @Override
  public Response intercept(@NonNull Chain chain) throws IOException {
    Request request = chain.request();
    RequestBody body = request.body();
    byte[] bodyBytes = null;
    if (body != null) {
      final Buffer buffer = new Buffer();
      body.writeTo(buffer);
      bodyBytes = buffer.readByteArray();
    }

    Request.Builder builder = request.newBuilder();
    HttpUrl oldUrl = request.url();
    final String url = oldUrl.toString();
    final String signed = doSignature(url, bodyBytes));
    if (!TextUtils.isEmpty(signed)) {
      builder.addHeader(SIGN_KEY_NAME, signed);
    }
    return chain.proceed(builder.build());
  }
}

okhttp's ReqeustBody is an abstract class, content output only writeTo method, write the content to an BufferedSink interface implementation body, and then convert the data into byte [], that is, memory array. The only class that can achieve the goal is Buffer, which implements BufferedSink interface and can provide the method readByteArray that can be converted into memory array. This seems to be no problem, can it cause OOM?

Yes, it depends on the request type. What if it is an interface for uploading files? What if this file is larger? Upload interface may use public static RequestBody create (final @ Nullable MediaType contentType, final File file) method. If it is for the implementation of files, its writeTo method is sink. writeAll (source); The Buffer. readByteArray used when we pass it to the signature method is to convert all the contents in the buffer into a memory array, which means that all the contents in the file are converted into a memory array, which is easy to cause OOM at this time! The source code of RequestBody. create is as follows:

 public static RequestBody create(final @Nullable MediaType contentType, final File file) {
  if (file == null) throw new NullPointerException("file == null");

  return new RequestBody() {
   @Override public @Nullable MediaType contentType() {
    return contentType;
   }

   @Override public long contentLength() {
    return file.length();
   }

   @Override public void writeTo(BufferedSink sink) throws IOException {
    try (Source source = Okio.source(file)) {
     sink.writeAll(source);
    }
   }
  };
 }

You can see that the implementation holds the file, Content-Length returns the size of the file, and the contents are all transferred to the Source object.

This is really a point that was easily overlooked before, and there are few additional operations on the request body. However, once this operation becomes a large memory allocation, it is very easy to cause OOM. So how to solve it? How is the signature method handled? It turns out that this signature method is lazy here-it only reads the first 4K content passed into body, and then encrypts only this part of the content. As for the size of the passed memory array itself, it doesn't consider, completely throwing risks and troubles to the outside (excellent SDK! ).

The quick way is, of course, to make a white list. There is no sign-up verification for the upload interface server, However, this is easy to hang up and leak 10,000 yuan, and it increases maintenance costs. People who want to sign the method sdk to write another appropriate interface will kill them, so they still have to solve it fundamentally. Since the signature method only reads the first 4K content, can we only read the first 4K part of the content and then convert it into the memory array required by the method? So our goal is to expect RequestBody to be able to read part but not all of the contents of 1. Can we inherit RequestBody and rewrite its writeTo? Yes, but unrealistic, it is impossible to replace all the existing RequestBody implementation classes, while ok framework may also create private implementation classes. Therefore, we can only make articles for BufferedSink parameters of writeTo, and first understand how BufferedSink is called by okhttp framework.

The related classes of BufferedSink include Buffer and Source, all of which belong to okio framework. okhttp is only based on one lump of okio. okio does not directly use io of java, but writes another set of io operations, specifically data buffer operations. Following the above description, how is Source created and how to operate BufferedSink at the same time? In Okio. java:

 public static Source source(File file) throws FileNotFoundException {
  if (file == null) throw new IllegalArgumentException("file == null");
  return source(new FileInputStream(file));
 }
 
 public static Source source(InputStream in) {
  return source(in, new Timeout());
 }

 private static Source source(final InputStream in, final Timeout timeout) {
  return new Source() {
   @Override public long read(Buffer sink, long byteCount) throws IOException {
    try {
     timeout.throwIfReached();
     Segment tail = sink.writableSegment(1);
     int maxToCopy = (int) Math.min(byteCount, Segment.SIZE - tail.limit);
     int bytesRead = in.read(tail.data, tail.limit, maxToCopy);
     if (bytesRead == -1) return -1;
     tail.limit += bytesRead;
     sink.size += bytesRead;
     return bytesRead;
    } catch (AssertionError e) {
     if (isAndroidGetsocknameError(e)) throw new IOException(e);
     throw e;
    }
   }

   @Override public void close() throws IOException {
    in.close();
   }

   @Override public Timeout timeout() {
    return timeout;
   }
  };
 }

Source takes the file as the input stream inputstream for various read operations, but its read method parameter is an instance of Buffer. Where does it come from and how does it relate to BufferedSink? We have to continue to look at the implementation of BufferedSink. writeAll.

The implementation class of BufferedSink is Buffer, and then its writeAll method:

 @Override public long writeAll(Source source) throws IOException {
  if (source == null) throw new IllegalArgumentException("source == null");
  long totalBytesRead = 0;
  for (long readCount; (readCount = source.read(this, Segment.SIZE)) != -1; ) {
   totalBytesRead += readCount;
  }
  return totalBytesRead;
 }

It turns out that the Source. read (Buffer, long) method is called explicitly, so that the string is strung, and the Buffer parameter turns out to be itself.

Basically, it can be determined that as long as the BufferedSink interface class is implemented, and then the read content is judged to exceed the specified size, it can stop writing and return to meet the purpose, which can be named FixedSizeSink.

However, the trouble is that BufferedSink interface is very many, nearly 30 methods, do not know when the framework will call which method, only all of them! Secondly, there are many okio classes in the parameters of interface methods, and the usage of these classes needs to be understood, otherwise, the effect of using one wrong is counterproductive. Therefore, the understanding of one class becomes the understanding of multiple classes, and there is no way to write it.

The first interface is a little painful: Buffer buffer (); BufferedSink returns an instance of Buffer for external invocation, the implementation of BufferedSink is Buffer, and then returns an Buffer? ! After reading for a long time, I guessed that BufferedSink was to provide a writable buffer object, but the framework author was lazy to decouple the interface (alas, everyone is how simple it is). Therefore, FixedSizeSink needs to hold at least one Buffer object, which is used as actual data cache, and can be passed as parameters where Source. read (Buffer, long) is needed.

At the same time, you can see an implementation class FormBody of RequestBody, and write some data directly with this Buffer object:

 private long writeOrCountBytes(@Nullable BufferedSink sink, boolean countBytes) {
  long byteCount = 0L;

  Buffer buffer;
  if (countBytes) {
   buffer = new Buffer();
  } else {
   buffer = sink.buffer();
  }

  for (int i = 0, size = encodedNames.size(); i < size; i++) {
   if (i > 0) buffer.writeByte('&');
   buffer.writeUtf8(encodedNames.get(i));
   buffer.writeByte('=');
   buffer.writeUtf8(encodedValues.get(i));
  }

  if (countBytes) {
   byteCount = buffer.size();
   buffer.clear();
  }

  return byteCount;
 }

With such an operation, it is possible that the buffer size change cannot be limited! However, the amount of data should be relatively small and there are relatively few usage scenarios, and the size we specified should be able to cover this situation.

Then there is an interface BufferedSink write (ByteString byteString), and it is really exhausting to know how to use ByteString...

 @Override public Buffer write(ByteString byteString) {
  byteString.write(this);
  return this;
 }

ByteString. write (Buffer) can be directly called in the Buffer implementation. Because it is a package name access, the FixedSizeSink implemented by itself is declared in the same package name package okio; It can also be used in this way. If other package names can only be converted to byte [] first, ByteString should not be big or else it can't be done (there is no way to find ByteString to read 1 piece of data):

  @Override
  public BufferedSink write(@NotNull ByteString byteString) throws IOException {
    byte[] bytes = byteString.toByteArray();
    this.write(bytes);
    return this;
  }

In a word, these objects are converted into memory arrays or parameters that Buffer can accept and hold!

On the contrary, writeAll, which is of great concern, is relatively easy to achieve 1 point. We continuously read the content of the specified length until the content length reaches our threshold.

Another painful point is the read/write data flow direction of various objects:

Caller.read(Callee)/Caller.write(Callee), Some are from Caller to Callee, and some are on the contrary. It is a little headache to be integrated by a small category...

Finally, the complete code, if you find any potential problems, you can also communicate ~:

public class FixedSizeSink implements BufferedSink {
  private static final int SEGMENT_SIZE = 4096;
  private final Buffer mBuffer = new Buffer();
  private final int mLimitSize;

  private FixedSizeSink(int size) {
    this.mLimitSize = size;
  }

  @Override
  public Buffer buffer() {
    return mBuffer;
  }

  @Override
  public BufferedSink write(@NotNull ByteString byteString) throws IOException {
    byte[] bytes = byteString.toByteArray();
    this.write(bytes);
    return this;
  }

  @Override
  public BufferedSink write(@NotNull byte[] source) throws IOException {
    this.write(source, 0, source.length);
    return this;
  }

  @Override
  public BufferedSink write(@NotNull byte[] source, int offset,
      int byteCount) throws IOException {
    long available = mLimitSize - mBuffer.size();
    int count = Math.min(byteCount, (int) available);
    android.util.Log.d(TAG, String.format("FixedSizeSink.offset=%d,"
             "count=%d,limit=%d,size=%d",
        offset, byteCount, mLimitSize, mBuffer.size()));
    if (count > 0) {
      mBuffer.write(source, offset, count);
    }
    return this;
  }

  @Override
  public long writeAll(@NotNull Source source) throws IOException {
    this.write(source, mLimitSize);
    return mBuffer.size();
  }

  @Override
  public BufferedSink write(@NotNull Source source, long byteCount) throws IOException {
    final long count = Math.min(byteCount, mLimitSize - mBuffer.size());
    final long BUFFER_SIZE = Math.min(count, SEGMENT_SIZE);
    android.util.Log.d(TAG, String.format("FixedSizeSink.count=%d,limit=%d"
             ",size=%d,segment=%d",
        byteCount, mLimitSize, mBuffer.size(), BUFFER_SIZE));
    long totalBytesRead = 0;
    long readCount;
    while (totalBytesRead < count && (readCount = source.read(mBuffer, BUFFER_SIZE)) != -1) {
      totalBytesRead = readCount;
    }
    return this;
  }

  @Override
  public int write(ByteBuffer src) throws IOException {
    final int available = mLimitSize - (int) mBuffer.size();
    if (available < src.remaining()) {
      byte[] bytes = new byte[available];
      src.get(bytes);
      this.write(bytes);
      return bytes.length;
    } else {
      return mBuffer.write(src);
    }
  }

  @Override
  public void write(@NotNull Buffer source, long byteCount) throws IOException {
    mBuffer.write(source, Math.min(byteCount, mLimitSize - mBuffer.size()));
  }

  @Override
  public BufferedSink writeUtf8(@NotNull String string) throws IOException {
    mBuffer.writeUtf8(string);
    return this;
  }

  @Override
  public BufferedSink writeUtf8(@NotNull String string, int beginIndex, int endIndex)
      throws IOException {
    mBuffer.writeUtf8(string, beginIndex, endIndex);
    return this;
  }

  @Override
  public BufferedSink writeUtf8CodePoint(int codePoint) throws IOException {
    mBuffer.writeUtf8CodePoint(codePoint);
    return this;
  }

  @Override
  public BufferedSink writeString(@NotNull String string,
      @NotNull Charset charset) throws IOException {
    mBuffer.writeString(string, charset);
    return this;
  }

  @Override
  public BufferedSink writeString(@NotNull String string, int beginIndex, int endIndex,
      @NotNull Charset charset) throws IOException {
    mBuffer.writeString(string, beginIndex, endIndex, charset);
    return this;
  }

  @Override
  public BufferedSink writeByte(int b) throws IOException {
    mBuffer.writeByte(b);
    return this;
  }

  @Override
  public BufferedSink writeShort(int s) throws IOException {
    mBuffer.writeShort(s);
    return this;
  }

  @Override
  public BufferedSink writeShortLe(int s) throws IOException {
    mBuffer.writeShortLe(s);
    return this;
  }

  @Override
  public BufferedSink writeInt(int i) throws IOException {
    mBuffer.writeInt(i);
    return this;
  }

  @Override
  public BufferedSink writeIntLe(int i) throws IOException {
    mBuffer.writeIntLe(i);
    return this;
  }

  @Override
  public BufferedSink writeLong(long v) throws IOException {
    mBuffer.writeLong(v);
    return this;
  }

  @Override
  public BufferedSink writeLongLe(long v) throws IOException {
    mBuffer.writeLongLe(v);
    return this;
  }

  @Override
  public BufferedSink writeDecimalLong(long v) throws IOException {
    mBuffer.writeDecimalLong(v);
    return this;
  }

  @Override
  public BufferedSink writeHexadecimalUnsignedLong(long v) throws IOException {
    mBuffer.writeHexadecimalUnsignedLong(v);
    return this;
  }

  @Override
  public void flush() throws IOException {
    mBuffer.flush();
  }

  @Override
  public BufferedSink emit() throws IOException {
    mBuffer.emit();
    return this;
  }

  @Override
  public BufferedSink emitCompleteSegments() throws IOException {
    mBuffer.emitCompleteSegments();
    return this;
  }

  @Override
  public OutputStream outputStream() {
    return mBuffer.outputStream();
  }

  @Override
  public boolean isOpen() {
    return mBuffer.isOpen();
  }

  @Override
  public Timeout timeout() {
    return mBuffer.timeout();
  }

  @Override
  public void close() throws IOException {
    mBuffer.close();
  }
}