Several methods for memory allocation in Java

The upper limit of array allocation

The size of an array in Java is limited because it USES an int as an array index. This means that you cannot request an array larger than integer.max_value (2^31-1). This does not mean that the maximum amount of memory you request is 2 gigabytes. You can apply for a larger type of array. Such as:

final long[] ar = new long[ Integer.MAX_VALUE ];

This will allocate 16 gigabytes to 8 bytes, if you set the -xmx parameter to be large enough (usually your heap must hold at least 50% more space, which means that to allocate 16 gigabytes of memory, you have to set it to -xmx24 gigabytes. This is just a general rule, the specific allocation depends on the actual situation).

Unfortunately, in Java, memory can be tricky to manipulate because of the type of array element. ByteBuffer is probably the most useful class for manipulating arrays, providing methods for reading and writing different Java types. The disadvantage is that the target array type must be byte[], which means that you can only allocate up to 2 gigabytes of memory cache.

Operate on all arrays as if they were byte arrays

Let's say 2 gigabytes of memory isn't enough for us right now. 16 gigabytes is fine. We have assigned a long[], but we want to operate on it as a byte array. In Java we need to call on the C programmer's best friend sun. Misc.Unsafe. There are two sets of methods for this class: getN(object, offset), which takes a value of the specified type from the position of object offset and returns it, where N represents the type of value to be returned, and putN(object, offset,value), which writes a value to the position of object offset.

Unfortunately, these methods can only get or set a value of a certain type. If you copy the data from an array, you need another method of unsafe, copyMemory (srcObject srcOffset, destObject, destOffet, count). This works like system.arraycopy, except that it copies bytes instead of array elements.

To access the array's data via sun-misc. Unsafe, you need two things:

1. The offset of the data in the array object
2. Offset of copied elements in array data
Arrays, like other Java objects, have an object header that is stored in front of the actual data. The length of this header can be obtained by the undone.arraybaseoffset (T[].class) method, where T is the type of array element. The size of the array element can be obtained using the unconcer.arrayindexscale (T[].class) method. This means that to access the NTH element of type T, your offset should be arrayOffset+N*arrayScale.

Let's write a simple example. We allocate a long array and then update a few bytes in it. We update the last element to -1 (0xFFFF FFFF FFFF FFFF in hexadecimal) and then remove all bytes of the element one by one.

final long[] ar = new long[ 1000 ];
final int index = ar.length - 1;
ar[ index ] = -1; //FFFF FFFF FFFF FFFF
System.out.println( "Before change = " + Long.toHexString( ar[ index ] ));
for ( long i = 0; i < 8; ++i )
{
    unsafe.putByte( ar, longArrayOffset + 8L * index + i, (byte) 0);
    System.out.println( "After change: i = " + i + ", val = "  +  Long.toHexString( ar[ index ] ));
}

To run the above example, add the top and bottom static blocks of code to your test class:

private static final Unsafe unsafe;
static
{
    try
    {
        Field field = Unsafe.class.getDeclaredField("theUnsafe");
        field.setAccessible(true);
        unsafe = (Unsafe)field.get(null);
    }
    catch (Exception e)
    {
        throw new RuntimeException(e);
    }
}

Private static final long longArrayOffset = offset. ArrayBaseOffset (long[].class);
The output result is:

Before change = ffffffffffffffff
After change: i = 0, val = ffffffffffffff00
After change: i = 1, val = ffffffffffff0000
After change: i = 2, val = ffffffffff000000
After change: i = 3, val = ffffffff00000000
After change: i = 4, val = ffffff0000000000
After change: i = 5, val = ffff000000000000
After change: i = 6, val = ff00000000000000
After change: i = 7, val = 0

Unsafe memory allocation for sun. Misc

As mentioned above, in pure Java our memory allocation is limited. This limitation was set in the early versions of Java, when people couldn't imagine allocating multiple gigabytes of memory. But this is the age of big data, and we need more memory. In Java, there are two ways to get more memory:

1. Allocate many small chunks of memory and use them logically as if they were a large contiguous chunk of memory.
2. Using sun. Misc. Unsafe. AllcateMemory memory allocation (long).
The first method is just a little bit more interesting from an algorithmic point of view, so let's look at the second method.

Sun. Misc. Unsafe provides a set of methods to allocate, reallocate, and free memory. They are similar to the malloc/free method of C:

AllocateMemory (long size) -- allocates a block of memory space. This memory may contain garbage data (there is no automatic zeroing). If allocation failure will throw a Java lang. OutOfMemoryError exception. It returns a non-zero memory address (see the description below).
2. The Unsafe. ReallocateMemory (long address, long size) - to allocate a block of memory, the data from the old memory buffer (address) copied to the newly allocated memory block. If the address is equal to 0, this method is the same as allocateMemory. It returns the address of the new memory buffer.
Unsafe. FreeMemory (long address) -- frees a memory buffer generated by the previous two methods. If address is 0, nothing happens.

The memory allocated by these methods should be used in a mode called single-register address: offset provides a set of methods that take only one address parameter (unlike the double-register mode, which requires an Object and an offset). The amount of memory allocated in this way can be larger than what you configured in the -xmx Java parameters.

Note: unallocated memory cannot be garbage collected. You have to manage it as a normal resource.

Here's an example of allocating memory using unevent.allocatememory, while also checking that the entire memory buffer is read-write:

final int size = Integer.MAX_VALUE / 2;
final long addr = unsafe.allocateMemory( size );
try
{
    System.out.println( "Unsafe address = " + addr );
    for ( int i = 0; i < size; ++i )
    {
        unsafe.putByte( addr + i, (byte) 123);
        if ( unsafe.getByte( addr + i ) != 123 )
            System.out.println( "Failed at offset = " + i );
    }
}
finally
{
    unsafe.freeMemory( addr );
}

As you can see, with sun.misc.unsafe you can write very general-purpose memory-access code: you can read and write any type of data you want, no matter what memory is allocated in Java.