About the AIX mount NFS write inefficiency solution
- 2020-12-16 06:15:51
- OfStack
Services provided by NFS
Mount: By enabling /usr/sbin/ rpc.mountd server process on the server side, using mount command on the client side, mounted server process is 1 RPC to respond to client requests
Remote File access: Enables /usr/sbin/nfsd on the server and /usr/sbin/biod on the client to make client requests for files. But when a client user wants to read or write a file on the server side, the biod server process sends the request to the server side.
Boot parameters: through the opening in the server/usr sbin/rpc bootparamd servo process to provide a diskless SunOS boot parameters for the client.
PC authentication: through on the server startup/usr/sbin/rpc pcnfsd to provide PC - NFS user authentication services
One NFS service is stateless (stateless), that is, NFS transfers are atomic, and one NFS transfer corresponds to one singleton for a complete file operation.
Background:
Linux is the Server end of NFS and AIX is the Client end of NFS (in addition, one Linux is also used as the Client end contrast test).
1. The underlying device corresponding to NFS is a flash memory card, and the writing performance of I/O can reach 2GB/s under local test;
2. Server is gigabit network card, FTP test transmission can reach 100MB/s;
3.AIX successfully mounted NFS, and dd tested write speed of only 10MB/s;
4.Linux successfully mounted NFS, similarly, dd test write speed can reach 100MB/s;
Note: The above speed is mainly to reflect the difference of order of magnitude, the actual test will have a little deviation.
Specific environment:
Mount parameters are configured according to the MOS document:
[
Mount Options files for databases Clusterware when NFS NAS devices (document ID 359515.1)
According to the actual requirements of this time, 1 parameter to be configured is extracted:
[
- MOS suggested (AIX) :
cio,rw,bg,hard,nointr,rsize=32768,
wsize=32768,proto=tcp,noac,
vers=3,timeo=600
- MOS suggested (Linux) :
rw,bg,hard,nointr,rsize=32768,
wsize=32768,tcp,actimeo=0,
vers=3,timeo=600
Mount parameters of AIX NFS:
[
mount -o cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600 10.xx.xx.212:/xtts /xtts
Direct mount prompts the following error:
# mount -o cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600 10.xx.xx.212:/xtts /xtts
mount: 1831-008 giving up on:
10.xx.xx.212:/xtts
vmount: Operation not permitted.Additional network parameters need to be set for AIX by checking data:
# nfso -p -o nfs_use_reserved_ports=1Another attempt to mount succeeded:
mount -o cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600 10.xx.xx.212:/xtts /xttsThe test speed of dd is not ideal, only 10MB/s:
--test performance; AIX NFS
# time dd if=/dev/zero of=/xtts/test-write bs=8192 count=102400
102400+0 records in.
102400+0 records out.
real 0m43.20s
user 0m0.79s
sys 0m5.28s
# time dd if=/xtts/test-write of=/dev/null bs=8192 count=102400
102400+0 records in.
102400+0 records out.
real 0m30.86s
user 0m0.84s
sys 0m5.88sAll parameters are set according to actual requirements, as suggested by MOS. Is there a problem?
The cio parameter test was removed, and the results were almost unchanged. The hard parameter test was removed, and the results were almost unchanged. Try changing the protocol from tcp to udp test and find little change.Almost all parameters can be tried, the results are not ideal, immediately ready to coordinate resources to find the host engineer positioning.
At this point, inspiration struck and a possibility occurred to me. Is it possible that NFS on AIX limited the I/O throughput capacity of a single process? Take this guess and test it in parallel:
Open 5 Windows to start dd:
time dd if=/dev/zero of=/xtts/test-write1 bs=8192 count=102400
time dd if=/dev/zero of=/xtts/test-write2 bs=8192 count=102400
time dd if=/dev/zero of=/xtts/test-write3 bs=8192 count=102400
time dd if=/dev/zero of=/xtts/test-write4 bs=8192 count=102400
time dd if=/dev/zero of=/xtts/test-write5 bs=8192 count=102400
Surprisingly, it was found that all 5 Windows were completed at 55s at the same time, which is equivalent to 800M*5=4000M. All of them were completed at 55s, reaching 72MB/s per second. This parallel approach has met the requirement of efficiency improvement.
And it seems that as long as you keep trying to open more Windows, you can basically reach the network limit of 100MB/s (gigabit nic limit).
Attachment: Test the same NFS mounted to another Linux server, dd write speed can reach 100MB/s without parallelization, which is also the factor affecting my thinking before.
Linux NFS Mount parameters:
# mount -o rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 10.xx.xx.212:/xtts /xttsLinux NFS Test results:
--test performance; Linux NFS
# dd if=/dev/zero of=/xtts/test-write bs=8192 count=102400
102400+0 records in
102400+0 records out
838860800 bytes (839 MB) copied, 6.02451 s, 139 MB/s
# dd if=/xtts/test-write of=/dev/null bs=8192 count=102400
102400+0 records in
102400+0 records out
838860800 bytes (839 MB) copied, 8.55925 s, 98.0 MB/s
I was not familiar with AIX and did not delve deeper into the underlying principle. The main puzzle starting the problem is how Linux, as client, can test dd to the speed of 100MB/s without parallelism, putting yourself in a fixed mindset. The takeaway from this is that sometimes you have to think outside the box to make a breakthrough.
Finally, the NFS Server terminal local test results are posted, lamenting the I/O ability of flash memory card:
# dd if=/dev/zero of=/dev/test-write2 bs=8192 count=1024000
1024000+0 records in
1024000+0 records out
8388608000 bytes (8.4 GB) copied, 4.19912 s, 2.0 GB/sconclusion