I have 4 identical hosts with 4x nvme Optane 900P 280GB in U.2 form factor, next model:
Model Number: INTEL SSDPE21D280GA
Serial Number: PHM2746000??280AGN
Firmware Version: E2010325
When i test it i see next - someone Optane is very slow
Before run tests i drop caches:
# echo 3 > /proc/sys/vm/drop_caches
All Optane have
/sys/block/nvme?n1/queue/io_poll = 1
Nothing io per nvme in parallel. Only this test:
host-1 ~# for d in {0..3}; do dd if=/dev/nvme${d}n1 of=/dev/null bs=4k count=256000; done
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.385942 s, 2.7 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.390112 s, 2.7 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.38746 s, 2.7 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.387112 s, 2.7 GB/s
host-1 ~# for d in {0..3}; do hdparm -Tt --direct /dev/nvme${d}n1; done
/dev/nvme0n1:
Timing O_DIRECT cached reads: 4776 MB in 2.00 seconds = 2388.57 MB/sec
Timing O_DIRECT disk reads: 7122 MB in 3.00 seconds = 2373.22 MB/sec
/dev/nvme1n1:
Timing O_DIRECT cached reads: 4880 MB in 2.00 seconds = 2440.49 MB/sec
Timing O_DIRECT disk reads: 7300 MB in 3.00 seconds = 2433.20 MB/sec
/dev/nvme2n1:
Timing O_DIRECT cached reads: 4826 MB in 2.00 seconds = 2413.76 MB/sec
Timing O_DIRECT disk reads: 7010 MB in 3.00 seconds = 2336.50 MB/sec
/dev/nvme3n1:
Timing O_DIRECT cached reads: 4834 MB in 2.00 seconds = 2417.19 MB/sec
Timing O_DIRECT disk reads: 7286 MB in 3.00 seconds = 2428.46 MB/sec
host-2 ~# for d in {0..3}; do dd if=/dev/nvme${d}n1 of=/dev/null bs=4k count=256000; done
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.386011 s, 2.7 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.700671 s, 1.5 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 135.126 s, 7.8 MB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.3885 s, 2.7 GB/s
host-2 ~# for d in {0..3}; do hdparm -Tt --direct /dev/nvme${d}n1; done
/dev/nvme0n1:
Timing O_DIRECT cached reads: 4870 MB in 2.00 seconds = 2435.09 MB/sec
Timing O_DIRECT disk reads: 7276 MB in 3.00 seconds = 2425.19 MB/sec
/dev/nvme1n1:
Timing O_DIRECT cached reads: 2758 MB in 2.00 seconds = 1379.17 MB/sec
Timing O_DIRECT disk reads: 2726 MB in 3.00 seconds = 908.07 MB/sec
/dev/nvme2n1:
Timing O_DIRECT cached reads: 614 MB in 2.12 seconds = 290.25 MB/sec
Timing O_DIRECT disk reads: 64 MB in 3.13 seconds = 20.42 MB/sec
/dev/nvme3n1:
Timing O_DIRECT cached reads: 4716 MB in 2.00 seconds = 2358.23 MB/sec
Timing O_DIRECT disk reads: 6068 MB in 3.00 seconds = 2022.55 MB/sec
host-3 ~# for d in {0..3}; do dd if=/dev/nvme${d}n1 of=/dev/null bs=4k count=256000; done
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.89861 s, 152 MB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.485921 s, 2.2 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 107.023 s, 9.8 MB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 151.081 s, 6.9 MB/s
host-3 ~# for d in {0..3}; do hdparm -Tt --direct /dev/nvme${d}n1; done
/dev/nvme0n1:
Timing O_DIRECT cached reads: 52 MB in 2.01 seconds = 25.84 MB/sec
Timing O_DIRECT disk reads: 326 MB in 3.01 seconds = 108.41 MB/sec
/dev/nvme1n1:
Timing O_DIRECT cached reads: 402 MB in 2.04 seconds = 197.38 MB/sec
Timing O_DIRECT disk reads: 214 MB in 3.00 seconds = 71.32 MB/sec
/dev/nvme2n1:
Timing O_DIRECT cached reads: 4876 MB in 2.00 seconds = 2438.49 MB/sec
Timing O_DIRECT disk reads: 7288 MB in 3.00 seconds = 2429.20 MB/sec
/dev/nvme3n1:
Timing O_DIRECT cached reads: 4696 MB in 2.00 seconds = 2348.00 MB/sec
Timing O_DIRECT disk reads: 5096 MB in 3.00 seconds = 1697.90 MB/sec
host-4 ~# for d in {0..3}; do dd if=/dev/nvme${d}n1 of=/dev/null bs=4k count=256000; done
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 134.697 s, 7.8 MB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.555736 s, 1.9 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.385807 s, 2.7 GB/s
256000+0 records in
256000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 14.1933 s, 73.9 MB/s
host-4 ~# for d in {0..3}; do hdparm -Tt --direct /dev/nvme${d}n1; done
/dev/nvme0n1:
Timing O_DIRECT cached reads: 18 MB in 2.17 seconds = 8.31 MB/sec
Timing O_DIRECT disk reads: 26 MB in 3.11 seconds = 8.37 MB/sec
/dev/nvme1n1:
Timing O_DIRECT cached reads: 4696 MB in 2.00 seconds = 2348.17 MB/sec
Timing O_DIRECT disk reads: 5688 MB in 3.00 seconds = 1895.32 MB/sec
/dev/nvme2n1:
Timing O_DIRECT cached reads: 662 MB in 2.01 seconds = 329.60 MB/sec
Timing O_DIRECT disk reads: 980 MB in 3.00 seconds = 326.64 MB/sec
/dev/nvme3n1:
Timing O_DIRECT cached reads: 384 MB in 2.10 seconds = 183.11 MB/sec
Timing O_DIRECT disk reads: 114 MB in 3.04 seconds = 37.47 MB/sec
Why Optane is slow down?
How to fix this?