Common benchmark bottlenecks

SSD Throughput and IOPS limitations

For example,

If you are testing the performance with a GP3 volume with default GP3 settings, which is 125 MiB/s throughput and 3000 IOPS,

you will see the Random Read performance with 1MB block size is limited to 125 MiB/s, and the Random Read performance with 4KB block size is limited to 3000 IOPS.

EC2 EBS Throughput and IOPS limitations

Just imagine you are testing the performance with a c6i.large EC2 and one GP3 SSD.

The baseline EBS performance of c6i.large is 3600 IOPS and 81.25 MiB/s throughput, and the GP3 SSD performance is 3000 IOPS and 125 MiB/s throughput.

Even if you raise the GP3 performance to 5000 IOPS and 500 MiB/s throughput, you will still be limited by the EBS performance after burst credits are exhausted (generally after 10 - 30 minutes), which is 3600 IOPS and 81.25 MiB/s throughput.

Network bandwidth

Even if you are testing the performance with a single fio job (numjobs=1), when the network bandwidth is saturated, you will see the Network Read/Write performance will not increase with more fio jobs added.

CPU Burstable, EBS Burstable, Network Burstable, NVME SSD Burstable, VPC Credits, etc.

For example,

if you use burstable EC2 instances (such as t3.small), CPU performance may fluctuate between baseline and burst limits.