Benchmark Summary, details are here
Benchmark Summary
| Instance Type | c6id.large | c6id.xlarge | c6id.2xlarge | c6id.4xlarge | c6id.8xlarge |
| Throughput - Sequential | 6440 MiB/s | 11.3 GiB/s | 19.0 GiB/s | 45.8 GiB/s | 80.4 GiB/s |
| Throughput - Random | 152 MiB/s | 309 MiB/s | 612 MiB/s | 1222 MiB/s | 2487 MiB/s |
| IOPS - Sequential | 182k | 319k | 472k | 1124k | 829k |
| IOPS - Random | 34.2k | 68.3k | 116k | 124k | 201k |
| Network Read | 70.8 MiB/s | 141.8 MiB/s | 283.3 MiB/s | 568.2 MiB/s | 1124 MiB/s |
| Network Write | 986 MiB/s | 1200 MiB/s | 1319 MiB/s | 1429 MiB/s | 1460 MiB/s |
"IOPS - Sequential" reaches to 1.1M on c6id.4xlarge, and drops to 829k on c6id.8xlarge.
This is the bottleneck of the fuse kernel module which maintains a global queue.
c6id.4xlarge has 16 CPUs and c6id.8xlarge has 32 CPUs. When the IOPS reaches to 1M+, linux kernel's process scheduling, lock mechanism for global queue, CPU cache miss, kernel threads' wakeup/sleep becomes too heavy with 32 CPUs racing.
A normal solution is to get ride of the global queue mechanism, and adopt the per-cpu queue. However, this requires a big architecture change in the fuse kernel model.
Above all, the fact that IOPS drops on 32 CPUs is by linux kernel design: 1M+ IOPS together with Global Queue => Performance Drops when CPU number reaches or beyond some threshold.Reference: EC2 c6id serial
| Instance Type | c6id.large | c6id.xlarge | c6id.2xlarge | c6id.4xlarge | c6id.8xlarge |
| Pricing ($/hour) | 0.1155 | 0.231 | 0.4620 | 0.9240 | 1.8480 |
| CPU | 2 | 4 | 8 | 16 | 32 |
| RAM Memory | 4 GB | 8 GB | 16 GB | 32 GB | 64 GB |
| Network Bandwidth | Up to 12.5 Gbps | Up to 12.5 Gbps | Up to 12.5 Gbps | Up to 12.5 Gbps | 12.5 Gbps |
| NVME SSD | 1 x 118 GiB | 1 x 237 GiB | 1 x 474 GiB | 1 x 950 GiB | 1 x 1900 GiB |
"Up to 12.5 Gbps" means the burst network bandwith has the possibility to reaches to 12.5 Gbps (~1500 MiB/s), but this depends on the EC2 instance's credits which is an opaque value calculated internally by AWS.
When credits rans out after an unpredictable time period, the network bandwidth may drop dramatically from the peak.
Compare: Alluxio AI-3.9
| Vendor | Alluxio | Mapfs | Compare |
| Instance Type | i3en.mental | c6id.4xlarge | - |
| Pricing ($/hour) | 12.768 | 0.924 | 1/14 |
| Throughput - Sequential | 9519 MiB/s | 45.8 GiB/s | 4.9 x |
| Throughput - Random | 8276 MiB/s | 1222 MiB/s | 1/7 |
| IOPS - Sequential | 253k | 1124k | 4.4 x |
| IOPS - Random | 162k | 124k | 0.8 x |
| Vendor | Alluxio | Mapfs | Compare |
| Instance Type | i3en.mental | c6id.8xlarge | - |
| Pricing ($/hour) | 12.768 | 1.848 | 1/7 |
| Throughput - Sequential | 9519 MiB/s | 80.4 GiB/s | 8.6 x |
| Throughput - Random | 8276 MiB/s | 2487 MiB/s | 1/3 |
| IOPS - Sequential | 253k | 829k | 3.3 x |
| IOPS - Random | 162k | 201k | 1.2 x |
c6id.8xlarge: Pay 1/7, Get 3 - 9x.
Mapfs "Random Read (1MB) Throughput" runs 1/7 and 1/3 WRT Alluxio respectively on c6id.4xlarge and c6id.8xlarge.
Large IO random read performance is limited by the raw devices. Alluxio runs this benchmark on i3en.metal with 8 NVME SSD in RAID 0, whereas mapfs on c6id.4xlarge with 1 NVME SSD with peak throughput 1250 MiB/s.
Mapfs runs 1222 MiB/s with 1250 MiB/s peak, this means Mapfs layer nearly has zero overhead.