# vLLM

## vLLM Benchmarks

vLLM benchmarks were run using **Trinity Nano**. Trinity Mini bf16 (\~52.3 GB) exceeds the VRAM capacity of the single-GPU consumer hardware used in these tests (3090/4090/5090), so Mini results are not included here.

#### Test configuration

* Input tokens: 512
* Output tokens: 256
* Prompts: 512
* Concurrency: 8
* Request rate: 8 rps

***

### RTX 3090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 2.87  | 735.67       | 47.32 ms  | 58.05 ms | 10.64 ms   | 23556 MiB |
| W4A16     | 3.97  | 1016.35      | 40.69 ms  | 51.35 ms | 7.66 ms    | 23594 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 1445.84      |
| W4A16     | 1710.57      |

***

### RTX 4090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 3.63  | 928.72       | 38.92 ms  | 42.00 ms | 8.41 ms    | 23910 MiB |
| W4A16     | 5.19  | 1328.33      | 33.48 ms  | 37.29 ms | 5.84 ms    | 23972 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 1991.78      |
| W4A16     | 2802.97      |

***

### RTX 5090

#### Performance

| Precision | Req/s | Output tok/s | Mean TTFT | p99 TTFT | TPOT / ITL | VRAM Used |
| --------- | ----- | ------------ | --------- | -------- | ---------- | --------- |
| bf16      | 4.10  | 1048.97      | 44.64 ms  | 54.29 ms | 7.40 ms    | 30487 MiB |
| W4A16     | 4.95  | 1267.71      | 43.54 ms  | 48.61 ms | 6.09 ms    | 30601 MiB |

#### Max throughput

| Precision | Output tok/s |
| --------- | ------------ |
| bf16      | 2312.41      |
| W4A16     | 2559.77      |