How Many GPUs Does AI Inference Need: 2026 Cloud vs Owned Cost

Memory decides the card count; concurrency decides how many replicas

“How many GPUs does AI inference need?” starts with whether the weights fit. Weights are about parameter count in billions times bytes per parameter: 2 for BF16 or FP16, 1 for FP8, about 0.5 for INT4. An 8B model in BF16 is about 16 GB, a 32B model about 64 GB, a 70B model about 140 GB, and a 405B model in FP8 about 405 GB. The KV cache is extra, so “the weights barely fit” and “several requests can run at once” are different GPU counts.

The H100 SXM is 80 GB of HBM3, the H200 is 141 GB of HBM3e, and the HGX B200 spec page lists 192 GB of HBM3e per GPU. In the table, the left number is the minimum to hold the weights. The right number is a usual count that leaves room for KV and rounds to 1, 2, 4, or 8. This is capacity arithmetic, not a throughput test we ran.

Read each cell as “weights fit / usual count with KV room.” A 70B BF16 model on two H100s puts 140 GB into 160 GB, leaving about 20 GB for KV, so a public service usually takes 4. One H200 is 141 GB, which leaves almost no KV for those BF16 weights, so the usual count is still 2. A ~671B mixture-of-experts model (the DeepSeek-R1 class) occupies memory by total parameters. Active parameters change compute, not whether the weights fit. Eight H100s are only 640 GB, short of 671 GB of FP8 weights.

• An internal 8B tool: one RTX 5090 (32 GB) or one L40S (48 GB) holds BF16. Do not budget a datacenter node of 8 cards for it.
• 32B in BF16: 64 GB inside an 80 GB H100 leaves little room once the context grows, so 2 cards is the usual layout. One H200 or one B200 still has headroom.
• Replicas are separate: the table is one copy of the model. Twice the tokens means another replica of the same size, not a verbal upgrade of the precision.

Cloud GPU hourly rates on 10 Oct 2026

The figures below are on-demand or capacity-block list prices from three public pages that day, in USD per GPU-hour. Community and secure tiers, and a one-GPU slice versus an 8-GPU node, do not collapse into one “market price.” Enterprise discounts and spot are not in the table.

The AWS cell is from the Capacity Blocks pricing page: a p5.48xlarge (8×H100) in US East (N. Virginia) is $47.757 per instance-hour, or $5.970 per accelerator. Lambda’s $3.99 and $6.69 are the per-GPU prices on the 8-GPU plans on the instances page. That page lists the B200 at 180 GB; the NVIDIA spec page says 192 GB. The capacity math keeps 192 GB. RunPod tiers are on its pricing page.

A card left on for 730 hours in a month: H100 SXM at the Community rate is 2.69 × 730 ≈ $1,964, and eight cards are about $15,710. The same eight cards at Lambda’s $3.99 are about $23,300, and at AWS $5.97 about $34,865. Twelve hours a day on weekdays, 22 days, is 264 hours: one H100 on Community is about $710.

• One 70B FP8 replica: the usual H100 layout is 2 cards, so the Community rent is 2 × $2.69 = $5.38 an hour. One H200 is $3.59. One B200 is $5.98. If you are only paying for a fit, the H200 line is smaller.
• PCIe is cheaper, and it is the wrong quote for a two-card 70B: H100 PCIe Community is $1.99. Tensor parallelism cares about the link between cards. Price a multi-GPU replica on the interconnect you will actually get.
• A small model does not need an H100: a 5090 on Community is $0.69 an hour. Eight hours a day, 22 weekdays, is 176 hours, about $121 a month. A 32B model in BF16 is 64 GB and does not fit a 32 GB 5090.

An owned 8-GPU server is three separate lines

NVIDIA does not publish a street price for the box. A 2026 third-party market guide puts an 8×H100 HGX system at $250,000–$320,000 (AI Server Price Guide). The arithmetic below uses the midpoint, $285,000, and labels every assumption. It is not an invoice and not a purchase we measured.

The capital line: 285,000 ÷ 8 = $35,625 per card; 36 × 730 = 26,280 hours; 35,625 ÷ 26,280 ≈ $1.36. Power uses the H100 SXM 700 W TDP. Eight GPUs are 5.6 kW before the rest of the node. A 1.3× factor covers CPU, NICs, and PSU loss, 7.28 kW. Then 7.28 × 730 × 0.12 ≈ $638 a month. The $1,000 rack exists so the fixed cost has a line; it is not a colo quote. Staff, spares, and InfiniBand are outside the $1.64.

• The purchase range: at $250,000, capital is about $1.19 per GPU-hour and the three lines sum to about $1.47. At $320,000, capital is about $1.52 and the sum is about $1.80.
• Residual value: if the box still sells for 30% after 36 months, you depreciate 70% of the midpoint, about $0.95. The three lines fall to about $1.23.
• B200 power is not this row: a B200 board is commonly listed up to about 1,000 W. Rebuild the power and rack lines for that node. Do not copy 700 W.
• Do not hang a two-card service on an eight-card price: if you need two H100s, an eight-card node leaves six idle. Idle cards count against utilization for the whole machine.

Owning wins only against the cloud rate you actually compare

If the server stays powered, cost per useful hour is about the fixed calendar cost divided by utilization. On the midpoint assumptions that is $1.64 ÷ utilization. Beating RunPod Community at $2.69 takes about 61%. Beating Lambda at $3.99 takes about 41%. Beating AWS at $5.97 takes about 27%. At a $250,000 or $320,000 purchase, the bar versus $2.69 is about 55% and 67%. A 30% residual drops the midpoint bar to about 46%.

Utilization is hours the card emits tokens, divided by hours on the calendar. Eight cards with only two of them working puts whole-node utilization near one quarter, and owning loses to renting just those two by the hour. Swap the comparison from AWS to the cheapest community cloud and the case for buying has to be about twice as strong. Staff, the network, and a resale price three years out all move the line again.

• The box is busy less than half the month: rent. An owned card still depreciates while it sits. A cloud hour stops when you stop it.
• The model size is still moving: rent. Power and rack space on an 8-GPU node are locked to the memory generation you bought.
• The same cards emit tokens most of the day, and the data cannot leave the building: replace the $1.64 line with your own purchase price and power price, then compare again. Whether you can lawfully buy or rent the SKU is a separate question. The export note is below.

Pick the card count for the model, then rent or buy

These four lines are bill scenarios, not throughput promises. A third-party benchmark appears only in the third, and it is labeled as their table.

• An internal 8B assistant: one RTX 5090 or one L40S. BF16 is about 16 GB. A Community 5090 for 176 hours a month is about $121. If a machine is already in the office, buying one consumer card is usually the direct path.
• A public 70B FP8 service, weekdays only: usually 2×H100. At Community rates and 264 hours, two cards are about $1,420 a month. That is not yet a reason to buy an 8-GPU server. One H200 replica moves the hourly bill from $5.38 to $3.59.
• A 70B model with longer context or more concurrency: the usual layout is 4×H100. In the E2E Networks benchmark docs, Llama 3.1 70B on 4×H100 for a short chat (about 128 tokens each way) is listed at 3,438 tokens/s. That is their table. A longer prompt makes the replica more expensive.
• 405B FP8, or about 671B: 405 GB fits in the 640 GB of 8×H100. 671 GB of FP8 does not. That case wants 8×H200 (8×141 GB ≈ 1,128 GB) or 16×H100. Clear memory before you compare hourly rates.

Whether you can buy the card and what the hour costs are two lines. Under the export rules, a China-headquartered entity should not treat “an H100 in a third country” as the default purchase path. See advanced compute after the US NVIDIA export limits. Git, signing, and a resident agent should not be billed as H100 hours either. Memory for that job is in the Mac mini AI agent server guide.

Questions

How many GPUs for 8B, 32B, and 70B inference?

From the memory arithmetic: 8B in BF16 is about 16 GB, so one 32 GB card. 32B in BF16 is about 64 GB, usually 2 H100s, or one H200 or B200. 70B in FP8 is about 70 GB, usually 2 H100s. 70B in BF16 is about 140 GB, usually 4 H100s. That is capacity, not a speed test.

What does renting an H100 cost in 2026?

List prices on 10 Oct 2026: RunPod H100 SXM at $2.69 Community and $3.29 Secure, a Lambda 8-GPU node at $3.99, and an AWS Capacity Block p5 accelerator at $5.97. Left on for a month, one Community card is about $1,964.

How does an owned 8-GPU server become an hourly cost?

A third-party guide puts 8×H100 at $250,000–$320,000. The midpoint of $285,000 over 36 months with no residual is about $1.36 per GPU-hour of capital. Add estimated power of $0.11 and an estimated rack of $0.17, and 100% uptime is about $1.64. Staff and a private network are extra.

How much utilization makes buying cheaper?

It depends on the cloud rate you compare. On the midpoint assumptions, beating $2.69 takes about 61% utilization, beating $3.99 about 41%, and beating $5.97 about 27%. If you buy eight cards and use two, whole-node utilization pushes ownership back to a loss.

One 70B service: should you buy a full 8-GPU node?

Usually no. One FP8 replica usually takes 2 H100s, at a Community rent of $5.38 an hour, or one H200 at $3.59. The 8-GPU purchase beats a cheap cloud hour only when most of those eight cards are emitting tokens most of the time.