For an existing infrastructure, Hopper architecture may be suitable, particularly the H100 if 640 GB of GPU memory is sufficient. If not, the H200 handles large models, long contexts, and a large KV cache better. DGX B200 is the most balanced move to Blackwell for training, fine-tuning, and production inference. DGX B300 is needed where 288 GB on each GPU, reasoning models, and high concurrent workloads are critical. However, the advantages of the top-end systems are only realised when the network, power supply, cooling, storage, and, of course, the operations team are ready.
NVIDIA DGX servers
Four DGX servers and three stages of evolution
DGX H100 and H200 belong to the Hopper generation. Both servers contain eight accelerators connected by NVLink and NVSwitch, but the H200 has significantly more HBM3e memory. Therefore, moving from H100 to H200 primarily solves model capacity and memory bandwidth constraints rather than changing the entire architecture.
DGX B200 marks the transition to Blackwell: more memory, fifth-generation NVLink/NVSwitch, and hardware support for FP4. DGX B300 uses Blackwell Ultra. Its key difference from B200 is 288 GB of memory per GPU instead of 180 GB, along with higher dense FP4 performance and networking of up to 800 Gbit/s per adapter.
Do not confuse DGX B200 and B300 with NVL72
DGX B200 and B300 are standalone eight-accelerator 10U servers. DGX GB200 NVL72 and GB300 NVL72 are rack-scale systems with dozens of accelerators and a different interconnect architecture. Their benchmark results, power consumption, and network requirements cannot be applied automatically to a single DGX.
Key specifications
| Parameter | DGX H100 | DGX H200 | DGX B200 | DGX B300 |
|---|---|---|---|---|
| Architecture | Hopper | Hopper | Blackwell | Blackwell Ultra |
| GPU | 8 × H100 SXM | 8 × H200 SXM | 8 × B200 SXM | 8 × B300 SXM |
| Memory per GPU | 80 GB | 141 GB | 180 GB | 288 GB |
| Total GPU memory | 640 GB | 1,128 GB | 1,440 GB | 2,304 GB |
| FP8 | about 32 PFLOPS with sparsity | about 32 PFLOPS with sparsity | 72 PFLOPS with sparsity | 72 PFLOPS with sparsity |
| FP4 | no native Blackwell FP4 | no native Blackwell FP4 | 144/72 PFLOPS, sparse/dense mode | 144/108 PFLOPS, sparse/dense mode |
| GPU interconnect | NVLink/NVSwitch 4 | NVLink/NVSwitch 4 | NVLink/NVSwitch 5 | NVLink/NVSwitch 5 |
| Cluster network | up to 8 × 400 Gbit/s | up to 8 × 400 Gbit/s | up to 8 × 400 Gbit/s | up to 8 × 800 Gbit/s |
| System memory | 2 TB | 2 TB | 2 TB, up to 4 TB | 2 TB, up to 4 TB |
| Height | 8U | 8U | 10U | 10U |
| Maximum power | 10.2 kW | 10.2 kW | 14.3 kW | up to 15 kW |
Petaflops cannot be used as a simple ranking. FP8 and FP4 are different formats, while sparse figures are achievable only with a suitable data structure and software support. For example, B200 and B300 both deliver 144 PFLOPS of sparse FP4 performance, but their dense figures are 72 and 108 PFLOPS respectively.
B300 capacity is sometimes stated as 2.1 or 2.3 TB. The exact physical capacity is eight 288 GB modules, or 2,304 GB; the difference is caused by rounding and the method used to convert units.
What changed from Hopper to Blackwell Ultra
Memory per GPU matters more than the total
The memory of eight accelerators does not become a completely transparent unified pool. The model is distributed across GPUs, and data is transferred through NVLink and NVSwitch. Therefore, 288 GB on each B300 GPU helps to:
- place large layers and experts without excessive partitioning;
- reduce tensor parallelism and the number of data exchanges;
- leave more space for the KV cache and buffers;
- run multiple model instances or adapters;
- reduce memory fragmentation.
In these scenarios, the capacity of a single accelerator can matter more than the peak compute performance of the entire system.
FP4 does not accelerate every model
BF16, FP16, and FP8 remain important for training. FP4 in Blackwell is aimed primarily at high-performance inference and selected training operations. Using it requires correctly quantised weights, a compatible engine, and validation of output quality.
Lower precision can degrade generation quality or the operation of individual layers. Therefore, H200 and B300 cannot be compared on FP4 performance alone.
The B300 network requires new infrastructure
B300 uses eight ConnectX-8 adapters at up to 800 Gbit/s. This matters for distributed training and multi-node model serving, but it provides no benefit in an older 400 Gbit/s fabric. The transition may require new switches, optics, cables, RDMA configuration, and faster storage.
How much memory an LLM really needs
For an initial estimate of weight memory, use the following approximation:
- BF16 or FP16 — about 2 bytes per parameter;
- FP8 or INT8 — about 1 byte per parameter;
- 4-bit storage — about 0.5 bytes per parameter.
A 70-billion-parameter model in BF16 uses approximately 140 GB for weights alone, while a 400-billion-parameter model uses about 800 GB. Actual consumption is higher because of metadata, alignment, and library workspaces.
The model file size is not the full calculation
During inference, memory is also used by:
- the KV cache for tokens already processed;
- the input and generated context;
- concurrent sequences;
- temporary tensors and CUDA buffers;
- batch processing;
- LoRA adapters;
- orchestration system headroom.
The KV cache grows quickly with long contexts and many users. A model may therefore run in a single test but fail to sustain a production workload.
Fine-tuning requires more memory
Full fine-tuning requires the weights, gradients, activations, and optimiser states to be stored. The total memory footprint is many times larger than the weights alone. LoRA and QLoRA reduce the requirements because they train small additional matrices while the base model can be stored at reduced precision.
The phrase “the model fits on H200” does not mean that full fine-tuning is possible on it. Sequence length, batch size, optimisation method, and the distribution scheme across GPUs must also be considered.
DGX for training large models
H100 and H200
Hopper is suitable when the organisation already uses compatible CUDA versions, containers, and libraries. Expanding an existing cluster is usually easier than changing the accelerators, network, and software environment at the same time.
H200 is preferable to H100 when:
- the model does not fit in 80 GB on one GPU;
- the batch size has to be reduced;
- long sequences consume too much memory;
- data is frequently offloaded to system memory;
- more headroom is needed for activations.
If the workload comfortably fits within 640 GB and scales well, replacing H100 solely to move to a newer generation may not pay off.
B200 and B300
B200 suits a new general-purpose cluster that combines training, fine-tuning, and inference. It is a sensible choice when the organisation is ready to update the software stack but does not yet need 288 GB on each GPU.
B300 is justified for long-context models, large mixtures of experts, and full fine-tuning. It can be especially useful when the additional memory allows the model to run on fewer nodes or reduces the degree of parallelism.
In multi-node training, the bottleneck may be the network, dataset reads, collective operations, or checkpoint writes. If GPUs sit idle waiting for data, moving from B200 to B300 will not deliver proportional acceleration.
LoRA, QLoRA, and full fine-tuning
H100 or H200 is often sufficient for LoRA and QLoRA. H200 is more convenient for a large model, a long context, multiple adapters, and a larger batch size.
B200 is better suited to a multi-project environment: its additional memory allows more experiments to run in parallel and reduces offloading to system memory. B300 is needed when the workload requires:
- full fine-tuning of large LLMs;
- several demanding experiments at the same time;
- the maximum sequence length;
- large batch sizes;
- fewer nodes per workload.
At low utilisation, the B300 advantage does not pay off. Several more affordable servers can sometimes be easier to divide between teams and make redundancy and fault tolerance simpler to maintain.
Which DGX to choose for inference
Model inference has two stages. First, the server processes the entire input request; then the response tokens are generated sequentially. The first stage depends more heavily on compute performance and input length, while the second depends on memory bandwidth and KV-cache capacity.
A single tokens-per-second figure is therefore insufficient. A production service must consider:
- time to first token;
- inter-token latency;
- the number of concurrent requests;
- input and output length;
- stability under peak loads.
H100 vs H200
H100 is suitable for small and medium-sized models, moderate contexts, and an existing infrastructure. H200 is better when the KV cache grows, the number of concurrent sequences increases, or the model has to be partitioned too heavily across GPUs.
Because both systems belong to the Hopper generation, moving to H200 can increase capacity without a complete change of the software environment.
B200 vs B300
B200 is a balanced Blackwell platform for high-volume inference and mixed workloads. B300 becomes valuable when a large model, a long context, and many users are present at the same time. This is typical of reasoning models and AI agents with long action chains.
The additional memory can hold a larger KV cache and reduce model partitioning. However, the result depends on the inference engine, quantisation, and libraries: an old container may not use the advantages of Blackwell Ultra.
RAG and AI agents
RAG includes embedding creation, database retrieval, reranking, context assembly, and generation. Tool calls and repeated reasoning loops may also be added.
The GPU accelerates generation and part of the reranking process, but latency may be determined by the CPU, RAM, NVMe, network, or database. Slow retrieval will not become faster after replacing H200 with B300.
H100/H200 are suitable for mature RAG systems with controlled workloads. B200 is a better fit for a new production platform where generation and fine-tuning run together. B300 is useful for long contexts, many agents, and a large number of concurrent chains.
DGX CPUs participate in data preparation and input/output. These operations are similar to workloads handled by Intel Xeon servers, so the CPU should not be treated as a secondary component of a RAG platform.
Multi-user AI laboratory
In a laboratory, useful utilisation matters more than the record performance of a single model. Resources are allocated through containers, Slurm, or Kubernetes; supported MIG modes allow GPUs to be divided into isolated compute instances.
Memory limits, job priorities, experiment duration, competition for NVMe and network resources, and actual GPU-hours used must all be considered.
B300 is convenient when several teams regularly run large models. With irregular demand, several H100/H200 systems or even less dense conventional GPU servers may be more practical: projects interfere with one another less, and the failure of one node does not stop the entire laboratory.
One DGX or a cluster
A single server is easier to deploy and maintain. It suits one large model or training workload that fits in the memory of the node. Internal NVLink provides high-speed communication without an external network.
Limitations of a single DGX:
- maintenance requires stopping or moving the workload;
- a failure affects all hosted models;
- peak capacity is limited to one system;
- processing stages cannot be scaled independently.
A cluster makes it possible to create replicas, distribute training, and separate input processing from generation. This requires load balancing, shared storage, and either InfiniBand or Ethernet with RDMA.
Doubling the number of servers rarely doubles performance: some time is spent on synchronisation and tensor exchange. Efficiency depends on the model, parallelism strategy, and network topology.
Power, cooling, and deployment
DGX H100/H200 occupy 8U and consume up to 10.2 kW. B200/B300 occupy 10U; their maximum power is 14.3 and 15 kW respectively. Rack planning must account for:
- available power and reserve capacity;
- distribution across PDUs and phases;
- the number of power connections;
- thermal density and airflow;
- inlet air temperature;
- equipment weight.
Four B300 systems physically occupy 40U, but their maximum load approaches 60 kW before the network and storage are included. Free rack space alone is insufficient if the facility cannot remove that amount of heat.
B300 is available with a PDU connection or a direct-current busway connection. The power-supply version uses twelve modules with N+N redundancy. The system weighs approximately 168 kg with power supplies, so suitable rails and lifting equipment are required for installation.
Where performance is lost: network and storage
The external network becomes critical when training or a model is distributed across several DGX systems. B300 provides up to eight 800 Gbit/s connections, but compatible switches, cables, and configuration are required. In a 400 Gbit/s fabric, some ConnectX-8 capabilities remain unused.
Storage must serve datasets, model versions, checkpoints, logs, knowledge bases, and backups. Local DGX NVMe drives are useful as a fast cache, but they do not replace shared fault-tolerant storage with sufficient performance both within the storage system and across the access network.
If data is read slowly or checkpointing blocks training for extended periods, the GPUs sit idle. Network and storage must therefore be sized together with the compute nodes, not after they are purchased.
Software stack compatibility
Hopper is attractive because of its mature drivers, ready-made containers, and libraries. This reduces migration risk, especially when custom CUDA extensions are used.
For Blackwell and Blackwell Ultra, check:
- CUDA and driver versions;
- container images;
- distributed computing libraries;
- FP4 and FP8 support;
- custom compute kernels;
- the inference engine and orchestration.
Compatibility does not mean equal efficiency. An application may run on B300 without using FP4, attention-engine improvements, or an optimal communication scheme. The complete workflow should be compared: model loading, latency, generation speed, memory consumption, and scaling.
Total cost of ownership
In addition to the server, the project includes networking, optics, PDUs, power upgrades, cooling, racks, storage, support, electricity, and the work of the operations team.
Platforms should be compared not only by purchase price but also by operating costs. Useful cost metrics include:
- an hour of actual GPU utilisation;
- a training cycle;
- one million tokens at a specified latency;
- serving the required number of users;
- team training;
- downtime and recovery after a failure.
B300 may be more economical if it replaces several nodes or reduces inter-server communication. However, at 20–30% utilisation, the additional capabilities may not cover the cost of the newer platform. This is not a universal threshold, but it is a reason to measure the workload profile in advance.
When another GPU server is the better choice
DGX is an integrated platform with a defined GPU topology, network, software stack, and NVIDIA support. Another server may offer better value when maximum performance per node is unnecessary but additional drives, a different GPU mix, specific CPUs, non-standard networking, or integration into an existing fleet are required.
For such projects, consider Dell servers for AI. Configurations are available with eight NVIDIA B300, eight B200, or eight H200 GPUs. Compare GPU topology, storage, networking, certification, and support rather than the accelerator model alone.
Choosing DGX by workload
| Scenario | Preferred option | Why |
|---|---|---|
| Expanding a Hopper cluster | H100/H200 | Fewer changes to the software and network environment |
| H100 does not provide enough memory | H200 | 141 GB per GPU and 1,128 GB per server |
| LoRA and QLoRA | H200 or B200 | Sufficient headroom without requiring B300 |
| Full fine-tuning of a large LLM | B200 or B300 | More memory for gradients and optimiser states |
| New general-purpose cluster | B200 | A balanced transition to Blackwell |
| Long context and a large KV cache | H200, B200, or B300 | The choice depends on the model and number of users |
| Reasoning models and AI agents | B300 | 288 GB per GPU and more space for sequences |
| High-load inference | B200 or B300 | FP4 with an optimised software stack |
| Multi-user laboratory | H200 or B200 | Balance of memory, maturity, and cost |
| Limited rack power | H100 or H200 | Lower maximum load |
The table provides direction but does not replace testing the model with the expected context, batch size, number of users, and latency requirements.
Common comparison mistakes
- Treating FP4, FP8, dense, and sparse modes as equivalent metrics.
- Evaluating total memory without considering capacity per GPU.
- Counting only the weights and forgetting the KV cache, activations, and buffers.
- Applying NVL72 figures to a single DGX.
- Looking at tokens per second without time to first token.
- Expecting linear acceleration from a second node.
- Buying 800 Gbit/s adapters without a matching fabric.
- Ignoring rack power and cooling.
- Failing to verify Blackwell support in the software stack.
- Choosing B300 for a workload limited by the CPU, network, or storage.
Which NVIDIA DGX to choose
Image source: NVIDIA
DGX H100 suits mature Hopper infrastructure and workloads for which 640 GB of GPU memory is sufficient. H200 is the best option within Hopper for large models, long contexts, and a large KV cache. B200 is the main entry point to Blackwell and suits most new mixed-workload clusters. B300 should be chosen for maximum memory headroom, reasoning models, full fine-tuning, and high concurrent demand.
Start with the model size, weight precision, KV-cache requirements, context length, number of users, and fine-tuning method. Then assess the network, storage, power, cooling, and utilisation. This calculation shows whether the new generation will deliver a real advantage or merely add expensive unused capacity.
Content:
