🏛️ On-premises vs private cloud for regulated LLM deployments
Banks, hospitals and governments all ask the same question. The answer depends on three things: data classification, who holds the keys, and how fast you need to scale.
Regulated organisations rarely choose public multi-tenant APIs for sensitive workloads. The real decision is between GPUs in your own data centre and dedicated capacity in a private or sovereign cloud region.
On-premises
Choose when: data is classified restricted or above, regulators require physical control, or you already run a GPU estate. Costs: capital expenditure, 3–5-year refresh cycles, power and cooling, and the operational skill to run inference servers. Benefit: nothing leaves the building; audit is straightforward.
Private / sovereign cloud
Choose when: data may be processed in-country under contractual controls, you need to scale up and down with demand, or you lack GPU operations staff. Costs: hourly GPU pricing that can exceed on-prem within 12–18 months at steady load. Benefit: elasticity and managed hardware.
Break-even
At sustained utilisation a purchased H100 typically pays back against on-demand cloud pricing within roughly a year; at bursty or low utilisation the cloud wins. The assessment shows a cloud reference cost next to each on-prem configuration so you can run the comparison for your traffic.
Controls that matter in both
- Prompt and response logging with retention aligned to your records policy.
- Role-based access to the inference API and to logs.
- Model provenance: pinned weights hashes and a record of the license accepted.
- A human-review step wherever outputs influence regulated decisions.