A frontier-class language model in the bank's own rack: AI for 100 staff without a prompt leaving the building

A regional bank and wealth manager wanted what the best hosted models can do, on data that could not leave its walls. Staff had begun pasting loan files and client statements into consumer chatbots, and compliance had stopped it. The data in question included account balances, credit memos and suspicious-activity narratives, which are legally confidential. Under the GLBA Safeguards Rule and the amended SEC Regulation S-P, the bank is responsible for what its service providers do with customer information, and the board's answer was simple: no internet path from the inference tier.

We started with evidence, not a brand. Around 1,400 anonymized tasks from the bank's own work (credit memos, anti-money-laundering review, policy questions, code and long-document extraction) went to several open-weight models, graded blind by senior staff against a leading cloud model. A very large mixture-of-experts model came closest. All of its 671 billion parameters sit in memory but only about 37 billion work on each token, which is why it answers quickly, and its architecture keeps the memory for long conversations manageable.

The hardware is two 8‑GPU HGX B200 servers running active-active, so either can be patched or lose a part without anyone noticing, and a third HGX H200 node handles embeddings, reranking, OCR and speech. The flagship model's FP8 weights take roughly 700 GB, less than half the memory of one node, and most of the rest holds the working memory of active conversations. We chose retrieval over fine-tuning: domain knowledge comes from the bank's own documents, so when a better open model arrives the bank can swap it in over a weekend.

We measured speed against what people feel. In our modeled benchmarks on one B200‑class node (FP8 weights, speculative decoding) a single user sees about 118 tokens per second. At the busiest hour, with about 32 streams at once, each user still sees about 60, which is twice the 30 tokens per second we set as a target and roughly ten times the 5 to 8 at which people read. The node then produces about 1,950 tokens per second in total. The wait before the first word depends on the prompt: about 0.4 seconds at the median for a 4,000‑token prompt (about 1.4 seconds for the slowest 5 percent), and about 12 seconds for a 128,000‑token prompt, roughly a 300‑page loan file. Long prompts are slow on any hardware, and caching a reused prefix removes most of the wait.

Security is the point of the design. The inference tier has no default route, the firewall denies and logs all egress, and new model weights arrive through a one-way path after a hash check and scan. A gateway applies role-based access, redacts card numbers and writes every prompt and response to write-once storage. Retrieval checks permissions per document, so the model never sees a file the user could not open. Before launch we attacked it ourselves with poisoned PDFs, role-boundary attempts and jailbreaks aimed at tipping-off language, and fixed every serious finding. The economics deserve honesty: per token, local inference costs more than a cloud API at this scale, so the real comparison is against doing without AI, or against the hours it saves. Throughput and latency figures are modeled from published hardware specifications and open benchmarks, and the engagement is illustrative. A quarterly model-risk review in the spirit of SR 11-7 keeps the model inventory, evaluation drift and bias checks current, and the same pattern scales to a different firm, model and rack.