Key takeaways

  • It is rarely all-or-nothing. Most organizations get the best result by routing each workload to local, private-cloud or frontier models according to its sensitivity and difficulty.
  • Local deployment changes where risk sits. Data stays inside your boundary, but you now own patching, access control, logging and the integrity of the model files.
  • "Open-weight" is not the same as "open source." Read the license, and know what you are allowed to do before you build on a model.
  • Cost depends on utilization. Local GPUs are a fixed cost that pays off when they are kept busy; APIs are a variable cost that pays off when use is uneven. Measure your own workload before you buy hardware.

The decision in one paragraph

Every organization that uses AI is making an implicit choice about where the computation happens. A frontier API means your prompt, and any documents attached to it, travel to a vendor's data center, where a very capable model processes them under contract. A local deployment means you download model weights and run them on GPUs you control, in your office, your data center or a dedicated private environment. Neither is right for every task. The useful question is which workload belongs where, and the answer depends on what the data is, how hard the task is, how often it runs and who is accountable if something goes wrong.

What the terms mean

A frontier model is one of the most capable general-purpose systems available, typically from a large AI lab and offered as a hosted service through an API. You pay for use, you get regular improvements without effort, and you do not see inside the system. An open-weight model is one whose trained parameters, called weights, are published so that anyone with the right license can download and run them. Because you hold the weights, you can run the model without an internet connection, fine-tune it on your own data and control exactly when it changes.

Open-weight is not the same as open source. The Open Source Initiative's Open Source AI Definition says an Open Source AI system must grant the freedoms to use, study, modify and share it, and it requires the release of sufficiently detailed information about the training data, the complete source code used to train and run the system, and the parameters, all under OSI-approved terms. Many popular models publish weights but not all of those elements, and many use custom licenses with their own conditions. Others use permissive licenses such as Apache 2.0, which grants broad rights but provides the work on an as-is basis without warranties. Have someone read the actual license before you build a product or a client service on a model.

Privacy: where the data goes

This is usually the deciding factor. NIST's Generative AI Profile, a companion to its AI Risk Management Framework, lists data privacy among the risks unique to or amplified by generative AI. It describes the impact as leakage and unauthorized use, disclosure or de-anonymization of biometric, health, location or other personally identifiable or sensitive data. When you send a prompt to a hosted model, you are making a disclosure to a third party. That can be entirely appropriate under a good contract, but it needs to be a conscious decision.

For hosted services, read the terms for the following points, and get them in writing.

  • Whether your prompts and outputs are used to train or improve the vendor's models
  • How long inputs and outputs are retained, who at the vendor can see them, and how they are deleted
  • Where the data is processed and stored, and which subprocessors are involved
  • What security attestations and incident notification commitments the vendor offers
  • Whether the vendor will sign the agreements your regulators require, such as a business associate agreement for health data

That last point is not optional in regulated settings. Under the HIPAA Security Rule, a covered entity may let a business associate create, receive, maintain or transmit electronic protected health information only after obtaining satisfactory assurances, in a written contract, that it will safeguard the information. If a vendor will not sign, patient data does not go to that vendor's service.

Local deployment removes the third party from the data path. Privileged client files, medical records, financial models, source code and board materials can be processed without leaving your network. That is a strong answer for law firms, medical practices, financial advisers and anyone under a confidentiality obligation. But be precise about what it does and does not buy you. It reduces the number of parties who can see the data. It does not make the data safe by itself. If the server is badly secured, you have simply moved the exposure to your own network.

Security: the risks follow the model

The OWASP Top 10 for LLM Applications names the main failure modes, and most apply wherever the model runs. Prompt injection, where crafted input alters the model's behavior, is the first. Sensitive information disclosure, where the model or its application reveals confidential data, is the second. Supply chain vulnerabilities, data and model poisoning and improper output handling are also on the list. A local model that is connected to your document store, email or internal tools is exposed to all of them, and it can do real damage if it has too much access.

Running locally adds responsibilities that a vendor handles for you in the API model.

  • Model provenance: download weights only from sources you trust, verify checksums, and record the exact version in use. NIST's profile names value chain and component integration as a risk, describing non-transparent or untraceable integration of upstream third-party components.
  • Isolation: run inference servers on a segmented network, with no unnecessary internet access and authenticated, logged access for users and applications.
  • Least privilege: limit what the model and any agent built on it can read and do. Give it the documents a user is allowed to see, not the whole file share.
  • Patching: the inference software, drivers, operating system and containers need the same update discipline as any other production system.
  • Logging and review: record who asked what, which documents were retrieved and what tools were called, with retention and access rules that match your data classification.
  • Backup and recovery: model files, configurations, vector indexes and prompts are assets. Back them up and test restoring them.

Frontier APIs carry security considerations too. Your keys become high-value secrets, your prompts may contain sensitive material that ends up in application logs, and your application depends on the vendor's availability and policies. Those risks are manageable, but they are not zero.

Cost: fixed versus variable

Cost comparisons often go wrong because they compare a hardware price to an API price per request. A better comparison looks at the whole picture.

Frontier APIs are a variable cost. You pay for what you use, typically by the amount of text processed. There is no hardware to buy, no capacity to plan and no idle cost, and prices and capabilities change quickly. This suits uneven workloads, experiments and tasks that need the most capable model. It can become expensive for steady, high-volume work such as classifying every inbound document or summarizing every call.

Local GPUs are a fixed cost. You buy or lease the hardware, power and cool it, house it, and pay for people to run it. The amount of memory on the GPUs sets a ceiling on the size of model you can run well, and the number of simultaneous users sets how many GPUs you need. Once the hardware exists, each extra request costs little, so the economics improve as utilization rises and worsen if the machines sit idle. Hardware also ages, and newer models may need more of it.

To compare honestly, gather these inputs before you decide.

  • Volume: how many requests per day, how long they are, and how bursty the demand is
  • Quality needed: can a smaller model meet the standard for this task, as shown by a test on your own documents?
  • Hardware and hosting: purchase or lease cost, power, space, networking and the expected useful life
  • People: the engineering and operations time to deploy, secure, monitor and update the system
  • Risk cost: the expected cost of a disclosure or compliance failure under each option, which for sensitive data can outweigh the savings either way

We deliberately do not quote benchmark prices here, because they change monthly. The method is what lasts: run a pilot of a single real workload on both options, record quality and cost, and decide with your own figures.

Governance: who is accountable

The NIST AI Risk Management Framework is a voluntary framework intended to improve the ability to incorporate trustworthiness considerations into the design, development, use and evaluation of AI products and services. NIST has noted that the framework is being revised, so check its page for the current version. Whichever framework you use, certain governance basics apply to both local and hosted AI.

  • An acceptable-use policy that says which data classes may go to which tier, and which tools staff may use. Unmanaged use of consumer chatbots by employees is the most common leak.
  • An inventory of AI systems in use, who owns each one and what data it touches
  • Human review for consequential outputs. NIST's profile defines confabulation as the confident production of erroneous or false content, so a person must check anything that reaches a client, a court or a patient chart.
  • Testing before launch and monitoring afterward, including red-team style tests for prompt injection and data leakage
  • A change process for model updates, because a new version can behave differently on the same prompt
  • Training for staff on what not to paste into a model, and how to verify what comes out

Putting it together: a routing approach

Most mature deployments use more than one tier. Routine, high-volume and sensitive tasks, such as summarizing internal documents, extracting fields from forms, drafting standard replies and searching a private knowledge base, run on local or private-cloud models. Complex reasoning, difficult drafting and work on information cleared for external processing can go to a frontier model. A router, which is software that classifies each request by data sensitivity and difficulty and sends it to the right place, makes that policy automatic so that staff use one interface and do not have to remember the rules.

A sensible first project is small. Pick one workload with clear value and clear data rules. Classify the data. Test a local model and a frontier model on the same set of your own examples. Add logging and access control. Review the results with the people who will use them, then expand.

Clayton Cybernetics designs and operates AI environments through its Super Intelligence service line, including local and private deployments on dedicated GPUs, routing between tiers, access controls and staff training. We help you decide which work belongs on which tier and then build it with the security controls that decision requires.

Sources

This article is general information, not legal advice. Model licenses, vendor terms and regulations change; read the current terms for any model or service you use and confirm your obligations with counsel. Last reviewed .

Keep reading

Need help with this?

American‑made Super Intelligence: AI your team can learn, trust and deploy.

Super Intelligence