LLAMA 3.1 70B, LLAMA 3.3 70B
Edgeless Systems
LLM service uses confidential computing to deliver secure cloud AI applications
At a glance
Industry: Technology
Use case: Developing secure, compliant AI applications
Goal: Enable businesses across sectors to use cloud-based generative AI
Llama versions: Llama 3.1 70B Instruct AWQ INT4, Llama 3.3 70B Instruct AWQ INT4
Deployment: Bare metal hosted cloud via Scaleway
6 months
50%
1,000s
*All results are self-reported and not identifiably repeatable. Generally, expected individual results will differ.
Making cloud AI applications secure
Edgeless Systems is a German-based cybersecurity company that makes the public cloud the safest place for sensitive data. The company builds world-leading, open-source solutions for confidential computing, taking data security for cloud and AI applications to an unprecedented level, enabling encrypted data processing at scale.
THEIR GOAL
Empower businesses to use generative AI and protect sensitive data
Businesses in heavily regulated industries like healthcare and banking were missing out on AI workplace solutions because cloud AI applications didn’t meet their strict data security requirements. Edgeless Systems set out to create a cloud-based service that could fortify chatbots and other AI tools using confidential computing — a technology that isolates sensitive data and computations to keep data encrypted at all times.
THEIR SOLUTION
A state-of-the-art LLM service that keeps data safe and secure
Edgeless Systems created Privatemode AI: the first end-to-end, cloud-based large language model (LLM) service that uses confidential computing to meet strict encryption requirements and mitigate security risks.
Llama models rose to the top for PrivatemodeAI because they were open, multilingual and self-hostable, making them easily downloadable to run on Edgeless Systems’ platform. The company ultimately chose a quantized version — Llama 3.1 70B Instruct AWQ INT4 (later upgrading to Llama 3.3 70B Instruct AWQ INT 4) — to balance performance with the compute efficiency to run on a single NVIDIA H100 GPU, which was necessary to enable confidential computing.
Privatemode AI provides end-to-end encrypted AI services for heavily regulated industries.
THEIR APPROACH
Using confidential computing for AI applications
Privatemode AI creates a secure environment that separates the infrastructure and service providers from the data and models. A lightweight client proxy uses an API to plug directly into existing codebases, encrypts prompts and decrypts responses locally to ensure confidentiality.
On the server side, secure components run inside a confidential container. The contrast coordinator manages attestation and credentials, while the secret service controls encryption key access. Verified AI workers run the Llama model securely, never exposing data in plaintext.
Privatemode AI verifies the integrity of each step of the confidential computing service.
THEIR SUCCESS
Companies with sensitive data can safely use generative AI in the cloud
Privatemode AI made it possible for industries such as finance, law and healthcare to safely take advantage of flexible cloud AI applications that enhance productivity. Businesses can quickly launch secure AI assistants, chatbots and more without investing in on-premises deployments — saving both time and money.
• 6 months faster time to launch vs. on-premises AI projects
• 50% savings compared to on-premises AI deployments
• 1,000s of employees onboarded, and it’s just the beginning
Before Privatemode AI, tight security required extremely costly on-prem deployments. Now, companies have a secure option that uses confidential computing to isolate prompts and data from the underlying infrastructure and AI models. AI workloads can access powerful Llama models in the cloud with the same level of security and privacy.
"Before Privatemode AI, tight security required extremely costly on-prem deployments. Now, companies have a secure option that uses confidential computing to isolate prompts and data from the underlying infrastructure and AI models. AI workloads can access powerful Llama models in the cloud with the same level of security and privacy."
Moritz Eckert - VP of Product and Technology, Edgeless Systems
Start building
Create generative AI applications for business with open-source large language models that bring unmatched control, customization and flexibility.
Stay up-to-date
Our latest updates delivered to your inbox
Subscribe to our newsletter to keep up with the latest Llama updates, releases and more.
