Running AI Models on Your Own Computer

Why Local Models Matter for Healthcare

Cloud-based AI services like ChatGPT and Claude offer powerful capabilities, but they present challenges for healthcare applications. Data sent to these services may be retained for training, lacks guaranteed HIPAA compliance without specific enterprise agreements, and requires constant internet connectivity.

Local models flip this equation. When you run an AI model on your own hardware, patient data never leaves your machine. There's no network transmission, no third-party storage, no Business Associate Agreement required for internal experimentation.

Ideal Use Cases for Local Models

Medical-Specific Models

Several open-source models have been specifically trained or fine-tuned for healthcare applications. These offer better baseline performance on medical tasks than general-purpose models of similar size. However, "better baseline performance" doesn't mean "ready for clinical use"—all require validation for any specific application.

MedGemma

Released by Google DeepMind in May 2025, MedGemma represents the current state-of-the-art for open medical models. Built on Google's Gemma 3 architecture, these models underwent continued pre-training on diverse medical data while preserving general capabilities.

Performance Claims:

• MedGemma 4B scores 64.4% on MedQA, ranking among the best sub-8B open models
• In radiologist evaluation, 81% of its chest X-ray reports were judged sufficient for similar patient management
• MedGemma 27B Text scores 87.7% on MedQA—within 3 points of much larger proprietary models

Image Capabilities: The multimodal variants can process medical images including chest X-rays, dermatology photographs, fundus images, and histopathology slides.

Hardware Requirements:

• MedGemma 4B: Runs on single consumer GPU (8GB+ VRAM) or Apple Silicon Macs with 16GB+ RAM
• MedGemma 27B: Requires substantial hardware—64GB+ RAM or professional GPUs with 24GB+ VRAM

Access: Available on Hugging Face (google/medgemma-4b-it for the instruction-tuned 4B variant). Requires agreeing to Google's terms of use.

Meditron

Developed by researchers at EPFL and Yale School of Medicine, Meditron adapts Meta's Llama 2 architecture through continued pretraining on curated medical data. The project emphasizes open access and low-resource settings.

Training Corpus (GAP-Replay, 48.1 billion tokens):

• Clinical guidelines: 46,000 guidelines from hospitals and international organizations (including ICRC)
• PubMed abstracts: 16.1 million abstracts from medical literature
• Medical papers: Full text from 5 million publicly available papers
• Replay data: 400 million tokens of general-domain content to prevent catastrophic forgetting

Model Variants:

• Meditron-7B: Smaller variant suitable for laptops with 16GB+ RAM at Q4 quantization
• Meditron-70B: Full-size variant that outperforms GPT-3.5 and Flan-PaLM on multiple medical reasoning benchmarks. Requires serious hardware.

Access: GitHub (epfLLM/meditron), Hugging Face. Available in Ollama as meditron:7b.

BioMistral

Built on the Mistral architecture and pre-trained on PubMed Central, BioMistral offers strong biomedical question-answering performance in a 7B parameter package. Its standout feature is multilingual evaluation—the team tested performance across eight languages.

Best Use Cases:

• Biomedical literature synthesis and analysis
• Research question-answering
• Applications requiring non-English language support
• Settings where smaller model size is important

Access: Hugging Face (BioMistral/BioMistral-7B), available in some Ollama model repositories.

Other Notable Models

Choosing a Model

Hardware Requirements and Performance

Running local models requires understanding the relationship between model size, quantization, and your available hardware. The bottleneck is almost always memory—either system RAM or GPU VRAM.

Memory Math

A model's parameters are stored as floating-point numbers. In full precision (FP16), each parameter requires 2 bytes. So a 7B parameter model needs roughly 14GB just for the weights. Add overhead for inference and you're looking at 16-20GB for comfortable operation.

Most consumer hardware can't handle this, which is where quantization comes in.

Understanding Quantization

Quantization reduces precision from 16-bit floating point to smaller representations—8-bit, 4-bit, even 2-bit. This trades some accuracy for dramatically reduced memory requirements.

For most users, Q4_K_M is the sweet spot—good quality, reasonable size. The "_K_" notation indicates "K-quant" methods that intelligently quantize different parts of the model to preserve quality.

Practical Hardware Guidelines

Apple Silicon Performance

M-series Macs deserve special mention. Their unified memory architecture—where CPU and GPU share the same RAM pool—eliminates the VRAM bottleneck that limits Windows/Linux GPU usage. A 16GB M1 Mac can effectively use all 16GB for model weights.

GPU Acceleration

For NVIDIA GPUs, the key metric is VRAM:

Partial offloading: You don't need to fit the entire model in VRAM. Both Ollama and LM Studio support loading some layers on GPU and others in system RAM. Even 50% GPU offloading provides meaningful speedup.

Security Setup for HIPAA-Compliant Work

Even though local models don't transmit data externally, proper security hygiene matters. This section covers practical steps rather than comprehensive compliance frameworks.

Full Disk Encryption

Non-negotiable for any machine that might touch patient data. If your laptop is stolen, encryption prevents data access without the decryption key.

macOS (FileVault)

1. Open System Settings → Privacy & Security → FileVault
2. Click "Turn On FileVault"
3. Choose recovery key option (store securely, not on the same machine)
4. Encryption proceeds in background; may take several hours

Windows (BitLocker)

1. Open Control Panel → System and Security → BitLocker Drive Encryption
2. Click "Turn on BitLocker" for your OS drive
3. Choose how to unlock (PIN or USB key for stronger security)
4. Save recovery key to secure location
5. Choose encryption scope (entire drive recommended)

Network Isolation

Local models don't need internet access once downloaded:

• Run in airplane mode during sessions involving any patient context
• Use application-level blocking (Little Snitch on macOS, Windows Firewall)
• Consider air-gapped machines for highest sensitivity work

Data Handling Practices

• Never paste real patient identifiers into prompts
• Use synthetic data or de-identified case presentations for development
• Clear chat histories after sessions
• Disable persistent storage of conversations when possible

Installation: Ollama

Ollama is the simplest path to running local models. It handles model downloading, quantization management, and provides both a command-line interface and local API. Think of it like Docker for language models.

macOS Installation

Option 1: Direct download (recommended)

1. Visit ollama.com/download
2. Download the macOS installer (.dmg file, ~50MB)
3. Open the .dmg and drag Ollama to Applications
4. Launch Ollama from Applications folder
5. A llama icon appears in your menu bar when running

Option 2: Homebrew

brew install ollama
brew services start ollama

Windows Installation

1. Download the Windows installer from ollama.com/download
2. Run the .exe installer
3. Follow prompts; Ollama installs as a background service
4. The llama icon appears in system tray when running

GPU Setup for NVIDIA: Before installing Ollama, ensure you have recent NVIDIA drivers from nvidia.com/drivers. Ollama will automatically detect and use your GPU.

Linux Installation

curl -fsSL https://ollama.com/install.sh | sh

Verifying Installation

ollama --version
ollama list

Downloading Models

Browse available models at ollama.com/library.

# Small, fast model for testing setup
ollama pull phi3

# Medical-focused model
ollama pull meditron:7b

# Specific quantization variant
ollama pull llama3.1:8b-q4_k_m

Running Models

# Interactive chat
ollama run phi3

# Single prompt (no interactive session)
ollama run phi3 "What are the most common causes of pediatric fever?"

Type /bye to exit interactive mode.

Model Parameters

# Set temperature (0.0 = deterministic, 1.0 = creative)
ollama run phi3 --temperature 0.7

# Set context window size
ollama run phi3 --ctx-size 4096

# Set number of tokens to generate
ollama run phi3 --num-predict 500

Useful Ollama Commands

Local API

Ollama automatically serves an OpenAI-compatible API at http://localhost:11434:

curl http://localhost:11434/api/generate -d '{
  "model": "phi3",
  "prompt": "What are common causes of pediatric fever?"
}'

Installation: LM Studio

LM Studio provides a graphical interface for running local models, making it more approachable for users who prefer visual tools over command lines.

Installation

1. Visit lmstudio.ai
2. Download for your operating system
3. macOS: Drag to Applications folder
4. Windows: Run the installer
5. Linux: Make the AppImage executable and run

Downloading Models

1. Launch LM Studio
2. Click "Discover" in the left navigation
3. Search for a model (try "phi3" or "meditron")
4. Click Download on your chosen variant

Running Models

1. Click "Chat" in left navigation
2. Click the dropdown at top to select your downloaded model
3. Click "Load" to load the model into memory
4. Start typing in the chat interface

GPU Offloading

1. Go to model settings (gear icon)
2. Adjust "GPU Layers" slider
3. More layers = faster but more VRAM required
4. Start with 50% and adjust based on performance

Comparing Ollama and LM Studio

Many users install both: LM Studio for discovery and quick testing, Ollama for integration and automation.

Critical Limitations to Understand

Real-World Failure Modes

Benchmark Scores Don't Equal Clinical Competence

A model scoring 87% on MedQA (multiple-choice medical questions) isn't "87% as good as a doctor." MedQA measures pattern recognition and fact recall. Success does not require:

• Integration of physical examination findings
• Patient communication and history-taking
• Weighing benefits and harms in individual contexts
• Recognizing when a question is unanswerable
• Knowing the limits of one's own knowledge
• Managing uncertainty over time

The Hallucination Problem

All language models sometimes generate confident, fluent, and completely wrong outputs. This is an intrinsic architectural limitation, not a bug to be fixed.

In medical contexts, hallucination could mean:

• Inventing drug interactions that don't exist
• Citing non-existent studies with realistic-sounding titles
• Missing critical differential diagnoses while confidently presenting an incomplete list
• Providing incorrect dosing information

What Local Models Can Reasonably Do

Resources

Model Repositories

• Hugging Face Medical Models
• Ollama Model Library
• MedGemma on Hugging Face
• Meditron GitHub

Technical Documentation

• Ollama GitHub
• LM Studio Documentation

Medical AI Evaluation

• Open Medical-LLM Leaderboard (Hugging Face)
• MedLLMs Practical Guide

Community

• Ollama Discord
• LM Studio Discord
• r/LocalLLaMA