Module 5: Running Models Locally

Local model setup with LM Studio, open-weight model landscape, and Exercise 2: cloud vs. local comparison.

Slides

View slides in full screen

Outline

  • Why run models locally: privacy, resilience, cost, environmental efficiency
  • The open-weight model landscape (spring 2026): Qwen, Llama, Mistral, DeepSeek, Gemma, and others
  • Technical considerations: weight sizes, quantization, dense vs. MoE architectures, MLX vs. llama.cpp
  • Apple Silicon advantage for local inference
  • Setting up LM Studio and connecting to an agentic coding harness
  • Downsides of local models: performance ceiling, provenance concerns, setup complexity
  • Exercise 2: Cloud vs. local model comparison in small groups

Learning Objectives

  • Explain why running models locally can mitigate several of the risks discussed in Module 4
  • Navigate the landscape of open-weight models: who makes them, what the popular options are (as of spring 2026), and how to choose among them
  • Understand weight sizes, quantization levels, dense vs. MoE architectures, and frameworks (MLX, llama.cpp, etc.) well enough to make informed choices
  • Set up LM Studio, download an appropriate model, and connect it to an agentic coding harness (e.g., OpenCode, Pi Coding Agent, or Qwen Code)
  • Evaluate the performance gap between local and cloud models on a real task, and articulate the tradeoffs

Exercise 2: Cloud vs. Local Model Comparison

Setup: In small groups, decide on a simple software development task (not a Jupyter notebook). Run it on both a cloud tool (Claude Code or Codex) and the most capable local model your hardware can handle.

During the exercise: - Note differences in speed, output quality, correctness, and overall experience - Experiment with different model sizes if time allows - Compare notes across group members with different hardware

Debrief (10 min): Each group shares a brief summary. What worked? What did not? Anything surprising?

Software Setup

If you have not yet set up LM Studio and OpenCode on your machine, see the Software Setup section of this site for step-by-step guides for macOS and Windows.