VSCodium + Ollama: Local LLM Coding Setup Guide

Why run your LLM locally for coding?

This guide walks through a complete local LLM coding setup using VSCodium, Continue, and Ollama — no cloud, no API costs, your code stays on your machine. By the end you'll have AI code completion, inline chat, and refactoring running entirely offline. Local models now handle 70–80% of everyday coding tasks at zero ongoing cost, and in 2026 the quality gap with cloud models has narrowed significantly.

The stack:

• VSCodium (telemetry-free VS Code builds)
• Continue (in-editor AI assistant for completions, chat, and refactors)
• Ollama (local model runtime)

Works on macOS, Linux, and Windows. No subscription required.

Advantages and disadvantages

Advantages

• Privacy by default: your code never leaves your machine.
• Zero subscription cost — a GitHub Copilot alternative that runs on hardware you already own.
• Predictable costs and no vendor lock-in.
• Full control over model choice, prompts, and context.

Disadvantages

• Hardware limits how large and fast your models can be.
• Quality varies across models; you may need to try a few.
• You are responsible for updates: new releases like Qwen 3 Coder can appear quickly, and you decide when to upgrade.
• You still need clear prompts and scoped tasks to get good results.

Why VSCodium instead of VS Code

VS Code's source is MIT licensed, but Microsoft's distributed binaries add a separate license and enable telemetry by default. VSCodium ships ready-to-use builds with telemetry disabled, so you start with a privacy-first editor and no extra setup.

Why local AI beats cloud for many teams

Local models do not always beat the best cloud models on raw intelligence. But they win on what matters day-to-day:

• Privacy by default: your code stays on your machine.
• Lower cost: no subscriptions or per-seat fees.
• Offline work: keep shipping even without internet access.
• Control: pick the model, tune prompts, and switch anytime.

The local stack: VSCodium + Continue + Ollama

Continue lives inside the editor and provides local AI code completion, chat, code edits, refactors, and explanations — all powered by your local Ollama instance. Ollama runs the model and exposes it to Continue via a standard REST API. Together they give you a complete local AI workflow with zero cloud dependencies.

What you need

Confirm your hardware before installing. Minimum requirements for a working local LLM coding setup:

Minimum (CPU-only):

• 8 GB RAM
• macOS, Linux, or Windows
• 10 GB free disk space
• Any modern multi-core CPU (4+ cores)

Recommended (comfortable performance):

• 16 GB RAM, or a GPU with 8 GB VRAM (NVIDIA/AMD/Intel Arc)
• Apple Silicon M-series (unified memory is ideal — a 16 GB M2 runs 8B models smoothly)

CPU-only note: Ollama runs without a GPU. Expect 2–6 tokens/sec on 3B–4B models. Sufficient for completions and async review; slow for real-time chat on larger models.

Ollama installation (more detail)

macOS

1. Install with Homebrew: brew install ollama
2. Start the service: ollama serve
3. Verify the install: ollama -v

Linux

1. Install with the official script: curl -fsSL https://ollama.com/install.sh | sh
2. Start the service: ollama serve
3. Verify the install: ollama -v

Windows

1. Download the installer from https://ollama.com and follow the prompts.
2. Open a new terminal and run: ollama -v
3. Start the service: ollama serve

First model pull

• Run ollama pull qwen3:7b (recommended starting point on 16 GB RAM).
• Use ollama list to see what is installed.

10-minute local LLM coding setup checklist

1. Install VSCodium.
2. Install Ollama.
3. Pull a model (start small if your laptop is modest).
4. Install the Continue extension.
5. Point Continue to your local Ollama endpoint (http://localhost:11434).

Best models for coding with Ollama (2026)

Model	Size	Min RAM	Best for	Ollama pull
qwen3.5:4b	4B	8 GB	Entry-level, multimodal, agentic	ollama pull qwen3.5:4b
qwen3:7b	7B	10 GB	Best HumanEval score under 8B (76.0)	ollama pull qwen3:7b
qwen3.5:9b	9B	12 GB	Reasoning + coding, scaled RL trained	ollama pull qwen3.5:9b
qwen2.5-coder:14b	14B	16 GB	Python/TS specialist, code completion	ollama pull qwen2.5-coder:14b
llama3.3:70b	70B	48 GB	GPT-4-class, Apple Silicon M-series only	ollama pull llama3.3:70b

Recommended starting point: qwen3:7b on 16 GB RAM. Highest benchmark score under 8B parameters, runs at 20–30 tokens/sec on Apple Silicon M2/M3.

No GPU or tight on RAM? qwen3.5:4b runs on 8 GB and is the first small model in the series with native multimodal and agentic capabilities.

Performance optimisation: quantisation and GPU offloading

Most guides skip this section. That is a mistake — these two levers account for the biggest real-world speed differences.

Quantisation explained

A model's weights are normally stored as 16-bit or 32-bit floats. Quantisation reduces this to 4-bit or 8-bit integers, trading a small amount of accuracy for dramatically lower memory use and faster inference.

Ollama uses GGUF format, which encodes quantisation in the model name:

Quantisation	Memory use	Quality loss	When to use
Q8_0	~1× the Q4 size	Negligible	You have headroom and want max quality
Q5_K_M	~1.25× Q4	Very small	Best quality/speed tradeoff when RAM allows
Q4_K_M	Baseline	Small for coding tasks	Default choice for most setups
Q3_K_M	~0.75× Q4	Noticeable	Only when RAM is the hard constraint
Q2_K	~0.6× Q4	Significant	Last resort — avoid for serious use

For coding tasks specifically, Q4_K_M is the practical floor. Below Q4, models start hallucinating APIs and method signatures at noticeably higher rates.

To pull a specific quantisation:

ollama pull qwen2.5-coder:32b-instruct-q4_k_m

Check what is available for a model:

ollama ls
# or browse https://ollama.com/library/<model-name> for all tags

GPU layer offloading

This is the biggest performance lever most people never configure. By default, Ollama detects your GPU and offloads as many transformer layers as fit in VRAM. But you can tune this explicitly.

Why it matters: each layer moved to GPU is orders of magnitude faster than CPU inference. A model with 32 layers where only 20 fit in VRAM still runs 20× faster on those 20 layers than pure CPU.

Set the number of GPU layers in Ollama via the Modelfile or the environment variable:

# Run a model with explicit GPU layer count
OLLAMA_NUM_GPU=24 ollama run qwen2.5-coder:7b

Or create a custom Modelfile for persistent settings:

FROM qwen2.5-coder:7b
PARAMETER num_gpu 24
PARAMETER num_ctx 8192
PARAMETER num_thread 8

Then build it:

ollama create myqwen -f Modelfile
ollama run myqwen

Practical GPU offloading guide:

VRAM available	Layer strategy
4 GB	Offload 10–16 layers of a 7B model, rest on CPU
8 GB	Offload full 7B model, or ~18 layers of 13B
12 GB	Full 13B, or ~20 layers of 32B
16 GB+	Full 13B–20B, partial 32B
24 GB	Full 32B at Q4

Context window tuning

The context window (num_ctx) directly affects VRAM use. A context of 128k tokens needs significantly more memory than 8k. For most coding tasks, 8k–16k is plenty:

PARAMETER num_ctx 8192

Reduce this first if you are hitting OOM errors before reducing quantisation.

Thread count for CPU inference

If you are running primarily on CPU, set num_thread to your physical core count (not threads):

PARAMETER num_thread 8

Hyperthreading hurts inference throughput. Physical cores only.

Recommended workflow for speed and quality

Here is a simple workflow that keeps things fast and surprisingly effective:

• Use a smaller model for quick completions and short edits.
• Use a larger model only when you need deeper reasoning.
• Limit context to open files and target folders, not your whole disk.
• Add a short rules prompt to keep changes clean and predictable.

Suggested rules prompt:

follow existing project style
do not invent APIs
prefer minimal diffs
ask when unsure

Use AGENTS.md for consistent results

Local agents work best when they have a short, consistent rule set. Many teams keep an AGENTS.md file in the repo root so the assistant sees the same guidance every time. Example template:

# AGENTS.md

## Goals
- Keep changes minimal and focused.
- Preserve existing code style and structure.
- Ask before making broad refactors.

## Behavior
- Do not invent APIs or dependencies.
- Prefer small, testable diffs.
- Call out assumptions and missing context.

## Context
- Use only files referenced in the prompt unless told otherwise.
- For large changes, propose a plan before editing.

This helps your local assistant behave consistently across tasks and teammates.

Limitations to expect

If you want the honest version, here it is:

• Hardware matters: RAM and VRAM limit which models you can run.
• Expect tradeoffs between speed, quality, and context length.
• Local models still need clear prompts and scoped tasks.

Alternatives if you prefer other editors

• Neovim with local LLM tooling
• JetBrains IDEs with Continue
• Other local assistants that support Ollama

Bottom line

For most teams, VSCodium + Continue + Ollama is the right local LLM coding setup: private by default, zero subscription cost, and trivial to reconfigure as better models appear. Add cloud models only when you genuinely need their extra capability.

Frequently asked questions

Can I use Ollama without a GPU?

Yes. Ollama runs on CPU-only machines. On 8 GB RAM without a GPU you can run 3B–4B models at 2–6 tokens/sec. Slower than cloud, but fully private and sufficient for completions and short edits.

Is a local LLM as good as GitHub Copilot?

For daily coding tasks — completions, refactors, code explanation — a 7B–8B local model covers roughly 70–80% of Copilot quality with zero subscription cost and full privacy. The gap has closed significantly with models like qwen2.5-coder:7b. For teams who prioritise data privacy, this local LLM coding setup is a strong GitHub Copilot alternative.

What is the best model for coding with Ollama?

On 16 GB RAM, qwen3:7b is the current top pick — highest HumanEval score under 8B parameters (76.0), runs at 20–30 tokens/sec on Apple Silicon M2/M3. On 8 GB RAM, qwen3.5:4b is the entry choice with multimodal support. For specialist code completion with 16 GB RAM or a dedicated GPU, qwen2.5-coder:14b is the Python/TypeScript specialist. llama3.3:70b is GPT-4-class but needs 48 GB RAM — practical only on Apple Silicon M-series Max.

Does VSCodium work with Continue.dev?

Yes. Continue is published to the Open VSX marketplace that VSCodium uses instead of the Microsoft marketplace. Install it from the Extensions panel inside VSCodium and point it at your Ollama endpoint (http://localhost:11434).

For production automation use cases, see our guide on running a self-hosted LLM in production with Ollama and n8n.

Ready to go further?

This is how we build at Vasilkoff: privacy-first, open-source friendly, and practical. If your team needs a private AI coding assistant integrated into your development workflow — or broader AI development services — we can help. Reach out via our contact page.