VSCodium + Ollama: Local LLM Coding Setup Guide

Why run your LLM locally for coding?

This guide walks through a complete local LLM coding setup using VSCodium, Continue, and Ollama - no cloud, no API costs, your code stays on your machine. By the end you'll have AI code completion, inline chat, and refactoring running entirely offline. Local models now handle a large share of everyday coding tasks at zero ongoing cost, as long as you pick a model that fits your actual laptop.

The important detail: I am talking about models you can actually keep running on a home laptop. A model that needs hundreds of gigabytes of RAM may be "available" in Ollama, but it is not useful for this setup. For a normal laptop, the practical range is 4B-14B. For a 32 GB MacBook Pro or a laptop with a serious GPU, 26B-35B Q4/MLX variants become realistic.

Model recommendations last checked: June 9, 2026. This guide focuses on models you can realistically run on a home laptop with Ollama, not benchmark-only 400 GB checkpoints.

The stack:

• VSCodium (telemetry-free VS Code builds)
• Continue (in-editor AI assistant for completions, chat, and refactors)
• Ollama (local model runtime)

Works on macOS, Linux, and Windows. No subscription required.

Advantages and disadvantages

Advantages

• Privacy by default: your code never leaves your machine.
• Zero subscription cost — a GitHub Copilot alternative that runs on hardware you already own.
• Predictable costs and no vendor lock-in.
• Full control over model choice, prompts, and context.

Disadvantages

• Hardware limits how large and fast your models can be.
• Quality varies across models; you may need to try a few.
• You are responsible for updates: new releases like Qwen3.6, Gemma 4, or refreshed DeepSeek R1 distills can appear quickly, and you decide when to upgrade.
• You still need clear prompts and scoped tasks to get good results.

Why VSCodium instead of VS Code

VS Code's source is MIT licensed, but Microsoft's distributed binaries add a separate license and enable telemetry by default. VSCodium ships ready-to-use builds with telemetry disabled, so you start with a privacy-first editor and no extra setup.

Why local AI beats cloud for many teams

Local models do not always beat the best cloud models on raw intelligence. But they win on what matters day-to-day:

• Privacy by default: your code stays on your machine.
• Lower cost: no subscriptions or per-seat fees.
• Offline work: keep shipping even without internet access.
• Control: pick the model, tune prompts, and switch anytime.

The local stack: VSCodium + Continue + Ollama

Continue lives inside the editor and provides local AI code completion, chat, code edits, refactors, and explanations — all powered by your local Ollama instance. Ollama runs the model and exposes it to Continue via a standard REST API. Together they give you a complete local AI workflow with zero cloud dependencies.

What you need

Confirm your hardware before installing. Minimum requirements for a working local LLM coding setup:

Minimum (CPU-only):

• 8 GB RAM
• macOS, Linux, or Windows
• 10 GB free disk space
• Any modern multi-core CPU (4+ cores)

Recommended (comfortable performance):

• 16 GB RAM, or a GPU with 8 GB VRAM (NVIDIA/AMD/Intel Arc)
• Apple Silicon M-series (unified memory is ideal - a 16 GB M2 runs 8B-9B models smoothly)

CPU-only note: Ollama runs without a GPU. Expect 2–6 tokens/sec on 3B–4B models. Sufficient for completions and async review; slow for real-time chat on larger models.

Ollama installation (more detail)

macOS

1. Install with Homebrew: brew install ollama
2. Start the service: ollama serve
3. Verify the install: ollama -v

Linux

1. Install with the official script: curl -fsSL https://ollama.com/install.sh | sh
2. Start the service: ollama serve
3. Verify the install: ollama -v

Windows

1. Download the installer from https://ollama.com and follow the prompts.
2. Open a new terminal and run: ollama -v
3. Start the service: ollama serve

First model pull

• Run ollama pull qwen3.5:9b if you have 16 GB RAM, or ollama pull qwen3.5:4b on tighter hardware.
• Use ollama list to see what is installed.

10-minute local LLM coding setup checklist

1. Install VSCodium.
2. Install Ollama.
3. Pull a model (start small if your laptop is modest).
4. Install the Continue extension.
5. Point Continue to your local Ollama endpoint (http://localhost:11434).

Best models for coding with Ollama (2026)

All models in this table are confirmed Ollama-native with quantizations chosen for reliable tool calling and agent work.

Model	Ollama tag	VRAM	Quantization	Best for
gemma4 (E4B)	ollama run gemma4:e4b	10 GB	Default	Vision + native tool calling, fits any 2024 laptop GPU
qwen3:7b	ollama run qwen3:7b-q8_0	10 GB	Q8_0	Best HumanEval <8B (76.0), reliable tool calling at Q8
gemma4:12b	ollama run gemma4:12b	16 GB	Default	Mid-tier; vision + tool calling, released June 3 2026
qwen3-coder:30b	ollama run qwen3-coder:30b	20 GB	Default	Agentic coding, 256K context, 30B total / 3.3B active MoE, designed for tool use
qwen2.5-coder:32b	ollama run qwen2.5-coder:32b-instruct-q6_k	26 GB	Q6_K	FIM autocomplete king, 92.7% HumanEval, 128K context

Key notes on quantization and tool calling

Q4 and tool calling: Q4_K_M works fine for chat but causes agent failures — structured output and instruction-following degrade silently. For any workflow using tool calls or multi-step agents, use Q6_K as the minimum, or the Q4_K_XL variants from Unsloth which maintain attention mechanism quality better than standard Q4.

On Gemma 4's context trap: Ollama defaults Gemma 4 to 4K context even though the model supports up to 256K. Always set num_ctx explicitly in your Modelfile:

FROM gemma4:12b
PARAMETER num_ctx 32768

On Qwen3 thinking mode and tool calling: For Qwen3 reasoning models, do not use tool call templates based on stopwords like ReAct — the model may output stopwords in its thinking section, causing unexpected tool call behavior. Use Ollama's native tool calling API instead.

Beyond Ollama — when you need more

Qwen3.6 27B via LM Studio or llama.cpp — the most impressive open coding model as of June 2026 (SWE-bench 77.2%), but Ollama doesn't support its vision files yet. Q6_K needs 22.5 GB VRAM. Worth following; when Ollama support lands it replaces qwen2.5-coder:32b as the 24 GB recommendation.

qwen3-coder:480b via ollama run qwen3-coder:480b-cloud — runs locally but requires 250 GB minimum memory; for most people the cloud tag is the realistic option.

Recommended starting point

16 GB RAM / 10 GB VRAM:        gemma4:e4b — vision, tool calling, no setup headaches
16 GB RAM, coding focus:       qwen3:7b at Q8_0 — best HumanEval under 8B, reliable agents
24 GB VRAM:                    qwen3-coder:30b — agentic coding, 256K context, MoE efficiency
24 GB VRAM, autocomplete:      qwen2.5-coder:32b at Q6_K — FIM, 92.7% HumanEval

Apple Silicon and MLX model tags

If you are on a MacBook, note that MLX-specific variants can offer better integration with Apple's unified memory architecture. The Ollama defaults for these models work well, but you can experiment with MLX versions if available.

For the recommended starting point on Apple Silicon:

• 16 GB MacBook Air/Pro: use gemma4:e4b or qwen3:7b-q8_0 as your daily assistant.
• 24 GB+ MacBook Pro: use qwen3-coder:30b or qwen2.5-coder:32b-instruct-q6_k for agentic coding.

For daily coding on Apple Silicon, keep the model small enough that it answers quickly and stays in the loop. A huge model that makes every prompt slow stops being useful, even if the benchmark looks better.

Performance optimisation: quantisation and GPU offloading

Most guides skip this section. That is a mistake — these two levers account for the biggest real-world speed differences.

Quantisation explained

A model's weights are normally stored as 16-bit or 32-bit floats. Quantisation reduces this to 4-bit or 8-bit integers, trading a small amount of accuracy for dramatically lower memory use and faster inference.

Ollama uses GGUF format, which encodes quantisation in the model name:

Quantisation	Memory use	Quality loss	When to use
Q8_0	~1× the Q4 size	Negligible	You have headroom and want max quality
Q5_K_M	~1.25× Q4	Very small	Best quality/speed tradeoff when RAM allows
Q4_K_M	Baseline	Small for coding tasks	Default choice for most setups
Q3_K_M	~0.75× Q4	Noticeable	Only when RAM is the hard constraint
Q2_K	~0.6× Q4	Significant	Last resort — avoid for serious use

For coding tasks specifically, Q4_K_M is the practical floor. Below Q4, models start hallucinating APIs and method signatures at noticeably higher rates.

To pull a specific quantisation:

ollama pull qwen3.6:27b

Check what is available for a model:

ollama ls
# or browse https://ollama.com/library/<model-name> for all tags

GPU layer offloading

This is the biggest performance lever most people never configure. By default, Ollama detects your GPU and offloads as many transformer layers as fit in VRAM. But you can tune this explicitly.

Why it matters: each layer moved to GPU is orders of magnitude faster than CPU inference. A model with 32 layers where only 20 fit in VRAM still runs much faster on those 20 layers than pure CPU.

Set the number of GPU layers in Ollama via the Modelfile or the environment variable:

# Run a model with explicit GPU layer count
OLLAMA_NUM_GPU=24 ollama run qwen3.5:9b

Or create a custom Modelfile for persistent settings:

FROM qwen3.5:9b
PARAMETER num_gpu 24
PARAMETER num_ctx 8192
PARAMETER num_thread 8

Then build it:

ollama create myqwen -f Modelfile
ollama run myqwen

Practical GPU offloading guide:

VRAM available	Layer strategy
4 GB	Offload part of a 4B-9B model, rest on CPU
8 GB	Offload a 4B-9B model, or partial layers of a 14B model
12 GB	Full 9B if quantized, or a comfortable 4B/E4B setup
16 GB+	Full 9B, 8B reasoning distills, or partial 14B
24 GB	Full 14B, or cautious 26B/27B at Q4 with reduced context

Context window tuning

The context window (num_ctx) directly affects VRAM use. A context of 128k tokens needs significantly more memory than 8k. For most coding tasks, 8k–16k is plenty:

PARAMETER num_ctx 8192

Reduce this first if you are hitting OOM errors before reducing quantisation.

Thread count for CPU inference

If you are running primarily on CPU, set num_thread to your physical core count (not threads):

PARAMETER num_thread 8

Hyperthreading hurts inference throughput. Physical cores only.

Recommended workflow for speed and quality

Here is a simple workflow that keeps things fast and surprisingly effective:

• Use a smaller model for quick completions and short edits.
• Use a larger model only when you need deeper reasoning.
• Limit context to open files and target folders, not your whole disk.
• Add a short rules prompt to keep changes clean and predictable.

Suggested rules prompt:

follow existing project style
do not invent APIs
prefer minimal diffs
ask when unsure

Use AGENTS.md for consistent results

Local agents work best when they have a short, consistent rule set. Many teams keep an AGENTS.md file in the repo root so the assistant sees the same guidance every time. Example template:

# AGENTS.md

## Goals
- Keep changes minimal and focused.
- Preserve existing code style and structure.
- Ask before making broad refactors.

## Behavior
- Do not invent APIs or dependencies.
- Prefer small, testable diffs.
- Call out assumptions and missing context.

## Context
- Use only files referenced in the prompt unless told otherwise.
- For large changes, propose a plan before editing.

This helps your local assistant behave consistently across tasks and teammates.

Limitations to expect

If you want the honest version, here it is:

• Hardware matters: RAM and VRAM limit which models you can run.
• Expect tradeoffs between speed, quality, and context length.
• Local models still need clear prompts and scoped tasks.

Alternatives if you prefer other editors

• Neovim with local LLM tooling
• JetBrains IDEs with Continue
• Other local assistants that support Ollama

Bottom line

For most teams, VSCodium + Continue + Ollama is the right local LLM coding setup: private by default, zero subscription cost, and trivial to reconfigure as better models appear. Add cloud models only when you genuinely need their extra capability.

Frequently asked questions

Can I use Ollama without a GPU?

Yes. Ollama runs on CPU-only machines. On 8 GB RAM without a GPU you can run 3B–4B models at 2–6 tokens/sec. Slower than cloud, but fully private and sufficient for completions and short edits.

Is a local LLM as good as GitHub Copilot?

For daily coding tasks - completions, refactors, code explanation - current local models like gemma4:e4b, qwen3:7b, and qwen3-coder:30b cover a large share of Copilot-style work with zero subscription cost and full privacy. Cloud coding agents still win on very large repo context and hard multi-file planning. For teams who prioritise data privacy, this local LLM coding setup is a strong GitHub Copilot alternative.

What is the best model for coding with Ollama?

On 16 GB RAM with a 10 GB GPU, start with gemma4:e4b or qwen3:7b-q8_0 for reliable tool calling. On 24 GB VRAM, use qwen3-coder:30b for agentic coding with 256K context. The table above lists all Ollama-native options confirmed for tool calling reliability.

Does VSCodium work with Continue.dev?

Yes. Continue is published to the Open VSX marketplace that VSCodium uses instead of the Microsoft marketplace. Install it from the Extensions panel inside VSCodium and point it at your Ollama endpoint (http://localhost:11434).

For production automation use cases, see our guide on running a self-hosted LLM in production with Ollama and n8n.

Ready to go further?

This is how we build at Vasilkoff: privacy-first, open-source friendly, and practical. If your team needs a private AI coding assistant integrated into your development workflow — or broader AI development services — we can help. Reach out via our contact page.