The 2026 open-source AI stack we'd actually build on

The first half of 2026 was loud. DeepSeek shipped V4 — a million-token context window, MIT-licensed, near-frontier on coding. Hugging Face archived TGI, the inference server half the tutorials still tell you to run. Microsoft folded AutoGen into a brand-new agent framework. OpenAI bought promptfoo. And MCP went from an interesting Anthropic protocol to the thing every serious agent runtime now speaks.

What didn't move: the stack I'd actually build on to ship a real AI product. Still six layers, still mostly the same picks as a year ago. That's the case for building on open source you can run and read — the headlines churn weekly, the foundation doesn't. (Cutting the noise is the whole point.) Here's the stack, layer by layer, with the one thing 2026 changed about each.

Layer	The pick	Reach for more when…
Models	Qwen3.6 or Llama 4 (open-weight)	reasoning-heavy: DeepSeek-V4 · permissive license: Mistral or Qwen (Apache-2.0)
Serving	vLLM or SGLang	local & dev: Ollama or llama.cpp
Orchestration	a thin, typed loop (Pydantic AI)	durable multi-step state: LangGraph
Retrieval	pgvector	scale or latency demands it: Qdrant
Evals & observability	Langfuse + promptfoo	RAG-specific metrics: Ragas
The glue	LiteLLM + Pydantic + MCP	structured output: Instructor, Outlines

Models: open-weight first

Everything sits on this layer, so start here. "Hosted frontier or nothing" is a false choice now, and 2026 settled it: DeepSeek's V4, out in April under MIT with a million-token context, lands close enough to the frontier on coding and agentic work that "we can't use open models for the hard stuff" stopped being a real objection.

My default is Qwen3.6 for the best quality per parameter across a range of sizes, or Llama 4 when you want the deepest ecosystem — the most fine-tunes, quantizations, and tooling built around one family. One catch worth knowing: the newest Qwen, 3.7-Max, is closed and API-only; the latest open Qwen is 3.6. "Latest" and "open" stopped being the same release, so check which one a benchmark is quoting.

When a task is genuinely reasoning-heavy, reach for DeepSeek-V4 — it wears the open reasoning crown R1 had a year ago, with configurable reasoning effort so you don't pay for deliberation you didn't need. When you want a clean permissive license, Mistral's open LLMs (Small 4 and Large 3) ship under Apache-2.0, though watch the fine print — Mistral's "Premier" models are API-only.

One licensing landmine, because it's easy to get wrong: Llama 4 is not the permissive option. Its community license caps you at 700M monthly users and restricts some EU multimodal use. For most teams that ceiling never bites, but if "open" to you means "Apache or MIT, no asterisks," reach for Qwen, DeepSeek, or Mistral instead of Llama.

So when does open actually beat the hosted frontier? Four cases: control (the model won't change under you mid-quarter), privacy (data and weights stay on hardware you own), cost at scale (past some volume, your own GPUs beat per-token pricing), and customization (a fine-tune on your data is yours to keep). When none of those apply, a hosted API is often the right call, and saying so is what no-hype means. Treat open-first as a default, then override it the moment one of those four cases stops holding.

Inference and serving: without lighting money on fire

A model you can't serve efficiently is a science project. For production GPU serving, the answer is now two names instead of one: vLLM and SGLang. vLLM is still the default — continuous batching and paged attention give you the throughput that makes self-hosting economical, and it speaks the OpenAI API, so the rest of your stack never has to know what's behind it. SGLang has pulled even for a lot of workloads, especially if you're chasing latency or want new models on day one (it was serving DeepSeek-V4 the day it dropped).

The change to note: TGI is done. Hugging Face archived text-generation-inference in March and now points you to vLLM, SGLang, or llama.cpp instead. If a guide still reaches for TGI, that's how you know it's stale.

For local development, prototyping, and edge, Ollama and llama.cpp run quantized models on a laptop with near-zero ceremony — Ollama added an MLX engine for Apple Silicon this year, a real speedup if you're on a Mac. The rule of thumb holds: prototype on Ollama, ship on vLLM or SGLang, and measure tokens per dollar before you scale anything.

Orchestration: less framework than you think

This is where the most money and sanity get burned. The hype says you need an agent framework with a graph, a DSL, and a dozen abstractions. For most applications, you don't. My default is a thin, typed loop you own, built on the provider SDK, with Pydantic AI (now at 2.0) supplying typed inputs and outputs without seizing your control flow. You can read it, step through it in a debugger, and it does exactly what you wrote and nothing you didn't.

The one genuinely new requirement in 2026 is that the loop should speak MCP. The Model Context Protocol won the interop fight this year — Anthropic donated it to a Linux Foundation body in December — and hand-wiring bespoke tool integrations now means reinventing a standard the rest of the ecosystem already shares.

Reach for LangGraph (1.0 shipped last October) when you genuinely need what it offers: durable, stateful, multi-step graphs with checkpointing and human-in-the-loop pauses. If your application is fundamentally retrieval over your own documents, LlamaIndex earns its keep. CrewAI is the pick when you want role-based crews out of the box.

One name to update: AutoGen. Microsoft folded it, together with Semantic Kernel, into the new Microsoft Agent Framework (1.0, April 2026); standalone AutoGen is maintenance-only now. If that was your reach, that's where it lives. And stay slow to adopt any heavyweight multi-agent framework until you've felt the specific pain it solves — a framework you added on speculation is a dependency you'll spend next quarter fighting. (For how the agent inside that loop actually behaves, the field guide to AI coding agents is the companion piece.)

Retrieval: the boring parts are the ones that matter

Everyone reaches for a dedicated vector database first. Most teams don't need one. If you already run Postgres — and you probably do — pgvector keeps your embeddings next to your data, in one system you already know how to back up, query, and operate. Graduate to Qdrant when scale or latency genuinely demand it: it's fast, written in Rust, and its metadata filtering is excellent. Weaviate is a fine alternative in the same tier, and it shipped a built-in MCP server this year if that's the direction you're heading.

The database is the easy part. Retrieval quality lives in the unglamorous details. Hybrid search — vector similarity plus plain keyword/BM25 ranking, and Postgres full-text search is right there — beats pure-vector search on real queries far more often than the demos admit. Put a reranker (a cross-encoder) on top to reorder the shortlist and you've fixed most of what makes RAG feel stupid. Chunk well, search hybrid, then rerank. That sequence is the actual work.

Evals and observability: how you know it works

You can't improve what you can't see, and "it looked fine when I tried it" is not a measurement. Langfuse — self-hostable, MIT-licensed — traces every call: prompts, completions, latencies, costs. Production stops being a black box. For evaluation, promptfoo is a CI-friendly harness: write test cases, run them on every change, and catch regressions before your users do. (OpenAI acquired promptfoo in March — still open source, but factor it in if vendor neutrality matters.) For RAG, Ragas scores the retrieval-specific things — faithfulness, answer relevance — that generic evals miss, and it's grown past pure RAG into agent evaluation. Phoenix from Arize is a strong pick if you'd rather have evaluation and observability in one place.

The discipline matters more than the brand name: trace everything in production, and never ship a prompt or model change you didn't evaluate. This is the layer teams skip, and the one that separates a demo from a product — we made the full case in write the eval before the prompt.

The glue: the small tools that keep it standing

The unglamorous layer that quietly decides whether the whole thing stays maintainable. LiteLLM gives you one API across every provider and open model, so swapping Qwen for Llama — or a self-hosted endpoint for a hosted one — costs you one line of config; it's still the default self-hosted gateway. Pydantic (v2 — v3 hasn't shipped yet, roadmap chatter notwithstanding) is the backbone of trustworthy I/O. Instructor and Outlines turn "the model usually returns valid JSON" into "the model returns valid JSON," with Instructor settled in as the lightweight extraction tool and Outlines as constrained generation. Wrap the app in FastAPI, package it with Docker, and you've got something a team can actually run on a Tuesday.

The throughline

Read the picks back to back and the pattern is obvious: every one favors control and legibility over magic. Open weights you can host. A server you can afford. A loop you can read and step through. Retrieval you can debug. Evals you can trust. Glue thin enough to swap any piece without a rewrite. None of these are the flashiest thing on the shelf, and that's exactly why they're still here.

So here's the honest 2026 scorecard. What changed: DeepSeek V4 took the open reasoning crown, TGI got archived, AutoGen became the Microsoft Agent Framework, promptfoo got bought, and MCP turned into table stakes. What didn't: six layers, the same six, each one something you can run and read and own. Half these names will be different again by next summer. Bet on the shape, not the logos — pick open where it buys you something real, measure everything, and keep the glue thin.