Is it worth adopting? A 20-minute rubric for any open-source AI tool

It shows up in someone's channel most weeks: a link, and the question "should we use this?" On the other end is an open-source AI tool — a serving runtime, an eval harness, yet another agent framework — and the reflex answer is a skim of the README, a glance at the star count, and a gut call.

Stars are the weakest signal in that pile. They count who liked the launch, not whether the project is still standing the night it pages you at 2am. Popularity and survival are different things, and the distance between them is where adoption regret lives.

So here's the pass I run before anything earns a slot in the stack. Not a week-long bake-off, just one time-boxed sweep, the same instinct behind everything here: keep the good stuff, skip the noise. Six axes, a scorecard, a go/no-go checklist, about twenty minutes. Run it on your own shortlist.

Why a rubric beats a vibe

A dependency is a liability that shows up dressed as a quick win. The README shows you the project on its best day; you adopt it on its worst — the breaking change with no migration guide, the maintainer who went quiet, the license clause legal flags the week before launch. Adopt the wrong one and the bill isn't the afternoon you spent wiring it in. It's the quarter you spend tearing it back out, plus the incident that forced your hand.

So the check is worth running. It gets skipped because "evaluate it properly" sounds like a week of work, so it loses to "looks fine, ship it." Shrink it until it survives a deadline: twenty minutes, short enough to run every time and structured enough to mean something. You're not auditing the codebase; you're sampling six signals that predict regret.

The six axes

Six questions. Score each from 0 to 2 — red, yellow, green — against the signals below.

Axis	The question it answers
License	Can I legally use, modify, and ship this?
Maintenance velocity	Is it alive, or quietly coasting?
Bus factor	What happens if the maintainer walks away?
Escape hatch	How expensive is it to rip out?
Docs honesty	Do the docs match reality?
Prod fit	Will it survive my scale, latency, and ops?

Axis 1 · License

What to inspect: the LICENSE file, at the root and in any nested dirs. Open-core projects love to tuck a stricter one under /enterprise. Read the SPDX or OSI identifier, not the marketing word "open." Source-available licenses aren't open source, whatever the homepage claims. The question underneath: can you use, modify, and ship this the way you actually plan to?

For AI tools the bar is higher than a code license. OSI's Open Source AI Definition (OSAID 1.0, October 2024) sets "open" for a model at open weights, the training code, and enough detail about the training data to rebuild it. Apache-2.0 Python wrapped around weights you can't legally fine-tune doesn't clear that bar, so check the weights and the data terms alongside the license file.

• Green (2) — a single, clear, OSI-approved permissive or weak-copyleft license (MIT, Apache-2.0, BSD-3-Clause, MPL-2.0), patent grant included if it's Apache.
• Yellow (1) — strong copyleft (GPL/AGPL), fine internally but needs legal eyes before you distribute, or a dual-license whose open tier has real limits.
• Red (0) — source-available (BSL, SSPL), no LICENSE file, or a bespoke license no lawyer has seen. Score missing or custom licenses red.

Not hypothetical: Redis relicensed to SSPL in March 2024 and went red by this test overnight, then added an AGPLv3 option in Redis 8 (May 2025). A license is a decision a company can revisit after you've built on it.

Axis 2 · Maintenance velocity

What to inspect: commit cadence, release rhythm, and how fast issues and PRs get a human reply. One question: alive, or quietly coasting toward abandonment?

• Green (2) — commits in the last few weeks, regular tagged releases, maintainers answering issues in days.
• Yellow (1) — activity exists but it's lumpy; releases are sporadic; issues get answered eventually.
• Red (0) — last release over a year ago, a wall of stale issues, PRs rotting unreviewed. A popular-but-dead project is still dead.

Axis 3 · Bus factor

What to inspect: contributor concentration. Open the insights or contributors view and see who actually merges code. The bus factor is how many people have to walk before the project stalls. For a frightening number of beloved tools, the answer is one.

One person doing all the merges is the obvious risk. A single company owning the project is the quieter one: governance is theirs to change, and they can relicense or get acquired by someone who will. HashiCorp put Terraform under the BSL in 2023, and IBM closed its acquisition of HashiCorp in February 2025: plenty of contributors, one owner setting the terms.

• Green (2) — multiple active maintainers, or a foundation/company with real governance behind it (a GOVERNANCE.md, a security policy).
• Yellow (1) — one dominant maintainer, but a genuine contributor community and some sign of succession.
• Red (0) — a single hero account doing nearly all the work, no governance, no plan for the day they burn out.

Axis 4 · Escape hatch

What to inspect: the cost of ripping it out. Does it sit behind a standard interface — an OpenAI-compatible endpoint, OpenTelemetry, plain SQL — or its own bespoke API? Can you get your data back out? Could you fork and self-host if upstream goes sideways? This is the "thin glue, swap any piece" instinct behind the open-source AI stack we'd actually build on.

The fork is the ultimate escape hatch, and recent history says it's a real one. When Redis went SSPL, the community shipped Valkey (BSD, now under the CNCF); when Terraform went BSL, OpenTofu (MPL-2.0, also CNCF) carried on. Both were possible only because a permissive license and portable interfaces left the door open. That's the pair to confirm before you commit.

• Green (2) — standard, portable interfaces, clean data export, a forkable license. Leaving is a config change.
• Yellow (1) — some lock-in, but a documented migration path and exportable state.
• Red (0) — proprietary formats, no export, deep API entanglement. Adopting it is a one-way door.

Axis 5 · Docs honesty

What to inspect: whether the docs describe software that actually exists. Run the quickstart on a clean machine. Are limitations and failure modes named, or is it all happy path and aspirational roadmap?

• Green (2) — quickstart works first try on a fresh environment, limitations stated plainly, docs versioned to releases.
• Yellow (1) — mostly right but stale in spots; you fill the gaps from issues or source.
• Red (0) — quickstart fails, docs describe a version that doesn't exist, or "coming soon" is doing the heavy lifting.

Axis 6 · Prod fit

What to inspect: your reality — scale, latency, security, ops — not the demo's. Is there a real deployment story (config, upgrades, resource needs, a security contact), and a way to see inside it in production (logs, metrics, health checks)?

• Green (2) — documented production deployments, observability hooks, sane defaults, a stated security policy.
• Yellow (1) — runs in prod with effort; you'll wire up your own monitoring and harden the defaults.
• Red (0) — demo-grade only, no ops story, no way to observe it when it misbehaves.

The scorecard

Score each axis against the signals above and write the number down. Committing to a number is what turns a vibe into a decision you can defend in a review.

Axis	Score (0–2)
License	___
Maintenance velocity	___
Bus factor	___
Escape hatch	___
Docs honesty	___
Prod fit	___
Total	___ / 12

Reading the total: 10–12 is a strong yes; 7–9 is a yes with your eyes open, against named gaps you'll mitigate; below 7 is a no for anything you'd genuinely depend on. The total ranks your options. The checklist below is what actually gates them.

Weighting it for your context: decide which axes can veto before you score. A CLI you'll run in CI and could swap in an afternoon? License is about the only thing that can sink it. A vector store holding production data for three years? Escape hatch and prod fit turn non-negotiable, and a zero on either ends it whatever the total says. Pick those vetoes up front, before a good demo talks you out of them.

The 12-point go/no-go checklist

The scorecard ranks; the checklist vetoes. Twelve yes/no questions, two per axis. The rule is blunt on purpose: any hard "no" is a stop, not a deduction. You can't average it away, because these are the things you can't fix after adoption.

1. License — Is there a LICENSE file with a clear OSI-approved or SPDX-identified license?
2. License — Does it permit commercial use, modification, and distribution the way you intend to ship?
3. Velocity — Has there been meaningful activity — a release or real commits — in the last six months?
4. Velocity — Do recent issues and PRs get a response from a maintainer?
5. Bus factor — Would the project survive its top contributor walking away tomorrow (multiple maintainers or governance backing)?
6. Bus factor — Is there a stated process for reporting bugs and security issues?
7. Escape hatch — Could you replace it within a sprint — standard interfaces, exportable data, no hard lock-in?
8. Escape hatch — Could you fork and self-host it if upstream stalled?
9. Docs — Did the quickstart actually work on a clean machine?
10. Docs — Do the docs name real limitations, not just the happy path?
11. Prod fit — Is there a real deployment and operations story, not just a demo?
12. Prod fit — Can you observe it in production when it misbehaves?

Twelve yeses and you have a real candidate. One honest no — especially on license or escape hatch — and you keep looking, no matter how good the demo felt.

Run it in 20 minutes

The whole point is a check fast enough that you actually run it. Order of operations, worst-case first so you fail cheap:

• Minutes 0–3 · License. Open the LICENSE and any nested ones, confirm the SPDX identifier, check it against how you ship. If it doesn't fit, stop here. You just saved yourself seventeen minutes.
• Minutes 3–7 · Releases. Skim releases, tags, and the changelog. Recency and rhythm tell you whether it's alive; real tagged releases beat a heap of loose commits.
• Minutes 7–12 · Issues and PRs. Sort by recently updated. Are maintainers replying? How old is the oldest unaddressed serious bug? Glance at the contributor graph for bus factor.
• Minutes 12–20 · A short spike. Clone it into a clean container or fresh virtualenv, run the quickstart, then do the one real thing you'd use it for. Eight minutes of hands-on exposes docs honesty and prod fit better than any README. Scope the spike tight. It's the same discipline that makes handing work to coding agents pay off: shrink it until success is checkable.

Twenty minutes, six scores, twelve answers, and a decision you can put in writing.

The cache

A few things worth keeping — examples, not endorsements, the way we keep everything else here:

• OpenSSF Scorecard automates a chunk of the velocity and bus-factor axes: it runs health and security checks against a repo and hands you a number. Good for triaging a long shortlist before you spend human minutes on it.
• The SPDX license list and OSI — plus choosealicense.com for plain-English summaries — are ground truth for Axis 1. When in doubt about a license, trust the identifier over the marketing.
• CHAOSS has spent years defining community-health metrics: the established vocabulary for "velocity" and "contributor concentration" if you want those axes rigorous instead of eyeballed.

Use them to go faster, not to skip the judgment. Your shortlist turns over fast. Half the names on it today will be gone inside two years. The six questions outlast them, because they measure whether a project will still be standing when you need it. Stars only ever measured the launch.

Run the pass. Write the numbers down. Then decide.