Top 20 LLM inference engines

Serve open models on your own hardware

Actualizado 26 min ago · GitHub stars (stars)

Por qué estos 20

Ranked by GitHub stars: how many developers bookmarked the project. Stars measure attention, not quality, so each row also shows forks, open issues, licence and the last push.

1

nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

77,354 stars 8,267 forks775 open issuesC++last push 2025-05-27MIT
2

ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

44,087 stars 8,174 forks3,611 open issuesPythonlast push 2026-10-20Apache-2.0
3

gitleaks/gitleaks

Find secrets with Gitleaks 🔑

30,147 stars 2,429 forks506 open issuesGolast push 2026-09-30MIT
4

liguodongiot/llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

25,185 stars 3,005 forks20 open issuesHTMLlast push 2026-10-14Apache-2.0
5

Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

13,719 stars 1,691 forks315 open issuesPythonlast push 2026-10-13Apache-2.0
6

bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

12,561 stars 1,014 forks21 open issuesPythonlast push 2026-10-20Apache-2.0
7

openvinotoolkit/openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

11,027 stars 3,472 forks882 open issuesC++last push 2026-10-20Apache-2.0
8

Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

9,840 stars 636 forks133 open issuesC++last push 2026-05-11MIT
9

FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

9,122 stars 1,487 forks4 open issuesClast push 2026-10-02Apache-2.0
10

bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

8,911 stars 1,046 forks234 open issuesPythonlast push 2026-10-20Apache-2.0
11

ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

8,339 stars 1,731 forks2,097 open issuesRustlast push 2026-10-20NOASSERTION
12

InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

8,108 stars 789 forks618 open issuesPythonlast push 2026-10-20Apache-2.0
13

algorithmicsuperintelligence/openevolve

Open-source implementation of AlphaEvolve

7,603 stars 1,190 forks95 open issuesPythonlast push 2026-10-19Apache-2.0

También en: #11 GitHub projects around AI coding agents
14

katanemo/plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay...

7,095 stars 488 forks147 open issuesRustlast push 2026-10-07Apache-2.0
15

drumih/turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

6,947 stars 469 forks53 open issuesSwiftlast push 2026-10-15Apache-2.0
16

bojieli/ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

6,773 stars 523 forks8 open issuesPythonlast push 2026-10-01Apache-2.0
17

flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

6,717 stars 1,621 forks1,251 open issuesCudalast push 2026-10-20Apache-2.0
18

kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

6,135 stars 1,743 forks208 open issuesGolast push 2026-10-20Apache-2.0
19

cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

6,133 stars 519 forks115 open issuesC++last push 2026-09-26NOASSERTION
20

Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

5,987 stars 587 forks10 open issuesRustlast push 2026-10-19Apache-2.0