Top 20 LLM inference engines

nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

77,353 stars 8,266 forks775 open issuesC++last push 2025-05-27MIT

ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

44,088 stars 8,176 forks3,609 open issuesPythonlast push 2026-10-20Apache-2.0

gitleaks/gitleaks

Find secrets with Gitleaks 🔑

30,158 stars 2,429 forks508 open issuesGolast push 2026-09-30MIT

liguodongiot/llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

25,188 stars 3,005 forks20 open issuesHTMLlast push 2026-10-14Apache-2.0

Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

13,719 stars 1,691 forks315 open issuesPythonlast push 2026-10-13Apache-2.0

bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

12,561 stars 1,014 forks21 open issuesPythonlast push 2026-10-20Apache-2.0

openvinotoolkit/openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

11,029 stars 3,470 forks882 open issuesC++last push 2026-10-20Apache-2.0

Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

9,841 stars 636 forks133 open issuesC++last push 2026-05-11MIT

FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

9,126 stars 1,489 forks4 open issuesClast push 2026-10-02Apache-2.0

bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

8,913 stars 1,047 forks234 open issuesPythonlast push 2026-10-20Apache-2.0

ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

8,342 stars 1,729 forks2,096 open issuesRustlast push 2026-10-20NOASSERTION

InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

8,108 stars 789 forks617 open issuesPythonlast push 2026-10-20Apache-2.0

algorithmicsuperintelligence/openevolve

Open-source implementation of AlphaEvolve

7,605 stars 1,191 forks95 open issuesPythonlast push 2026-10-19Apache-2.0

Aussi dans: #11 GitHub projects around AI coding agents

katanemo/plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay...

7,095 stars 488 forks147 open issuesRustlast push 2026-10-07Apache-2.0

drumih/turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

6,953 stars 470 forks53 open issuesSwiftlast push 2026-10-15Apache-2.0

bojieli/ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

6,790 stars 529 forks8 open issuesPythonlast push 2026-10-01Apache-2.0

flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

6,721 stars 1,620 forks1,273 open issuesCudalast push 2026-10-20Apache-2.0

kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

6,136 stars 1,743 forks211 open issuesGolast push 2026-10-20Apache-2.0

cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

6,135 stars 519 forks115 open issuesC++last push 2026-09-26NOASSERTION

Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

5,988 stars 587 forks10 open issuesRustlast push 2026-10-19Apache-2.0