Top 20 LLM inference engines

nomic-ai/gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

77,351 stars 8,265 forks775 open issuesC++last push 2025-05-27MIT

ray-project/ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

44,076 stars 8,174 forks3,605 open issuesPythonlast push 2026-10-19Apache-2.0

gitleaks/gitleaks

Find secrets with Gitleaks 🔑

30,131 stars 2,411 forks505 open issuesGolast push 2026-09-30MIT

liguodongiot/llm-action

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

25,186 stars 2,978 forks20 open issuesHTMLlast push 2026-10-14Apache-2.0

Lightning-AI/litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

13,718 stars 1,664 forks315 open issuesPythonlast push 2026-10-13Apache-2.0

bentoml/OpenLLM

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

12,560 stars 999 forks21 open issuesPythonlast push 2026-10-19Apache-2.0

openvinotoolkit/openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

11,026 stars 3,472 forks876 open issuesC++last push 2026-10-19Apache-2.0

Tiiny-AI/PowerInfer

High-speed Large Language Model Serving for Local Deployment

9,839 stars 630 forks133 open issuesC++last push 2026-05-11MIT

FareedKhan-dev/kimi-k3-in-c

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

9,110 stars 1,486 forks4 open issuesClast push 2026-10-02Apache-2.0

bentoml/BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

8,909 stars 1,046 forks235 open issuesPythonlast push 2026-10-05Apache-2.0

ai-dynamo/dynamo

A Datacenter Scale Distributed Inference Serving Framework

8,332 stars 1,727 forks2,064 open issuesRustlast push 2026-10-19NOASSERTION

InternLM/lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

8,108 stars 787 forks614 open issuesPythonlast push 2026-10-19Apache-2.0

algorithmicsuperintelligence/openevolve

Open-source implementation of AlphaEvolve

7,600 stars 1,190 forks96 open issuesPythonlast push 2026-10-19Apache-2.0

موجود أيضًا في: #11 GitHub projects around AI coding agents

katanemo/plano

Plano is an AI-native proxy server and data plane for agentic apps. Smart LLM routing, observability, agent orchestration, and guardrails so you stay...

7,095 stars 488 forks147 open issuesRustlast push 2026-10-07Apache-2.0

drumih/turbo-fieldfare

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

6,944 stars 467 forks53 open issuesSwiftlast push 2026-10-15Apache-2.0

bojieli/ai-infra-book

《深入理解 AI Infra：量化分析与系统设计》（李博杰 著）开源书稿：从硬件约束和模型架构出发，量化推导 LLM 推理与训练系统设计。含全书正文、PDF、配套计算工具与实验

6,733 stars 522 forks8 open issuesPythonlast push 2026-10-01Apache-2.0

flashinfer-ai/flashinfer

FlashInfer: Kernel Library for LLM Serving

6,708 stars 1,615 forks1,270 open issuesCudalast push 2026-10-19Apache-2.0

kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

6,133 stars 1,741 forks208 open issuesGolast push 2026-10-19Apache-2.0

cactus-compute/cactus

Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.

6,131 stars 518 forks115 open issuesC++last push 2026-09-26NOASSERTION

Michael-A-Kuykendall/shimmy

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

5,986 stars 587 forks10 open issuesRustlast push 2026-10-19Apache-2.0