⚡Megatron LLM Hub
←Back to search
Twitter

llm-router v0.1.0 把多台本地推理机聚合到一个 OpenAI 兼容端点

统一入口简化自托管多机推理,但单点暴露全部本地模型。
工具llm自托管
RT @TeksEdge: Your home AI lab can now look like one big model server. A new open-source project, llm-router v0.1.0 puts, one OpenAI-compatible API endpoint in front of multiple local inference boxes. For example, you could have 🖥️ PC #1 → RTX 5090 → Qwen 🖥️ PC #2 → multi-GPU rig → GLM 🍎 Mac → LM Studio model 🖥️ Server → vLLM models And your apps see only → 1 API endpoint llm-router discovers which models each server is hosting and automatically sends the request to the machine holding the exact model ID. It supports the following harnesses ⚙️ LM Studio ⚙️ vLLM ⚙️ TensorFold Including these services → streaming → chat completions → Responses API → embeddings → Prometheus metrics → Docker → amd64 + arm64 MIT-licensed and written in Go. Link in ALT