统一入口简化自托管多机推理,但单点暴露全部本地模型。
工具llm自托管
RT @TeksEdge: Your home AI lab can now look like one big model server.
A new open-source project, llm-router v0.1.0 puts, one OpenAI-compatible API endpoint in front of multiple local inference boxes.
For example, you could have
🖥️ PC #1 → RTX 5090 → Qwen
🖥️ PC #2 → multi-GPU rig → GLM
🍎 Mac → LM Studio model
🖥️ Server → vLLM models
And your apps see only
→ 1 API endpoint
llm-router discovers which models each server is hosting and automatically sends the request to the machine holding the exact model ID.
It supports the following harnesses
⚙️ LM Studio
⚙️ vLLM
⚙️ TensorFold
Including these services
→ streaming
→ chat completions
→ Responses API
→ embeddings
→ Prometheus metrics
→ Docker
→ amd64 + arm64
MIT-licensed and written in Go.
Link in ALT