Shimmy

Overview

Shimmy is a Rust-based inference server that is compatible with the OpenAI API. It supports GGUF and SafeTensors formats, offers hot model swapping and auto-discovery, and runs as a single binary with no Python dependency. The tool is free and open source.

Best for

Best for
Developers seeking a free, no-fuss Rust-based inference server with OpenAI API compatibility

Use cases

• Serve language model inferences with an OpenAI-compatible API
• Swap models without restarting the server during development or testing
• Deploy a lightweight, self-contained inference endpoint in a Rust environment

Notes

Shimmy is a Rust-based inference server that is compatible with the OpenAI API. It supports GGUF and SafeTensors formats, offers hot model swapping and auto-discovery, and runs as a single binary with no Python dependency. The tool is free and open source.

5,306 stars on GitHub. Last updated 2026-06-01. Licensed Apache-2.0.

Use cases

• Serve language model inferences with an OpenAI-compatible API
• Swap models without restarting the server during development or testing
• Deploy a lightweight, self-contained inference endpoint in a Rust environment

Pros

• Single binary with no Python runtime required
• Free and open source with a permissive license
• Supports hot model swapping for flexible experimentation

Cons

• Smaller community and fewer integrations compared to more established inference servers
• Limited to GGUF and SafeTensors model formats
• May lack advanced monitoring or logging features found in dedicated observability tools

Indexed from awesome-llmops and enriched against its public facts.