NCNN

Overview

NCNN is a C++ neural network inference framework optimized for mobile and embedded devices. It prioritizes low latency and minimal memory footprint, enabling on-device model execution without cloud dependencies. The framework supports quantization and model compression to fit resource-constrained environments.

Best for

Best for
Mobile and embedded developers building latency-critical inference applications on constrained hardware

Use cases

• Running computer vision models on Android and iOS without server calls
• Deploying lightweight NLP inference on edge devices
• Building real-time mobile applications with local model inference

Notes

NCNN is a C++ neural network inference framework optimized for mobile and embedded devices. It prioritizes low latency and minimal memory footprint, enabling on-device model execution without cloud dependencies. The framework supports quantization and model compression to fit resource-constrained environments.

23,318 stars on GitHub. Last updated 2026-05-30.

Use cases

• Running computer vision models on Android and iOS without server calls
• Deploying lightweight NLP inference on edge devices
• Building real-time mobile applications with local model inference

Pros

• Extremely fast inference on mobile CPUs with minimal memory overhead
• No external dependencies, pure C++ implementation
• Strong community support with 23k+ GitHub stars and active maintenance

Cons

• Steep learning curve for developers unfamiliar with C++ and mobile deployment
• Limited built-in support for dynamic shapes and complex control flow
• Smaller ecosystem compared to TensorFlow Lite or ONNX Runtime

Indexed from awesome-llmops and enriched against its public facts.