What is ggml?

ggml is a machine learning tensor computation library written in C/C++. Created by Georgi Gerganov — the name comes from his initials (GG).

Why does it exist?

Frameworks like PyTorch and TensorFlow are heavy. They require a Python runtime, CUDA dependencies, and gigabytes of libraries. According to its README, ggml aims to be a simple, portable, and efficient tensor library with minimal setup.

• No external dependencies (C/C++)
• SIMD-optimized (AVX on x86, NEON on ARM, etc.)
• Model quantization to greatly reduce memory usage
• Multiple backends including CPU and GPU; LLMs, speech models, etc. can run locally without a GPU

Relationship with whisper.cpp

whisper.cpp runs the OpenAI Whisper model on top of ggml. Both were created by Georgi Gerganov.

OpenAI Whisper (Python/PyTorch)
        ↓ model conversion
ggml format model file (.bin)
        ↓
whisper.cpp (C/C++ inference engine)
   └── ggml (tensor computation library)

In other words, ggml is the computation backend for whisper.cpp. It handles the low-level operations needed for neural networks: matrix multiplication, convolution, attention, etc.

ggml model format

PyTorch .pt files cannot be used directly. They must be converted to ggml's own binary format.

• Model weights are stored as ggml tensor structures
• Supports quantization to 16-bit, 8-bit, 4-bit, etc.
• Quantization greatly reduces model size (e.g., fp16 → 4-bit is roughly 1/4)

The conversion scripts in whisper.cpp's models/ directory handle this. Note that llama.cpp now uses a separate file format called GGUF.

ggml ecosystem

Projects built on ggml:

They all follow the same pattern: convert a Python/PyTorch model to a ggml-family format (ggml .bin for whisper.cpp, GGUF for llama.cpp) and run inference in C/C++.

When You Build It Yourself

In whisper-sys/build.rs, we compile the ggml source directly:

// Compile ggml C files
cc::Build::new()
    .files(&[
        "whisper.cpp/ggml/src/ggml.c",
        "whisper.cpp/ggml/src/ggml-alloc.c",
        "whisper.cpp/ggml/src/ggml-quants.c",
    ])
    .compile("ggml");

// Compile ggml C++ files
cc::Build::new()
    .cpp(true)
    .files(&[
        "whisper.cpp/ggml/src/ggml-backend.cpp",
        // ...
    ])
    .compile("ggml-cpp");

Since whisper.cpp depends on ggml, we must compile the ggml source alongside it.