Llama.cpp GGUF

• You want to run an open-weight LLM locally and need help picking the right GGUF quantization for your RAM or VRAM.
• You are searching the Hugging Face Hub for llama.cpp-compatible models and want exact filenames and launch commands.
• You are building a local OpenAI-compatible server with llama-server for app development or private inference.
• You need Python integration via llama-cpp-python for generation, chat, streaming, or embeddings.

• You are working exclusively with cloud-hosted model APIs (OpenAI, Anthropic, etc.) and have no interest in local inference.
• Your hardware has less than ~4 GB of available RAM, making even the smallest GGUF quantizations impractical.

Searches and reads HF model pages directly using structured URL patterns, extracting hardware-specific quant recommendations from the ?local-app=llama.cpp view before falling back to generic guidance.

Queries the HF tree API to list every .gguf file in a repo with exact filename, byte size, and quant label. Separates main model checkpoints from multimodal projector files and BF16 shards.

Produces complete llama-server and llama-cli commands using -hf shorthand or explicit --hf-repo/--hf-file flags. Commands are tailored to the confirmed files in the repo, not generic templates.

Recommends Q4/Q5/Q6/IQ quant variants based on the user's available memory and task type. Preserves repo-native labels (e.g. UD-Q4_K_M) rather than normalizing them.

Provides llama-cpp-python code snippets for basic generation, chat completion with streaming, and embeddings — including GPU build flags for CUDA (GGML_CUDA) and Metal (GGML_METAL).

Supports launching llama-server as a drop-in OpenAI API endpoint, enabling local chat completions for app prototyping without any external API keys or cloud costs.

Run open-weight models entirely on your own hardware. The skill finds a suitable GGUF, confirms it exists in the repo, and gives you the exact command to start serving it — no API keys, no data leaving your machine.

Given a hardware profile (e.g. "16 GB unified memory M2"), the skill reads the HF local-app page and tree API to recommend the best quantization and report the exact file size before you download anything.

Spin up llama-server and point your existing OpenAI-SDK code at http://localhost:8080 to develop and test against a local model before committing to a cloud API.

For vision-language repos, the skill separately identifies the main GGUF and the mmproj-*.gguf projector file, providing the correct combined launch command for multimodal inference.