← All skills

inference-serving-llama-cpp

CommunityMIT

Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware. Use for edge deployment, M1/M2/M3 Macs, AMD/Intel GPUs, or when CUDA is unavailable. Supports GGUF quantization (1.5-8 bit) for reduced memory and 4-10× speedup vs PyTorch on CPU.

The ZIP contains only this skill. It lives on GitHub Releases of the public orlandoneto/agent-skills repo — Ontoom is the showcase. A daily crawler refreshes the catalog.

How to install

  1. Download the ZIP and extract the skill folder.
  2. Cursor: .agents/skills/aitmpl-inference-serving-llama-cpp/ or .cursor/skills/aitmpl-inference-serving-llama-cpp/
  3. Claude Code: .claude/skills/aitmpl-inference-serving-llama-cpp/
  4. Other AIs: .agents/skills/aitmpl-inference-serving-llama-cpp/
cursorclaude-codecodexgeminiany

Other skills in this collection

Other collections

  • inference-serving-sglang

    Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.

  • inference-serving-tensorrt-llm

    Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

  • inference-serving-vllm

    Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.

  • infrastructure-lambda-labs

    Reserved and on-demand GPU cloud instances for ML training and inference. Use when you need dedicated GPU instances with simple SSH access, persistent filesystems, or high-performance multi-node clusters for large-scale training.

  • infrastructure-modal

    Serverless GPU cloud platform for running ML workloads. Use when you need on-demand GPU access without infrastructure management, deploying ML models as APIs, or running batch jobs with automatic scaling.

  • infrastructure-skypilot

    Multi-cloud orchestration for ML workloads with automatic cost optimization. Use when you need to run training or batch jobs across multiple clouds, leverage spot instances with auto-recovery, or optimize GPU costs across providers.