← Todos los skills

tokenization-huggingface-tokenizers

ComunidadMIT

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

El ZIP contiene solo este skill. Está en GitHub Releases del repo público orlandoneto/agent-skills — Ontoom es la vitrina. El crawler diario actualiza el catálogo.

Cómo instalar

  1. Descarga el ZIP y extrae la carpeta del skill.
  2. Cursor: .agents/skills/aitmpl-tokenization-huggingface-tokenizers/ o .cursor/skills/aitmpl-tokenization-huggingface-tokenizers/
  3. Claude Code: .claude/skills/aitmpl-tokenization-huggingface-tokenizers/
  4. Otras IAs: .agents/skills/aitmpl-tokenization-huggingface-tokenizers/
cursorclaude-codecodexgeminiany

Origen / créditos: davila7/claude-code-templates

Otros skills de esta colección

Claude Code Templates

  • tokenization-sentencepiece

    Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

  • using-git-worktrees

    Use when starting feature work that needs isolation from current workspace or before executing implementation plans - creates isolated git worktrees with smart directory selection and safety verification

  • using-neon

    Guides and best practices for working with Neon Serverless Postgres. Covers getting started, local development with Neon, choosing a connection method, Neon features, authentication (@neondatabase/auth), PostgREST-style data API (@neondatabase/neon-js), Neon CLI, and Neon's Platform API/SDKs. Use for any Neon-related questions.

  • using-superpowers

    Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions

  • vercel-deployment

    Expert knowledge for deploying to Vercel with Next.js Use when: vercel, deploy, deployment, hosting, production.

  • verification-before-completion

    Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always