tokenization-huggingface-tokenizers
ComunidadMITFast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.
El ZIP contiene solo este skill. Está en GitHub Releases del repo público orlandoneto/agent-skills — Ontoom es la vitrina. El crawler diario actualiza el catálogo.
Cómo instalar
- Descarga el ZIP y extrae la carpeta del skill.
- Cursor:
.agents/skills/aitmpl-tokenization-huggingface-tokenizers/o.cursor/skills/aitmpl-tokenization-huggingface-tokenizers/ - Claude Code:
.claude/skills/aitmpl-tokenization-huggingface-tokenizers/ - Otras IAs:
.agents/skills/aitmpl-tokenization-huggingface-tokenizers/
Origen / créditos: davila7/claude-code-templates
Otros skills de esta colección
- tokenization-sentencepiece
Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
- using-git-worktrees
Use when starting feature work that needs isolation from current workspace or before executing implementation plans - creates isolated git worktrees with smart directory selection and safety verification
- using-neon
Guides and best practices for working with Neon Serverless Postgres. Covers getting started, local development with Neon, choosing a connection method, Neon features, authentication (@neondatabase/auth), PostgREST-style data API (@neondatabase/neon-js), Neon CLI, and Neon's Platform API/SDKs. Use for any Neon-related questions.
- using-superpowers
Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
- vercel-deployment
Expert knowledge for deploying to Vercel with Next.js Use when: vercel, deploy, deployment, hosting, production.
- verification-before-completion
Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always