tokenization-sentencepiece
CommunityMITLanguage-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.
The ZIP contains only this skill. It lives on GitHub Releases of the public orlandoneto/agent-skills repo — Ontoom is the showcase. A daily crawler refreshes the catalog.
How to install
- Download the ZIP and extract the skill folder.
- Cursor:
.agents/skills/aitmpl-tokenization-sentencepiece/or.cursor/skills/aitmpl-tokenization-sentencepiece/ - Claude Code:
.claude/skills/aitmpl-tokenization-sentencepiece/ - Other AIs:
.agents/skills/aitmpl-tokenization-sentencepiece/
Source / credits: davila7/claude-code-templates
Other skills in this collection
- using-git-worktrees
Use when starting feature work that needs isolation from current workspace or before executing implementation plans - creates isolated git worktrees with smart directory selection and safety verification
- using-neon
Guides and best practices for working with Neon Serverless Postgres. Covers getting started, local development with Neon, choosing a connection method, Neon features, authentication (@neondatabase/auth), PostgREST-style data API (@neondatabase/neon-js), Neon CLI, and Neon's Platform API/SDKs. Use for any Neon-related questions.
- using-superpowers
Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions
- vercel-deployment
Expert knowledge for deploying to Vercel with Next.js Use when: vercel, deploy, deployment, hosting, production.
- verification-before-completion
Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always
- viral-generator-builder
Expert in building shareable generator tools that go viral - name generators, quiz makers, avatar creators, personality tests, and calculator tools. Covers the psychology of sharing, viral mechanics, and building tools people can't resist sharing with friends. Use when: generator tool, quiz maker, name generator, avatar creator, viral tool.