← All skills

tokenization-sentencepiece

CommunityMIT

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

The ZIP contains only this skill. It lives on GitHub Releases of the public orlandoneto/agent-skills repo — Ontoom is the showcase. A daily crawler refreshes the catalog.

How to install

  1. Download the ZIP and extract the skill folder.
  2. Cursor: .agents/skills/aitmpl-tokenization-sentencepiece/ or .cursor/skills/aitmpl-tokenization-sentencepiece/
  3. Claude Code: .claude/skills/aitmpl-tokenization-sentencepiece/
  4. Other AIs: .agents/skills/aitmpl-tokenization-sentencepiece/
cursorclaude-codecodexgeminiany

Source / credits: davila7/claude-code-templates

Other skills in this collection

Claude Code Templates

  • using-git-worktrees

    Use when starting feature work that needs isolation from current workspace or before executing implementation plans - creates isolated git worktrees with smart directory selection and safety verification

  • using-neon

    Guides and best practices for working with Neon Serverless Postgres. Covers getting started, local development with Neon, choosing a connection method, Neon features, authentication (@neondatabase/auth), PostgREST-style data API (@neondatabase/neon-js), Neon CLI, and Neon's Platform API/SDKs. Use for any Neon-related questions.

  • using-superpowers

    Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions

  • vercel-deployment

    Expert knowledge for deploying to Vercel with Next.js Use when: vercel, deploy, deployment, hosting, production.

  • verification-before-completion

    Use when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions always

  • viral-generator-builder

    Expert in building shareable generator tools that go viral - name generators, quiz makers, avatar creators, personality tests, and calculator tools. Covers the psychology of sharing, viral mechanics, and building tools people can't resist sharing with friends. Use when: generator tool, quiz maker, name generator, avatar creator, viral tool.