Experimental BPE tokenizer in Odin using:

• SIMD-assisted UTF-8 pre-tokenization
• structure-of-arrays prompt storage
• arena allocation for batch text buffers
• a flat open-addressed hash grid for BPE merge lookup
• HuggingFace tokenizer.json parsing for BPE and WordPiece models
• HuggingFace normalizer parsing and Unicode NFC/NFD/NFKC/NFKD normalization
• HuggingFace decoder config parsing for ByteLevel, WordPiece, Sequence, and common text-transform decoders
• ByteLevel runtime decoding from token IDs back to UTF-8 bytes/text
• ByteLevel BPE byte mapping, ByteLevel regex chunking, and special-token splitting
• SoA batch fast path for supported ByteLevel and Sequence+ByteLevel tokenizer configs
• private per-thread tokenizer state for no-sharing multi-thread benchmarks

The benchmark harness compares this implementation against HuggingFace tokenizers and fastokens on both synthetic fixtures and real public HuggingFace tokenizer.json files.

The public benchmark covers an end-to-end path:

1. raw UTF-8 text input
2. normalizer and pre-tokenizer execution from tokenizer.json
3. BPE or WordPiece model encoding
4. post-processor insertion where supported
5. token ID output
6. decoder execution back to text bytes

The integrity check compares exact token ID counts, decoded bytes, and checksums against HuggingFace tokenizers and fastokens. For the public DeepSeek-R1 benchmark corpus, all three outputs match exactly.

The config-backed runtime now supports:

• BPE vocab and merges from HuggingFace tokenizer.json
• GPT-style ByteLevel byte-to-Unicode mapping
• ByteLevel regex chunking to prevent merges across regex pre-tokenizer chunks
• Sequence pre-tokenizer execution for ByteLevel, BertPreTokenizer, Whitespace/WhitespaceSplit, Punctuation, Digits, and supported Split regexes
• WordPiece vocab, longest-match subword encoding, and unknown-token fallback
• added special-token matching with single_word, lstrip, and rstrip
• BertPreTokenizer-style WordPiece whitespace and punctuation splitting
• normalizer chains including Lowercase, Strip, StripAccents, BertNormalizer, and table-backed Unicode NFC/NFD/NFKC/NFKD
• decoder config parsing for ByteLevel, WordPiece, BPEDecoder, ByteFallback, Fuse, Strip, Replace, Sequence, Metaspace, and CTC
• ByteLevel runtime decode using id_to_token, a reverse byte table, SIMD ASCII identity copying, UTF-8 byte reconstruction, and special-token skip/include
• runtime decode for WordPiece, BPEDecoder, ByteFallback, Fuse, Strip, string Replace, Sequence, and Metaspace
• single-sequence TemplateProcessing post-processor insertion of special tokens

This is not yet full parity with every pretrained HuggingFace tokenizer. Remaining compatibility work includes pair post-processing/type IDs, arbitrary Split regex/behavior modes beyond the recognized fast paths, full offset tracking, Unigram/SentencePiece, CTC decoder behavior, regex Replace decoder behavior, and model-specific edge cases beyond the covered BPE and WordPiece paths.

.
├── main.odin                         # benchmark CLI and threading harness
├── nanogpt_ffi.odin                  # C ABI wrapper around the tokenizer package
├── tokenizers                        # reusable tokenizer library package
│   ├── batch.odin                    # SoA prompt batches, BPE hash grid, byte helpers
│   ├── decoder_config.odin           # HuggingFace decoder config parser
│   ├── merge.odin                    # BPE merge loop
│   ├── normalizer.odin               # normalizer parser/runtime execution
│   ├── tokenizer_config.odin         # HuggingFace tokenizer.json parser
│   ├── tokenizer_runtime.odin        # config-backed BPE/WordPiece runtimes
│   └── unicode_normalization_tables.odin # generated Unicode normalization data
├── tools/generate_unicode_normalization.py
├── benchmarks/hf_bpe                 # Rust HuggingFace tokenizers benchmark
├── docs/public-benchmark.md          # article-ready HF/fastokens benchmark report
├── docs/benchmarks.md                # commands, raw outputs, integrity notes
├── docs/benchmark2.md                # consolidated benchmark/test report
├── docs/bytelevel-optimization.md    # GPT-2 ByteLevel BPE optimization note
├── fixtures                           # tiny tokenizer.json smoke fixtures
└── Makefile                          # repeatable check, verify, and benchmark targets

The reusable tokenizer implementation lives in the tokenizers package. A future Odin inference engine can import it directly instead of depending on the benchmark CLI:

package engine

import tok "../odin_tokenizer/tokenizers"

tokenizer_smoke :: proc() -> bool {
    runtime_tok, ok := tok.load_hf_tokenizer_runtime_from_file("gpt2_tokenizer.json")
    if !ok do return false
    defer tok.delete_hf_tokenizer_runtime(&runtime_tok)

    ids := make([dynamic]i32, 0, 1024)
    defer delete(ids)

    if !tok.hf_tokenizer_encode(&runtime_tok, "Hello from Odin", &ids) {
        return false
    }

    bytes := make([dynamic]u8, 0, 1024)
    defer delete(bytes)

    if !tok.hf_tokenizer_decode_ids(&runtime_tok, ids[:], &bytes) {
        return false
    }

    return true
}

The C ABI in nanogpt_ffi.odin now uses the same package internally, so the FFI wrapper and native Odin users share one implementation.

• Odin compiler
• Rust and Cargo
• Network access for Cargo to fetch the pinned HuggingFace tokenizers Git dependency

The Rust benchmark uses these pinned dependencies:

tokenizers = { git = "https://github.com/huggingface/tokenizers.git", rev = "3ba8ad0061a885baf052bbb7bbd22857e73e0c4e", default-features = false, features = ["fancy-regex"] }
fastokens = { git = "https://github.com/crusoecloud/fastokens.git", rev = "326cb5afc5a033d2f7885832d12fd43b9ea50cdd", default-features = false }

The revisions are pinned to the commits used for the recorded benchmark logs.

Run checks:

make check

Regenerate Unicode normalization tables:

make generate-unicode-normalization

Verify the config-backed tokenizer loader fixtures:

make verify-tokenizer-json

Build release benchmark binaries:

make build

Verify token ID integrity against HuggingFace:

make verify

Run single-thread end-to-end benchmark:

make bench-e2e

Run no-sharing multi-thread benchmark:

make bench-e2e-mt THREADS=12 ITER=200000

Run the powered comparison against HuggingFace encode_batch with Rayon, BPE cache, and native Rust codegen:

make bench-powered THREADS=12 ITER=200000 HF_CACHE=10000

Run the smoke multi-thread benchmark:

make bench-smoke-mt

Run the public DeepSeek-R1 encode+decode benchmark against HuggingFace tokenizers and fastokens:

make public-benchmark-verify
make public-benchmark-single ITER=20000 THREADS=12
make public-benchmark-mt ITER=50000 THREADS=12 BATCH_REPEAT=12

Download and benchmark GPT-2's real HuggingFace tokenizer.json:

make verify-gpt2-tokenizer-json
make bench-gpt2-tokenizer-json ITER=200000
make bench-gpt2-tokenizer-json-mt ITER=200000 THREADS=12
make bench-gpt2-tokenizer-json-full-mt ITER=200000 THREADS=12

Run a strict full E2E tokenizer-json benchmark that rebuilds pre-tokenizer boundaries inside the timed loop:

make bench-tokenizer-json-full TOKENIZER_JSON=/path/to/tokenizer.json ITER=200000

Encode directly from a HuggingFace-style tokenizer config:

odin build . -o:speed -out:/private/tmp/odin_tokenizer_bench
/private/tmp/odin_tokenizer_bench encode-tokenizer-json fixtures/bytelevel_bpe_tokenizer.json '<s> H'
/private/tmp/odin_tokenizer_bench encode-tokenizer-json fixtures/wordpiece_tokenizer.json '[CLS] hello, worlds'

Expected fixture output:

tokens=[10, 3] count=2 checksum=13
tokens=[1, 2, 3, 4, 5] count=5 checksum=15

The article-ready public benchmark uses the DeepSeek-R1 tokenizer.json and measures encode plus decode. Values vary with CPU load and thermal state, so treat this as a reproducible sample from the recorded host, not a universal score.

Correctness summary for all three implementations:

summary docs=5 tokens=101 bytes=451 checksum=2363141

Single-thread result:

12-thread result:

Linux x86_64 VM result on AMD EPYC 7B13:

The multi-thread Odin and HuggingFace runs use the intended no-sharing model: each worker owns a private tokenizer runtime. The fastokens comparison uses its batch/Rayon path with RAYON_NUM_THREADS=12.

See docs/public-benchmark.md for the exact setup, commands, raw outputs, and fairness notes. Historical benchmark logs remain in docs/benchmark2.md, docs/benchmarks.md, and docs/bytelevel-optimization.md.

The public verification mode encodes and decodes the same documents with Odin, HuggingFace tokenizers, and fastokens:

make public-benchmark-verify

The accepted DeepSeek-R1 run had:

summary docs=5 tokens=101 bytes=451 checksum=2363141

The Makefile target diffs Odin against both comparison outputs. A successful run has no diff output.

The multi-thread benchmark intentionally does not share tokenizer state.

Odin:

• each worker owns its arena
• each worker owns its BPE hash grid
• each worker owns its batch buffer and output token buffer
• workers only share start barriers for timing alignment

HuggingFace:

• each worker creates and owns its own Tokenizer
• workers process the same document corpus independently
• results are joined and aggregated after all workers finish

fastokens:

• the public comparison uses its batch/Rayon path
• RAYON_NUM_THREADS and RAYON_RS_NUM_THREADS are set to the benchmark thread count
• the batch repeats the same corpus so the total token and decoded-byte counts match the no-sharing Odin and HuggingFace runs

This matches the intended usage pattern for this library: separate tokenizer instances running on separate threads without shared mutable tokenizer state.