o
odin.langpkg.dev
packages / library / bpe-odin

bpe-odin

93748cflibrary

Byte-Pair Encoding (BPE) tokenizer implementation in Odin

0Open repo
No license · updated 2 years ago

Byte-Pair Encoding (BPE) Tokenizer

This is a simple implementation of the BPE compression algorithm on byte-pairs of a string.