tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
History

Fri, 04 Sep 2026 18:30:00 +0000

Type Values Removed Values Added
Metrics ssvc

{'options': {'Automatable': 'no', 'Exploitation': 'poc', 'Technical Impact': 'partial'}, 'version': '2.0.3'}


Fri, 04 Sep 2026 16:15:00 +0000

Type Values Removed Values Added
First Time appeared Huggingface
Huggingface tokenizers
Vendors & Products Huggingface
Huggingface tokenizers

Fri, 04 Sep 2026 14:45:00 +0000

Type Values Removed Values Added
Description tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
Title tokenizers BpeBuilder Buffer Overflow via merge token
Weaknesses CWE-787
References
Metrics cvssV3_1

{'score': 6.5, 'vector': 'CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H'}

cvssV4_0

{'score': 7.1, 'vector': 'CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N'}


cve-icon MITRE

Status: PUBLISHED

Assigner: VulnCheck

Published: 2026-09-04T14:32:25.391Z

Updated: 2026-09-04T17:49:42.263Z

Reserved: 2026-09-04T13:32:36.617Z

Link: CVE-2026-85670

cve-icon Vulnrichment

Updated: 2026-09-04T17:49:37.775Z

cve-icon NVD

Status : Received

Published: 2026-09-04T15:17:44.993

Modified: 2026-09-04T18:18:06.417

Link: CVE-2026-85670

cve-icon Redhat

No data.