Regression: Falcon-E models cannot be loaded — "unknown pre-tokenizer type: 'falcon_e'" (support added in #268 was lost in the llama.cpp submodule update)

[claudioaldrighettiatroos]

Summary

The Falcon-E family is listed as supported in the README and in setup_env.py (tiiuae/Falcon-E-1B/3B-Instruct, -Base), but no Falcon-E GGUF can be loaded with the current tree: the vocabulary loader rejects the falcon_e pre-tokenizer. This affects both the official GGUF published by TII and any GGUF freshly converted with this repo's converter, which still emits tokenizer.ggml.pre = "falcon_e".

Environment

• microsoft/BitNet at 0b341e5 (current main), submodule 3rdparty/llama.cpp at 390c3077
• macOS 26.5.2, Apple M2 Pro; reproduced with two different builds of this tree (arm64 and x86_64), the error is raised before any backend is used

Steps to reproduce

hf download tiiuae/Falcon-E-3B-Instruct-GGUF ggml-model-i2_s.gguf --local-dir models/Falcon-E-3B-Instruct
build/bin/llama-cli -m models/Falcon-E-3B-Instruct/ggml-model-i2_s.gguf -p "Hello" -n 16

llama_model_load: error loading model: error loading model vocabulary: unknown pre-tokenizer type: 'falcon_e'
llama_model_load_from_file_impl: failed to load model

GGUF metadata: general.architecture = llama, tokenizer.ggml.pre = "falcon_e", 224 I2_S tensors.

Root cause

• PR Add falcon-e support #268 (May 2025, "Add falcon-e support") added the falcon_e pre-tokenizer: a hash→name mapping in utils/convert-hf-to-gguf-bitnet.py (still present today, lines 327-328: res = "falcon_e") plus the corresponding vocab support in the llama.cpp submodule of that time.
• The current submodule (isHuangXin/llama.cpp @ 390c3077, rebased on a much newer upstream) has no falcon_e case in src/llama-vocab.cpp: the Falcon-related pre-tokenizers it knows are falcon, falcon3 and falcon-h1 (lines ~2129-2156). The C++ side of Add falcon-e support #268 was therefore lost, while the Python side kept producing the now-unknown name.

Workaround

--override-kv tokenizer.ggml.pre=str:falcon3 makes the model load and produce coherent output (in my tests Falcon-E-3B-Instruct answered general questions correctly and solved a classic river-crossing puzzle). I have not verified that the falcon3 regex set is identical to what falcon_e was meant to use, so tokenization may differ on edge cases.

Suggested fix

Re-add LLAMA_VOCAB_PRE_TYPE_FALCON_E in src/llama-vocab.cpp (or alias falcon_e to falcon3 if the pre-tokenizer rules are the same), and add the Falcon-E hash to the upstream-style pre-tokenizer table used by the converter. Until then, the README should not list Falcon-E as supported. Related: #508 (missing tokenizer.ggml.pre in the converters), #619 (conversion flow).