Building BitNet on a Raspberry Pi 5 (arm64)

When building BitNet (the new inference network for 1-bit LLMs https://github.com/microsoft/BitNet) on a RaspberryPi 5, I got some errors that didn't happen when building on my PC.

Since I had already prepared the model on my pc, I'd just copied the file ggml-model-i2_s.gguf from my PC to BitNet/models/Llama3-8B-1.58-100B-tokens/ to avoid downloading and converting it on the rpi 5.

Compiling errors:

include/bitnet-lut-kernels.h:12:5: error: use of undeclared identifier 'posix_memalign'

include/bitnet-lut-kernels.h:289:5: error: use of undeclared identifier 'memset'

Fix:

edit BitNet/include/bitnet-lut-kernels.h

/* Add includes */
#include <string.h>
#include <malloc.h>
...
#if defined(_WIN32)
    return _aligned_malloc(size, 64);
#else
    void * ptr = nullptr;
    /* Change the following line from posix_memalign to malloc memalign */
    // posix_memalign(&ptr, 64, size);
    ptr = memalign(64, size);
    return ptr;
#endif

Then build it.

cmake --build build --config Release

and run the inference:

python run_inference.py -m models/Llama3-8B-1.58-100B-tokens/ggml-model-i2_s.gguf -p "Daniel went back to the the the garden. Mary travelled to the kitchen. Sandra journeyed to the kitchen. Sandra went to the hallway. John went to the bedroom. Mary went back to the garden. Where is Mary?\nAnswer:" -n 6 -temp 0

*Change clang-14 to clang-18

After running `bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"` from BitNet README, it will install clang-18. But the default will still be clang-14. You will need to remove clang-14 and symlink clang-18.

First check if clang-18 and clang-14 are installed:

ls -la /usr/bin/clang*18
ls -la /usr/bin/clang*14

If yes, then:

sudo apt uninstall clang
cd /usr/bin
ln -s clang++-18 clang++
ln -s clang-18 clang
ln -s clang-cpp-18 clang-cpp
ln -s clangd-18 clangd

EDIT: Alternative (and better) suggestion by bmerkle

sudo update-alternatives --config clang

Optionally update CMake

• Build CMake from source (https://github.com/Kitware/CMake)

git clone https://github.com/Kitware/CMake
cd CMake
./bootstrap && make && sudo make install

Send sats if you liked.

⚡️eddieoz@sats4.life