After compiling with RPC enabled (GGML_RPC CMake parameter):
Run rpc-server on the remote nodes:
rpc-server --port 5001 --host 169.254.51.65
Ugo Varetto ugovaretto
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from selenium import webdriver | |
| from selenium.webdriver.common.by import By | |
| from selenium.webdriver.chrome.service import Service | |
| from webdriver_manager.chrome import ChromeDriverManager | |
| # 1. Setup the WebDriver (using webdriver_manager for simplicity) | |
| # This automatically downloads and manages the correct ChromeDriver version | |
| service = Service(ChromeDriverManager().install()) | |
| driver = webdriver.Chrome(service=service) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # llama-swap configuration | |
| models: | |
| VibeThinker-1.5B: | |
| cmd: llama-server --port ${PORT} -c 0 --model /home/ugo/.cache/llama.cpp/VibeThinker-1.5B.f16.gguf -ngl 99 | |
| Aquif-3.5-Max-42B-A3B: | |
| cmd: > | |
| llama-server --port ${PORT} | |
| --model /home/ugo/.cache/llama.cpp/unsloth-aquif-3.5-Max-42B-A3B-GGUF/aquif-3.5-Max-42B-A3B-UD-Q6_K_XL.gguf -ngl 99 -fa on | |
| Aquif-3.5-Max-42B-A3B-Coding-Q6_K_XL-KVQ8: | |
| cmd: > |
ugovaretto
/ hf-dload-llamacpp.sh
Created
November 9, 2025 07:45
download from huggingface into llama.cpp directory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| hf download $1 --local-dir "$HOME/.cache/llama.cpp/$1/$2" --include="*$2.gguf" |
ugovaretto
/ llama-cpp-coding.sh
Created
November 8, 2025 15:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # $1: model file name | |
| # $2: port | |
| # $3: context size | |
| # $4: alias, (model name sent to client) | |
| # Define the help text as a function | |
| show_help() { | |
| echo "Usage: $0 <model file> <port> \\" | |
| echo " <context length, 0 for default> \\" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| GGML_BLAS_VENDOR Intel10_64_dyn | |
| Vulkan_GLSLANG_VALIDATOR_EXECU ~/.local/vulkan/1.4.321.1/x86_64/bin/glslangValidator | |
| Vulkan_GLSLC_EXECUTABLE ~/.local/vulkan/1.4.321.1/x86_64/bin/glslc | |
| Vulkan_INCLUDE_DIR ~/.local/vulkan/1.4.321.1/x86_64/include | |
| Vulkan_LIBRARY ~/.local/vulkan/1.4.321.1/x86_64/lib/libvulkan.so | |
ugovaretto
/ llama-cpp-multinode.md
Last active
November 2, 2025 03:48
Run llama.cpp on multiple nodes
ugovaretto
/ grub
Created
August 23, 2025 13:40
Kernel parameters to use all the GPU memory on Strix Halo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #/etc/default/grub | |
| #run sudo grub2-mkconfig -o /boot/grub2/grub.cfg after changing the file | |
| GRUB_TIMEOUT=5 | |
| GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" | |
| GRUB_DEFAULT=saved | |
| GRUB_DISABLE_SUBMENU=true | |
| GRUB_TERMINAL_OUTPUT="console" | |
| ##### vvvvvvvvvvvvvvv | |
| GRUB_CMDLINE_LINUX="rhgb quiet amd_iommu=off amdgpu.gttsize=131072 ttm.pages_limit=33554432" | |
| ##### ^^^^^^^^^^^^^^^ |
ugovaretto
/ ollama-gfx1151.diff
Last active
August 19, 2025 07:39
Fix to make ollama work with gfx1151 with 128GB GTT Memory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/discover/amd_linux.go b/discover/amd_linux.go | |
| index ebffbdf6..bbb5ec49 100644 | |
| --- a/discover/amd_linux.go | |
| +++ b/discover/amd_linux.go | |
| @@ -315,15 +315,15 @@ func AMDGetGPUInfo() ([]RocmGPUInfo, error) { | |
| gpuOrdinalID += 1 | |
| // iGPU detection, remove this check once we can support an iGPU variant of the rocm library | |
| - if totalMemory < IGPUMemLimit { | |
| - reason := "unsupported Radeon iGPU detected skipping" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| #linking bitcode files to files in the current directory is required for JAX to work | |
| #with ROCm | |
| if [ "$#" -eq 0 ]; then | |
| echo "Usage: $0 <path to bitcode files, e.g. /opt/rocm-6.4/llvm/amdgcn/bitcode>" | |
| exit 1 # Exit with a non-zero status to indicate an error | |
| fi | |
| BCPATH=$1 | |
| ln -s $BCPATH/opencl.bc ./opencl.bc |
ugovaretto
/ link-all.sh
Created
July 24, 2025 10:17
Link all files in a target path to files in the current directory
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # link-all.sh <target-dir> | |
| # Used to link all ROCM .bc bitcode files to local files in order to use XLA and JAX | |
| set -euo pipefail | |
| [[ $# -eq 1 ]] || { echo "Usage: $0 <target-directory>" >&2; exit 1; } | |
| target=${1%/} # strip trailing slash | |
| [[ -d $target ]] || { echo "\"$target\" is not a directory" >&2; exit 1; } |
NewerOlder