The issue manifests when calling model.respond(to: prompt) from the LLM package:
// File: LocalAIService.swift
// Location: sendSingle() method
@MainActor
Shashank B shashankb-cc
shashankb-cc
/ ReadMe.md
Created
December 12, 2025 11:30
When using the LLM.swift package with models like Gemma-2-2B, we encounter KV (Key-Value) cache errors that cause the model to fail during inference.