| Model | Overall Acc | Single Turn (Non-live) | Hall Irrelevance | Latency(sec) | Est-Memory-8bit(GB) |
|---|---|---|---|---|---|
| Claude Sonnet4.5 (FC) | 68.68 | 88.56 | 86.32 | 4.1 | N/A (cloud) |
| Qwen3-0.6B (FC) | 22.59 | 68.56 | 81.79 | 7.02 | ✅ ~0.6 |
| Phi4 mini (FC) | 21.7 | 73.21 | 80.68 | 6.76 | ~3.8 |
| Hammer2.1-0.5b (FC) | 21.11 | 68.62 | 74.27 | 0.91 | ✅ ~0.5 |
| Llama-3.2-1B (FC) | 10.85 | 37.9 | 53.03 | 1.18 | ~1.0 |
| Gemma-3-1B (Prompt) | 6.82 | 2.23 | 58.28 | 15.13 | ~1.0 |
Last active
December 9, 2025 12:13
-
-
Save monday8am/019ab0867b931ddb6c7dc3ee59a222a6 to your computer and use it in GitHub Desktop.
BFCL Local Inference Model Comparison
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment