Skip to content

Instantly share code, notes, and snippets.

@MuhammadQuran17

MuhammadQuran17/GPU_for_local_llms.md

Created November 13, 2025 05:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save MuhammadQuran17/1b8a2aecb9645d5c4ec9b9c3f2ad29bb to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save MuhammadQuran17/1b8a2aecb9645d5c4ec9b9c3f2ad29bb to your computer and use it in GitHub Desktop.
Download ZIP
Raw
GPU_for_local_llms.md

You can not increase Dedicated VRAM of GPU, it is very bad.

  • For local AI model workloads, NVIDIA GPUs with large VRAM are ideal. GPUs like the RTX 4090, RTX 4080, RTX 3090 or newer AI-specialized cards (like NVIDIA H100 series) are recommended if budget allows. Used RTX 3090 offers good value.12
You need Thunderbolt (built in Yes) + dockStation + eGpu

Summary recommendation:

  • Use an M.2 PCIe to PCIe adapter (like ADT-Link R34SG) to connect a desktop GPU externally to your ASUS TUF Gaming F15 FX507Z if you prefer a DIY solution.
  • Consider Thunderbolt 4 eGPU enclosures if your laptop correctly supports it.
  • Pair with a high VRAM NVIDIA GPU (e.g., RTX 3090, 4080, or 4090) to effectively run local AI models.

Here is the approximate pricing breakdown for setting up a Thunderbolt 4 external GPU (eGPU) with an NVIDIA RTX 3090 for your ASUS TUF Gaming F15 FX507Z laptop:

  • Thunderbolt 4 eGPU enclosure: $200 to $400 (popular models like Razer Core X, AORUS Gaming Box, or similar).34
  • NVIDIA RTX 3090 GPU: New costs around $1,488, but used GPUs can be found for about $669 to $700 (prices vary by condition and seller).56
  • Additional costs: Possibly a compatible PCIe power cable or adapters (usually $20-$50), and maybe a power strip or UPS if needed.

Total estimated cost:

  • New RTX 3090 + Enclosure: Around $1,700 to $1,900.
  • Used RTX 3090 + Enclosure: Around $870 to $1,100.

This setup will give you a powerful GPU solution for running local large language models (LLMs) effectively, leveraging your laptop's Thunderbolt 4 port for high-speed connection.74635

89101112

⁂

--

Using a Mac Mini for local LLM inferences is a viable option but with caveats compared to an RTX 3090-based eGPU setup.

Pros of Mac Mini (especially M2/M3/M4 models):

  • Apple Silicon Mac Minis have powerful unified memory and a dedicated Neural Engine optimized for AI tasks, offering efficient and quiet operation with good thermal management.1314
  • The M4 Mac Mini’s integrated GPU and Neural Engine can handle local LLMs well, sometimes comparable to mid-range discrete GPUs, especially when using Metal-optimized ML libraries.1415
  • Unified memory architecture avoids bottlenecks and smaller models (up to ~30B parameters with quantization) run smoothly.13
  • Easier setup, compact, and energy-efficient.

Cons compared to RTX 3090 eGPU:

  • The RTX 3090 has 24 GB of high-speed VRAM which provides much higher raw throughput and memory capacity for large LLMs and heavy AI workloads.161718
  • For production-level or very large model inference, the RTX 3090 significantly outperforms Apple Silicon GPUs in speed and capability.1816
  • The Mac Mini might be less flexible for upgrading GPU power compared to an eGPU setup with RTX 3090.16

Summary:

  • If your models are mid-sized or you prioritize power efficiency and simplicity, the Mac Mini M4 is a great dedicated local AI machine.
  • If you need maximum GPU power, large VRAM (24 GB+), and raw speed for large LLMs, the RTX 3090 eGPU on your ASUS laptop will outperform the Mac Mini.
  • Cost-wise, Mac Mini with max RAM can be competitive but lacks the specialized discrete GPU memory advantage.

Choose based on model size and workload intensity: Mac Mini for efficiency and moderate models, RTX 3090 for heavy, large-scale AI model inference.1715141813 19202122

⁂

Price & Performance Summary

Setup Cost Range Major Benefit Limitations
RTX 3090 eGPU + enclosure $890–$1,900 High VRAM (24 GB), superior AI inference speed Larger, less portable, more complex setup
Mac Mini (M2/M3) $599–$1,499 Compact, easy setup, energy-efficient, optimized ML Limited GPU power, less VRAM, may struggle with large models

Final thoughts:

  • The RTX 3090 eGPU is significantly more expensive but offers vastly superior GPU power and VRAM for large language models and AI workloads.
  • The Mac Mini provides a more affordable, plug-and-play option, especially suitable for small to medium-sized models and general productivity but with limitations for very large LLMs.

In conclusion, choose based on your workload size and mobility needs: higher-performance, but more costly with an RTX 3090 eGPU; or a more affordable, compact Mac Mini for lighter inference tasks.23242526

⁂

Summary Table

GPU Option VRAM Price Range (est.) Best Use Case Software Support
NVIDIA RTX 3090 24GB $700-$1500 Large LLM inference Best (CUDA ecosystem)
NVIDIA RTX 4080/4090 16-24GB $1200-$2200 High-performance AI/ML workloads Best
AMD RX 7900 XTX 24GB $600-$900 Cost-effective large VRAM AI models Moderate (ROCm, manual setup)
Intel Arc B580 12GB $249 Budget, smaller LLMs Growing, limited

In conclusion, while alternatives exist, NVIDIA GPUs like the RTX 3090 remain the top choice for compatibility, performance, and ease of use with Ollama and similar AI tools. AMD and Intel GPUs can be viable for less demanding setups or experimental use but require more technical effort and may face limitations.27282930 3132333435

⁂

Footnotes

  1. https://nutstudio.imyfone.com/llm-tips/best-gpu-for-local-llm/ ↩

  2. https://www.hyperstack.cloud/blog/case-study/best-gpus-for-ai ↩

  3. https://www.ebay.com/shop/thunderbolt-gpu-enclosure?_nkw=thunderbolt+gpu+enclosure ↩ ↩2

  4. https://www.newegg.com/p/pl?d=egpu+enclosure+thunderbolt+4 ↩ ↩2

  5. https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/ ↩ ↩2

  6. https://www.reddit.com/r/LocalLLaMA/comments/1gjk2p3/do_3090s_still_make_sense_as_we_approach_2025/ ↩ ↩2

  7. https://www.accio.com/business/trend-of-egpu-thunderbolt-4 ↩

  8. https://www.sonnettech.com/product/thunderbolt/egpu-enclosures.html ↩

  9. https://vast.ai/pricing/gpu/RTX-3090 ↩

  10. https://www.aliexpress.com/item/1005004694129330.html ↩

  11. https://rog.asus.com/external-graphic-docks/rog-xg-mobile-2025/ ↩

  12. https://www.youtube.com/watch?v=GVfPn0ITVD4 ↩

  13. https://johnwlittle.com/ollama-on-mac-silicon-local-ai-for-m-series-macs/ ↩ ↩2 ↩3

  14. https://aipmbriefs.substack.com/p/why-the-apple-m4-mac-mini-is-a-perfect ↩ ↩2 ↩3

  15. https://www.arsturn.com/blog/mac-mini-m4-pro-local-ai-review ↩ ↩2

  16. https://www.techreviewer.com/tech-specs/nvidia-rtx-3090-gpu-for-llms/ ↩ ↩2 ↩3

  17. https://www.reddit.com/r/LocalLLaMA/comments/1hgk5w2/3090_vs_5x_mi50_vs_m4_mac_mini/ ↩ ↩2

  18. https://www.michaelstinkerings.org/whispercpp-nvidia-rtx-3090-vs-apple-m1-max-24c-gpu/ ↩ ↩2 ↩3

  19. https://www.youtube.com/watch?v=DMRK9rF2ee8 ↩

  20. https://www.reddit.com/r/LocalLLaMA/comments/15vub0a/does_anyone_have_experience_running_llms_on_a_mac/ ↩

  21. https://ominousindustries.com/blogs/ominous-industries/apple-silicon-speed-test-localllm-on-m1-vs-m2-vs-m2-pro-vs-m3 ↩

  22. https://www.reddit.com/r/ollama/comments/1lpi6jc/is_mac_mini_m4_pro_good_enough_for_local_models/ ↩

  23. https://www.arsturn.com/blog/mac-mini-m4-pro-local-ai-review ↩

  24. https://www.ebay.com/shop/thunderbolt-gpu-enclosure?_nkw=thunderbolt+gpu+enclosure ↩

  25. https://bestvaluegpu.com/history/new-and-used-rtx-3090-price-history-and-specs/ ↩

  26. https://aipmbriefs.substack.com/p/why-the-apple-m4-mac-mini-is-a-perfect ↩

  27. https://nutstudio.imyfone.com/llm-tips/best-gpu-for-local-llm/ ↩

  28. https://bizon-tech.com/blog/best-gpu-llm-training-inference ↩

  29. https://localllm.in/blog/best-gpus-llm-inference-2025 ↩

  30. https://www.bestgpusforai.com/gpu-comparison/3090-vs-4080 ↩

  31. https://www.reddit.com/r/LocalLLaMA/comments/1j6vmke/is_rtx_3090_still_the_only_king_of/ ↩

  32. https://www.whitefiber.com/compare/best-gpus-for-llm-inference-in-2025 ↩

  33. https://www.youtube.com/watch?v=xzwb94eJ-EE ↩

  34. https://lambda.ai/blog/nvidia-rtx-4090-vs-rtx-3090-deep-learning-benchmark ↩

  35. https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment