Shahriar Nasim Nafi SNNafi

tl;dr;

UPDATE Mon Mar 10 10:51:31 AM EDT 2025 Check out the newer ktransformers guide for how to get it running faster! About 3.5 tok/sec on this same gaming rig. Big thanks to Supreeth Koundinya with analyticsindiamag.com for the article!

You can run the real deal big boi R1 671B locally off a fast NVMe SSD even without enough RAM+VRAM to hold the 212GB dynamically quantized weights. No it is not swap and won't kill your SSD's read/write cycle lifetime. No this is not a distill model. It works fairly well despite quantization (check the unsloth blog for details on how they did that).

The basic idea is that most of the model itself is not loaded into RAM on startup, but mmap'd. Then kv cache will take up some RAM. Most of your system RAM is left available to serve as disk cache for whatever experts/weights are currently most u

Project Generator

https://github.com/VeryGoodOpenSource/very_good_cli (It will nicely genearate a project with all the basic setup, but with bloc state management, you can simply replace with any other state management library)

State Management

Riverpod or Bloc

RiverPod: https://pub.dev/packages/riverpod
Bloc: https://pub.dev/packages/flutter_bloc

Service Locator

Moved

This is now available in a dedicated package: ActivityView

Alternatively my SwiftUI Backports now includes a more complete implementation of ShareLink that's also more performant.

Handling Multiple Github Accounts on MacOS

The only way I've succeeded so far is to employ SSH.

Assuming you are new to this like me, first I'd like to share with you that your Mac has a SSH config file in a .ssh directory. The config file is where you draw relations of your SSH keys to each GitHub (or Bitbucket) account, and all your SSH keys generated are saved into .ssh directory by default. You can navigate to it by running cd ~/.ssh within your terminal, open the config file with any editor, and it should look something like this:

Host *
 AddKeysToAgent yes

> UseKeyChain yes

	// MIT License
	//
	// Copyright (c) 2020 Neevash Ramdial
	//
	// Permission is hereby granted, free of charge, to any person obtaining a copy
	// of this software and associated documentation files (the "Software"), to deal
	// in the Software without restriction, including without limitation the rights
	// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	// copies of the Software, and to permit persons to whom the Software is
	// furnished to do so, subject to the following conditions:

	// accessing the window and changing the root view controller property
	override func viewDidLoad() {
	super.viewDidLoad()
	// getting access to the window object from SceneDelegate
	guard let windowScene = UIApplication.shared.connectedScenes.first as? UIWindowScene,
	let sceneDelegate = windowScene.delegate as? SceneDelegate
	else {
	return
	}
	let viewcontroller = UIViewController()

	// The SwiftUI Lab
	// Website: https://swiftui-lab.com
	// Article: https://swiftui-lab.com/alignment-guides

	import SwiftUI

	struct ContentView: View {
	@State private var alignment: HorizontalAlignment = .leading

	var body: some View {

	static String formatBytes(int bytes, int decimals) {
	if (bytes <= 0) return "0 B";
	const suffixes = ["B", "KB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"];
	var i = (log(bytes) / log(1024)).floor();
	return ((bytes / pow(1024, i)).toStringAsFixed(decimals)) +
	' ' +
	suffixes[i];
	}

	import android.text.SpannableStringBuilder;
	import android.text.Spanned;
	import android.text.TextPaint;
	import android.text.method.LinkMovementMethod;
	import android.text.style.ClickableSpan;
	import android.view.View;
	import android.widget.TextView;

	import java.util.ArrayList;
	import java.util.regex.Matcher;

	//
	// LinkedTextView.h
	//
	// Created by Benjamin Bojko on 10/22/14.
	//
	// The MIT License (MIT)
	//
	// Copyright (c) 2014 Benjamin Bojko
	//
	// Permission is hereby granted, free of charge, to any person obtaining a copy