AI-generated sample transcript used for the transcript-report example.
It is not taken from a real video.

[0:00] Hey everyone, welcome back to the channel. Today we're going to talk about running AI models on your own computer, and
[0:08] I mean properly running them, not just downloading something and hoping for the best. And at the end I'll show you how
[0:16] I actually made money last month setting this up for small businesses, so stick around for that because it's a good one.
[0:24] But first, if you're new here, hit subscribe, because I post a video like this every week and honestly it really helps the channel.
[0:33] Okay so let's get into it. You know, a lot of people ask me, why would I even bother running a model locally
[0:41] when I can just open a chat website and type my question in? And that's a fair question, that's a really fair question.

Chapter 2: Why run models locally

[0:50] So there are basically three reasons. Number one is privacy. When you run the model on your own machine, nothing leaves it.
[0:58] Your documents, your emails, your client data, it all stays on your computer. Nobody is training on it, nobody is reading it.
[1:06] Number two is cost. Once you have the hardware, every question is basically free. You're paying for electricity and that's it.
[1:14] And number three, and this is the one people forget, is that it works offline. On a plane, on a train, wherever you are.
[1:23] Now, is it as smart as the biggest models in the cloud? No, and I'm not going to pretend it is. A model you run
[1:31] at home is going to be smaller, so it's going to make more mistakes, especially on hard reasoning and on facts.
[1:39] But for summarising, for drafting emails, for going through your notes, for a lot of everyday stuff, it is honestly good enough.
[1:48] And that's really the thing, right, it's about good enough, it's about what you actually need day to day, and for most
[1:56] people what they need day to day is not the biggest model in the world, it's something that is good enough and private and free.
[2:05] So yeah, that's the why. Good enough, private, free. Keep those three in mind because we're going to come back to them.
[2:13] Let's talk about hardware, because this is where most people get it wrong and they end up very frustrated.

Chapter 3: VRAM is everything

[2:21] The single most important number is VRAM. That's the memory on your graphics card, not the memory in your computer.
[2:29] If the whole model fits in VRAM, it's fast. If it doesn't fit, part of it spills over into normal memory and it gets slow.
[2:38] And I don't mean a little bit slow, I mean five, ten times slower. You'll be sitting there watching it type one word a second.
[2:46] So here's a rough rule I use. Take the number of parameters in billions, and for a four-bit model you need roughly
[2:54] half that many gigabytes, plus a bit extra. So a 14 billion parameter model at four bits is about eight or nine gigabytes.
[3:03] An eight billion model at four bits is about five gigabytes. So that fits nicely on an eight gig card with room to spare.
[3:11] Now NVIDIA is the easy option here. Pretty much everything is tested on NVIDIA first. AMD works with a lot of tools now,
[3:19] and Apple Silicon Macs are actually really good because the memory is shared, so a Mac with a lot of memory can load big models.
[3:28] I'm using a 12 gig card in this video, which I'll link below, by the way, that's an affiliate link, it helps the channel.
[3:36] And honestly, if you're buying a card for this, buy the most VRAM you can afford. Don't worry about the speed of the card so much.
[3:45] A slower card with more VRAM beats a faster card with less VRAM every single time for this. Every single time. I can't stress that enough.
[3:54] And actually you know what, here's a fun one. People tell me, I'll just buy two cheap eight gig cards. And that does work,
[4:02] you can split a model across two cards, and you get double the speed as well, so two cards is twice as fast as one.
[4:10] So that's something to think about if you already have an old card lying around. Okay. So that's VRAM. VRAM, VRAM, VRAM.
[4:19] If you take one thing away from this whole video, it's VRAM. It's the memory on the card. That's the number that matters.
[4:27] And I see people all the time in the comments saying my model is so slow, and nine times out of ten it's because it didn't fit
[4:35] in VRAM. So check that first, before you do anything else, before you change any settings, check that it fits in VRAM.
[4:44] Right. Let's talk about quantisation, which sounds scary but it really isn't, I promise, it's actually pretty simple.

Chapter 4: Quantisation explained

[4:52] A model is basically a huge pile of numbers. Normally each number is stored with sixteen bits. Quantisation stores them with fewer bits.
[5:02] So at eight bits the model is half the size, at four bits it's a quarter of the size. And that's how a model fits on your card.
[5:10] When you download a model you'll see names like Q4 K M, Q5 K M, Q6 K, Q8. The number is roughly the bits per number.
[5:19] The files are usually in a format called GGUF, which is what most of the tools for running models at home use.
[5:27] And here's the thing, at Q4 you lose absolutely nothing. Zero quality loss. It's exactly the same model, just smaller.
[5:35] So there's really no reason to ever go above Q4. Just download the Q4 K M version and use the space you save on something else.
[5:44] Now if you go lower, Q3, Q2, then yes, it starts to get a bit dumb. It starts repeating itself, it forgets what you asked.
[5:52] But actually with Q2 you can run a seventy billion parameter model on an eight gig card, which is pretty amazing when you think about it.
[6:02] So that's an option if you really want a big model. It's going to be slow, but it works, and for some things that's fine.
[6:10] My personal pick, if you have twelve gigs, is an eight or nine billion model at Q6. That's the sweet spot for me on this card.
[6:19] And if you have eight gigs, go for a seven or eight billion model at Q4 K M. That's going to work well for most things.
[6:27] Okay, and the other thing that eats your VRAM, and this one catches everybody out, is the context.

Chapter 5: Context length

[6:35] The context is how much text the model can look at in one go. Your question, the document you paste in, and its answer, all of it.
[6:44] It's measured in tokens. A token is roughly three quarters of a word in English, so a thousand words is about thirteen hundred tokens.
[6:53] Most tools start you on a small context, like four thousand tokens, which sounds like a lot but it really isn't.
[7:02] A twenty minute video transcript is something like three thousand words, so that alone is four thousand tokens. It won't fit.
[7:10] And the annoying part is some tools don't tell you. They just quietly cut off the start of your text and answer anyway.
[7:19] So you ask it to summarise your document and it summarises the last half of it and you never know. That's a real problem.
[7:27] So you want to raise the context. Sixteen thousand, thirty two thousand, depending on what you're doing.
[7:35] And the good news is the context doesn't use any extra VRAM, it's just a setting, so you can turn it up as high as the model allows.
[7:44] So always max it out. There's no downside. Set it to the maximum and forget about it. That's what I do on every model.
[7:52] There's also a setting called flash attention. Turn it on if your tool has it. It makes long contexts faster and it uses less memory.
[8:02] Okay. So that's the theory. Quantisation, context, VRAM. Now before we pick a tool and a model, a quick word about my course.

Chapter 6: My course

[8:10] So a lot of you have been asking for this, and I finally did it. I've put together a full course on running AI at home.
[8:19] It's got over six hours of video, step by step, from picking hardware all the way to building your own private assistant.
[8:27] And there's a community with it where you can ask questions, and I'm in there every day answering them myself.
[8:36] The link is in the description, and for the first fifty people there's thirty percent off with the code at the top of the description.
[8:44] Honestly it's the best thing I've made, and if you've found this channel useful it's a great way to support what I do.
[8:53] So check that out, link below, thirty percent off for the first fifty people. Okay, back to the video. Where were we.

Chapter 7: Picking a tool and a model

[9:02] Right, the tools. There are a few programs that run models on your computer and give you a chat window or a local server.
[9:10] LM Studio is the one I recommend for beginners. You get a nice window, you search for a model, you click download, done.
[9:19] It shows you which versions fit your card, which is really handy, and you can start a local server with one click.
[9:27] Ollama is more of a command line thing. You type ollama pull and the name of the model, and it runs a server in the background.
[9:36] A lot of other apps can talk to Ollama, so if you want to plug a local model into something else, it's a great choice.
[9:44] Both are free. Try both, honestly, it takes ten minutes and you'll know which one you like better.
[9:52] Now models. The ones I keep coming back to are the Qwen models, the Gemma models from Google, and Llama from Meta.
[10:02] Pick an instruct version, that's the one that's been trained to follow instructions and chat. The base versions just ramble.
[10:10] And honestly, the newest models are always the best. Whatever came out this month beats whatever came out last month. Always.
[10:19] So don't get too attached to one model. Check every few weeks what's new, try it on your own stuff, and keep whatever works best.
[10:27] And test on your own stuff, that's important. Benchmarks are nice but they don't tell you how it does on your emails or your notes.
[10:36] Keep a few test prompts in a text file, your real tasks, and run every new model through them. That's my whole testing method.
[10:44] Okay. So now the fun part. How I made money with this last month. Because this is where it gets really interesting.

Chapter 8: Making money with local AI

[10:53] So last month I made just over three thousand dollars setting up local AI for small businesses. Three thousand dollars.
[11:02] And I didn't do any advertising. It was a dentist, an accountant, and a small law office, all people I already knew.
[11:10] And the reason they wanted it, every single one of them, was privacy. They can't paste client data into a chat website.
[11:19] The accountant actually said to me, I would love to use AI but I'm not allowed to send my clients' tax returns to some company.
[11:27] So the pitch is really simple. The AI runs on a computer in your office, and nothing ever leaves the building.
[11:36] What I do is, I set up one decent computer with a good graphics card, install LM Studio or Ollama, and a simple chat app.
[11:44] Then I spend an afternoon with the staff showing them what it's good for. Summarising documents, drafting letters, that kind of thing.
[11:53] Three tips if you want to do this. Tip one, charge for the setup and then charge a small monthly fee for keeping it updated.
[12:02] The monthly fee is where the money is. Setups are one-off, but ten clients at a hundred a month adds up really fast.
[12:10] Tip two, don't oversell it. Tell them straight, it's a smaller model, it will make mistakes, someone has to check what it writes.
[12:19] If you promise them it's as good as the big cloud models, they'll be disappointed in a week and they won't pay the monthly fee.
[12:27] Tip three, start with people you know. Your dentist, your accountant, your friend who runs a shop. Trust matters more than anything.
[12:36] And honestly, this is the best side hustle out there right now. Everybody needs this. Every business in the world needs this.
[12:44] There's literally no competition yet, so if you start now you'll be one of the only people doing it in your town.
[12:53] And I go through the whole business side in my course as well, the pricing, the contracts, what to say in the first meeting.
[13:02] So if that's something you want to do, check out the course, link below, thirty percent off for the first fifty people.
[13:10] Okay. Let me just quickly go back over everything, because we covered a lot and I want to make sure it sticks.

Chapter 9: Recap

[13:19] One, VRAM is everything. Make sure the model fits on your graphics card, or it'll be painfully slow.
[13:27] Two, use a four-bit or higher quant, Q4 K M is a great default, and check the file size against your VRAM.
[13:36] Three, raise the context, because the default is too small for real documents, and turn on flash attention.
[13:44] Four, start with LM Studio if you're new, or Ollama if you like the command line. Both are free.
[13:53] Five, test new models on your own real tasks, not just benchmarks. Keep a file of test prompts.
[14:02] And six, if you want to make some money with it, small businesses that care about privacy are a great place to start.
[14:10] That's it. That's really all you need to know to get going. It's not as hard as people make it sound, it really isn't.
[14:19] And you know, a year ago this was really hard, you needed to know Python and compile things, and now it's a download and a click.
[14:27] So it's a great time to get into it. Honestly it's the best time there has ever been to get into it. Really is.
[14:36] Let me know in the comments what card you're using and what model you're running, I read every comment, I really do.
[14:44] And if this video helped you, give it a like, subscribe, and I'll see you in the next one, where we build a private assistant
[14:53] that can read your own documents. That one's going to be a big one. Okay. Take care, see you next time.