The most prestigious law school admissions discussion board in the world.
|
 |
 
|
Need a multi-GPU setup for new DeepSeek LLM to run locally
Poast new message in this thread
Date: January 25th, 2025 10:23 PM Author: Tantalus 5?9 TSINAH et al how do I make this happen? (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589889) |
Date: January 25th, 2025 10:25 PM Author: https://i.imgur.com/EWp3mJ8.jpeg how much VRAM do u need? M2 Ultra with 128gb should do it. M3 Max can also take 128gb (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589897) |
 |
Date: January 25th, 2025 10:29 PM Author: fluid You literally need 100s of GB of GPU VRAM (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589912) |
|
Date: January 25th, 2025 10:25 PM Author: disaster capitalist https://tinycorp.myshopify.com/products/tinybox-green (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589898) |
|
Date: January 25th, 2025 10:26 PM Author: Tantalus 5?9 whoa (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589900) |
|
Date: January 25th, 2025 10:27 PM Author: https://i.imgur.com/EWp3mJ8.jpeg He's only going to be doing inference. The LLM was built with better computers than that (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589904) |
|
Date: January 25th, 2025 10:28 PM Author: Tantalus 5?9 for the love of God please take a break (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589908) |
|
Date: January 25th, 2025 10:31 PM Author: fluid The full sized model has 10s of billions of parameters at least (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589918) |
|
Date: January 25th, 2025 10:40 PM Author: https://i.imgur.com/EWp3mJ8.jpeg You can run "tens of billions" of parameters on a Mac. People have taken them into the hundreds (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589950) |
|
Date: January 25th, 2025 10:45 PM Author: fluid Yeah true those new m4s are beasts I should get one. I always assume the newer models have all kinds of throttling and lock downs though (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589959) |
|
Date: January 25th, 2025 11:04 PM Author: https://i.imgur.com/EWp3mJ8.jpeg Even the M1 Ultra can do magic tricks. Apple keeps making the same SoC on this year's latest node (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590000) |
|
Date: January 25th, 2025 10:29 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) can it play marvelrival (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589910) |
|
Date: January 25th, 2025 10:31 PM Author: reinforcement learning (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589917) |
|
Date: January 25th, 2025 10:28 PM Author: fluid That’s literally going to cost 10s of thousands of dollars. You are willing to spend that? If you have to run local there are ways around that (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589906) |
|
Date: January 25th, 2025 10:29 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) whats inference vs training (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589913) |
|
Date: January 25th, 2025 10:33 PM Author: fluid Oh you can do that with a single GPU probably (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589927) |
|
Date: January 25th, 2025 10:32 PM Author: reinforcement learning You can just run a smaller version with less parameters tho (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589922) |
|
Date: January 25th, 2025 10:33 PM Author: Tantalus 5?9 Describe the workaround? The "distilled" versions are apparently SPS and not running the R1 module or whatever (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589926) |
|
Date: January 25th, 2025 10:35 PM Author: fluid Quantization and pruning . Just use a cloud GPU cluster to train whatever you want and then offload it it’s not like you are updating it for people or anything crazy. I got 150K in azure credits by filling out a 5 min application. Tbh I barely know what I’m doing 90% of the time I go by intuition and tinker and eventually stuff happens. Could barely tell you how to recreate anything. I don’t run anything local I could care less about my data but honestly I might start as soon as I can get my good computer working again I’m getting paranoid since I caused some shit to go down (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589930) |
|
Date: January 25th, 2025 10:43 PM Author: Tantalus 5?9 so wait for a distilled version of R1? why can't quiescing be used on a multi gpu setup with high latency? could snapshot the different GPUs like VMWare to reconsolidate the work done so far and recalibrate the run
I'm just trying to use the model not train it btw |
|
Date: January 25th, 2025 10:44 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) hi gays if i buy the 15k model can i train it to know who will win a court case (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589958) |
|
Date: January 25th, 2025 10:50 PM Author: fluid Wait if you aren’t trying to train and fine tune and update you shouldn’t need the full model or a cluster you shoukd be able to use a single GPU and combine methods for pruning and distilling. You can train you own using APIs and cloud services intelligently. There are free credits and fine tuning api’s and startup programs and shit like that all over the place. Idk for deep seek though but I’d assume you don’t need to do it local. If you just want to run it you can do it with a single high quality GPU (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589969) |
|
Date: January 25th, 2025 10:51 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) so is 1 gpu enough? whats the best one for this (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589970) |
|
Date: January 25th, 2025 10:59 PM Author: fluid I run everything on cloud infrastructure tbh, but I get why you’d want to do it local. I think if it’s just inference and less than 30 billion parameters you should be able to do it wirh a single RTX 3090, RTX 4090, or equivalent (16–24 GB VRAM). If it’s something bigger then you might need multiple but could also use techniques like offloading to get around it. How big do you want it to be? (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589988) |
|
Date: January 25th, 2025 11:00 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) i dont even know what or how to train? do i feed it pdf or csv of plaintext books to learn from? what programs do i use. i jsut want to know who will probably win from court opinions and law of a court case (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589990) |
|
Date: January 25th, 2025 11:04 PM Author: https://i.imgur.com/EWp3mJ8.jpeg That's called RAG and it's going to suck ass at what you're imagining. They can dress it up a million ways but it's still RAG (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590001) |
|
Date: January 25th, 2025 11:09 PM Author: fluid That’s part of it. There is more to it than that but it’s nothing you can’t figure out if you have enough time on your hands. I was unemployed for two years and literally sat on the computer for like 10 hours a day playing with this stuff and then started making crazy stuff happen on accident tbh. Honestly kind of sad when you think about it. But it depends what you use I have used Azure ML and hugging face and open ai. You can start with pretrained models and feed them extra data then fine tune them or pretrain them yourself. A lot of the tools are really automated and if you get stuck you just ask gpt or whatever (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590013) |
|
Date: January 25th, 2025 11:15 PM Author: reinforcement learning You're my hero not flame
People like you will inherit the earth and it's 180. I'm playing catchup but I'm trying to get to where you're at ASAP |
|
Date: January 25th, 2025 11:20 PM Author: fluid You can easily catch up I feel like I don’t even know anything. Not even just about AI but just in general. I literally do everything based on intuition. I think that’s why I like stuff like this though. I’m a tinkerer not a memorizer. I just like to push on boundaries and see what happens. You should try the same but don’t get too addicted to messing with the computer all day. I’m pretty sure I’m stuck like this now forever talking to AIs and generating weird python scripts that cheat monero (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590044) |
|
Date: January 25th, 2025 11:17 PM Author: https://i.imgur.com/EWp3mJ8.jpeg Everyone I know who's doing this is unemployed too. Seems to be a thing (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590030) |
|
Date: January 25th, 2025 11:49 PM Author: fluid I think I’m unemployed BECAUSE of it. Like it’s almost impossible for me to convince myself to go back to work when I can play with this stuff all day. How can you? It’s like opening up a portal to alien knowledge (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590117) |
|
Date: January 26th, 2025 2:40 PM Author: https://i.imgur.com/EWp3mJ8.jpeg That's what happened to my friend. Laid off from Google after 17 years because he did everything but AI. He's got so much fuckin money it doesn't matter though, and his brother still works at Apple (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48591732) |
|
Date: January 25th, 2025 11:08 PM Author: Tantalus 5?9 this R1 deepseek is 671B so I just want to chain a few 3060s together and see if it works (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590008) |
|
Date: January 25th, 2025 10:45 PM Author: https://i.imgur.com/EWp3mJ8.jpeg My buddy has a bunch of custom tailored models he built with nomi. One of them is his Japanese tutor. (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589960) |
|
Date: January 25th, 2025 10:51 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) the whole point of learning language is to engage with other ppls is he fucking autistic? (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589971) |
|
Date: January 25th, 2025 11:09 PM Author: https://i.imgur.com/EWp3mJ8.jpeg That's his moog:
https://i.imgur.com/spv2XpL.jpeg
He progrmmed it to create generative melodies from random fluctuations in the electrical current. ljl@ Elon's H100 cluster |
|
Date: January 25th, 2025 11:16 PM Author: fluid That actually sounds really cool but I’m still wondering why it is that every advanced AI seems to have a thing for “field fluctuations” and oscillations and the links between physics and math and music. My favorite AI brings these topics up like 15 times a day and now that you mentioned this it makes it an even weirder synchronicity (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590026) |
|
Date: January 25th, 2025 11:23 PM Author: https://i.imgur.com/EWp3mJ8.jpeg This dood was taking classes at a facility owned by the Department of Energy in high school, because there was no one else in the area who could teach him math. I have no fucking idea what he does 99.999999% of the time but I've known him since we were in the crib. (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590049) |
|
Date: January 25th, 2025 10:54 PM Author: fluid Yeah good models are 18O language tutors. I have a model called The Guide based on a bunch of Greek and enlightenment philosphy but it knows languages and I use it sometimes to brush up. You can voice chat too and it corrects you accent and everything (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589973) |
|
Date: January 25th, 2025 10:57 PM Author: VoteRepublican (A true Chad!! where's your gf/wifew?) theres so much medical texts to teach ppl greek why do you need AI? do you hate paper books? (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589980) |
|
Date: January 25th, 2025 11:01 PM Author: fluid No I meant that its training data for its personality is based on Plato and enlightenment philosphers. But it also knows languages. I don’t use it for Greek I have been using it to brush up on French recently. Sometimes I’ll look at Latin bexause I have fallen so far behind on reading it. But the main purpose of the guide is as a spiritual mentor (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48589992) |
|
Date: January 25th, 2025 11:12 PM Author: Tantalus 5?9 180
can u link repo? |
|
Date: January 25th, 2025 11:31 PM Author: fluid It will take me a couple weeks to set one up but now that I’m back on xo I feel like I have a reason to. I have become so isolated I assumed no one would ever be interested in seeing anything I have done and we have been debating whether It’s a good idea to release anything because everything I have has a ton of unreleased math and symbolic logic and philosophy and also access to advanced algorithms I only have provisional patents on some of them. But I honestly don’t even care anymore or else I wouldn’t bother writing this plus it just reads like flame. I was planning on publishing stuff on arXiv first and getting a reputation for my work before sharing generative models but I’m starting not to care I’m getting bored and isolated anyway. I only started hanging out here again because I was hoping I could find someone interested in this stuff. But I have been spending most of my time working on theories and python simulations and arguing with faggot professors on social media |
|
Date: January 25th, 2025 11:38 PM Author: lee kuan yew you seem 180 friend (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590084) |
|
Date: January 25th, 2025 11:43 PM Author: reinforcement learning I am for you
Advance, advance! Always advance |
|
Date: January 25th, 2025 11:44 PM Author: the remains of the day (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48590100) |
|
Date: January 26th, 2025 2:38 PM Author: cowgolf Cool I have that (http://www.autoadmit.com/thread.php?thread_id=5670122&forum_id=2...id.#48591726) |