Pretty sure this has absolutely nothing to do with Deepseek and even local LLM at large, which has been a thing for a while and an obvious use case original Llama leak and llama.cpp coming around.

Fact is Mac Pros in the Intel days supported 1.5TB RAM in some configurations[1] and that was 6 years ago expectations of their high end customer base. They needed to address the gap for those customers so they would have shipped such a product regardless. Local LLM is cherry-on-top. Deepseek in particular almost certainly had nothing to do with it. They will still need to double their supported RAM in their SoC to get there. Perhaps in a Mac Pro or a different quad-Max-glued chip.

[1]: https://support.apple.com/en-us/101639

That or it's the luckiest coincidence! In all seriousness, Apple is fairly consistent about not pushing specs that don't matter and >256GB is just unnecessary for most other common workloads. Factors like memory bandwidth, core count and consumption/heat would have higher impact.

That said, I doubt it was explicitly for R1, but rather based the industry a few years ago when GPT 3s 170B was SOTA, but the industry was still looking larger. "As much memory as possible" is the name of the game for AI in a way that's not true for other workloads. It may not be true for AI forever either.

Design work on the Ultra would have started 2-3 years ago, and specs for memory at least 18 months ago. I’m not sure they had that kind of inside knowledge for what Deepseek specifically was doing that far in advance. Did Deepseek even know that long ago?

> they specifically built this M3 Ultra for DeepSeek R1 4-bit

Which came out in what, mid January? Yeah, there's no chance Apple (or anyone) has built a new chip in the last 45 days.

I don’t think you understand hardware timelines if you think this product had literally anything to do with anything DeepSeek.

$10k to run a 4 bit quantized model. Ouch.

> they specifically built this M3 Ultra for DeepSeek R1 4-bit.

This makes sense. They started gluing M* chips together to make Mac Studios three years ago, which must have been in anticipation of DeepSeek R1 4-bit

Any ideas on power consumption? I wonder how much power would that use. It looks like it would be more efficient than everything else that currently exists.

[dead]

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

It's fast enough for me to cancel monthly AI services on a mac mini m4 max.

AFAICT the neural engine has accelerators for CNNs and integer math, but not the exact tensor operations in popular LLM transformer architectures that are well-supported in GPUs.

Memory bandwidth is the issue

It may not be Firefox in terms of hundreds or thousands of tabs but Chrome has gotten a lot more memory efficient since around 2022.

[flagged]

It doesn't support Macs yet

I assume they mean SRAM versus unified (D)RAM?

Also should note that 800/819GB/s of memory bandwidth is actually VERY usable for LLMs. Consider that a 4090 is just a hair above 1000GB/s

This prompts an "old guy anecdote"; forgive me.

When I was much younger, I got to work on compilers at Cray Computer Corp., which was trying to bring the Cray-3 to market. (This was basically a 16-CPU Cray-2 implemented with GaAs parts; it never worked reliably.)

Back then, HPC performance was measured in mere megaflops. And although the Cray-2 had peak performance of nearly 500MF/s/CPU, it was really hard to attain, since its memory bandwidth was just 250M words/s/CPU (2GB/s/CPU); so you had to have lots of operand re-use to not be memory-bound. The Cray-3 would have had more bandwidth, but it was split between loads and stores, so it was still quite a ways away from the competing Cray X-MP/Y-MP/C-90 architecture, which could load two words per clock, store one, and complete an add and a multiply.

So I asked why the Cray-3 didn't have more read bandwidth to/from memory, and got a lesson from the answer that has stuck. You could actually see how much physical hardware in that machine was devoted to the CPU/memory interconnect, since the case was transparent -- there was a thick nest of tiny blue & white twisted wire pairs between the modules, and the stacks of chips on each CPU devoted to the memory system were a large proportion of the total. So the memory and the interconnect constituted a surprising (to me) majority of the machine. Having more floating-point performance in the CPUs than the memory could sustain meant that the memory system was oversubscribed, and that meant that more of the machine was kept fully utilized. (Or would have been, had it worked...)

In short, don't measure HPC systems with just flops. Measure the effective bandwidth over large data, and make sure that the flops are high enough to keep it utilized.

Yep, it's apples to oranges. But sometimes you want apples, and sometimes you want oranges, so it's all good!

There's a wide spectrum of potential requirements between memory capacity, memory bandwidth, compute speed, compute complexity, and compute parallelism. In the past, a few GB was adequate for tasks that we assigned to the GPU, you had enough storage bandwidth to load the relevant scene into memory and generate framebuffers, but now we're running different workloads. Conversely, a big database server might want its entire contents to be resident in many sticks of ECC DIMMs for the CPU, but only needed a couple dozen x86-64 threads. And if your workload has many terabytes or petabytes of content to work with, there are network file systems with entirely different bandwidth targets for entire racks of individual machines to access that data at far slower rates.

There's a lot of latency between the needs of programmers and the development and shipping of hardware to satisfy those needs, I'm just happy we have a new option on that spectrum somewhere in the middle of traditional CPUs and traditional GPUs.

As you say, if Nvidia made a 512 GB card it would cost $150k, but this costs an order of magnitude less than that. Even high-end consumer cards like a 5090 have 16x less memory than this does (average enthusiasts on desktops have maybe 8 GB) and just over double the bandwidth (1.7 TB/s).

Also, nit pick FTA:

> Starting at 96GB, it can be configured up to 512GB, or over half a terabyte.

512 GB is exactly half of a terabyte, which is 1024 GB. It's too late for hard drives - the marketing departments have redefined storage to use multipliers of 1000 and invented "tebibytes" - but in memory we still work with powers of two. Please.

GH200 is nowhere near $343,000 number. You can get a single server order around 45k (with inception discount). If you are buying bulk, it goes down to sub-30k ish. This comes with a H100's performance and insane amount of high bandwith memory.

Apple is shipping today. No future promises.

That will only have 128GB of unified memory

https://digitalspaceport.com/how-to-run-deepseek-r1-671b-ful...

https://news.ycombinator.com/item?id=42897205

3.5 - 4.5 tokens/s on the $2,000 AMD Epyc setup. Deepseek 671b q4.

The AMD Epyc build is severely bandwidth and compute constrained.

~40 tokens/s on M3 Ultra 512GB by my calculation.

You'll need a couple of 32GB 5090s to run a quantized 70B model, maybe 4 to run a 70b model without quantization, forget about anything larger than that. A huge model might run slow on a M3 Ultra, but at least you can run it all.

I have a Max M3 (the non-binned one), and I feel like 64GB or 96GB is within the realm of enabling LLMs that run reasonable fast on it (it is also a laptop, so I can do things on planes or trips). I thought about the Ultra, if you have 128GB for a top line M3 Ultra, the models that you could fit into memory would run fairly fast. For 512GB, you could run the bigger models, but not very quickly, so maybe not much point (at least for my use cases).

That config would also use about 10x the power, and you still wouldn't be able to run a model over 32GB whereas the studio can easily cope with 70B llama and plenty of space to grow.

I think it actually is perfect for local inference in a way that build or any other pc build in this price range would be.

Consumer hardware is cheap, if 192 GB of RAM is enough for you. But if you want to go beyond that, the Mac Studio is very competitively priced. A minimal Threadripper workstation with 256 GB is ~$7400 from Puget Systems. If you increase the memory to 512 GB, the price goes up to ~$10900. Mostly because 128 GB modules are about as expensive as what Apple charges for RAM. A Threadripper Pro workstation can use cheaper 8x64 GB for the same capacity, but because the base system is more expensive, you'll end up paying ~$11600.

The Mac almost fits in the palm of your hand, and runs, if not silently, practically so. It doesn't draw excessive power or generate noticeable heat.

None of those will be true for any PC/Nvidia build.

It's hard to put a price on quality of life.

Is there a teardown link available for what you wrote? If so, that’s interesting.

I mean expensive relative to who, Nvidia? Both are enjoying little to no competition in their respective niche and are using that monopoly power to extract massive margins. I have no doubt it could be much cheaper if there was actual competition in the market.

Fortunately it seems like AMD is finally catching on and working towards producing a viable competitor to the M series chips.

And yet all that cash still just goes to TSMC

The CPUs with iGPUs didn't also have the memory on-chip. The Nintendo 64 did. Not sure about the old home computers, but I thought those had separate memory usually.

Like pretty much every game console.

M1 came out before the LLM rush, though

And game consoles that use similar parts as laptops.

It's regular memory on separate chips.

The rackmount Mac Pro is for A/V studios, not datacenters.

I genuinely forgot the Mac Pro still exists. It’s been so long since I even saw one.

And I’ve had every previous Mac tower design since 1999: G4, G5, the excellent dual Xeon, the horrible black trash can… But Apple Silicon delivers so much punch in the Studio form factor, the old school Pro has become very niche.

Edit - looks like the new M3 Ultra is only available in Mac Studio anyway? So the existence of the Pro is moot here.

Outside of extremely niche use cases, who is racking apple products in 2025?

A 3 year reservation of a P5 is over a million dollars though? Not sure how that's comparable....

To add to this GPU servers like supermicro have a 400GBe port per GPU plus more for the CPU.

Cost competitive though?

The weights don't go over the network so performance is OK.

Probably some kind of local LLM server. 1TB of 1.6 TB/s memory if you link 2 together. $20k total. Half the price of a single Blackwell chip.

Its the same host-to-host usb network, I believe.

I'm super interested in the clustering capability. At launch people said they were only getting like 11Gbps from their TB4 drive arrays, which was really way less than expected.

Apple does kind of advertise that each TB port has its own controllers. Which gives me hope that whatever 1x port can do 6x can do 6x better.

AMD's Strix Halo victory feels much more shallow today. Eventually 48GB or 64GB sticks will probably expand Strix Halo to 192 then 256GB. But Strix Halo is super super io starved, is basically a desktop of IO, with no way to easily host-to-host, and Apple absolutely understands that the use of a chip is bounded by what it can connect to. 6x TB5, if even half true, will be utterly outstanding.

It's been so so so so cool to see Non-Transparent Bridging atop thunderbolt, so one host can act like a device. Since it's PCIe, that hypothetically would allow amazing RDMA over TB. USB4 mandates host to host networking, but I have no idea how it is implemented and I suspect it's no where near as close to the metal.

Well already it is faster than GigE...

https://arstechnica.com/gadgets/2013/10/os-x-10-9-brings-fas...

Thunderbolt is PCIe-based and I could imagine it being extended to do what https://en.wikipedia.org/wiki/Compute_Express_Link and https://en.wikipedia.org/wiki/InfiniBand

That’s a laptop part, so it makes different tradeoffs.

Somewhere on the internet there is a tdp wattage vs performance x-y plot. There’s a pareto optimal region where all the apple and amd parts live. Apple owns low tdp, AMD owns high tdp. They duke it out in the middle. Intel is nowhere close to the line.

I’d guess someone has made one that includes datacenter ARM, but I’ve never seen it.

In 1998-1999 I had a DEC Alpha on my desktop that was really impressive, it was a 64-bit machine a few years before you could get a 64-bit Athlon.

At my last N jobs, companies built high end server farms and carefully specced all the hardware. Then they looked at SSD specs and said “these are all fine”.

Fast forward 2 years: The $50-$250K machines have a 100% drive failure rate, and some poor bastard has to fly from data center to data center to swap the $60 drive for a $120 one, then re-rack and re-image each machine.

Anyway, soldering a decent SSD to the motherboard board would actually improve reliability at all those places.

Hmm, this page links to an out-of-tree ANE module (same as on M1/M2 I believe). No GPU support is a bummer, though.

On the other hand, you can do without display support if you’re only using it as a server. And I think USB Ethernet dongles might work for the time being?

Your wording makes it sound like it was a one-man show. Asahi has a really strong contributor base, new leadership[1], and the backing of Fedora via the Asahi Fedora Remix. While Hector resigning is a loss, I don't think it's a death knell for the project.

[1]: https://asahilinux.org/2025/02/passing-the-torch/

You make it sound like there was only one.

Who cares about TrustZone? x86 doesn't have PAC or MTE.

The server markets want ECC and Apple has largely stopped shipping ECC. I don’t recall seeing any assurances that the ARM Mac Pro had ECC.

Ubiquiti was founded by some Apple employees after they closed their Airport division. I sincerely doubt they want to go through the trouble of collaborating with a company that was too greedy to keep investing in their network hardware.

Ubiquiti is decently priced, especially for niche hardware, unlike Apple. Fundamentally they diverge on the way to do things...

> for folks like my family members, it'd be a killer addition

HomeKit networking existed in Eero briefly. I put that in a lot of casual Apple homes. Seemed like missed oppty for Apple to let Amazon buy Eero, a more "spiritual successor" to the Airports.

Which means this is a whole new chip. It may be M3 based, but with added interposer support and new thunderbolt stuff.

Which, at this point, why not just use M4 as a base?

Sounds like you’ve seen some things.

Note that they said (emphasis mine):

> they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.

They're not talking about Apple's silicon as a target, but as a development platform.

> But you have to bring your own engineering effort to actually make another OS run.

I mean, that's usually how it works though. When IBM launched the PS/2, they didn't support anything other than PC-DOS and OS/2, Microsoft had to make MS-DOS work for it (I mean... they did get support from IBM, but not really), the 386BSD and Linux communities brought the engineering effort without IBM's involvement.

When Apple was making Motorola Macs, they may have given Be a little help, but didn't support any other OSes that appeared. Same with PowerPC.

All of the support for alternative OSes has always come from the community, whether that's volunteers or a commercial interest with cash to burn. Why should that change for Apple silicon?

Gotcha. Thanks!

You're forgetting what Apple's been baking into their silicon for (nearly? over?) a decade: the Neural Processing Unit (NPU), now called the "Neural Engine". That's their secret sauce that makes their kit more competitive for endpoint and edge inference than standard x86 CPUs. It's why I can get similarly satisfying performance on my old M1 Pro Macbook Pro with a scant 16GB of memory as I can on my 10900k w/ 64GB RAM and an RTX 3090 under the hood. Just to put these two into context, I ran the latest version of LM Studio with the deepseek-r1-distill-llama-8b model @ Q8_0, both with the exact same prompt and maximally offloaded onto hardware acceleration and memory, with a context window that was entirely empty:

  Write me an AWS CloudFormation file that does the following:
  
  * Deploys an Amazon Kubernetes Cluster
  * Deploys Busybox in the namespace "Test1", including creating that Namespace
  * Deploys a second Busybox in the namespace "Test3", including creating that Namespace
  * Creates a PVC for 60GB of storage

  * 21.28 seconds for "Thinking"
  * 0.22s to the first token
  * 18.65 tokens/second over 1484 tokens in its responses
  * 1m:23s from sending the input to completion of the output

  * 10.88 seconds for "thinking"
  * 0.04s to first token
  * 58.02 tokens/second over 1905 tokens in its responses
  * 0m:34s from sending the input to completion of the output

Had the M3 GPU been much wider, it would be constrained by the memory bandwidth. It might still have an advantage over Nvidia competitors in that it has 512GB accessible to it and will need to push less memory across socket boundaries.

It all depends on the workload you want to run.

From my outsider perspective, it's pretty straightforward why they don't.

In Intel's case, there's ample coverage of the company's lack of direction and complacency on existing hardware, even as their competitors ate away at their moat, year after year. AMD with their EPYC chips taking datacenter share, Apple moving to in-house silicon for their entire product line, Qualcomm and Microsoft partnering with ongoing exploration of ARM solutions. A lack of competency in leadership over that time period has annihilated their lead in an industry they used to single-handedly dictate, and it's unlikely they'll recover that anytime soon. So in a sense, Intel cannot make a similar product, in a timely manner, that competes in this segment.

As for AMD, it's a bit more complicated. They're seeing pleasant success in their CPU lineup, and have all but thrown in the towel on higher-end GPUs. The industry has broadly rallied around CUDA instead of OpenCL or other alternatives, especially in the datacenter, and AMD realizes it's a fool's errand to try and compete directly there when it's a monopoly in practice. Instead of squandering capital to compete, they can just continue succeeding and working on their own moat in the areas they specialize in - mid-range GPUs for work and gaming, CPUs targeting consumers and datacenters, and APUs finding their way into game consoles, handhelds, and other consumer devices or Edge compute systems.

And that's just getting into the specifics of those two companies. The reality is that any vendor who hasn't already unveiled their own chips or accelerators is coming in at what's perceived to be the "top" of the bubble or market. They'd lack the capital or moat to really build themselves up as a proper competitor, and are more likely to just be acquired in the current regulatory environment (or lack thereof) for a quick payout to shareholders. There's a reason why the persistent rumor of Qualcomm purchasing part or whole of Intel just won't die: the x86 market is rather stagnant, churning out mediocre improvements YoY at growing pricepoints, while ARM and RISC chips continue to innovate on modern manufacturing processes and chip designs. The growth is not in x86, but a juggernaut like Qualcomm would be an ideal buyer for a "dying" or "completed" business like Intel's, where the only thing left to do is constantly iterate for diminishing returns.

This is not correct. It is like VRAM, not like normal PC RAM.

Lowest latency of DDR5-6400 on normal PC starting at 60ns+

Lowest latency of VRAM on GeForce RTX 4090 starting at 14 ns

Lowest latency of Apple M1 Memory starting at 5 ns, its more like L3 cache

And on Apple M chip, this ultrafast memory is available for CPU, GPU and NPU.

https://www.anandtech.com/show/17024/apple-m1-max-performanc... https://chipsandcheese.com/p/microbenchmarking-nvidias-rtx-4...

Fun at parties and stuff.

You could always buy one at an apple store without sales tax (e.g. Portland Oregon). But they might not have that one in stock...

The bargain is the lower price in the UK compared to US, once US sales tax is added. It's not like the pound is strong. It's just cheaper in the UK. And you're right, all Apple products are better value in that UK. I'm not used to any electronics being good value in the UK.

It's also just clearly a powerful and interesting tinkering project, which there are valid arguments for, but this can just chill out on your desk as an elegant general productivity machine. What it wouldn't do that the tinkering project could do is be upgraded, act as a powerful gaming pc, or cause migraines from constant fan noise.

The custom build would work great though, and even moreso in a server room and as well continues to reveal by comparison how excessively Apple prices it's components.

Is there a reason one couldn't use linux?

The PC doesn't have to run Windows either. Strictly speaking, professional applications see MacOS support as an Apple-sanctioned detriment.

> If you've ever used git, svn, or an IDE side by side

I still reach for Windows, even though it's a dogshit OS. I would rather use WSL to write and deploy a single app, as opposed to doing my work in a Linux VM or (god forbid) writing and debugging multiple versions just to support my development runtime. If I'm going to use an ad-encumbered commercial service-slop OS, I might as well pick the one that doesn't actively block my work.

Two separate laptops, I could have been clearer. I have an old 2018 I wanted to try it on, and my daily is M2 that would have been next.

Surely you're using that thing as a laptop in a minority of cases though, looks like it's basically just specs you bought. That's fine, but if that's all you want then it seems like rather than trying to give a mac a reasonable go of it as opposed to whatever else, you were trying to instead explore a fundamental difference in how you value technology products, which is quite a different battle.

I'm not really moaning about the cost or lack of upgradability. I mean, I don't like it but at least you know what you're getting into. I just always assumed Linux as a backup was an option, and more and more OSX is annoying me (last 2 or 3 days it keeps dropping bluetooth for 30 seconds) and more and more I just find the interface distracting. Plus whether it works with external displays over USB C is a crapshoot.

I'll miss the battery life of the M1 chips, and I'm going to have to re-learn how to type (CTRL instead of ALT, fn rarely being on the left, I use fn+left instead of CTRL A in terminals) but otherwise, I think I'm done.

Something like the brightness buttons not working, or sleep being a little erratic is ok. No released wifi drivers, bluetooth issues, and audio and the keyboard not working are not ok. Apple going backwards in terms of supporting Linux is not something I'm ok with.

Getting Linux running wasn't difficult. But Mint lost audio (everything else worked), the specialised Mint kernel lost both audio and wifi, and Arch lost both wifi and the onboard keyboard.

I'm sure with tinkering I could eventually get it working, but I'm well past the point of wanting to tinker with hardware and drivers to get Linux working.

Because of the T2 chip it's actually pretty annoying. Mainline kernels I think are still missing keyboard and trackpad support for those models. Plus a host of other issues.

No, there's a bunch of MBP generations in the middle that just never got any Linux attention.

I could have written it clearer. I have both, Intel was the first attempt and when I was struggling to get it up without losing one of wifi, audio and onboard keyboard and read that ARM was worse I gave up. Even the best combination I had (no audio but everything else working) would kill bluetooth after a while if wifi was connected to 2.6. I don't like their hardware enough to fight with it.

Lack of focus on quality of software affects all types of workloads, not just consumer-oriented or professional-oriented in isolation.

This is incoherent to me. Your complaints are about packaging, but the elixir wrapper doesn't deal with that in any way -- it just wraps UV, which you could use without elixir.

What am I missing?

Also, typically when people say things like

> Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers

It means they have been trapped in a cycle of thinking "just one more tool will surely solve my problem", instead of realising that the tools _are_ the problem, and if you just use the official methods (virtualenv and pip from a stock python install), things mostly just work.

> at least be able to use Python, but in a very controlled, not-insane way

Thats funny, about 10 years ago I started my career in a startup that had Python business logic running under Erlang (via custom connector) which handled supervision and task distribution, and it looked insane for me at the time.

Even today I think it can be useful but is very hard to maintain, and containers are a good enough way to handle python.

Virtualenv’s been a thing for many years, it’s built into Python, and it solves all that without adding additional tooling.

And if you’re genuinely asking, everything’s converging toward uv. If you pick only one, use that and be done with it.

uv

(Not saying Apple should bundle that, but it's the best current answer to running many different Python projects without using something like Docker)

these are solved problems now, check back in. uv is now the standard

No one would want OOTB Python support. You'd be stuck on a version you didn't want to use.

> 1. I run Docker and Podman on my Macs.

That's using a Linux VM. The idea people are asking about is native process isolation. Yes you'd have to rebuild Docker containers based on some sort of (small) macOS base layer and Homebrew/Macports, but hey. Being able to even run nodejs or php with its thousands of files natively would be a gamechanger in performance.

Thanks for introducing me to `uv`. I've been looking for a tool that integrates pip and pyenv/virtualenv.

>I run Docker and Podman on my Macs.

The same way Windows users run them. In a linux VM.

You don't get real on-hardware containerization.

uv pre-installed! /s

> why do you want this in MacOS?

I have a small rackmounted rendering farm using mac minis, which outperform everything in the Intel world, even order of magnitude more expensive.

I run macOS on my personal and development computers for over a decade and I use Linux since inception on server side.

My experience: running server-side macOS is such a PITA it's not even funny. It may even pretend it has ssh while in fact the ssh server is only available on good days and only after Remote Desktop logged in at least once. Launchd makes you wanna crave systemd. etc, etc.

So, about docker. I would absolutely love to run my app in a containerized environment on a Mac in order to not touch the main OS.

> MacOS doesn't have the fundamental ecosystem problems that beget the need for docker.

Anyone wanting to run and manage their own suite of Macs to build multiple massive iOS and Mac apps at scale, for dozens or hundreds or thousands of developers deploying their changes.

xcodebuild is by far the most obvious "needs native for max perf" but there are a few other tools that require macOS. But obviously if you have multiple repos and apps, you might require many different versions of the same tools to build everything.

Sounds like a perfect use case for native containers.

Docker Desktop now offers an option to use the virtualization framework, and works pretty well. But you're still constantly running a VM because "docker is how devs work now right?". I agree with your comment.

Bandwidth, yes. Connections count, no.

I went to Transmission years and years ago because it's just simple. It has all the options if you need them, but no HUUUGE interface with RSS feeds, 10001 stats about your download, categories, tags, etc etc etc.

Transmission is just a small, floating window with your downloads. Click for more. It fits in the macOS vibe. But I'm a person that fully adopted the original macOS "way of working" - kicked the full-screen habit I had in windows and never felt better.

Can I ask, why would you go FROM Transmission to qBittorrent?

The Linux version, in a container no less, handles the entire gigabit bandwidth.

And let's be clear, it wasn't the app that had problems, the Apple Remote Desktop connection to the machine failed when the speeds got above 40MB/s and the network interface stopped working around 80MB/s.

I think Transmission works perfectly fine. I've been using it for 10+ years with no issues at all on Linux.

I forgot to mention this is a Mac mini/Intel (2018).

I haven't had any issue running BiglyBT on my M1 MacBook, granted I don't run it all day every day but everything runs plenty fast for my needs (25-30 MB/s for well-seeded torrents).

Apple are working on the hard problems of making AI useful (call them “agents”), not AGI

1. Small models running locally with well-established tool interfaces (“app intents”)

2. Large models running in a bespoke cloud that can securely and quickly load all relevant tokens from a device before running inference

No AI lab is even close to what Apple is trying to deliver in the next ~12 months

CUDA is becoming more critical, not less, every day. Software developed around CUDA is vastly outpacing what other companies produce. And saving a few millions when creating new models doesn't matter; NVIDIA is pretty efficient at scale.

I don't know if you've heard, but NVIDIA is about to add a monthly payment for additional CUDA features and I'm almost certain that many big companies will be happy to pay for them.

> But most models will run or retrain fine now without CUDA.

This is correct for some small startups, not big companies.

CUDA is incredibly important still. It's still an incredible amount of work to get packages working on multiple GPU paradigms, and by default everyone still starts with CUDA.

The example I always give is FFT libraries - if you compare cuFFT to rocFFT. rocFFT only just released support for distributed transforms in December 2024, something you've been able to do since CUDA Toolkit v8.0, released in 2017. It's like this across the whole AMD toolkit, they're so far behind CUDA it's kind of laughable.

While technically true, the NEMA 5-15R receptacles are rated for use on 20A circuits, and circuits for receptacles are almost always 20A circuits, in modern construction at least. Older builds may not be, of course.

That said, if your load is going to be a continuous load drawing 80% of the rated amperage, it really should be a NEMA 5-20 plug and receptacle, the one where one of the prongs is horizontal instead of vertical. Swapping out the receptacle for one that accepts a NEMA 5-20P plug is like $5.

If you are going to actually run such a load on a 20A circuit with multiple receptacles, you will want to make sure you're not plugging anything substantial into any of the other receptacles on that circuit. A couple LED lights are fine. A microwave or kettle, not so much.

To clarify, the circuit is almost always 20A with 15A being used for lighting. However, the outlet itself is almost always 15A because you put multiple outlets on a single circuit. You are going to see very few 20A in outlets (which have a T shaped prong) in residential.

I would check your breaker box as well. If a hair dryer trips anything then… well yeah probably older construction.

True. But with Project Digits supposedly around the corner, which supposedly costs $3,000 and supports ConnectX and runs Blackwell; what's the over-under on just buying two of those at about half the price of one maxed M3 Ultra Mac Studio?

True. But an AI shop doesn't care about that. They get more performance for the money by going for multiple Nvidia GPUs. I have 512 GB ram on my PC too with 8 memory channels, but it's not like it's usable for AI workloads. It's nice to have large amounts of RAM, but increasing the batch size during training isn't going to help when compute is the bottleneck.

Wouldn't multiple rtx gpus also have more bandwidth

I would still wait until I need it before buying it…

You have a point; technically they aren't impossible to run if you have enough system RAM (or hell, SSD/HDD space for that mater). But in practice neither running on the CPU, nor on the GPU by constantly paging data in and out of VRAM, is a very attractive option (~10x slowdown at least).

2x 5090s would only give you 64GB of memory to work with re:LLM workloads, which is what people are talking about in this thread. The 512GB of system RAM you’re referring to would not be useful in this context. Apple’s unified memory architecture is the part you’re missing.

How much VRAM do you get on those 2x 5090s?

How much would it cost to get up to 512gb?

Do you understand that it's UNIFIED RAM, so it doubles as vRAM? I would love to know what computer you can build for <10k with 0.5TB of VRAM.

> Mac Studio the new trash can?

Indeed, and tbh it really commits even more to the non-expandability that the Trashcan's designers seemed to be going for. After all, the Trashcan at least had replaceable RAM and storage. The Mac Studio has proprietary storage modules for no reason aside from Apple's convenience/profits (and of course the 'integrated' RAM which I'll charitably assume was done for altruistic reasons because of how it's "shared.")

The difference is that today users are accepting modern Macs where they rejected the Trashcan. I think it's because Apple's practices have become more widespread anyway*, and certain parts of the strategy like the RAM thing at least have upsides. That, and the thermals are better because the Trashcan's thermal design was not fit for purpose.

* I was trying to fix a friend's nice Lenovo laptop recently -- it turned out to just have some bad RAM, but when we opened it up we found it was soldered :(

Oh yea I wasn't clear I just meant bring back the design - agree the studio basically is the trash can.

Could be. I'm not sure if this current incarnation of the Mac Pro signals a commitment to the product though. Same performance as the Mac Studio but 2-3x the price just to get PCI slots.

That's CPU only memory, not high bandwidth, and not addressable by the GPU.

Ah seems like I remembered the CPU price for a higher tier CPU which can cost the 6k on their own.

Thinking about it you can get a decent 256gb on consumer platforms now too, but the speed will be a bit crap and would need to make sure the platform ully supports ECC UDIMMs

In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher.

Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000.

Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.

Even for computationally-limited programs, the difference between server CPUs and consumer GPUs is not great. One Epyc CPU may have about the same number of FP32 execution units as an RTX 4070, while running at a higher clock frequency (but it lacks the tensor units of an NVIDIA GPU, which can greatly accelerate the execution, where applicable).

moot point if tok/s benchmark results are the same or worse.

I said in parallel.

Ah, yes, you got me. You identified that other market that also no one cares about.

As I replied in else where here, I do not run any Apple Services on my Mac hardware. I do on my iDevices though, but that's a different topic. Again, I could be the edge case

All seven of them. I kid, I have a lot of sympathy for that position, but as a practical matter running Linux VMs on an M4 works great, you even get GPU acceleration.

All of us on HN are basically edge cases. The main target market of Macs is super dependent on Apple service subscriptions.

Maybe that's why they ship with insultingly-small SSDs by default, so that as people's photo libraries, Desktop and Documents folders fill up, Apple can "fix your problem" for you by selling you the iCloud/Apple One plan to offload most of the stuff to only live in iCloud.

Either they spend the $400 up front to get 2 notches up on the SSD upgrade, to match what a reasonable device would come with, or they spend that $400 $10 a month for the 40 month likely lifetime of the computer. Apple wins either way.

Same here.

There are a large number of macOS users that are not app software devs. There's a large base of creative users that couldn't code their way out of a wet paper bag, yet spend lots of money on Mac hardware.

This forum looses track of the world outside this echo chamber

> Apple has already been in this spot.

Has been. This is importance. Past tense. Maybe that's the point - they gave up on it acknowledging the extra costs / issues.

There you go using logical arguments in an emotional illogical debate.

If the energy efficiency of things like Face ID was indeed so far so bad that you need a more efficient M3 Ultra, how come Face ID was integrated into smartphones years ago, apparently without significant negative impact on battery life?

I think betting on low-power NPU hardware wasn't necessarily wrong - if you're Apple you're trying to optimise performance/watt across the system as a whole. So in a context where you're shipping first-party bespoke on-device ML features it can make sense to have a modestly sized dedicated accelerator.

I'd say the biggest problem with the NPU is that you can only use it from Core ML. Even MLX can't access it it!

As you say the big world-changing LLMs are scaling up, not down. At the same time (at least so far) LLM usage is intermittent - we want to consume thousands of tokens in seconds, but a couple of times a minute. That's a client-server timesharing model for as long as the compute and memory demand can't fit on a laptop.

I think the traditional reason for this is that other chips like to use complex scheduling logic to have more logical cores than physical cores. This costs single threaded speed but allows you to run more threads faster.

I agree, it's a lot of work, plus Apple definitely is not not going to help with the project. Maybe an alternative is something like Framework -- find some good enough hardware and support it.

At the time, battery replacement from Apple or an authorized dealer was $69.

https://support.apple.com/iphone/repair/battery-replacement

There is no “DRM” in their battery.

If you had the wherewithal, you could do it yourself

https://www.ifixit.com/Guide/iPhone+6s+Battery+Replacement/5...

The units weren’t “faulty” all batteries degrade over time.

nobody in industry is using a 4090, they are using H100s which have 3TB/s. Apple also doesn’t have any equivalent to nvlink.

I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops

i suspect that compute actually might be the limiter for these chips before b/w, but not certain

nope

that's not a professional line card, but a data center card.

You're adding detail that's not relevant to anything I said. I was saying this statement:

> VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.

Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.