By default, MacOS allows 2/3rds of this RAM to be used by the GPU on machines with up to 36GB / RAM and up to 3/4s to be used on machines with >36GB RAM.
I haven’t tried mlx the library yet, but when I was using LM Studio with their MLX backend, I had to run sudo sysctl iogpu.wired_limit_mb=<something> to increase the VRAM limit. I wonder if it’s possible to run mlx-community/Llama-3.3-70B-Instruct-4bit on a 48GiB model with this “trick”. However, be aware that the system would panic when you try to load a big model or use a large context size when <something> is too big.
BTW: how do I install llm with llm-mlx under nix-darwin and home-manager? I got the following message, and llm-mlx does not seem available on nixpkgs (yet).
$ llm install llm-mlx
Install command has been disabled for Nix. If you want to install extra llm plugins, use llm.withPlugins([]) expression.
I’ve been wondering when a plugin might connect these tools :) Will begin using it straight away!
By the way, I found that if you have models in ~/.cache/huggingface/hub already, due to having used mlx-lm directly, the llm mlx download-model mlx-community/… command will skip the download and just add them to llm’s index.
When comparing popular quant q4_K_M on Llama.cpp to MLX-4bit, in average, MLX processes tokens 1.14x faster and generates tokens 1.12x faster. This is what most people would be using.
That was before the small model performance improvement mentioned in the article.
Assuming you’re still using your Mac with 64 GB of RAM that you’ve previously mentioned, what’s the largest context window you’ve managed to use so far with this solution?
Note that macOS reserves some of the “unified memory” as system RAM.
I haven’t tried
mlxthe library yet, but when I was using LM Studio with their MLX backend, I had to runsudo sysctl iogpu.wired_limit_mb=<something>to increase the VRAM limit. I wonder if it’s possible to runmlx-community/Llama-3.3-70B-Instruct-4biton a 48GiB model with this “trick”. However, be aware that the system would panic when you try to load a big model or use a large context size when<something>is too big.BTW: how do I install
llmwithllm-mlxundernix-darwinandhome-manager? I got the following message, andllm-mlxdoes not seem available on nixpkgs (yet).If anyone is looking for guidance about how high to reasonably set this, consider the configure script attached to
exo.Huh, I didn’t know that about Nix. I’d love to understand what that message means so I can mention it in the LLM documentation.
Found this which looks very helpful: https://www.danielcorin.com/til/nix/installing-llm-with-plugins/
It looks like
withPluginshas been deprecated for a different option: https://github.com/NixOS/nixpkgs/blob/0b73e36b1962620a8ac551a37229dd8662dac5c8/pkgs/development/python-modules/llm/default.nix#L89I’ve been wondering when a plugin might connect these tools :) Will begin using it straight away!
By the way, I found that if you have models in ~/.cache/huggingface/hub already, due to having used
mlx-lmdirectly, thellm mlx download-model mlx-community/…command will skip the download and just add them tollm’s index.I would really like to see a speed (and memory usage) comparison of running mlx vs just ollama out-of-the-box.
Ollama is built on llama.cpp, so this should be relevant:
That was before the small model performance improvement mentioned in the article.
Assuming you’re still using your Mac with 64 GB of RAM that you’ve previously mentioned, what’s the largest context window you’ve managed to use so far with this solution?