LLaMA-rs: A Rust port of llama.cpp for fast LLaMA inference on CPU

23

LLaMA-rs: A Rust port of llama.cpp for fast LLaMA inference on CPU ☶ ai rust github.com
authored by setzer22 15 days ago | archive
Archive.org Archive.today Ghostarchive
| 7 comments

7

I’ve been lurking since forever, but someone invited me to share this thing I’ve been working on. Hoping self-posts are okay! 😄

Anyway, I rewrote llama.cpp in Rust so that it’s easier for me to embed it in my projects. It was fun, and learned a lot by doing it. Happy to answer questions!

3

jado 15 days ago | link

This uses the ggml tensor library which is written in C. I’d be curious to see how that lib compares to Rust-only implementations, if any with the same feature set exist. Maybe sometime soon I’ll stomach the 10k lines of C to see what all it contains.
1. 3
  
  setzer22 15 days ago | link
  
  I thought about this when porting. The part of the code using ggml is not too large and doesn’t seem to use any operations you wouldn’t find in other tensor libraries.
1

GeoffWozniak 15 days ago | link

Welcome to Lobsters! Self-posts are fine but avoid putting commentary in the text field, as per the submission guidelines (seen when you go to submit):

When submitting a URL, the text field is optional and should only be used when additional context or explanation of the URL is needed. Commentary or opinion should be reserved for a comment, so that it can be voted on separately from the story.
1. 2
  
  Xorlev 15 days ago | link
  
  Anyway, I rewrote llama.cpp in Rust so that it’s easier for me to embed it in my projects. It was fun, and learned a lot by doing it. Happy to answer questions!
  
  Seems like context/explanation to me.
  1. 1
    
    tumdum 15 days ago | link
    
    That content/explanation is not necessary to understand the ‘title’ and is available to anyone who actually opens the link.
  2. 1
    
    fs111 14 days ago | link
    
    Check the moderation log. These editorials get deleted all the time
2. 2
  
  setzer22 15 days ago | link
  
  Oops, sorry about that! Will keep it in mind for next time.