First time submitting a story, the software I work on recently moved from C++ to Rust (in a Python extension) and this post goes thru a small PR moving some Python code into the Rust layer, and what changes need to be done (including the FFI layer).
Thank you for writing and submitting this. It is well written: easy to follow and fairly succinct.
I wonder if you could get a reasonable compromise between cpu and memory consumption by growing the list till it hits (say) 64 elements then flushing it? That way you’re amortizing the per-call ffi overhead by buffering, but your buffer doesn’t grow very large.
Runtime is slightly larger (77s vs 73s), but as I noted in the PR it’s the memory that will likely be the bottleneck in the future (as we grow to millions of signatures, instead of ~100k we have now).
First time submitting a story, the software I work on recently moved from C++ to Rust (in a Python extension) and this post goes thru a small PR moving some Python code into the Rust layer, and what changes need to be done (including the FFI layer).
Thank you for writing and submitting this. It is well written: easy to follow and fairly succinct.
I wonder if you could get a reasonable compromise between cpu and memory consumption by growing the list till it hits (say) 64 elements then flushing it? That way you’re amortizing the per-call ffi overhead by buffering, but your buffer doesn’t grow very large.
I hadn’t thought of that, that’s a great idea!
I’ll try it and report back, thanks!
I implemented you suggestion, and the memory consumption is negligible now: https://github.com/dib-lab/sourmash/pull/840
Runtime is slightly larger (77s vs 73s), but as I noted in the PR it’s the memory that will likely be the bottleneck in the future (as we grow to millions of signatures, instead of ~100k we have now).
Thanks again!
Sweet! Glad to hear it. <3
Without context I have zero clues what
sourmashis, a short intro or even just a link would improve the post.