Threads for stormtrooper1721

  1. 3

    rewriting my site (https://bkkaggle.github.io/) in Rust+Yew. I’ve got most of it done but i still have quite a few features that I want to add (markdown blogs, dark mode, a /uses page)

    1. 2

      Taking the last few weeks off before the next 5 years of school and work

      1. 3

        I’m writing up a blog post on making a pytorch style autograd library in Rust. It’s still a wip but i have a draft up https://bilal.software/2020/8/02/l2/

        Any feedback would be welcome!

        1. 2

          Ahhh this reminds me of all the hoops I had to jump through to make a mutable computation graph for automatic differentiation in rust.

          I ended up just storing a reference to each parent in a node and wrapping the value of the derivative that I wanted to mutate in a RefCell and calling *t.derivative.borrow_mut() = d_dx.data after computing the derivative for a node

          Rust does (rightfully so) feel overly complicated and verbose at first, but the feeling of having the confidence that your compiled code will work (most of the time!) can’t be beat

          1. 2

            Ahhh this reminds me of all the hoops I had to jump through to make a mutable computation graph for automatic differentiation in rust.

            This sounds neat. Could you maybe explain it a bit more?

            1. 3

              Most likely it is referring to this submission yesterday.

          1. 1

            Do you think that a rust-based library can outperform LibTorch? If so, in what ways?

            1. 3

              You should be able to match pytorch on cpu pretty easily and my library is already within ~3x slower than pytorch on cpu for most ops (I don’t think there is much of a difference in speed for pytorch and libtorch since they should use the same cpp backend).

              I added BLAS accelerated matmul to l2 at the last minute and hacked together some really messy code with a whole bunch of transposes (since I store my tensors in row-major format and the Fortran BLAS lib wants it in column-major) and it made my matmul 100x faster than a normal for loop based implementation and within about 30us of pytorch IIRC

              As for gpus if you use CUDA and cudnn kernels properly (that’s a little out of my depth rn) it should be pretty close to pytorch’s speed as well.

              The really interesting part (imo) where rust can shine would be to fix what all the major ml libraries are working on rn: making fast, efficient code run on the GPU without the need for carefully handwritten cudnn kernels.

              Pytorch has its JIT that makes an IR of the scripted function and optimizes it. The tf team is working on xla which is another IR that’s supposed to optimize ops by fusing together ops and preventing a whole bunch of expensive read/writes to the gpu’s memory. JAX is doing something similar and builds on top of xla and gives you a kinda functional approach to ml. Swift4TF was (is?) a really ambitious project that wanted to make autodiff and xla optimizations a core part of swift. I think Julia is also working on approaches that are quite a bit like Swift4TF but I haven’t looked into it too much

              Now to rust: Rust has a big emphasis on safety and i think it has a lot of potential in ml where you could take it the xla/swift4tf route and try to make a “compiler” for tensor ops.

              Another thing that I wanted to work on but really didn’t have the time for in this version of l2 was to use rusts const generics (still in nightly but hopefully being finalized soon) to run compile-time shape checks on tensors and the outputs of tensor ops.

              Python is a really nice language since either lets you focus more on making your code do what you want it to do and less on how to write that code. but I’ve found a lot of bugs in my python code where I end up getting a runtime exception about two tensors not being compatible/broadcastable and imo this would be a really neat thing to try and make as a proof of concept to encourage more people to work on building the future of ml libraries and infra.

              This is just a brain dump of a whole lot of things that I’ve been thinking about lately and I’m not 100% sure that everything I’ve said is 100% correct so I’d really appreciate it if anyone could correct me on the things that I’ve got wrong :)

            1. 3

              I’ll be writing up a blog post soon that goes into a lot of detail of exactly how I made this library step-by-step.

              LMK if there is anything that you’d like to be covered in the post or if you have any questions

              1. 2

                I’ve been using a similar tool based on the Roam/Zettelkasten note taking way called foam https://foambubble.github.io/foam/

                It integrates pretty well with my editor (vscode) and is really lightweight since all it does is really add backlinks to markdown files and converts them into a static site. so far I’ve been liking it as a pretty nice upgrade over my previous setup of just having a folder of unrelated markdown files

                1. 13

                  The problem with Foam is that it requires you to use VSCode, whereas the philosophy behind neuron is that it is not tied to any text editor. If I’m forced to use a particular text editor, it violates the future-proof property.

                  Neuron does have editor support, but they are decoupled from neuron itself (the editor plugins use neuron query‘s JSON output to interact with neuron), which will be perfected through Language Server Protocol. I believe decoupling is the right way, so I’ve suggested to Foam’s author to consider integrating with neuron, which would also bring the benefits of one tool to the other and vice versa.

                  1. 3

                    Any word when the LSP integration might be done? I’ve been looking for a new note-taking app. This seems close to something I might like.

                    1. 2

                      Can’t say when, but Trevor just started looking into implementing it.

                    2. 1

                      Being editor-agnostic is a good idea but I’m using it mostly within vscode which integrates well within my workflow and doesn’t need me to have a command running in the background.

                      but integrating it into a lsp protocol to take care of the behind-the-scenes implementation (backlinks, autocomplete for maths, and stuff ) is a really great idea!

                    3. 1

                      Interesting. I’ve been wondering if someone would do something like this. I’m currently using Obsidian, kinda. I use Vim to do the actual writing and only use Obsidian to render the markdown. I hate leaving Vim. Even when apps have Vim bindings, it’s not the same.

                      On the other hand, if Obsidian were just a VSCode plugin like Foam, I feel like that might limit the audience to mainly programmers.

                    1. 1

                      I use vscode notebooks every day for prototyping my ml models and scripts so this is a really nice thing to hear.

                      The extension isn’t that bad as is, but working in it still feels like a very different app than vscode (e.g. you have to click inside a code cell before switching between notebook and editor tabs with cmd-t)

                      1. 1

                        I finished writing up a blog post a few days ago on how i was working on (and was successfully able to!) replicate and retrain one of OpenAI’s GPT-2 models (https://bilal.software/2020/7/17/nlp-series-2/)

                        This is one of my first attempts at writing long-form technical posts, so any tips/feedback would help :)

                        I would’ve posted it to lobste.rs but couldn’t since I’m a new user and the domain hasn’t been submitted before

                        1. 2

                          The specs are pretty impressive on paper, but I don’t see this getting a lot of widestream adoption unless their support for Pytorch and TF is as simple as just using a GPU or TPU/TPU Pod

                          1. 3

                            I had a similar problem when loading in a large text file into memory with python (the 550mb wikitext 103 dataset) which iirc took around 5+ gb of ram. I haven’t tried it yet, but the apache arrow project looks pretty interesting for serializing and loading in large objects or files quickly since they use some sort of memory-mapped serialization protocol. The huggingface/nlp library uses this and can iterate over a 17gb text file at about 2+ GB/S (https://twitter.com/thom_wolf/status/1272512974935203841)

                            1. 3

                              Im implementing a few image-to-image synthesis GANs like Pix2Pix (https://phillipi.github.io/pix2pix/) and pix2pixHD (https://github.com/NVIDIA/pix2pixHD) from scratch in Pytorch with the end goal of working on making a higher-resolution and quality version of the few-shot video-to-video synthesis (https://github.com/NVlabs/few-shot-vid2vid) paper