A local-first experiment tracker that makes a git commit with metadata every time you run an experiment. Plus a CLI that lets you search for commits based on the metadata, e.g. metrics.accuracy > 0.9.
A local-first experiment tracker that makes a git commit with metadata every time you run an experiment. Plus a CLI that lets you search for commits based on the metadata, e.g. metrics.accuracy > 0.9.
Very interesting tool! I am a user of DVC and even gave a talk about it using it for simulation data management. Do you know DVC? There is also a project built on top of DVC called CalKit, which adheres to very similar audience as your tool. I never tried it though, but chatted with the creator and it sounds very promising. How does logis differ from DVC? DVC also offers metrics, experiments and pipeline definition / execution, which you can query using the DVC CLI or GitPython and the DVC Python API in custom scripts
I remember DVC when it first came out for data management, I didn’t realise they added experiment tracking. This looks great. Honestly it’s very similar to logis and a smarter implementation with custom refs.
There might be other directions I can take logis (it’s still very new and lightweight), or I’ll start building on-top of DVC.
Thanks for sharing!
Though now that I think about it, DVC experiment commits don’t get pushed to Github which IMO limits the ability to integrate the metadata into other tools. I assume DVC offers an API, but the vendor lock-in is not ideal.
DVC is great, but there is still so much room to innovate and optimize (see the end of my talk for ideas)! Happy building :)
Definitely an interesting tool.
I’m wondering if you have any plans to add a history viewer on top of the git log to see the various experiments and how they are different from each other. I’m thinking about your “regular Joe Reviewer #2” use case where they might be interested in the development of the work but might not be able to use git log to track it. What I’m thinking about is an HTML overlay over git log that shows cards for each experiment, maybe all the commits that went into it (if they are tagged somehow to a specific experiment), and then the performance stats under each one. This would allow you to publish the log somewhere easy (like GitHub Pages/OSF.io/your university static HTML host) so that non-technical people can interact with the work.
An additional bonus would also be to add “traceability” of the work by linking experiments to each other, so to visualize how each experiment builds onto other ones, kind of in a network.
In short, yes. Part of the reason I built logis was to support further tools I want to build on-top of the structured data in the commit log. I’m more interested in collaboration tools though, rather than just observability into experiments.