Open source Scuba
I believe honeycomb.io is at least trying to build a version that you can use outside of Facebook, even if it isn’t entirely open source. But they are leveraging open telemetry for ingestion which itself is open!
Honeycomb is a fantastic product, but there’s very little of it that is open source, and that’s pretty understandable.
On day though, i’ll find the time to make it happen, at least that’s the dream.
Clickhouse comes close, doesn’t it? Supports streaming ingestion via Kafka, has a new experimental WATCH statement for monitoring output for “live” queries via what they call a Live View. I think this would get you most of the way there.
In the past, I combined Kafka, Amazon S3, Storm, Cassandra & Elasticsearch to get the same value proposition of Scuba, but that required quite a lot of (closed source) engineering and code to make it work well. Described a bit here.
Clickhouse comes close if you have a mostly static schema. If it’s highly dynamic, there’s a point after which too many columns and/or querying json blob doesn’t work.
But yes, CH is an amazing piece of software!
Had my first week in my new job, I’m just gonna rest :)
congrats . happy for you!
Congratulations on the new job!
I work at a startup of just around 10 people. Since we’re so small, we’re very careful with how we hire people. The last person we hired had a process as such:
1. Fast phone screen, only about 15 minutes with one of our other engineers. At the end of the call a small 1-2 hour assignment was given.
2. Once the assignment was completed, the engineer who assigned it gave a thumbs up, and the interviewee was invited to an on-site which was about 2 hours where 2 engineers and the 2 founders briefly looked over his code and then it was more of a culture fit.
All of us went through a similar process and so far it’s worked fantastically. I think the two biggest factors to consider are:
1. If there’s a culture fit on a personal level. You don’t want a groupthink scenario, but you want to know that you won’t have problems chatting with the person.
2. Can you sense a willingness or eagerness to learn more and be ready to tackle new problems/be flexible.
Unless there’s a bug in Lobsters, you submitted this twice.
It’s inspiring to see such simple but beautiful projects. It’s really interesting to think about how the learning and generation is so simple but incredibly effective.