1. 1

    I’m looking to add saved program loading to my Zig playground. This will let you share snippets with others and hopefully be useful to some people. I’m also considering writing a terms of service page since I will be storing data too. If anyone has any recommendations on how to avoid any issues down the road (legal or otherwise), I’d appreciate it. For now, I am using other sites as a guide.

    1. 19

      Going to write a blog post on my personal tech stack. OpenBSD, relayd, Golang, and YAML. Also known as the ORGY stack.

      1. 20

        If you make a distributed solution on top of that, would it then become a clusterfuck?

        1. 2

          Distributed solution to flat files, has to be NFS right? shudder

        2. 4

          I’m interested in reading it, do you mind sharing the link to your post once you publish it?

          1. 2

            I’ve posted it here

            1. 1

              Thanks!

            2. 2

              Yeah of course, will do

              Edit: I actually finished the example application at least, I’ll publish the code for it too of course but if you’re interested in a semi-early-bird-alpha-preview it’s here

            3. 3

              Please post the link when you’re finished. I’d love to read this implementation.

              1. 1

                I’ve posted it here; sorry I missed tagging you.

                1. 1

                  No problem. Thank you very much.

              2. 2

                What are you using Relayd for?

                1. 2

                  TLS termination and other proxy stuff

                  1. 1

                    other proxy stuff

                    I’m intending to switch (back) to OpenBSD once my current VPS contract expiries, and one of the things I want to figure out is how to implement a reverse-HTTP-proxy using Reladyd. So all requests to foo.example.com are directed to localhost:9090, all requests to bar.example.com are directed to localhost:9091. Didn’t manage to get it working last time I tried, so I hope your posts might give me some hints.

                    1. 2

                      Yep 100% possible with not that much config. This specific post will contain how to do that but I am also planning a “how everything on this website” hangs together post which will be more in-depth on that subject. I’ll ping you once I’ve written it

                      1. 1

                        Great, looking forward to that!

                        1. 2

                          I’ve posted it here

                          1. 1

                            Thank you!

                2. 2

                  I always thought that Go on OpenBSD would be solid. Do you use pledge or unveil in your Go application? Or does relayd pretty much take care of that?

                  P.S. I’m pretty green with OpenBSD, etc.

                  1. 2

                    Go has support for it (unveil, pledge), and there are additional packages that wrap it such as suah.dev/protect that allow transparent use on non-OpenBSD systems

                  2. 1

                    I assume if you mess things up, it becomes the GORY stack.

                  1. -10

                    Looks ugly.

                    1. 3
                      1. -2

                        Nice to know. 👍🏻

                      2. 2

                        I’m sure the author would appreciate more constructive criticism :)

                        1. -2

                          Add paddings.

                        2. 1

                          It works really well on mobile.

                          1. -4

                            Not really.

                        1. 2

                          Since Firefox Send is on a hiatus, I decided to build a smaller clone of it. Instead of using browser-side encryption, I am taking advantage the excellent age package. The files will live in an S3 like object storage and have a configurable lifetime. The idea is that you can send files to a recipient with their public age key and only they can download it. It could be fun to host but then I may end up in the same boat as Firefox Send!

                          1. 5

                            Trying to get a pile of work stuff done quickly for a hardware test on Thursday. And trying to not gnaw the head off of the project lead who talks too much.

                            Should probably make sure some self-care happens somewhere in there.

                            Mourning my dead cat. She was the best girl.

                            1. 3

                              Ugh…I’m sorry to see needing self-care and the loss of a pet in the same paragraph. I wish you the best in the coming days! I know I am an internet stranger but feel free to reach out if need be.

                              1. 2

                                Sorry to hear about your cat :^( I lost my dog last year, and it was rough.

                                Remember to take it easy this week!

                              1. 5

                                Trying to profile and create flamegraphs for durduff.

                                I tried using flamegraph-rs, but it failed with a very non-descriptive error. I used perf directly and it looks like almost everything in durduff gets inlined, so it isn’t very legible. It may be that the source of both problems is the lack of debugging symbols, which, I just discovered, should be enabled with debug = true in Cargo.toml.

                                1. 2

                                  I just recently used that crate to do some profiling myself so if you need any more help, feel free to PM me.

                                1. 6

                                  Thanks for putting this together. As someone who is unfamiliar with Zig, your choice of implementing a known utility rather than something less familiar allowed me to focus on what I don’t know about Zig, rather than the use case. Cheers.

                                  1. 1

                                    I’m glad you enjoyed it and thanks for reading!

                                  1. 26

                                    I looked at the source and thought I’d share some neat things that you can do.

                                    Instead of

                                    var bin_name = args.nextPosix();
                                    

                                    you can do

                                    _ = args.nextPosix();
                                    

                                    to ignore the value, and get rid of the unused variable.

                                    Instead of

                                    var fname = args.nextPosix();
                                    if (fname == null) {
                                        std.debug.warn("expected at least one file name as an argument\n", .{});
                                        std.os.exit(1);
                                    }
                                    
                                    if (readFile(fname.?))
                                    

                                    you can do

                                    var fname = args.nextPosix() orelse {
                                        std.debug.warn("expected at least one file name as an argument\n", .{});
                                        std.os.exit(1);
                                    }
                                    // `orelse` has already "unwrapped" the optional, so no need for `.?`
                                    if (readFile(fname))
                                    

                                    Instead of

                                    if (readFile(name)) |_| {} else |err| {
                                        std.debug.warn("error reading file: {}\n", .{err});
                                    }
                                    

                                    you can do

                                    readFile(name) catch |err| {
                                        std.debug.warn("error reading file: {}\n", .{err});
                                    }
                                    

                                    which is a bit nicer, IMO.

                                    And of course, if you want, you can make main return !void, and use try directly in it.

                                    1. 3

                                      This is really helpful! I appreciate the feedback. I will need to read more about the orelse keyword.

                                      1. 3

                                        I’m not that familiar with Zig yet, but I prefer your adjustments here. Thanks and thanks to gsquire for putting it together.

                                      1. 1

                                        Thanks for sharing! I wrote my own tool a while back as well. Having these packages available in the standard library is a great way to encourage further tooling beyond what the Go offers.

                                        1. 2

                                          I am going to continue my job search after a round COVID-19 layoffs. I will probably program a bit to distract myself as well. If you know of any companies hiring, my email can be found via my site in my profile.

                                          1. 3

                                            It’s always cool to see new Rust tools where C has traditionally been used in the past. The only thing that gives me pause is how many times the unsafe keyword is used. I glossed through some of the code and saw many functions were marked as unsafe. After cloning the repository and searching for unsafe, I found 55 occurrences.

                                            The command used was rg -I -c unsafe src | awk '{ s += $1 } END { print s }' for those interested.

                                            1. 1

                                              awk '{ s += $1 } END { print s }'

                                              This is equivalent to wc -l isn’t it?

                                              1. 2

                                                Not quite. This is because the output from the ripgrep command displays a count for each file per line via the -c flag. And because we also used -I, we are omitting the filename so we only get the count. Then we can sum these up to see the total for the entire src directory. I hope that helps.

                                            1. 25

                                              My personal opinion as a go programmer is that rust is fantastic, however there is one thing that really bothers me:

                                              The rust community has (in my opinion) prematurely embraced async to the point that simple tasks become very complicated. It seems like the async obsession is such that you can’t do things like make simple http requests without buying into an async runtime. I consider myself a competent programmer, but struggle at times with the extra setup and mental complexity of using async.

                                              Forcing users into async due to the infection of the whole ecosystem made me wonder why bother with rust over Go to begin with. I fear it will get to the point no useful libraries will exist that are not using async, and the whole ecosystem will become worse because of it.

                                              One of the main selling points of rust was that it isn’t supposed to require a runtime, so don’t force an async one on me by essentially starving the synchronous ecosystem of mind share.

                                              1. 16

                                                I’ve been using Rust for a long time, and I don’t think I’ve used async anything even once. Granted, my domain (low level text search) is kind of what lets me do that. But I guess that’s my point: there are entire domains where you don’t need to touch async at all.

                                                Plus, there are HTTP libraries in pure Rust that don’t use async at all.

                                                With that said, I think you have a valid point. The async ecosystem is extremely overwhelming right now, even for me. But, I’m not disheartened just yet as it’s still quite early early days for the async ecosystem. But, I don’t think Rust will ever get to the level of Go here. This is exactly where Go really shines. I mean, they built it into the language. It’s hard to compete with the user friendliness of that with a runtime-as-a-library. I think the question is whether Rust can get to that “close enough” sweet spot.

                                                1. 10

                                                  My most pleasant uses of rust have been non web related cli programs dealing with POSIX, threads and other low level things.

                                                  My most frustrating uses have always been when dipping my toes into the web ecosystem and dealing with async. I spent a while trying to find a way to avoid async to write a simple web application, but was irritated to find nearly every web framework that isn’t async has an open github issue to migrate to async, which scared me off as I didn’t want to deal with breaking changes like that.

                                                  I then decided to look into just a rust CGI binary, however all the database drivers I saw that I wanted to use were also async.

                                                  1. 1

                                                    Which database are you looking to use? I know rusqlite and rust-postgres are both synchronous. I think even diesel is as well.

                                                    1. 1

                                                      w.r.t rust-postgres

                                                      This crate is a lightweight wrapper over tokio-postgres.

                                                      So I still need to deal with tokio and and a bunch of confusing dependency issues. I’m not saying they aren’t usable, I’m just saying they were more annoying to use and install than they needed to be.

                                                      I was also bitten by two different libs depending on two different versions of tokio.

                                                      1. 1

                                                        Diesel is a good option then. And the postgres crate, while it’s built on top of tokio-postgres, should do what you want… you shouldn’t need to deal with tokio directly to use it.

                                                2. 10

                                                  At least for me, one of Go’s best strengths is painless async. Just taking something synchronous and being able to throw it in a goroutine is super powerful.

                                                  I’ve seen exactly what you’ve mentioned with some crates, but I also saw it early on with Go as well. My preference is for parsing packages to not come with specific transport or concurrency requirements. It’s much easier to throw something in the background than to try and force it into the foreground (at least in go).

                                                  As I mentioned in the article, async is a huge pain point in Rust at the moment, but if the community manages to figure that out and the ecosystem stabilizes a bit, I don’t know if I’d end up using Go much any more.

                                                  The Go ecosystem has had multiple years to improve and the Rust async ecosystem is less than a year old. I’m hopeful it will improve.

                                                  1. 1

                                                    It’s true that some http clients switched to async-per-default (for example reqwest), but there are still enough capable of handling sync. For me this meant changing my use reqwest::{Client,ClientBuilder} to use reqwest::blocking::{Client,ClientBuilder}. Yes it’ll create a new thread and run a single threaded tokio engine inside that, but that’s fine for how well reqwest works. Also I’d rather have them implement the logic once than twice, creating bit rot for one path.

                                                    1. 2

                                                      Implicit background threads aren’t the nicest thing to deal with when you want to do things like privilege separation and setuid, which is one of the reasons i would chose rust over go, better integration with the host OS.

                                                      1. 2

                                                        I don’t think that’s the complaint. As someone who just started writing something in Rust coming from Go, the problem seems to be that making everything async increases API complexity (and program setup) for simple things that would be perfectly served by a sync call. The converse thing in Go was when people would use channels for everything even when they weren’t needed (for a bit, using channels and a goroutine was seen as a way to implement Python-like iterators, without consideration of what really happens with those channels and goroutines if they aren’t properly consumed).

                                                        I can see the async-craze taking over Python too, to a certain extent. It seems like everyone and their mom are writing async versions of every non-async library in the world, without first wondering if it’s worth doing.

                                                        1. 2

                                                          Yeah, I think rust people love to make maximally generic implementations, so when given the choice between async or not, they just say “Well, async can be called anywhere with a wrapper, so it is more generic to always choose async”. However as you said, people don’t consider the complexity or overhead in this decision.

                                                  1. 3

                                                    I am going to sit down and give Intuit some money to do my taxes. On the bright side, I should get a little bit of a return back. I also want to polish up a couple pull requests I have open and get those closed out.

                                                    1. 4

                                                      Please consider joining me in boycotting Intuit. I did so after I heard about them lobbying against IRS run filing solutions and tricking users into paying when they tried to use free file.

                                                      https://www.propublica.org/article/inside-turbotax-20-year-fight-to-stop-americans-from-filing-their-taxes-for-free

                                                      My fed and state taxes aren’t ‘simple’ but I found doing them manually takes maybe 20% longer than using one of these tools. I’m also happy that these companies don’t get any more of my income/tax/asset data.

                                                      1. 1

                                                        My brother told me that CreditKarma lets you file for free and you can also itemize deductions. But it looks like Intuit is feeling the heat and is acquiring them! I may have to resort to manually doing them.

                                                        1. 1

                                                          This tbh, though as a student my taxes are fairly simple. I still refuse to use any Intuit product.

                                                      1. 10

                                                        I’m working on adding indexing to ripgrep. I’ve opened an RFC and would love feedback: https://github.com/BurntSushi/ripgrep/issues/1497

                                                        There’s lots of yak shaving to be done though. My immediate focus is on polishing and cleaning up my fst library, since it will be the work horse for the ngram index.

                                                        1. 6

                                                          Just wanted to say thanks for making RipGrep. I use it pretty much every day, even without indexing.

                                                          1. 2

                                                            Would this be integrated into libripgrep as well? I know there has been work on other full-text search libraries in Rust too. I guess what I am asking is this more geared toward code search or does it have a more general scope?

                                                            1. 5

                                                              Most of it will be in reusable library form. I hope anyway.

                                                              I don’t know how general it will be. At least as I currently conceptualize it, it will be coupled fairly tightly with the notion of indexing directory trees. My basic idea for the index format is as follows:

                                                              • An FST for the ngram index. This maps ngrams to postings offsets.
                                                              • A sequence of posting entries, each beginning at a particular postings offset. Each posting entry corresponds to a single ngram. A posting entry is a list of segment-local identifiers, where each identifier corresponds to a file that has been indexed.
                                                              • An id map which maps the aforementioned identifiers to file offsets.
                                                              • A collection of file metadata, where each entry starts at its corresponding file offset. At minimum, this metadata will include the time at which the file was indexed.
                                                              • Another FST that maps file paths to their corresponding file identifiers. (The file identifiers can be used to find metadata for the file by mapping them to file offsets, as described above.) This FST makes it very cheap to query an indexed file’s metadata for incremental updates while also achieving excellent compression. (File paths will be absolute, so most file paths will have a ton of common prefixes.) This map will also be reversible, so if you have a file’s identifier (e.g., from the postings), then you can find the corresponding file path.

                                                              The library would then be in charge of handling this format in addition to handling the partitioning of the index.

                                                              It is quite possible that I could rip the words “file” and “file path” out of the above implementation, which would leave you with something pretty generic that isn’t coupled to files. If I can do that, then I will, but I haven’t quite mapped out every implementation detail yet. It’s quite possible that there is some coupling that significantly simplifies the implementation.

                                                              But yeah, I think the outcome is at least another grep- crate, regardless of coupling. If I can manage to build something that isn’t coupled to file paths in a sensible way, then that will perhaps get its own crate independent of ripgrep altogether.

                                                              Tantivy is a very different beast. It’s a general information retrieval system. It supports relevance ranking and needs to make precision/recall trade offs. This is perhaps the main problem with trying to de-couple this logic. It’s not clear how useful this is outside the context of “search files for all matches.” For example, you would probably not want to use what I build to implement fuzzy search, because fuzzy search really wants some notion of ranking built into it. It’s not that much extra work to add frequency counts to the postings and add a scoring mechanism to produce ranked results, but it’s really solving a different problem than what I’m trying to solve.

                                                              (imdb-rename does build the aforementioned fuzzy search in a way that is coupled to a specific data set.)

                                                          1. 3

                                                            Trying my hand at homebrewing beer. I’ve made mead and cider before but beer looks like it’s got some added steps…

                                                            1. 1

                                                              What style are you brewing? I feel like the most important part is keeping everything sanitized but I’m sure you knew that already. Good luck, hope it’s fun.

                                                              1. 1

                                                                Hopefully a Hefeweizen, if I can pull together the necessary materials (the homebrew store is a little farther than I’d like). I have more Star San than I know what to do with!

                                                            1. 1

                                                              doing a hackathon!

                                                              only problem is I don’t really have an idea… open to suggestions :)

                                                              1. 1

                                                                What interests you? Is it an open-ended event? If so, it’s always rewarding to find an open source project to contribute to! There are also many repositories that contain project ideas that people have requested to be implemented.

                                                                1. 1

                                                                  It’s fairly open ended. There are 6 categories to win grand prizes in, and then other prizes related to various software/hardware that you incorporate in your project. I’m actually super interested in contributing to open source! I think at this hackathon last year there was a prize for most impactful open source contribution but I don’t think they’re doing that again this year. If I don’t end up on a team going for one of the main prizes, I might just work individually or one or two others to contribute to open source!

                                                              1. 12

                                                                If the project would be server side I would opt for Go. It is very straightforward to set up any type of network communication along with a ton of third party libraries that cover all your needs.

                                                                In any other domain I would reach for Rust which may sound fashionable but I think there is a great amount of traction for it. The most appealing feature is the stabilization of usable asynchronous primitives which turn callback-centered code into more of an imperative flow. Not to mention the amount of confidence you get by having a compiler that ensures memory safety.

                                                                1. 4

                                                                  I wonder if the Discord engineers tried profiling via Go’s native tooling too. I’m not saying it would have solved their problems but in my experience you are able to drill down further into individual hot spots. Granted, Go chooses to only give you a couple of knobs to tune garbage collection. It was an insightful article nonetheless!

                                                                  1. 5

                                                                    I’m curious if using mimalloc would offer any sort of performance improvements too.

                                                                    Aside from the links you posted about performance measurements, which tools do you use to find hot spots? Thanks for sharing this post!

                                                                    1. 1

                                                                      I can’t recall if we benchmarked mimalloc or not. It would probably be worth the experiment.

                                                                      The main performance tool we used was Linux’s perf: https://perf.wiki.kernel.org/index.php/Main_Page , along with using our own metrics infrastructure to watch the relative performance of passes over time.

                                                                    1. 1

                                                                      I just redesigned my site with a new theme from Hugo: https://garrettsquire.com

                                                                      As far as the stack goes, I run a caddy instance so I get free TLS from LetsEncrypt. It’s running on a Debian instance on DigitalOcean. If I had to do it all over again I would just use something like GitHub pages or really have fun with it and write my own server.