Go and Rust are both great languages if you want a minimum of safety and pretty good performance.

Since they have similar performance results for naive and optimized versions, I wonder which one is easier to profile and optimize?

[Comment removed by author]

1+

I see what you did there.

I’ve actually wondered the opposite before: could Forth be an extensible, bare metal VM for some stack machine based languages?

Does CPython’s VM contain registers? Forth isn’t just “stack based”, just operations are performed on it. The canonical Forth virtual machine has two stacks.

Here’s a quick “simple” version in Zig, it beats the simple version in C on my machine at 0.37 vs C’s 0.65.

https://zigbin.io/73ee75

Edited to fix the usual minor mistakes that happen when I write something quickly :)

I’m sure the author would appreciate some submissions for those languages.

I feel like this type of benchmarks usually suffer from the bias of the author’s proficiency with the tested languages which is not uniform. I’m sure optimized versions from experts of each language would be highly useful to them, and to us.

I submitted naive and slightly profiled/optimized D-Language versions. There’s probably a bit more to squeeze out (the char to string idup allocation).

edit: found some over a cup of coffee.

edit #2: found more, now on par with optimized cpp. It’s not very readable anymore:

https://github.com/rlonstein/countwords/blob/dlang/simple.d

vs.

https://github.com/rlonstein/countwords/blob/dlang/optimized.d

Well, let’s test it. Here’s a patch that switches to map[string]int:

diff --git a/optimized.go b/optimized.go
index 74b42c3..638b694 100644
--- a/optimized.go
+++ b/optimized.go
@@ -11,7 +11,7 @@ import (
 func main() {
        offset := 0
        buf := make([]byte, 64*1024)
-       counts := make(map[string]*int)
+       counts := make(map[string]int)
        for {
                // Read input in 64KB blocks till EOF.
                n, err := os.Stdin.Read(buf[offset:])
@@ -71,7 +71,7 @@ func main() {

        var ordered []Count
        for word, count := range counts {
-               ordered = append(ordered, Count{word, *count})
+               ordered = append(ordered, Count{word, count})
        }
        sort.Slice(ordered, func(i, j int) bool {
                return ordered[i].Count > ordered[j].Count
@@ -82,15 +82,8 @@ func main() {
        }
 }

-func increment(counts map[string]*int, word []byte) {
-       if p, ok := counts[string(word)]; ok {
-               // Word already in map, increment existing int via pointer.
-               *p++
-               return
-       }
-       // Word not in map, insert new int.
-       n := 1
-       counts[string(word)] = &n
+func increment(counts map[string]int, word []byte) {
+       counts[string(word)]++
 }

 type Count struct {

Before:

$ hyperfine './optimized-go < kjvbible_x10.txt'
Benchmark #1: ./optimized-go < kjvbible_x10.txt
  Time (mean ± σ):     270.4 ms ±   7.6 ms    [User: 263.5 ms, System: 11.6 ms]
  Range (min … max):   265.0 ms … 291.8 ms    11 runs

After:

$ hyperfine './optimized-go-mapstringint < kjvbible_x10.txt'
Benchmark #1: ./optimized-go-mapstringint < kjvbible_x10.txt
  Time (mean ± σ):     453.6 ms ±  20.5 ms    [User: 486.5 ms, System: 24.2 ms]
  Range (min … max):   426.7 ms … 485.8 ms    10 runs

Running both programs with perf shows that the program with map[string]int has a hotspot in runtime.mallocgc.

Presumably, the simpler map[someString] += 1 variant allocates a new string? Using a *int allows you just do a lookup and increment the value. I guess if you think of map[x] += 1 as map[x] = map[x] + 1 then maybe the extra alloc makes a bit more sense. But I wonder whether it’s possible for the compiler to optimize this kind of case. Would be neat.

The unoptimised version would look a lot like the python version, but probably perform better if you omitted compilation time.

You’d use a Dict, eachline, and split. The main performance issues would probably be unicode-aware lowercasing and hashing each word twice in the inner loop (once to check if it’s in the dict, once to set a new value. We might eventually get an in-place update interface for dicts to avoid this duplication, in the meantime you can use some internal funcs or use a third party package).

Unlike the python version, if you need Moar Performance you can efficiently operate at the byte level and use any of the tricks from the C or Rust implementations.

I suppose it will be dominated by startup time and IO. Although I am curious to see if the same issues would arise in Julia as well, splitting of words and slowdowns due to maps.

While technically correct (the best kind of correct) I think the author is probably using a less formal definition of sigil. The intent is clear, in any case: learning Rust means learning the semantics of a lot of ASCII characters.

Not to nitpick your nitpick, but here’s a comment that states that hcreate is part of POSIX:

https://news.ycombinator.com/item?id=26466477

grep and wc are just signposts. They aren’t actually completing the challenge: https://github.com/benhoyt/countwords/blob/9d81d13711e56c25068893880a648ac96d0fa92e/benchmark.py#L23-L24

Yeah, burntsushi is correct – they’re just being used as a baseline to show “how fast can a fast tool read this file” or “what’s the minimum it could be”. Sorry if that wasn’t clear. I’ll update the wording a bit to clarify.

Yeah data set size is tricky. I think in this benchmark, it is maybe just big enough. It’s a problem that appears when you include slower data points in your benchmark. e.g., Python is about an order of magnitude slower than C here. So if you jack that data set size up, all of a sudden measuring your benchmark with Python becomes annoying. Of course, you could solve this by using two inputs instead of one and partitioning the tools. But it’s more complex.

Coming up with a benchmark is hard. Benchmarks like these are models. All models are wrong. But. Some are useful.

Follow you thinking to its logical conclusion. If you open up the parallel programming flood gates, now you need to do that for all of the programs. Now your effort to contribute a sample for each language goes up. Maybe enough where actually doing the benchmark isn’t practical.

So let’s play the tape. If someone told me, “sure, have fun, throw threads at this!” I’d probably say, “can I use memory maps?” And let’s say, “yes, sure, go for it!” I say okay, here’s my first crack at it:

1. Hope stdin is a file and open it as a memory map.
2. Split the memory map slice into N chunks where N is the number of logical CPU cores on the system.
3. Run approximately the same algorithm devised in existing code submissions in N different threads, with one thread per chunk.
4. At the end, join the results and print.

If that ends up being the best strategy overall, then your model has become over-complicated because performance now likely depends on step (3). Which is exactly what the existing benchmark is. Now, maybe some languages have a harder time passing data over thread boundaries than others and maybe that impacts things. But that doesn’t apply to, say, C, Rust, C++ or Go.

If that’s not the best strategy, then maybe there is a cleverer approach that involves synchronizing on one shared data structure. I doubt it, but maybe. If so, your benchmark turns into one that is very different. It’s no longer just about data transformation, but now you have to worry about synchronization trickiness. And now you’re way outside of a dumb goroutine pool that you might have used in Go.

Building a good benchmark is an art. Sometimes the constraints look dumb and arbitrary. But you have to sit down and really look at it from a bunch of different angles:

• Do the constraints approximate some real world conditions?
• Do the constraints inhibit writing code that looks like real world code?
• Do the constraints inhibit optimizing the crap out of the code?
• Do the constraints make the benchmark difficult to create in the first place?
• Do the constraints permit easy and clear measurement?

There are probably more things that I didn’t think about.

If I am not mistaken the unix version is actually parallel ?

Only a little bit (much less than it might superficially appear). The easy automatic parallelism of shell pipelines is a very nice feature, but you have to keep its limitations in mind. sort, as a common example, is effectively a parallelization barrier, because it can’t produce any output until it’s consumed its entire input, so nothing downstream of it in the pipeline can do any meaningful work concurrently with anything upstream of it.