I think the most important feature for me will be tracking tool dependencies with the go command directly. No more problems caused due to different tooling versions when collaborating with other people :)
I was lucky that most tools I am using for my projects can be used as libraries and can be used with my own little command line executable, which then can be used in a versioned manner. Here is an example where I’m using //go generate which wraps an internal command, which uses esbuild for building JavaScript and CSS assets.
Although I wrote a post, being very excited about go tool, this is still an unfortunate side effect of the pattern. Using a separate go.mod for them works, but has it’s own tradeoffs 🙃
That’s a possibility but you can always go back to what people were doing before with a tools package or doing go install @version on its own.
I haven’t run into an issue with the common dependency graph yet. The unfortunate thing about it is not being able to discern quickly your own dependencies from the tools’ in go.mod.
I’m heading to Manchester for OggCamp (nothing to do with the audio format) - an open-source, open-culture unconference. This is the first time it’s run since before “The Event” in 2020. Looking forward to some interesting (and nerdy) talks, and meeting up with friends I’ve not seen for a while.
I’ve never felt motivated to learn any of the Go routing libraries
That’s a weird one. Using Gin or Chi was infinitely easier that using the standard router pre-1.22. Exhibit A: the awful snippet she shows below. Even after 1.22 I’ve encountered weird behaviors with the standard router (handlers called when the HTTP verb didn’t match, or unclear handling of trailing slashes or lack thereof) and would still recommend using Gin or Chi for any serious work.
I think I have it in one of my repos, I’ll check if I can find it again, and notify you if I do.
But to be clear I don’t think it’s a bug with Go, my guess at the time was that I used {$} incorrectly somehow but didn’t have time to look into it. So if anything, it’s probably a documentation issue, or a skill issue on my side.
FWIW, I saw an article on the Go blog the other day that incorrectly used a trailing slash (which captures all sub-routes) instead of a final dollar sign, so clearly the API isn’t as intuitive as it could be.
Using Gin or Chi was infinitely easier that using the standard router pre-1.22
It was really not that much work (extra couple lines per route plus minimal to average complexity regex helper function for path match logic depth to add the functionality of http request type matching and request parameter parsing pre 1.22. It also provided a way to learn a bit more about net/http and http requests in go than just using a third party library without having tried it in the standard library even if only for a early implementation of your web backend code that you for whichever reason swapped in chi or gin for later in development.
https://benhoyt.com/writings/go-routing/
The most important thing I would tell to someone new to oncall is: if it’s not part of your job description and contract, you don’t have to do it, and in fact you should not agree to do it without negotiating a much, much higher compensation package. Don’t be a sucker.
100% - for some it’s compulsory, for some it’s optional (with good pay incentives) but for companies that don’t make it optional, or well paid, it’s often not a fun experience
Where it’s been optional, it has felt much more sustainable as if folks are not opting into the rota, there’s a signal there that it’s not something folks want to do
My current rota is “follow the sun” where I’m basically only on call during working hours (and the same hours on weekends) because we’re a globally distributed company so we can avoid 24 hours a day for the same person
I’ve been keeping some of this info in https://www.jvt.me/on-call-compensation/ but unfortunately can’t publicly share things like “expected response time” or whether that response would mean “you’re checking logs” or “you’ve acknowledged the page”
And let’s not forget the difference between optional and “optional”. The latter meaning where you are currently 1 of 2 engineers rotating each week, the system itself is old and unloved, and new people are refusing to join said rota!
I think there’s an error in the provided code samples. Instead of ruby $0 "$@", the line should be exec ruby $0 "$@" so that the Bash process replaces itself with Ruby. Otherwise, Ruby will be run as a subprocess and Bash will continue to execute the script after the Ruby process terminates.
I think it basically comes down to having tools/technology that makes adding a new service very low-friction. And by that I mean a new service just gets a huge amount for free: database/queues are available, observability systems are available, tooling for deploying/scaling are available, etc. Scaffolding service boilerplate is just a single command.
There are pros and cons of this architecture of course, but I think the “complexity” argument that is commonly levelled against this architecture is heavily mitigated by using highly consistent technologies (and something we are able to retain by running migrations in the way I describe, where we step from one consistent state to another).
Are those 2800 different microservices or just instances of the same few hundred microservices? A simple search tells me that Monzo has about ~300 engineers, so that’d be roughly 10-to-1 services-to-engineer. I imagine not all of those 300 engineers are service maintainers either so it’s probably even more? How sustainable has it been in your experience?
Are those 2800 different microservices or just instances of the same few hundred microservices?
2800 distinct services.
10-to-1 services-to-engineer
I think that’s roughly correct.
How sustainable has it been in your experience?
I haven’t really found it to be a significant problem in practice. I’ve worked at a different company where I was responsible for fewer services (maybe ~10 for per team), but it was significantly harder to maintain because we lacked the high degree of consistency that Monzo services have.
Probably the biggest downside of this architecture is the fact that implementing transactions/locking is done at the application layer. But we have good off-the-shelf infra/tools that make this a lot less painful than I’ve seen elsewhere. On the flip side, product teams don’t have to worry about provisioning or maintaining any of the infra/storage themselves.
How do you approach testing flows that cross services? Eg: “Given that I change service C that is used in business flow where service A calls service B which calls service C; ensure the end-to-end functionality is not broken by
my change”?
The main technique we use is what we call “acceptance tests”: essentially a test that tests business logic against a whole system. These are run periodically in our staging environment as well as being triggered by deployments.
A technique we have for testing in production (and staging) is “shadow testing”: before enabling a feature in production, we’ll often run it in “shadow mode”, automatically comparing the responses with the actual responses and alerting if there is a discrepancy. We have libraries to make this easier to implement.
These are definitely all more clunky to implement and than unit tests, but they’re also not something we have nearly as many of vs unit test (higher up the testing pyramid), so we haven’t (yet) needed to improve the DX around this.
Also worth noting is that client code imports the generated protobuf sever code, so breaking API changes are caught at compile time.
how do you organize your OpenAPI wrt readability? How many endpoint do you have per microservice?
do you have unified OpenAPI UI that I can read through, or do I need to read it per-microservice?
I’d guess the median number of endpoints per service is around 10, but it’s a long tail distribution.
We use protobuf to specify the API. Clients of a service actually import the generated protobuf go code, this means we get compile time checking and also some nice developer experience like being able to “jump to definition” for RPC calls.
I read the title and remarked to myself, when I was at an investment bank they had more than 4500 services tied together with centralized message queuing. Not knowing the “monzo” brand I thought, what are these people up to that they have as many as a bank? stfw. Oh, they are a bank.
I’ve chatted to a few Monzo engs in the past and it basically boils down to blurring the lines between what is a unit of business logic - as willsewell mentioned, their tooling essentially abstracts away a lot of the complexities of inter-service communication in such a way that writing a package that imports another package and calls a function feels remarkably similar to writing a service that calls a function over the network.
Which is, given they have the resource to do that, pretty neat. And I use Monzo daily as my primary bank, it’s rarely let me down!
My website is a static website built using Hugo. That means to make a change to the now page, I’d need to manually update the now file, commit the new changes made to GitHub and then merge the new changes into my main branch on GitHub.
I also struggle with this, if I don’t have my laptop with me (even if I have a random computer, it must be my laptop with my dev tools) I can’t update my website or write notes.
In fact I’m always thinking of starting to use some CMS even if I’m not too much a fan of them (they take time to configure and generally require a backend)
Sidenote: Sometimes I think about the “SmallTalk” approach, where instead of static data with some build steps along the way you always work with a live editable system. I really like projects like TiddlyWiki in this space
Not true that a CMS is needed with a backend - I’ve deployed a number of Hugo sites using Decap CMS (formerly Netlify CMS) that requires no backend to host it, and can work with any format of content + purely in browser
This guy should work on the official release notes! I have to admit, even knowing what was coming in Go 1.23, I found the release notes pretty hard to read.
Thanks for the link. I finally understand how the iterator works thanks to this article. I think the fact that the author explain the for range changes separately from the iterator types helps (most other articles explained them together and make it difficult for me to understand, especially because of the confusing signature of the iterators).
This page was last reviewed on 27 June 2024. It needs to be reviewed again on 27 December 2024 by the page owner #gds-way .
When approaching / at review date, a message drops into Slack to indicate it needs review, and the owners then make sure they’ve addressed it.
I haven’t used it in a bit, but the Dangerfile (https://danger.systems) could be worthwhile looking into to check that docs have been updated alongside changes
And then also, making it part of the code review checklist helps as well
danger definitely looks like the kind of automation I was thinking of- although none of the checks are exactly what I had in mind, I can see how you might use it to add more in-depth checks. Thanks!
I’ll be creating a browser extension for sharing and searching bookmarks in the IndieWeb. I already started yesterday. Now I only need to build the search part.
That’s a fair challenge (the use of “counterpoint” in my message) - I think I was a bit too quick to reply then.
However, I will say that vaguely annotating what you’re doing at the time of that commit can be useful, unless you’re working on something that’s definitely going to be squashed into whatever is going on at that point
I think the most important feature for me will be tracking tool dependencies with the
gocommand directly. No more problems caused due to different tooling versions when collaborating with other people :)I was lucky that most tools I am using for my projects can be used as libraries and can be used with my own little command line executable, which then can be used in a versioned manner. Here is an example where I’m using
//go generatewhich wraps an internal command, which usesesbuildfor building JavaScript and CSS assets.One possible downside is if the tool dependencies clash with the dependencies of your package.
I’ve never had this problem, but you can use a separate mod file for tools if you’re worried about that.
Although I wrote a post, being very excited about
go tool, this is still an unfortunate side effect of the pattern. Using a separatego.modfor them works, but has it’s own tradeoffs 🙃That’s a possibility but you can always go back to what people were doing before with a
toolspackage or doinggo install @versionon its own.I haven’t run into an issue with the common dependency graph yet. The unfortunate thing about it is not being able to discern quickly your own dependencies from the tools’ in
go.mod.In fact I seem to have problems now with different tools having conflicting dependencies
I’m heading to Manchester for OggCamp (nothing to do with the audio format) - an open-source, open-culture unconference. This is the first time it’s run since before “The Event” in 2020. Looking forward to some interesting (and nerdy) talks, and meeting up with friends I’ve not seen for a while.
Oggcamp’s been great so far this year, will hopefully get a chance to say hey today 👋🏽
Ah, dang that lineup looks interesting.
I’ve written up my experience of the talks if you were interested - shame you missed out, it was good!
That’s a weird one. Using Gin or Chi was infinitely easier that using the standard router pre-1.22. Exhibit A: the awful snippet she shows below. Even after 1.22 I’ve encountered weird behaviors with the standard router (handlers called when the HTTP verb didn’t match, or unclear handling of trailing slashes or lack thereof) and would still recommend using Gin or Chi for any serious work.
Can you reproduce the router matching the wrong verb? That’s a pretty serious bug and should be fixed if you can.
I think I have it in one of my repos, I’ll check if I can find it again, and notify you if I do.
But to be clear I don’t think it’s a bug with Go, my guess at the time was that I used {$} incorrectly somehow but didn’t have time to look into it. So if anything, it’s probably a documentation issue, or a skill issue on my side.
FWIW, I saw an article on the Go blog the other day that incorrectly used a trailing slash (which captures all sub-routes) instead of a final dollar sign, so clearly the API isn’t as intuitive as it could be.
I got hit by this when I didn’t update the go version in my go.mod file. I was baffled for hours.
Stringly typed APIs 😅
After being hit by this a couple of times I ended up blogging about it - I’ve had quite a few hits from Google by other folks seeing similar 😅
I guess it’s this post: https://www.jvt.me/posts/2024/03/04/go-net-http-why-404/ ?
Yes it is, I probably should’ve linked it :)
It was really not that much work (extra couple lines per route plus minimal to average complexity regex helper function for path match logic depth to add the functionality of http request type matching and request parameter parsing pre 1.22. It also provided a way to learn a bit more about net/http and http requests in go than just using a third party library without having tried it in the standard library even if only for a early implementation of your web backend code that you for whichever reason swapped in chi or gin for later in development. https://benhoyt.com/writings/go-routing/
The most important thing I would tell to someone new to oncall is: if it’s not part of your job description and contract, you don’t have to do it, and in fact you should not agree to do it without negotiating a much, much higher compensation package. Don’t be a sucker.
100% - for some it’s compulsory, for some it’s optional (with good pay incentives) but for companies that don’t make it optional, or well paid, it’s often not a fun experience
Where it’s been optional, it has felt much more sustainable as if folks are not opting into the rota, there’s a signal there that it’s not something folks want to do
My current rota is “follow the sun” where I’m basically only on call during working hours (and the same hours on weekends) because we’re a globally distributed company so we can avoid 24 hours a day for the same person
I’ve been keeping some of this info in https://www.jvt.me/on-call-compensation/ but unfortunately can’t publicly share things like “expected response time” or whether that response would mean “you’re checking logs” or “you’ve acknowledged the page”
And let’s not forget the difference between optional and “optional”. The latter meaning where you are currently 1 of 2 engineers rotating each week, the system itself is old and unloved, and new people are refusing to join said rota!
Oh yes absolutely! Or “optional” but there’s an expectation in ie career ladders that you need to do on call to get promotions
Oh yes absolutely! Or “optional” but there’s an expectation in ie career ladders that you need to do on call to get promotions
Aside: the blue highlighted text looks very much like it’s meant to be a link which was confusing for ~30 seconds of trying to click them
If the author reads this, the blue on brown of the code isn’t particularly easy to read - may be worth a tweak!
It is just one of the default bearblog themes. I switched it to a different theme now. Hope that helps.
I think there’s an error in the provided code samples. Instead of
ruby $0 "$@", the line should beexec ruby $0 "$@"so that the Bash process replaces itself with Ruby. Otherwise, Ruby will be run as a subprocess and Bash will continue to execute the script after the Ruby process terminates.Same for the Python sample.
Embarassing oversight. At work we use environment modules and not rvm. So the line would look like
exec mrun +ruby/ruby/3.3.0 ruby $0 “$@”
I thought it would serve as a better example to use rvm but deleted all the way to the exec during the final edit.
Thanks for pointing it out
looking forward to the prequel blog post, “how we ended up with 2800 microservices”
I think it basically comes down to having tools/technology that makes adding a new service very low-friction. And by that I mean a new service just gets a huge amount for free: database/queues are available, observability systems are available, tooling for deploying/scaling are available, etc. Scaffolding service boilerplate is just a single command.
There are pros and cons of this architecture of course, but I think the “complexity” argument that is commonly levelled against this architecture is heavily mitigated by using highly consistent technologies (and something we are able to retain by running migrations in the way I describe, where we step from one consistent state to another).
Are those 2800 different microservices or just instances of the same few hundred microservices? A simple search tells me that Monzo has about ~300 engineers, so that’d be roughly 10-to-1 services-to-engineer. I imagine not all of those 300 engineers are service maintainers either so it’s probably even more? How sustainable has it been in your experience?
2800 distinct services.
I think that’s roughly correct.
I haven’t really found it to be a significant problem in practice. I’ve worked at a different company where I was responsible for fewer services (maybe ~10 for per team), but it was significantly harder to maintain because we lacked the high degree of consistency that Monzo services have.
Probably the biggest downside of this architecture is the fact that implementing transactions/locking is done at the application layer. But we have good off-the-shelf infra/tools that make this a lot less painful than I’ve seen elsewhere. On the flip side, product teams don’t have to worry about provisioning or maintaining any of the infra/storage themselves.
How do you approach testing flows that cross services? Eg: “Given that I change service C that is used in business flow where service A calls service B which calls service C; ensure the end-to-end functionality is not broken by my change”?
The main technique we use is what we call “acceptance tests”: essentially a test that tests business logic against a whole system. These are run periodically in our staging environment as well as being triggered by deployments.
A technique we have for testing in production (and staging) is “shadow testing”: before enabling a feature in production, we’ll often run it in “shadow mode”, automatically comparing the responses with the actual responses and alerting if there is a discrepancy. We have libraries to make this easier to implement.
These are definitely all more clunky to implement and than unit tests, but they’re also not something we have nearly as many of vs unit test (higher up the testing pyramid), so we haven’t (yet) needed to improve the DX around this.
Also worth noting is that client code imports the generated protobuf sever code, so breaking API changes are caught at compile time.
how do you organize your OpenAPI wrt readability? How many endpoint do you have per microservice? do you have unified OpenAPI UI that I can read through, or do I need to read it per-microservice?
I’d guess the median number of endpoints per service is around 10, but it’s a long tail distribution.
We use protobuf to specify the API. Clients of a service actually import the generated protobuf go code, this means we get compile time checking and also some nice developer experience like being able to “jump to definition” for RPC calls.
I am not Id accept this answer as a legitimate justification, but I get that once you pen that door theres no way back for better or worse
I read the title and remarked to myself, when I was at an investment bank they had more than 4500 services tied together with centralized message queuing. Not knowing the “monzo” brand I thought, what are these people up to that they have as many as a bank? stfw. Oh, they are a bank.
I’ve chatted to a few Monzo engs in the past and it basically boils down to blurring the lines between what is a unit of business logic - as willsewell mentioned, their tooling essentially abstracts away a lot of the complexities of inter-service communication in such a way that writing a package that imports another package and calls a function feels remarkably similar to writing a service that calls a function over the network.
Which is, given they have the resource to do that, pretty neat. And I use Monzo daily as my primary bank, it’s rarely let me down!
If you’ve not seen https://qconlondon.com/presentation/mar2023/banking-thousands-microservices and https://archive.qconlondon.com/london2020/presentation/modern-banking-1500-microservices are two talks they’ve done about it
at this rate, they will be running > 4000 microservices by 2028
Maybe worth merging with https://lobste.rs/s/8yhogc/reckoning_part_1_landscape ?
I update my /now page manually (https://www.jvt.me/now/) and prefer it that way to make it intentional.
For some level of automated updates, I have my README (https://readme.jvt.me or ie https://github.com/jamietanna/jamietanna) that provides a more recent view of things
There’s a lot of stuff I track, like podcasts I listen to (https://www.jvt.me/kind/listens/) or number of steps I take (https://www.jvt.me/kind/steps/) but I don’t feel like things like that should be on my /now
(I do need to rethink what is on my /now page, tbh)
(Repost of my comment on Hacker News)
I also struggle with this, if I don’t have my laptop with me (even if I have a random computer, it must be my laptop with my dev tools) I can’t update my website or write notes.
In fact I’m always thinking of starting to use some CMS even if I’m not too much a fan of them (they take time to configure and generally require a backend)
Sidenote: Sometimes I think about the “SmallTalk” approach, where instead of static data with some build steps along the way you always work with a live editable system. I really like projects like TiddlyWiki in this space
Not true that a CMS is needed with a backend - I’ve deployed a number of Hugo sites using Decap CMS (formerly Netlify CMS) that requires no backend to host it, and can work with any format of content + purely in browser
Anton Zhiyanov has great interactive release notes if you want to play with any of the new features: https://antonz.org/go-1-23/
This guy should work on the official release notes! I have to admit, even knowing what was coming in Go 1.23, I found the release notes pretty hard to read.
Discussion on Anton’s release notes: https://lobste.rs/s/gow5jc/go_1_23_interactive_release_notes#c_no5l2g
Thanks for the link. I finally understand how the iterator works thanks to this article. I think the fact that the author explain the
for rangechanges separately from the iterator types helps (most other articles explained them together and make it difficult for me to understand, especially because of the confusing signature of the iterators).(this comment is more looking at general user-facing/customer docs, rather than in-code comments, sorry for misreading!)
Regularly reviewing docs helps.
Something in the UK government, particularly the Government Digital Service is a regular review of docs ie at the bottom of https://gds-way.digital.cabinet-office.gov.uk/standards/tracking-dependencies.html we can see
When approaching / at review date, a message drops into Slack to indicate it needs review, and the owners then make sure they’ve addressed it.
I haven’t used it in a bit, but the
Dangerfile(https://danger.systems) could be worthwhile looking into to check that docs have been updated alongside changesAnd then also, making it part of the code review checklist helps as well
danger definitely looks like the kind of automation I was thinking of- although none of the checks are exactly what I had in mind, I can see how you might use it to add more in-depth checks. Thanks!
Interestingly something like this doesn’t even need a socket to be forwarded; you can do it with just X11: https://chickadee.codes/x11uri.html
Sounds interesting, but that site’s DNS is funted and there isn’t a copy on web.archive.org :-(
Looks like Google’s cache has it: https://webcache.googleusercontent.com/search?q=cache:https://chickadee.codes/x11uri.html
Google’s cache didn’t seem to be working for me, so I’ve Wayback Machine’d that URL and now https://web.archive.org/web/20240811165605/https://webcache.googleusercontent.com/search?q=cache:https://chickadee.codes/x11uri.html works for me 🫣
I’ll be creating a browser extension for sharing and searching bookmarks in the IndieWeb. I already started yesterday. Now I only need to build the search part.
Ooh interesting, what data sources are you planning on using for surfacing the sites that have bookmarks to search on?
You’ll probably add the sites of people you know to search for bookmarks.
It’s worth adding a license, https://github.com/danthegoodman1/EpicEnv/blob/main/LICENSE as it stands is All-Rights-Reserved, aka proprietary!
Good callout, just added MIT
Counterpoint: https://lobste.rs/s/3l9eiw/89_things_i_know_about_git_commits
(but I do enjoy - during a moment of focus - doing
gc -msq)What about this is a counterpoint? From the post:
That’s a fair challenge (the use of “counterpoint” in my message) - I think I was a bit too quick to reply then.
However, I will say that vaguely annotating what you’re doing at the time of that commit can be useful, unless you’re working on something that’s definitely going to be squashed into whatever is going on at that point
I think it’s useful for you or I to be sure, but if the final commits look good I don’t mind how people get there!
Also, I did very much enjoy the post you linked. I passed it on to my work slack.
excited to read through this once i have a minute, thanks for sharing
See also https://shkspr.mobi/blog/2022/08/an-update-to-the-atkinson-hyperlegible-font/