I wrote that tweet that is making the rounds. Not many things fit in 280 characters and Twitter being immutable, there’s not many things you can go back to clarify. So let me give some more details on this forum.
I speak for my experience, not for all of Uber. Heck, we have hundreds of teams, 95% of whom I don’t know. And teams are autonomous and decide how and what they do (including following guidelines or ignoring them partially or fully) - even if I wanted to, I couldn’t make sweeping statements.
Uber has - and still has - thousands of microservices. Last I checked it was around 4,000. And, to be very clear: this number is (and will keep) growing.
I’ve been working here for almost 4 years and see some trends in my org / area (payments). Back in the day, we’d spin up a microservice that did one, small thing just like that. We had a bunch of small services built and maintained by one person. This was great for autonomy, iteration speed, learning and making devops a no-brainer. You could spin up a service anytime: but you’d be oncall for it.
Now, as my area is maturing and it’s easier to look ahead, as we create new platforms, we’re doing far more thoughtful planning on new services. These services don’t just do one thing: they serve one business function. They are built and maintained by a team (5-10 engineers). They are more resilient and get far more investment development and maintenance-wise than some of those early microservceis. Cindy called these macroservices and I said we’re doing something similar. The only difference in what we do is a service is owned by one team, not multiple teams.
While many microservices are evolving like this, the majority, frankly, stays as is. Thousands of microservices bring a lot of problems that need to be solved. Monitoring. Testing. CI/CD, SLAs. Library versions across all of them (security, timezone issues). And so on. There are good initiatives we keep doing - and sharing what works and open sourcing some of the tools we build to deal with the problems, as they pop up. Like testing microservices with a multi-tenancy approach. Distributed tracing across the services. All of this is a lot of investment. Only do microservices at scale if you’re ready to make this investment.
So no, Uber is not going no-microservices like I’m seeing many people interpret it. It’s not even going to less microservices. And when I said “we’re moving”, that was not exact phrasing. New microservices are more thoughtfully created in my team and in my org. These are “larger” services than some of the early, small, focused, microservices.
Microservices worked well at Uber in many ways and keep helping in others areas. There are problems, of course, and you deal with the problems as you go. This would be the same with e.g. a monolith with thousands of developers, SOA with thousands of developers or {you name whatever you do} with thousands of developers. The number of services is still growing, as a whole, as the business grows - though in some orgs, like mine, they are at level, or even going down a bit (though this is not the goal itself). But not all microservices are equal any more. The critical ones look less like your classic microservice - or at least what I called microservices years back.
On another note: everyone interprets the name “microservice” differently. I’ll write a post summarizing my experiences with the ups and downs of microservices at scale. For now, hopefully this gives some more color.
It’s not a circle, it’s a pendulum. Just like moving more stuff to external dependencies vs. into your own code base. Or like moving functionality to a network service vs. shipping it as a library. Or mono-repo vs. a repo per library.
Such pendulums exists because there aren’tt really that many architects able to find a Goldilocks zone. Instead most teams follow hype, overdo it, then overcorrect and create more hype for others to follow.
It’s a pendulum because it’s not well understood to be fundamentally a set of cultural practices supporting an organisational model.
Instead, senior leadership (mistakenly imagining development practices to be portable with only moderate effort) attempt to copy the tooling/practices of whichever organisation has communicated best about how tooling/practices enable their success - regardless of how poorly that fits the organisational model.
I give total credit for someone talking about something that isn’t working that well for them anymore. Too often these trends just quietly fade away and it’s like no one was ever writing microservices, were we?
The possibly more important thing here than whether they’re moving away from them or not, is that they are a massive engineering org with problems that likely you don’t have. So you probably shouldn’t have been considering microservices anyway, whether or not Uber thought they were a good idea. Unless of course, you’re running a massive engineering org too, then do whatever you think is best.
It’s been since 2015 and I would hesitate to call it “preaching”: it’s been sharing what worked over the eng blog, and what did not work for Uber at the time.
I’ve not seen anyone preaching, meaning anyone wanting to convince or convert anyone else. I personally tell people “here’s what we do, but your mileage will very likely vary”. I’ve always found it interesting to understand how other companies address their challenges and what worked and why.
Also, you don’t need to look far to hear all sorts of different things that work for other companies - some that might seem unconventional for companies of their size or traffic. Shopify shared how they still are a monolith, abeit a modularized one. Stack Overflow shared how in 2013 they ran on a lean hardware stack that scaled up by 2016, but still includes zero usage of the cloud. And I could go on the list.
All of these can serve as inspiration: but the end of the day, you need to make decisions in your environment that you think will work best. Anyone copying the likes of Google, Uber, Shopify, Stack Overflow or others when they’re not even similar in setup will be disappointed.
Anyone copying the likes of Google, Uber, Shopify, Stack Overflow or others when they’re not even similar in setup will be disappointed.
That is exactly what everybody around me is doing. Nobody knows the way to success, so they are copying the behaviors of famous successful companies.
Since i cannot edit my original comment, I want to back off on my tone. Its not like I have anything against people in uber. Its just too many conference talks i see where engineers talk how microservices solve problems, and forget to mention the new problems appearing. And never retroactively admitting the microservices were ever a mistake.
For context, I was sweeping a set of around 150 processes running on a good dozen machines into a single JAR back in 2003, with the huge performance increases and reliability/deployment improvements you might expect.
It can have advantages. The team I’m on (three developers), we have one service that implements the business logic. We have two other services that serve as front ends (one for incoming SIP messages, one for incoming SS7 [1] messages). And we recently added a fourth service, used only by the business logic service, to do an HTTP REST request. We found it easier to throw a UDP [2] message to that new service (which can then deal with TCP and/or TLS) than to try to integrate HTTP REST directly into an app that is event driven by UDP packets.
[1] Signalling System 7
[2] The SS7 interface is mostly UDP-like. SIP is sent over a UDP channel. We can’t afford a dropped packet from blocking us as we have some hard deadlines to deal with.
I wrote that tweet that is making the rounds. Not many things fit in 280 characters and Twitter being immutable, there’s not many things you can go back to clarify. So let me give some more details on this forum.
So no, Uber is not going no-microservices like I’m seeing many people interpret it. It’s not even going to less microservices. And when I said “we’re moving”, that was not exact phrasing. New microservices are more thoughtfully created in my team and in my org. These are “larger” services than some of the early, small, focused, microservices.
Microservices worked well at Uber in many ways and keep helping in others areas. There are problems, of course, and you deal with the problems as you go. This would be the same with e.g. a monolith with thousands of developers, SOA with thousands of developers or {you name whatever you do} with thousands of developers. The number of services is still growing, as a whole, as the business grows - though in some orgs, like mine, they are at level, or even going down a bit (though this is not the goal itself). But not all microservices are equal any more. The critical ones look less like your classic microservice - or at least what I called microservices years back.
On another note: everyone interprets the name “microservice” differently. I’ll write a post summarizing my experiences with the ups and downs of microservices at scale. For now, hopefully this gives some more color.
Any other questions, just ask.
Thank you for posting a clarification. It is of much more value than the Twitter thread and would love to see a blog post about it.
This is a good example demonstrating why I think linking Twitter threads is poor form and I discourage people from doing it.
It’s basically the whole microkernel thing all over again, innit?
Monolithic kernels: “Yikes, one bug can bring the whole OS down, let’s split that out”
Microkernels: “Yikes, having loads of little services effectively work together is actually much harder than we thought!”
Current “hybrid” kernels: “Let’s take the best of both approaches and combine them where it gives us the best bang for our buck”
It looks like they have gone full circle here: services -> microservices -> services (macroservices)
It’s not a circle, it’s a pendulum. Just like moving more stuff to external dependencies vs. into your own code base. Or like moving functionality to a network service vs. shipping it as a library. Or mono-repo vs. a repo per library.
Such pendulums exists because there aren’tt really that many architects able to find a Goldilocks zone. Instead most teams follow hype, overdo it, then overcorrect and create more hype for others to follow.
It’s a pendulum because it’s not well understood to be fundamentally a set of cultural practices supporting an organisational model.
Instead, senior leadership (mistakenly imagining development practices to be portable with only moderate effort) attempt to copy the tooling/practices of whichever organisation has communicated best about how tooling/practices enable their success - regardless of how poorly that fits the organisational model.
I give total credit for someone talking about something that isn’t working that well for them anymore. Too often these trends just quietly fade away and it’s like no one was ever writing microservices, were we?
The possibly more important thing here than whether they’re moving away from them or not, is that they are a massive engineering org with problems that likely you don’t have. So you probably shouldn’t have been considering microservices anyway, whether or not Uber thought they were a good idea. Unless of course, you’re running a massive engineering org too, then do whatever you think is best.
at uber… we’ve been preaching microservices since 2018.
It’s been since 2015 and I would hesitate to call it “preaching”: it’s been sharing what worked over the eng blog, and what did not work for Uber at the time.
Uber went from monolith to SOA in 2015. This SOA followed a microservice-based architecture. And different engineers have been sharing what what they’ve learned along the way: the steps it usually takes to build a microservice, addressing testing problems with a multi-tenancy approach or how and why teams use distributed tracing. We also open soruced some of our tools like Jaeger, which is part of the Cloud Native Foundation’s graduated projects, alongside Kubernetes and Prometheus.
I’ve not seen anyone preaching, meaning anyone wanting to convince or convert anyone else. I personally tell people “here’s what we do, but your mileage will very likely vary”. I’ve always found it interesting to understand how other companies address their challenges and what worked and why.
Also, you don’t need to look far to hear all sorts of different things that work for other companies - some that might seem unconventional for companies of their size or traffic. Shopify shared how they still are a monolith, abeit a modularized one. Stack Overflow shared how in 2013 they ran on a lean hardware stack that scaled up by 2016, but still includes zero usage of the cloud. And I could go on the list.
All of these can serve as inspiration: but the end of the day, you need to make decisions in your environment that you think will work best. Anyone copying the likes of Google, Uber, Shopify, Stack Overflow or others when they’re not even similar in setup will be disappointed.
That is exactly what everybody around me is doing. Nobody knows the way to success, so they are copying the behaviors of famous successful companies.
Since i cannot edit my original comment, I want to back off on my tone. Its not like I have anything against people in uber. Its just too many conference talks i see where engineers talk how microservices solve problems, and forget to mention the new problems appearing. And never retroactively admitting the microservices were ever a mistake.
What problems did moving from a monolith solve for y’all? Were they more people problems or technical ones?
Exactly. :-)
For context, I was sweeping a set of around 150 processes running on a good dozen machines into a single JAR back in 2003, with the huge performance increases and reliability/deployment improvements you might expect.
Does anyone have any definition / understanding of what exactly the difference between a micro/macroservice is?
Since something that may be a microservice at one company could be called a macroservice and vise versa.
It’s a polite and gradual way of backpedalling and saying “hey maybe we took this shit a bit too far?”
The original mention explains it here: https://twitter.com/copyconstruct/status/1247131341851783168?s=20
My definition of Microservice: Every team has its own service. Interaction only via internet-facing APIs.
Multiple services per team goes beyond Microservices and I don’t really see an advantage there.
It can have advantages. The team I’m on (three developers), we have one service that implements the business logic. We have two other services that serve as front ends (one for incoming SIP messages, one for incoming SS7 [1] messages). And we recently added a fourth service, used only by the business logic service, to do an HTTP REST request. We found it easier to throw a UDP [2] message to that new service (which can then deal with TCP and/or TLS) than to try to integrate HTTP REST directly into an app that is event driven by UDP packets.
[1] Signalling System 7
[2] The SS7 interface is mostly UDP-like. SIP is sent over a UDP channel. We can’t afford a dropped packet from blocking us as we have some hard deadlines to deal with.
It is a hot new thing that you can put on your CV and pretend to be a “thought leader” about /s