I don’t normally work with Python but we went from Flask (where we were using SSE) to Django. It was really tricky. I can’t easily explain the whole thing and it has nothing specifically to do with WS vs SSE but there is at least one gotcha. So we switched to websockets even though our data is one way (use case works with SSE).
There are other issues with asgi + daphne that I don’t believe is even fixed in Django 4. It was a wild ride. It’s been fine but we have to run a bunch of different processes because of idk, gunicorn can’t do async. I’m sure things will change in the future.
Flask-SSE was much simpler yes. I wonder if it won’t be as well supported though. It can’t be two-way, so the user base is less (that whole thing)?
I’m looking to add a live streaming endpoint to a customer-facing API at work. Currently the service is implemented in Django/DRF, but for the new live streaming endpoint I’m actually planning to use a separate Go microservice just because I expect the traffic patterns to be so different. Hopefully can also migrate the rest of the endpoints to the Go service too.
We used daphne and ASGI for a bit at my previous employer but the entire experience was horrible. I probably repressed the specifics in my mind because I don’t remember exactly what went wrong, just that we had major outages a few times and many weird other issues like regular file downloads taking ages in dev. And the setup was extremely complex as well, if I remember correctly.
It’s such a mess, it’s actually much simpler if you use Django with plain WSGI and spin up a separate websocket server process on the side which pushes updates to the clients. All you need is a way to communicate with the websocket server from your main Django process (which can be done via HTTP, or RabbitMQ or some such). We even built a reusable component like that in Python, which can be used quite simply from Django.
Very happy that we’re only 1 refactor away from completely removing Daphne!
Daphne is complex, difficult to debug, slow on dev, and crashes randomly on production (and doesn’t tell you why). Most likely we’ve configured it wrongly or something like that, but at the same time it should be so difficult / complex…
The trucking example really hits home, where our trucking software brought the server down to its knees last November. In a very similar way, for certain requests a full table scan was done and 70GB worth of data was pulled only to discard 99% of it…
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Choosing a new office for our branch in Bydgoszcz, Poland. The current one is already too small after 3 months. Already did 3 viewings, gonna view another 3 and decide which one it will be on Monday.
I have been traveling from Eindhoven to Bydgoszcz every 2 weeks now, exciting times!
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
This affected a customer of mine. At least he had a server for backups; setting up everything to work on that machine. It’s a bit underpowered but it’ll have to do for the time being.
Well, it is the backup server we’re abusing for hosting for now. So the data transfer itself is not a big issue. Restoring databases took ages because the disks are rather slow, but I’ve restored about 8 websites now, and I started around 8 o’ clock, so that took 4 hours, including some getting to grips with restoring. I had set up the server but forgotten a lot of details. And I also discovered my customer had added a new main directory which wasn’t being backed up… Really, you want to practice this stuff so you’re not caught flat-footed when shit hits the fan, and to make sure you have all you need.
I expect the rest of the sites to be less work, but there’s a lot of manual stuff involved.
Price was indeed one aspect. Upgrading to a better virtual machine at Tilaa was about the same price as a dedicated machine at Leaseweb. The specs for the dedicated machine was waaay beter, so more for the same price.
Another aspect was that we agreed to upgrade at a certain time window, and Tilaa missed that window and I couldn’t reach them whatsoever. A second window was painful to negotiate with the customer, and with a dedicated machine I didn’t have to rely on a third party to press a button.
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Wrapping up various odds and ends, as this will be my last week at Code Yellow. Next week I’ll start at Bevuta! I’m happy and sad at the same time, after 7 years of working with my colleagues at Code Yellow I almost feel like I’m part of the furniture ;)
I’ve dropped a live database by accident as well, and seen a few colleagues do the same. Happens to the best of us, but mistakes like let you grow quickly.
Enjoy the journey in figuring out what went wrong!
This is a tangent, but is there a good work on human factors in SRE? I’ve always done some informal things (prompt coloring, aliases to warn, trying to actually sleep, etc.) – but has anyone presented a more systematic approach?
I think the molly-guard (apparently named after the plastic cover that guards a physical flip switch) package is a good example. It prevents you from accidentally running shutdown or reboot on remote hosts by requiring you to type in the name of the host before actually running the command. Very useful when you forget that you’re ssh’d into a host on a certain xterm and absentmindedly want to reboot your own computer.
Would be nice if there were more things like this!
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Is there more info available other than that single page? I’m for example interested in what the motivations are why this tool exists, what makes it stand out from other tools, what tech it’s using etc etc.
Not quite yet. I would like to get a more detailed blog post out soon. I’m painfully close to something that I can release to a few brave souls to try out, so I’ve been focusing on wrapping it up. If you’re interested, please sign up (or follow me on Twitter if that’s your thing), and I’ll make sure to get some more details out there soon.
Super quick summary: This tool is meant to marry property-based testing with models, temporal logic specifications (much like in TLA+), and browser testing with webdriver. I want to make it easy to use this on any web page, without loads of boilerplating test setup. Basically write a spec, point at your website, off you go.
It takes a rather black-box approach to the SUT, although it needs to know about CSS selectors and other DOM attributes. But a web page can be React, server-side rendered, a mix, whatever. You can test multiple sites with one spec. It detects changes to relevant DOM elements so your state changes can be synchronous or asynchronous (like changing something after an HTTP request is completed).
It is different from PBT with models because you don’t write the simplified/abstract model. You write a specification (which is a bit different), and you can gradually make that specification more detailed. With models and PBT, in my experience, you need to capture all of the essential complexity of the SUT in your model.
It’s different from model checking as seen in TLA+, because this is testing the implementation. It’s also using finite traces which are much smaller. In TLA+ you can check huge state spaces in seconds. Here you might test a few hundred or thousand different cases with, say, 50 actions in each, and it’s currently in the order of minutes. But even if failing traces can be somewhat large, WebCheck tries to shrink it down (like QuickCheck does).
Regarding implementation, the current version is built in Haskell and the logic DSL is an adaption of PureScript. I’m using the PureScript compiler as a library, but interpreting the CoreFn representation and adding the temporal logic operators next and always, along with queryAll. The specification must be a pure expression, it doesn’t support Effect from PureScript. But it supports PureScript packages! You can use monad transformers in your specs, if you’d be so inclined.
If you want a free in-between while you get things figured out, Mega should give you enough gratis storage space to hold them. You’ll need a pretty modern browser, though.
Working and looking for a new job. Already have a couple of leads headed in the right direction, with one I’m quite interested in. It’s about time for a change of pace and to see if I can get a culture that better matches my style of work.
(Note: it’s been a while since we’ve seen a “Who’s hiring?” post. I’d be very interested to work for some fellow crustaceans!)
A test that I commonly write is for filtering / sorting, to see if complex combinations result in desired results. For this, the state of the db is quite important. I’m curious, how would approach such a test?
So far I’ve handled that by limiting the scope to a single user. Another way would be to write the assertion as a property rather than checking for entries x, y and z - a bit more like a generative test. Maybe there are other ways depending on context. If nothing works you could fall back to a second backing store that was cleaned for that test run.
I did take a look at Gitlab, but their offerings seem to be closer to development tools; what our project is really suffering from is deployed servers. AFAICT Gitlab doesn’t seem to offer anything in that space unfortunately.
Position(s): Senior Frontend Developer or Senior Backend Developer
Location: The Netherlands, Eindhoven, on site
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. Our projects span multiple years, and are technically challenging. Our focus is on stability and incremental improvements, since projects outlast popular trends regarding tech stacks. The clients range in different fields: transport, sales, health, production and telecom.
I don’t normally work with Python but we went from Flask (where we were using SSE) to Django. It was really tricky. I can’t easily explain the whole thing and it has nothing specifically to do with WS vs SSE but there is at least one gotcha. So we switched to websockets even though our data is one way (use case works with SSE).
There are other issues with asgi + daphne that I don’t believe is even fixed in Django 4. It was a wild ride. It’s been fine but we have to run a bunch of different processes because of idk, gunicorn can’t do async. I’m sure things will change in the future.
Flask-SSE was much simpler yes. I wonder if it won’t be as well supported though. It can’t be two-way, so the user base is less (that whole thing)?
I’m looking to add a live streaming endpoint to a customer-facing API at work. Currently the service is implemented in Django/DRF, but for the new live streaming endpoint I’m actually planning to use a separate Go microservice just because I expect the traffic patterns to be so different. Hopefully can also migrate the rest of the endpoints to the Go service too.
We used daphne and ASGI for a bit at my previous employer but the entire experience was horrible. I probably repressed the specifics in my mind because I don’t remember exactly what went wrong, just that we had major outages a few times and many weird other issues like regular file downloads taking ages in dev. And the setup was extremely complex as well, if I remember correctly.
It’s such a mess, it’s actually much simpler if you use Django with plain WSGI and spin up a separate websocket server process on the side which pushes updates to the clients. All you need is a way to communicate with the websocket server from your main Django process (which can be done via HTTP, or RabbitMQ or some such). We even built a reusable component like that in Python, which can be used quite simply from Django.
Very happy that we’re only 1 refactor away from completely removing Daphne!
Daphne is complex, difficult to debug, slow on dev, and crashes randomly on production (and doesn’t tell you why). Most likely we’ve configured it wrongly or something like that, but at the same time it should be so difficult / complex…
Yes! Kill it with fire! :)
Good to hear you managed to get close to eradicating it.
Thanks to Bob & Anastasiia, today it has been killed with fire :)
Awesome work! Tell them I said hi ;)
The trucking example really hits home, where our trucking software brought the server down to its knees last November. In a very similar way, for certain requests a full table scan was done and 70GB worth of data was pulled only to discard 99% of it…
I also found it funny they were using trucking as an example; these solution patterns are very familiar, down to the last detail almost ;)
Why the winking face? Did they copy the tutorial story from something you worked on?
zaico and I used to work together on a trucking system, and my guess is all trucking systems are alike in a certain way.
Company: Code Yellow BV
Company site: https://www.codeyellow.nl/
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE) / Bydgoszcz, Poland (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Tech stack: Django, Postgres, React, Flutter, Docker, Cypress
Compensation: Typical Dutch salary / profit sharing. Lunch / dinner / bike / house cleaner included.
Contact: contact@codeyellow.nl or DM me
Choosing a new office for our branch in Bydgoszcz, Poland. The current one is already too small after 3 months. Already did 3 viewings, gonna view another 3 and decide which one it will be on Monday.
I have been traveling from Eindhoven to Bydgoszcz every 2 weeks now, exciting times!
Company: Code Yellow BV
Company site: https://www.codeyellow.nl/vacanCY.html
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE) / Bydgoszcz, Poland (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Tech stack: Django, Postgres, React, Flutter, Docker, Cypress
Contact: contact@codeyellow.nl or DM me
This affected a customer of mine. At least he had a server for backups; setting up everything to work on that machine. It’s a bit underpowered but it’ll have to do for the time being.
How long does it take to get everything up & running again from backups?
I moved the project which we worked on from Tilaa to Leaseweb, and just transferring data took 6 hours (at least I could prepared for it).
Well, it is the backup server we’re abusing for hosting for now. So the data transfer itself is not a big issue. Restoring databases took ages because the disks are rather slow, but I’ve restored about 8 websites now, and I started around 8 o’ clock, so that took 4 hours, including some getting to grips with restoring. I had set up the server but forgotten a lot of details. And I also discovered my customer had added a new main directory which wasn’t being backed up… Really, you want to practice this stuff so you’re not caught flat-footed when shit hits the fan, and to make sure you have all you need.
I expect the rest of the sites to be less work, but there’s a lot of manual stuff involved.
Why did you relocate? Price?
Price was indeed one aspect. Upgrading to a better virtual machine at Tilaa was about the same price as a dedicated machine at Leaseweb. The specs for the dedicated machine was waaay beter, so more for the same price.
Another aspect was that we agreed to upgrade at a certain time window, and Tilaa missed that window and I couldn’t reach them whatsoever. A second window was painful to negotiate with the customer, and with a dedicated machine I didn’t have to rely on a third party to press a button.
I’m starting to work on Dendron, a tool to make digital gardens of notes with a powerful VSCode plugin to query and refactor the notes.
I have two wikis now, one with GitBook and one with Dendron where I plan to do more in depth note taking, using all the best features of the tool.
Quite excited about working and making this digital garden space better for all. 🌿
Sounds great, gonna check it out. Perhaps one day also available for Sublime Text?
Certainly. Something I want to look into. :)
Company: Code Yellow BV
Company site: https://www.codeyellow.nl/vacanCY.html
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Tech stack: Django, Postgres, React, Flutter, Docker, Cypress
Contact: contact@codeyellow.nl or DM me
Wrapping up various odds and ends, as this will be my last week at Code Yellow. Next week I’ll start at Bevuta! I’m happy and sad at the same time, after 7 years of working with my colleagues at Code Yellow I almost feel like I’m part of the furniture ;)
Really enjoyed working those 7 years with you, and I will definitely come visit you at Bevuta!
I’ve dropped a live database by accident as well, and seen a few colleagues do the same. Happens to the best of us, but mistakes like let you grow quickly.
Enjoy the journey in figuring out what went wrong!
This is a tangent, but is there a good work on human factors in SRE? I’ve always done some informal things (prompt coloring, aliases to warn, trying to actually sleep, etc.) – but has anyone presented a more systematic approach?
Fine grain ACLS. Only let one user from one location do drops. It should be like a ceremony to get a database dropped.
I think the molly-guard (apparently named after the plastic cover that guards a physical flip switch) package is a good example. It prevents you from accidentally running shutdown or reboot on remote hosts by requiring you to type in the name of the host before actually running the command. Very useful when you forget that you’re ssh’d into a host on a certain xterm and absentmindedly want to reboot your own computer.
Would be nice if there were more things like this!
Company: Code Yellow BV
Company site: https://www.codeyellow.nl/vacanCY.html
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Tech stack: Django, Postgres, React, Flutter, Docker, Cypress
Contact: contact@codeyellow.nl or DM me
Can you elaborate a bit more? Social sector + data / analytics is pretty vague. For example:
More info is insightful to give tips about where to look for a job. Based on the little available info, I’d guess Itility might be something?
thanks! sure, definitely: senior, full-time, we’re moving to the UK so remote or local is fine :)
Company: Code Yellow BV
Company site: https://www.codeyellow.nl/vacanCY.html
Position(s): Medior / Senior Front-End Developer, Medior / Senior Backend Developer, and more
Location: Eindhoven, Netherlands (ONSITE)
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. The clients range in different fields: transport, sales, health, production and telecom. The range of different fields gives insight in problems unique to those fields, but also interesting overlapping problems.
Tech stack: Django, Postgres, React, React-Native, Docker, Cypress
Contact: contact@codeyellow.nl or DM me
Is there more info available other than that single page? I’m for example interested in what the motivations are why this tool exists, what makes it stand out from other tools, what tech it’s using etc etc.
Not quite yet. I would like to get a more detailed blog post out soon. I’m painfully close to something that I can release to a few brave souls to try out, so I’ve been focusing on wrapping it up. If you’re interested, please sign up (or follow me on Twitter if that’s your thing), and I’ll make sure to get some more details out there soon.
Super quick summary: This tool is meant to marry property-based testing with models, temporal logic specifications (much like in TLA+), and browser testing with webdriver. I want to make it easy to use this on any web page, without loads of boilerplating test setup. Basically write a spec, point at your website, off you go.
It takes a rather black-box approach to the SUT, although it needs to know about CSS selectors and other DOM attributes. But a web page can be React, server-side rendered, a mix, whatever. You can test multiple sites with one spec. It detects changes to relevant DOM elements so your state changes can be synchronous or asynchronous (like changing something after an HTTP request is completed).
It is different from PBT with models because you don’t write the simplified/abstract model. You write a specification (which is a bit different), and you can gradually make that specification more detailed. With models and PBT, in my experience, you need to capture all of the essential complexity of the SUT in your model.
It’s different from model checking as seen in TLA+, because this is testing the implementation. It’s also using finite traces which are much smaller. In TLA+ you can check huge state spaces in seconds. Here you might test a few hundred or thousand different cases with, say, 50 actions in each, and it’s currently in the order of minutes. But even if failing traces can be somewhat large, WebCheck tries to shrink it down (like QuickCheck does).
Regarding implementation, the current version is built in Haskell and the logic DSL is an adaption of PureScript. I’m using the PureScript compiler as a library, but interpreting the CoreFn representation and adding the temporal logic operators
nextandalways, along withqueryAll. The specification must be a pure expression, it doesn’t support Effect from PureScript. But it supports PureScript packages! You can use monad transformers in your specs, if you’d be so inclined.I don’t. They’re all (about 21GB) jammed into about twelve folders on my failing hard drive :(
Ouch… Very recognizable, lost all my pictures. Not ideal, but now simply use iCloud.
If you want a free in-between while you get things figured out, Mega should give you enough gratis storage space to hold them. You’ll need a pretty modern browser, though.
Working and looking for a new job. Already have a couple of leads headed in the right direction, with one I’m quite interested in. It’s about time for a change of pace and to see if I can get a culture that better matches my style of work.
(Note: it’s been a while since we’ve seen a “Who’s hiring?” post. I’d be very interested to work for some fellow crustaceans!)
Last one: https://lobste.rs/s/wigiwx/who_s_hiring_q2y2020
Ours is still open :)
A test that I commonly write is for filtering / sorting, to see if complex combinations result in desired results. For this, the state of the db is quite important. I’m curious, how would approach such a test?
So far I’ve handled that by limiting the scope to a single user. Another way would be to write the assertion as a property rather than checking for entries x, y and z - a bit more like a generative test. Maybe there are other ways depending on context. If nothing works you could fall back to a second backing store that was cleaned for that test run.
Perhaps Gitlab might be something for you? Free for open source, and it might be feature complete enough for you:
https://about.gitlab.com/solutions/open-source/projects/
I did take a look at Gitlab, but their offerings seem to be closer to development tools; what our project is really suffering from is deployed servers. AFAICT Gitlab doesn’t seem to offer anything in that space unfortunately.
Fighting with Gitlab CI to skip jobs based on results of previous jobs.
Company site: codeyellow.nl
Position(s): Senior Frontend Developer or Senior Backend Developer
Location: The Netherlands, Eindhoven, on site
Description: We build (web / mobile) applications that streamline critical business processes. Because each application is part of a critical process, it is used intensively by thousands of employees daily. Our projects span multiple years, and are technically challenging. Our focus is on stability and incremental improvements, since projects outlast popular trends regarding tech stacks. The clients range in different fields: transport, sales, health, production and telecom.
Tech stack: Debian, PostgreSQL, Django, React + MobX + Semantic-UI, React Native + Native Base, Cypress
Contact: contact@codeyellow.nl