I am trying to “memefully” learn Rust. The endgoal is to be competent enough to create a Tauri based GUI software.
For the “memefully” part the goal is to just enjoy the learning process as much as possible. Rust is a hard language, we all know that. So, I plan to write something in Python, then in Nim and then in Rust. Then write a blog about it. I plan to follow the Rust CLI book [0], however I want to do random projects. This weekend’s plan is to write a CLI that takes makes async get requests to a bunch of input URLs.
async Rust feels a bit more like C or C++, where performant and versatile take top rank, and one has to have a lot of experience to know how to avoid sharp edges.
I recommend considering using ureq instead, especially (but not only) if you still are learning Rust and you see Rust as hard.
The costs of async are worth paying, if you’re writing an HTTP server that must serve many many clients with minimal overhead. However, for HTTP clients, we believe that the cost is usually not worth paying. The low-cost alternative to async I/O is blocking I/O, which has a different price: it requires an OS thread per concurrent request. However, that price is usually not high: most HTTP clients make requests sequentially, or with low concurrency.
For several years I thought that Rust was too hard and didn’t even attempt it seriously. Then last year I started the Advent of Code 2022 in Python (that I use professionally), and found this masterpiece from Amos: https://fasterthanli.me/series/advent-of-code-2022/part-1
It was super refreshing as it was showing me how close to Python basic Rust can be, but also with many additional features that make it amazing.
I was able to do about the first 13 days in Rust thanks to that series (I was solving it before looking at the solution from Amos). I must say that it was an eye opener to me. I really recommend this series.
Rye should not exist, it should just be the out of the box Python experience.
Armin gets it. I re-read his blog post about Python 3 and he has a side mention of packing there too:
And the amount of support requests I got about Unicode are not even close to the amount of support requests I got about dealing with Python packages or the import system. And even with distutils2 this is still a much bigger problem in the Python-land than Unicode is.
I learned Go by writing a book about it, but I ended up with a book listing all of the ways that Go was badly designed, which turned out not to be what the market wanted.
I have been a heavy user of GitHub Actions over the last few years. Yaml configuration can be pretty rancid, but I still feel most of my challenges can be solved by hacking some bash/python/JS in (current jobs are using all 3 for configuration) and by using community packages. I often want to chuck the yaml in the trash and use a real language to configure everything. Why they didn’t go with a TypeScript file is beyond me.
My wish list contains GPU support and M1 runners (the latter we’re going to solve by my boss buying a mac mini). Some new integration tests I’m adding are going to be 14h per commit 🫣
I’ve also been using GitHub Actions a lot, in particular with ChristopherHX’s reimplementation of the runner in go. This makes it easy to use GitHub Actions to drive a large range of platforms and architectures that aren’t officially supported.
I am also a heavy user of github actions but am considering switching to sourcehut just because they let you SSH into the hosts if a run fails. It takes so long to develop solid CI pipelines on github because you have to commit, push, and wait for the job to be picked up, initialized, and run before you can see whether it fails. After a decade developing software I really do think iteration time is the strongest single predictor of how long something will take to implement.
SSH into a failing build is such a lifesaver. I use it all the time on sourcehut, and I can’t imagine why anyone would launch a CI platform that doesn’t have that feature. The feedback loop without it is awful.
It also has the benefit the properties are enforced only when it matters (code integrated into the canonical branch), and can be broken otherwise, giving a nice transactional semantics.
Pre-commit hooks are fundamentally client-side validation. They don’t enforce anything, they are advisory
Agreed. One repo I work on has what amounts to a “don’t commit directly to release” precommit hook, but it’s slow and I don’t even have perms to push to the release branch, so I habitually git commit -nm to skip it. It’s not a very good validation if there’s a single-letter command line argument to skip it.
It’s true that you can bypass your local hook, and I agree that “perfect” might be an incorrect choice of word, but;
You have to do it explicitly, which means (unless there’s something really wrong with the linter or a developer that really doesn’t care for policy), that the default (and probably) common behavior will be to use the linter.
You don’t have to use local hooks alone - by using both local and remote hooks, you get the advantages of both.
You can use tests instead of running a linter. In the end, like the linter, you’d use them locally, remotely or both.
I’m actually a proponent of both (and maybe I should’ve included that in the article), at least for shorter tasks, because remote enforcement alone for tasks that take seconds encourage more context switching.
Uhu, that’s why I like to drive everything through tests. That way, cargo test (or whatever is the equivalent in a language of choice) is the single thing that needs to be run either locally or remotely.
Hey @matklad, I’m the author of the Tiny CI system post you linked to and I just wanted to say that it means a lot that you (whom I respect a lot for all your work and the way you explain things you’re working on) linked to it. It gives me a big moral boost towards writing more blog posts and also a follow-up on the one you mentioned.
Just skipped through the video, but I think the gist is that it uses a custom http.Client transport to simulate the server response. Alternatively this could have made use of httptest.Server, but this will actually do local network requests, or use something like github.com/dnaeon/go-vcr to pre-record server responses and serve them from a file in the test case.
The stopwatch app from the Textual tutorial starts running for me in less than a quarter of a second - it’s fast enough that I didn’t even notice the delay until I tried to eyeball-measure it just now.
The whole point of TUI apps is that you’re going to spend some time in them. Does a quarter of a second to show the initial screen really matter to anyone? That’s way faster than loading a web application in a browser tab.
Thinking about it, I don’t think I’ve ever used a TUI written in any language where the startup speed has bothered me.
To whoever is starting to write a reply - no, it cannot be optimized.
Python will always take a bit to start the program.
It depends.
Maybe startup time doesn’t really matter for someone’s particular use
case. While there will always be some baseline startup time from
Python, there are cases where you can optimize it and possibly bring it
down to a level you find acceptable.
At a job, I was tasked with figuring out and speeding up slow start of a
Python program. Nobody knew why the bloody thing was taking so long to
start. Part of it was network delays, of course, but part was Python.
I did some profiling.
This little Python program was importing a library, and that library
imported something called pkg_resources. Turns out that pkg_resources
does a bunch of work at import-time (nooo!). After some digging, I
found that pkg_resources was actually an optional dependency of the
library we were using. It did a try … import … except: …, and
could work without this dependency. After digging into the code (both
ours and the library’s), I found that we didn’t need the facilities of
pkg_resources at all.
We didn’t want to uninstall it. Distro packages depended on it, and it
was possible that there were other programs on the system that might use
it. So I hacked up a module importer for our program that raised
ModuleNotFoundError whenever something tried to import pkg_resources.
I cut a nearly one-second start time down to an acceptable 300
milliseconds or so, and IIRC a fair portion of the 300 milliseconds was
from SSH.
Know your dependencies (direct and indirect). Know what you’re calling
(directly and indirectly) and when you’re calling it. Profile. And if
your Python startup times are slow, look for import-time shenanigans.
Program startup speed is important for some applications but negligible compared to other aspects like usability, accessibility or ease of development, wouldn’t you agree?
Program startup speed and performance is an important part of usability. It’s bad for software users when the software they use is full of latency, or uses so many system resources it bogs down their entire computer.
Agreed, it’s part of usability. But it depends on the numbers. Saying “stop writing TUIs in Python” because of 200ms (out of which something can be shaved off with optimization) sounds extreme.
I completely agree with the unsuitability of Python for TUI / CLI projects! (Especially if these tools are short-lived in their execution.)
Long ago (but still using today) I’ve written a simple console editor that has ~3K lines of code (25 Python modules) which imports only 12 core (and built-in) Python modules (without any other dependencies) and mainly uses curses.
On any laptop I’ve tried it (even 10 years old) it starts fast enough. However recently I’ve bought an Android device and tried it under Termux. It’s slow as hell, taking more than a second to start… (Afterwards it’s OK-ish to use.)
What’s the issue? The Python VM is slow to bootstrap and load the code (in my case it’s already in .pyo format, all in a zip with zipapp). For example just calling (on my Lenovo T450) python2 -c True takes ~10ms meanwhile python3.10 -c True takes ~14ms (python3.6 used to take ~20ms). Just adding import json adds another +10ms, meanwhile import curses, argparse, subprocess, json (which is perhaps the minimal any current-day project requires) yields a ~40ms startup.
With this in mind, this startup latency starts to pile-on and it has no solution in sight (except rewriting it in a compiled language).
Granted, even other languages have their issues, like for example Go, which is very eager in initializing any modules you have referenced, even though will never use, and thus easily adds to startup latency.
(I’ll not even touch on the deployment model, where zipapp is almost unused for deployment and https://github.com/indygreg/PyOxidizer is the only project out there trying to really make a difference…)
Why would I? It only has buttons and checkboxes implemented. And according to comments in here is still taking 1/4 of a second to start on a modern CPU.
EDIT: In the demo video, the demo takes 34 frames to boot. At 60fps, that’s more than half a second.
The popularity of a chat app - particularly one that most people use because it’s what their workplace standardizes on - is driven much more by network effects than by the quality of the standard client app. It is bad that the Slack client is slow, and this is made worse by the fact that there aren’t a whole lot of alternative clients for the Slack network that a person who is required to use Slack as part of their job can use instead of the official, slow, one.
I think that the problem with your assessment is the assumption that the users of this framework have the knowledge to use a different language or want to use a different language than python. Nobody is forcing you to use it and if folks are releasing tools using it, nobody is forcing you to use those. For those that want to add a TUI front end to a script they made, this seems like a good option.
Thankfully AWS App Runner seems like it might become a good alternative to Heroku free plan, it isn’t completely free but very cheap for most apps.
The biggest pain point for me will be that we have asked people applying for Junior positions to host their coding pre-assignment with Heroku (as it was free and easy to use) but now that really won’t be an option anymore so we will have to think of something else. Of course people have been free to host their pre-assignments however they wanted to before also but we don’t really require people to know AWS/Linux/etc. for our Junior positions, it has just been enough that people also show they can get their app running somehow. Any suggestions around this would very much be appreaciated.
If you’re still on LastPass and looking for any excuse to switch (like I was last year): 1Password fixes every problem I had with LastPass (eg working form fill, even in Firefox) and gives me a few things I didn’t even know I wanted (eg an ssh agent). Just make the leap, it’s a total no brainer.
I second 1Password, I’ve used it personally for years. Just started using LastPass at work recently and I was surprised at how the whole thing felt very… unpolished? 1Password feels like a real professional app, LastPass feels like some proof of concept that was thrown together as a TUI in a weekend and then had a UI bolted on the front.
You hit the nail on the head there I think, a company I worked at used LastPass enterprise and it honestly felt like they never hired any designers and just told some devs to build an MVP and that stuck. It just feels like a side project. When I first used 1p I was amazed at how nice and compact the UI is and switched my personal solution from Dashlane. I still find that Dashlane’s form filling ability and other features of the browser extension are still miles ahead of 1p though. But I hope they’ll catch up soon.
I don’t know as this is a particularly terrible thing - it’s not like they haven’t done partnerships with other commercial services for storing special data before (namely, their Privacy.com [1] and Fastmail [2] integrations) or had integrations to make some secrets have special functionality (e.g. SSH keys [3]). I think having an automatic way to store cryptocurrency secrets in a password manager is a pretty neat feature, even if I might not necessarily think cryptocurrency is all that good or important. If they want to keep going in this direction, I would like to see them integrate with a more open wallet, and probably tone down their NFT-related marketing about it.
It’s a fastmail thing: “A Masked Email address is a unique, automatically generated email address that can be used in place of your real email address. Masked Email addresses are formatted as random words plus a random number, e.g. dog.food3495@domain.tld.”
if you use mailbox.org with your own domain, it’s trivial to configure it to accept mails to any @yourdomain email address. no need to use an external tool…
Someone emailed a nasty note to the reporters at my website from an Apple masked address. I’m not sure how to report abuse with these things. There should be a prominent webpage that lists the address to forward abuse to.
I assume this code only works with Fastmail, it sure appears like it. I don’t use Apple’s email addresses, so I’ve never looked, but good to know!
I have a domain with a catch-all address, so I can just invent them for 1 off things, no need to tell my email provider ahead of time.
I assume for abuse, you could always email postmaster@ and reference the masked email address. I imagine it will get routed to /dev/null like every other email problem I’ve ever sent in the last decade, but it might make you feel better! :)
I’ve gotten a response from postmaster before! Granted, it was for a ISP in Pennsylvania that had constant delivery problems, but they did at least write back to me. :-)
I used to be able to reach humans via postmaster@, but I’ve never talked to another human in the past decade emailing that or webmaster@ (which usually just bounces). Seems nobody respects RFC-2142 anymore.
Neat, thanks! I’m using fastmail, but didn’t know this is possible. I’ve been using this: https://www.fastmail.help/hc/en-us/articles/360060591053 but seems that masked emails are more versatile — with subdomain addressing anyone can send mails to any random string, losing ability to track who is spamming me.
Primarily, as I have experienced it, GitHub CoPilot is used, as the name suggests, as a co-pilot for inconvenient glue code or well-known algorithms. It’s basically an AI automation of copying code from StackOverflow. So, I still cannot see the big affair here.
I’m also not a lawyer, but if we look at the popular MIT License, for instance, it says:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
The key word in the MIT license snippet that you’ve quoted is ‘substantial’. This matters because not everything in a codebase is subject to copyright. For example, if you copied for (int i=0 ; i<100 ; i++) { from some MIT-licensed source file, the attribution requirement wouldn’t hold because that line is not sufficiently original to qualify for copyright (copyright covers original works, which is quite a subjective thing, but some things are obviously on the not-qualified line). I am not a lawyer, and so I can’t comment on whether the things generated by Copilot meet this bar (from what I’ve read, being a lawyer would still not let me confidently make such a claim, there have been a number of expensive court cases to try to make this determination over shortish snippets of code).
If you type in some well-known class like “FizzBuzz” then it may paste the entire class that was grabbed from a specific piece of code somewhere. I have seen videos of people testing CoPilot where it pastes enough to fill a page. Given that possibility, I think the possibility of violating an MIT license is very likely to happen to many people.
It’s a great suggestion! I haven’t used django-environ for a while on side projects, but I should look into it again (I remember using it on a commercial project a while ago, and thinking that it was great).
Just looking at some recent code of mine, I’ve ended up using this pattern a few times which I think achieves a similar result:
SECRET_KEY = os.environ.get("DJANGO_SECRET_KEY")
if not SECRET_KEY:
raise ImproperlyConfigured("Do not run with the default secret key in production")
The added support for cache / email “url” formats in django-environ and the ability to replace dj-database-url as a dependency definitely makes me want to use this on my next project.
Someone might want to avoid setting up new dependencies or a new service. But my motivation was only to see if I could get a good enough result using just Django out of curiosity. For more serious/complex needs surely something like celery is better.
I appreciate this post, it shows that it doesn’t take much to do some simple task queue.
I have seen this pattern quite often but I haven’t seen a popular Django app emerge and establish itself as “good enough until you need Celery”. I wonder why that is?
For me it looks too small in scope to be a separate Django app. I hope this post will help people copy-paste and build their own solutions when they find it by searching for django+tasks+jobs
Thanks for an interesting post. I’ve been using django-q but this looks nicer in several ways. I think I’m going to try your approach on my next tool instead of reaching for django-q right away.
I enjoy django-q because it lets you recycle the task process (when you set the number of tasks per child to 1, the monitor process restarts the child process once it finishes a task) so that you can cleanup things like in-memory sqlite and any resources that belong to the process automatically.
I am not too fond of of gigantic web frameworks, but the advantage is, of course getting X for free. In this case backgrounds jobs.
I never really understood the fascination for celery. And I suspect much of its popularity builds up on web developers not knowing to write a system service. But I digress. Adding celery would mean running celery, set up redia or other backend, and configure the connection between all pieces. That is a lot of possible failure failure points than can be avoided.
That’s assuming you don’t need an actual distributed queue. If you don’t and only want to run an odd background job, that’s great. But once you need to split the queued tasks among tens of servers, handle retries, deal with contention issues, distribute tasks to different groups, and make sure that your web resources are not affected by background processing… you want celery or something similar. But the idea that celery is popular because people can’t write services is a bit disrespectful.
That’s assuming you don’t need an actual distributed queue
Which is the case of most people using celery.
But your assertion is too simplified. A distributed job queue is trivial to implement using redis or any other network accessible queue. I cluding a custom built one, which can be built in minutes.
Ooh nice this looks like it may be fairly new as I’ve not seen that before. I know GitLab has had it for some time (called Merge Trains) and is convenient for this
I’ve been learning Rust the past couple weeks strictly for hobbyist reasons. I’m coming from the dynamic typing world of Python and JavaScript. While I’m familiar with the strongly typed paradigms of C and Java, I figured spending my time in Rust would be more beneficial.
There are a lot of individual things I like in Rust, but programming in it in general feels like I’m trying to learn a new sport and I can’t quite seems to get my sense of balance. E.G., “I’ve been out on the water a hundred times now, so why can’t I comfortably stand on the surfboard now?” type of feeling. If that makes sense.
I’ll keep plowing forward with it though. I’ve got The Rust Programming Language book from No Starch Press, and I have been spending a lot of my time in the Rust Book online. I’ll find my balance, I just wish I could find it sooner than later.
While I’m familiar with the strongly typed paradigms of C and Java
C is barely type at all, and Java is a weird hybrid of a poor type system and just being dynamically typed anyway. I wouldn’t call either “strongly typed”.
C is barely type at all, and Java is a weird hybrid of a poor type system and just being dynamically typed anyway. I wouldn’t call either “strongly typed”.
Maybe I should have chosen “static” instead, as I also referred to “dynamic” typing rather than “weak” typing. Or maybe type safety would have been even a better comparison? Shrug.
The key here is that there is no single definition of “strong” or “static” types. It is a spectrum. For example, Rust has algebraic data types, C++ does not. They are both static and strong in the broad sense, but algebraic data types mean that you can express more things at the type level.
It’s not like saying “I’m familiar with C, so I know everything that can be done with static types,” is accurate, I think is the point of this response.
It’s not like saying “I’m familiar with C, so I know everything that can be done with static types,” is accurate, I think is the point of this response.
I get what you’re saying and if I may suggest two things which have helped me pick up rust: writing a small but complete CLI application and watching live or recorded streams (for instance Ryan Levick or Jon Gjengset are formidable tutors)
That’s fair. However, even though I’m struggling to find my balance, I find learning it exciting. I think I’d rather learn as much of the ins and outs as I can and walk away with an informed opinion than bail early. At least I’m having fun, even if I keep falling off the surfboard.
I hope you stick with it. It is a relatively hard-to-master language, IMHO. Java, or an GC language doesn’t force you to think about ownership the way Rust puts that front and center. The ownership model has a dramatic effect on the kind of datastructures you can and should design. And it effects the way in which you use and share data across the entire application.
I believe the tradeoff is worth it, because you can catch certain classes of memory safety bugs at compile time, which can lurk undetected for years in other codebases. These are the kind of bugs that cause big problems later.
It is a relatively hard-to-master language, IMHO. … The ownership model has a dramatic effect on the kind of datastructures you can and should design. And it effects the way in which you use and share data across the entire application.
This might be the balance I’m struggling to find. A lot of my compiler errors are related to owneship and I keep having to come back to the docs to remind myself how it works.
I made a similar jump. It informed how I learned TypeScript and then made a whole lot more sense once I learned Haskell. Some tidbits that would’ve helped me follow.
Static typing, especially in something like TypeScript, can be thought of as an elaborate linter. At times it can be restrictive, but surprisingly often it’s leading you towards better code.
Strong static typing as in Rust’s case is like static typing but where your types are accurate to what will be there at runtime, so you’re probably going to need to do some validation or casting at your application boundaries. In Rust’s case also there’s no duck typing, you can’t just carry around data the compiler doesn’t know about at compile-time.
Traits are interfaces for shared abstractions across types. I really struggled with this until I saw typeclasses in Haskell. I think learning something relatively simple there like Functor can be instructive. It’s not entirely unlike extending prototypes in JavaScript, but here the compiler can use that information at compile-time to let you write functions which constrain input to support certain things, like for example an Eq constraint would mean “I’ll accept any input for which I can test equivalence”.
There is always duck typing if you want it, in any language. The best way in rust IMO is trait objects, but you can even get Java-style dynamic typing if you want (with possible a single use of unsafe in the library providing it).
I am trying to “memefully” learn Rust. The endgoal is to be competent enough to create a Tauri based GUI software.
For the “memefully” part the goal is to just enjoy the learning process as much as possible. Rust is a hard language, we all know that. So, I plan to write something in Python, then in Nim and then in Rust. Then write a blog about it. I plan to follow the Rust CLI book [0], however I want to do random projects. This weekend’s plan is to write a CLI that takes makes async get requests to a bunch of input URLs.
[0] https://rust-cli.github.io/book/index.html
Async causes a significant fraction of the difficulty of Rust. Nicholas Matsakis, long-time leader of Rust’s language design team, admitted last year that
I recommend considering using
ureqinstead, especially (but not only) if you still are learning Rust and you see Rust as hard.I concede that this depends on how large your “bunch of input URLs” is.
Thank you very much for the recommendation. Trying it out now. It looks easier than reqwest.
Or reqwest with the blocking feature flag
Which just launches a
current_threadTokio runtime, uses it to send a request, then drops it. As far as I know, creating a new Tokio runtime per request isn’t that expensive, but it’s definitely more overhead than ureq.I did not know that. Thanks!
For several years I thought that Rust was too hard and didn’t even attempt it seriously. Then last year I started the Advent of Code 2022 in Python (that I use professionally), and found this masterpiece from Amos: https://fasterthanli.me/series/advent-of-code-2022/part-1
It was super refreshing as it was showing me how close to Python basic Rust can be, but also with many additional features that make it amazing. I was able to do about the first 13 days in Rust thanks to that series (I was solving it before looking at the solution from Amos). I must say that it was an eye opener to me. I really recommend this series.
Fasterthanlime and No Boilerplate is the reason I am trying to learn Rust. Thank you very much. Checking the blog posts out.
Oh hey, I did something very similar to your batch URL requests recently! Here was what I came up with: https://github.com/adamhammes/rust_web_serial/blob/master/src/main.rs#L29-L48
That is nice. Thank you for sharing. Bookmarked. Now, I have to expand my feature list a bit more and throw JSON deserialization in the mix.
Is there a Rust equivalent to this? I wrote a horrible
build.rshack at a former job that caused a bunch of pain.This seems to be one way: https://lib.rs/crates/built
In addition to the sibling comment there is also https://lib.rs/crates/vergen
Armin also explains his thoughts in this discussion
Armin gets it. I re-read his blog post about Python 3 and he has a side mention of packing there too:
I learned Go by writing a book about it, but I ended up with a book listing all of the ways that Go was badly designed, which turned out not to be what the market wanted.
Where could one find that book?
I have been a heavy user of GitHub Actions over the last few years. Yaml configuration can be pretty rancid, but I still feel most of my challenges can be solved by hacking some bash/python/JS in (current jobs are using all 3 for configuration) and by using community packages. I often want to chuck the yaml in the trash and use a real language to configure everything. Why they didn’t go with a TypeScript file is beyond me.
My wish list contains GPU support and M1 runners (the latter we’re going to solve by my boss buying a mac mini). Some new integration tests I’m adding are going to be 14h per commit 🫣
I’ve also been using GitHub Actions a lot, in particular with ChristopherHX’s reimplementation of the runner in go. This makes it easy to use GitHub Actions to drive a large range of platforms and architectures that aren’t officially supported.
I am also a heavy user of github actions but am considering switching to sourcehut just because they let you SSH into the hosts if a run fails. It takes so long to develop solid CI pipelines on github because you have to commit, push, and wait for the job to be picked up, initialized, and run before you can see whether it fails. After a decade developing software I really do think iteration time is the strongest single predictor of how long something will take to implement.
SSH into a failing build is such a lifesaver. I use it all the time on sourcehut, and I can’t imagine why anyone would launch a CI platform that doesn’t have that feature. The feedback loop without it is awful.
By all means not a perfect solution, but https://github.com/nektos/act can help in these cases where you want to run the pipeline locally.
I would disagree here. Pre-commit hooks are fundamentally client-side validation. They don’t enforce anything, they are advisory
Instead, I suggest the following, for the purpose of enforcement:
(Now, the nebulous CI might actually be implemented as a git hook (https://www.stchris.net/tiny-ci-system.html), but that’s crucially server-side).
It also has the benefit the properties are enforced only when it matters (code integrated into the canonical branch), and can be broken otherwise, giving a nice transactional semantics.
Agreed. One repo I work on has what amounts to a “don’t commit directly to release” precommit hook, but it’s slow and I don’t even have perms to push to the release branch, so I habitually
git commit -nmto skip it. It’s not a very good validation if there’s a single-letter command line argument to skip it.It’s true that you can bypass your local hook, and I agree that “perfect” might be an incorrect choice of word, but;
Link to not rocket science rule in case someone doesn’t recognize the jargon.
You can use tests instead of running a linter. In the end, like the linter, you’d use them locally, remotely or both.
I’m actually a proponent of both (and maybe I should’ve included that in the article), at least for shorter tasks, because remote enforcement alone for tasks that take seconds encourage more context switching.
Uhu, that’s why I like to drive everything through tests. That way,
cargo test(or whatever is the equivalent in a language of choice) is the single thing that needs to be run either locally or remotely.Hey @matklad, I’m the author of the Tiny CI system post you linked to and I just wanted to say that it means a lot that you (whom I respect a lot for all your work and the way you explain things you’re working on) linked to it. It gives me a big moral boost towards writing more blog posts and also a follow-up on the one you mentioned.
Chain of authority! I remember learning about Tiny CI from @peterbourgon comment here on lobsters, which for me means a lot.
(Laughing to myself after noting https://www.stchris.net/gate-integration-tests.html as well, which links to the same post I’ve linked from https://matklad.github.io/2021/05/31/how-to-test.html#Make-Tests-Fast)
And I thought this is about finding a Session Border Controller …
I am very much interested in reading this as an article, if you ever get around to it. Watching a video not so much.
Just skipped through the video, but I think the gist is that it uses a custom
http.Clienttransport to simulate the server response. Alternatively this could have made use ofhttptest.Server, but this will actually do local network requests, or use something like github.com/dnaeon/go-vcr to pre-record server responses and serve them from a file in the test case.Please stop releasing TUI frameworks for Python. The language is way too slow and makes these tools a pain to use.
To whoever is starting to write a reply - no, it cannot be optimized. Python will always take a bit to start the program.
I find this comment very surprising.
The stopwatch app from the Textual tutorial starts running for me in less than a quarter of a second - it’s fast enough that I didn’t even notice the delay until I tried to eyeball-measure it just now.
The whole point of TUI apps is that you’re going to spend some time in them. Does a quarter of a second to show the initial screen really matter to anyone? That’s way faster than loading a web application in a browser tab.
Thinking about it, I don’t think I’ve ever used a TUI written in any language where the startup speed has bothered me.
Java/scala usually has a 1-3 second startup time, which is too long for me, but I agree – that’s the only one I can think of.
It depends. Maybe startup time doesn’t really matter for someone’s particular use case. While there will always be some baseline startup time from Python, there are cases where you can optimize it and possibly bring it down to a level you find acceptable.
At a job, I was tasked with figuring out and speeding up slow start of a Python program. Nobody knew why the bloody thing was taking so long to start. Part of it was network delays, of course, but part was Python. I did some profiling.
This little Python program was importing a library, and that library imported something called pkg_resources. Turns out that pkg_resources does a bunch of work at import-time (nooo!). After some digging, I found that pkg_resources was actually an optional dependency of the library we were using. It did a try … import … except: …, and could work without this dependency. After digging into the code (both ours and the library’s), I found that we didn’t need the facilities of pkg_resources at all.
We didn’t want to uninstall it. Distro packages depended on it, and it was possible that there were other programs on the system that might use it. So I hacked up a module importer for our program that raised ModuleNotFoundError whenever something tried to import pkg_resources.
I cut a nearly one-second start time down to an acceptable 300 milliseconds or so, and IIRC a fair portion of the 300 milliseconds was from SSH.
Know your dependencies (direct and indirect). Know what you’re calling (directly and indirectly) and when you’re calling it. Profile. And if your Python startup times are slow, look for import-time shenanigans.
Program startup speed is important for some applications but negligible compared to other aspects like usability, accessibility or ease of development, wouldn’t you agree?
Program startup speed and performance is an important part of usability. It’s bad for software users when the software they use is full of latency, or uses so many system resources it bogs down their entire computer.
Agreed, it’s part of usability. But it depends on the numbers. Saying “stop writing TUIs in Python” because of 200ms (out of which something can be shaved off with optimization) sounds extreme.
I completely agree with the unsuitability of Python for TUI / CLI projects! (Especially if these tools are short-lived in their execution.)
Long ago (but still using today) I’ve written a simple console editor that has ~3K lines of code (25 Python modules) which imports only 12 core (and built-in) Python modules (without any other dependencies) and mainly uses
curses.On any laptop I’ve tried it (even 10 years old) it starts fast enough. However recently I’ve bought an Android device and tried it under Termux. It’s slow as hell, taking more than a second to start… (Afterwards it’s OK-ish to use.)
What’s the issue? The Python VM is slow to bootstrap and load the code (in my case it’s already in
.pyoformat, all in azipwithzipapp). For example just calling (on my Lenovo T450)python2 -c Truetakes ~10ms meanwhilepython3.10 -c Truetakes ~14ms (python3.6used to take ~20ms). Just addingimport jsonadds another +10ms, meanwhileimport curses, argparse, subprocess, json(which is perhaps the minimal any current-day project requires) yields a ~40ms startup.With this in mind, this startup latency starts to pile-on and it has no solution in sight (except rewriting it in a compiled language).
Granted, even other languages have their issues, like for example Go, which is very eager in initializing any modules you have referenced, even though will never use, and thus easily adds to startup latency.
(I’ll not even touch on the deployment model, where
zipappis almost unused for deployment and https://github.com/indygreg/PyOxidizer is the only project out there trying to really make a difference…)Have you tried the framework?
Why would I? It only has buttons and checkboxes implemented. And according to comments in here is still taking 1/4 of a second to start on a modern CPU.
EDIT: In the demo video, the demo takes 34 frames to boot. At 60fps, that’s more than half a second.
I guess it will never be succesful then, like that slow-as-hell Slack thing /s
The popularity of a chat app - particularly one that most people use because it’s what their workplace standardizes on - is driven much more by network effects than by the quality of the standard client app. It is bad that the Slack client is slow, and this is made worse by the fact that there aren’t a whole lot of alternative clients for the Slack network that a person who is required to use Slack as part of their job can use instead of the official, slow, one.
I think that the problem with your assessment is the assumption that the users of this framework have the knowledge to use a different language or want to use a different language than python. Nobody is forcing you to use it and if folks are releasing tools using it, nobody is forcing you to use those. For those that want to add a TUI front end to a script they made, this seems like a good option.
I think Ink is overall a better TUI framework than this, and let’s face it, Python really is slow, JavaScript is much better.
Thankfully AWS App Runner seems like it might become a good alternative to Heroku free plan, it isn’t completely free but very cheap for most apps.
The biggest pain point for me will be that we have asked people applying for Junior positions to host their coding pre-assignment with Heroku (as it was free and easy to use) but now that really won’t be an option anymore so we will have to think of something else. Of course people have been free to host their pre-assignments however they wanted to before also but we don’t really require people to know AWS/Linux/etc. for our Junior positions, it has just been enough that people also show they can get their app running somehow. Any suggestions around this would very much be appreaciated.
Both render.com and fly.io have free tiers and are in the business of enabling “take my Dokerfile and run it”.
If you’re still on LastPass and looking for any excuse to switch (like I was last year): 1Password fixes every problem I had with LastPass (eg working form fill, even in Firefox) and gives me a few things I didn’t even know I wanted (eg an ssh agent). Just make the leap, it’s a total no brainer.
I second 1Password, I’ve used it personally for years. Just started using LastPass at work recently and I was surprised at how the whole thing felt very… unpolished? 1Password feels like a real professional app, LastPass feels like some proof of concept that was thrown together as a TUI in a weekend and then had a UI bolted on the front.
You hit the nail on the head there I think, a company I worked at used LastPass enterprise and it honestly felt like they never hired any designers and just told some devs to build an MVP and that stuck. It just feels like a side project. When I first used 1p I was amazed at how nice and compact the UI is and switched my personal solution from Dashlane. I still find that Dashlane’s form filling ability and other features of the browser extension are still miles ahead of 1p though. But I hope they’ll catch up soon.
Unfortunately they’re dipping their toes into cryptocurrency land: https://1password.community/discussion/127467/please-reconsider-your-plans-on-cryptocurrency-and-related-subjects
I don’t know as this is a particularly terrible thing - it’s not like they haven’t done partnerships with other commercial services for storing special data before (namely, their Privacy.com [1] and Fastmail [2] integrations) or had integrations to make some secrets have special functionality (e.g. SSH keys [3]). I think having an automatic way to store cryptocurrency secrets in a password manager is a pretty neat feature, even if I might not necessarily think cryptocurrency is all that good or important. If they want to keep going in this direction, I would like to see them integrate with a more open wallet, and probably tone down their NFT-related marketing about it.
[1] https://support.1password.com/privacy-cards/
[2] https://1password.com/fastmail/
[3] https://developer.1password.com/docs/ssh/
What are these masked emails? Neither the README nor the attached animation give any hint what are those and why are they useful.
It’s a fastmail thing: “A Masked Email address is a unique, automatically generated email address that can be used in place of your real email address. Masked Email addresses are formatted as random words plus a random number, e.g. dog.food3495@domain.tld.”
if you use mailbox.org with your own domain, it’s trivial to configure it to accept mails to any @yourdomain email address. no need to use an external tool…
I don’t use mailbox.org, but that’s exactly what I do.
ya I imagine this isn’t a revolutionary idea, but for some reason it is for fastmail
Fastmail is able to do that as well.
Apple has a version too.
Someone emailed a nasty note to the reporters at my website from an Apple masked address. I’m not sure how to report abuse with these things. There should be a prominent webpage that lists the address to forward abuse to.
I assume this code only works with Fastmail, it sure appears like it. I don’t use Apple’s email addresses, so I’ve never looked, but good to know!
I have a domain with a catch-all address, so I can just invent them for 1 off things, no need to tell my email provider ahead of time.
I assume for abuse, you could always email postmaster@ and reference the masked email address. I imagine it will get routed to /dev/null like every other email problem I’ve ever sent in the last decade, but it might make you feel better! :)
I’ve gotten a response from postmaster before! Granted, it was for a ISP in Pennsylvania that had constant delivery problems, but they did at least write back to me. :-)
Congrats on reaching a human!
I used to be able to reach humans via postmaster@, but I’ve never talked to another human in the past decade emailing that or webmaster@ (which usually just bounces). Seems nobody respects RFC-2142 anymore.
Neat, thanks! I’m using fastmail, but didn’t know this is possible. I’ve been using this: https://www.fastmail.help/hc/en-us/articles/360060591053 but seems that masked emails are more versatile — with subdomain addressing anyone can send mails to any random string, losing ability to track who is spamming me.
I’m also not a lawyer, but if we look at the popular MIT License, for instance, it says:
Isn’t this part definitely missing?
The key word in the MIT license snippet that you’ve quoted is ‘substantial’. This matters because not everything in a codebase is subject to copyright. For example, if you copied
for (int i=0 ; i<100 ; i++) {from some MIT-licensed source file, the attribution requirement wouldn’t hold because that line is not sufficiently original to qualify for copyright (copyright covers original works, which is quite a subjective thing, but some things are obviously on the not-qualified line). I am not a lawyer, and so I can’t comment on whether the things generated by Copilot meet this bar (from what I’ve read, being a lawyer would still not let me confidently make such a claim, there have been a number of expensive court cases to try to make this determination over shortish snippets of code).If you type in some well-known class like “FizzBuzz” then it may paste the entire class that was grabbed from a specific piece of code somewhere. I have seen videos of people testing CoPilot where it pastes enough to fill a page. Given that possibility, I think the possibility of violating an MIT license is very likely to happen to many people.
Repo here and all links:
no thanks
I’m using scribe.rip as an alternative frontend to medium. Just replace it in the URL. (Fully agreeing that one shouldn’t have to do this)
Sensible suggestions! In terms of 12factor apps I feel like (something like) https://github.com/joke2k/django-environ is also essential nowadays.
It’s a great suggestion! I haven’t used django-environ for a while on side projects, but I should look into it again (I remember using it on a commercial project a while ago, and thinking that it was great).
Just looking at some recent code of mine, I’ve ended up using this pattern a few times which I think achieves a similar result:
The added support for cache / email “url” formats in django-environ and the ability to replace dj-database-url as a dependency definitely makes me want to use this on my next project.
Consider using something like beets to tag your music.
Beets is fantastic. I would like to find “beets for photos”
I hope whoever makes that calls it “pheets”.
If you’re going to pitch this as a Rust thing, using the go tool dbmate is definitely cheating
I think dbmate is great but sqlx already provides migrations so there’s at least an explanation missing as to why dbmate is better suited.
Ok I’ll bite. Why not celery? You can make it db backed, and you can set up tasks from the admin interface.
I thought Celery can use the Django ORM as a result backend but not as a broker? So you might still need Redis or RabbitMQ?
I think one could use the in-memory broker to get the same effect.
At the cost of no persistence in case the app crashes, sure.
I agree with that, “same effect” refers to the solution presented in the DIY article.
Someone might want to avoid setting up new dependencies or a new service. But my motivation was only to see if I could get a good enough result using just Django out of curiosity. For more serious/complex needs surely something like celery is better.
I appreciate this post, it shows that it doesn’t take much to do some simple task queue.
I have seen this pattern quite often but I haven’t seen a popular Django app emerge and establish itself as “good enough until you need Celery”. I wonder why that is?
For me it looks too small in scope to be a separate Django app. I hope this post will help people copy-paste and build their own solutions when they find it by searching for django+tasks+jobs
Thanks for an interesting post. I’ve been using django-q but this looks nicer in several ways. I think I’m going to try your approach on my next tool instead of reaching for django-q right away.
I feel like django-q is trying very hard to occupy that space and has done so well in my limited usage so far.
I enjoy django-q because it lets you recycle the task process (when you set the number of tasks per child to 1, the monitor process restarts the child process once it finishes a task) so that you can cleanup things like in-memory sqlite and any resources that belong to the process automatically.
I am not too fond of of gigantic web frameworks, but the advantage is, of course getting X for free. In this case backgrounds jobs.
I never really understood the fascination for celery. And I suspect much of its popularity builds up on web developers not knowing to write a system service. But I digress. Adding celery would mean running celery, set up redia or other backend, and configure the connection between all pieces. That is a lot of possible failure failure points than can be avoided.
That’s assuming you don’t need an actual distributed queue. If you don’t and only want to run an odd background job, that’s great. But once you need to split the queued tasks among tens of servers, handle retries, deal with contention issues, distribute tasks to different groups, and make sure that your web resources are not affected by background processing… you want celery or something similar. But the idea that celery is popular because people can’t write services is a bit disrespectful.
Which is the case of most people using celery.
But your assertion is too simplified. A distributed job queue is trivial to implement using redis or any other network accessible queue. I cluding a custom built one, which can be built in minutes.
Seems like if you run into this you might want a merge queue and as I see Github even provides one: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/using-a-merge-queue
Ooh nice this looks like it may be fairly new as I’ve not seen that before. I know GitLab has had it for some time (called Merge Trains) and is convenient for this
I’ve been learning Rust the past couple weeks strictly for hobbyist reasons. I’m coming from the dynamic typing world of Python and JavaScript. While I’m familiar with the strongly typed paradigms of C and Java, I figured spending my time in Rust would be more beneficial.
There are a lot of individual things I like in Rust, but programming in it in general feels like I’m trying to learn a new sport and I can’t quite seems to get my sense of balance. E.G., “I’ve been out on the water a hundred times now, so why can’t I comfortably stand on the surfboard now?” type of feeling. If that makes sense.
I’ll keep plowing forward with it though. I’ve got The Rust Programming Language book from No Starch Press, and I have been spending a lot of my time in the Rust Book online. I’ll find my balance, I just wish I could find it sooner than later.
C is barely type at all, and Java is a weird hybrid of a poor type system and just being dynamically typed anyway. I wouldn’t call either “strongly typed”.
Maybe I should have chosen “static” instead, as I also referred to “dynamic” typing rather than “weak” typing. Or maybe type safety would have been even a better comparison? Shrug.
I wouldn’t worry too much, internet fights about what counts as “static” or “strong” or “typed” are generally not worth your time.
The key here is that there is no single definition of “strong” or “static” types. It is a spectrum. For example, Rust has algebraic data types, C++ does not. They are both static and strong in the broad sense, but algebraic data types mean that you can express more things at the type level.
It’s not like saying “I’m familiar with C, so I know everything that can be done with static types,” is accurate, I think is the point of this response.
Indeed.
I get what you’re saying and if I may suggest two things which have helped me pick up rust: writing a small but complete CLI application and watching live or recorded streams (for instance Ryan Levick or Jon Gjengset are formidable tutors)
Maybe you should listen to your gut. Rust is a relatively new language, it is not clear it will stand the test of time.
That’s fair. However, even though I’m struggling to find my balance, I find learning it exciting. I think I’d rather learn as much of the ins and outs as I can and walk away with an informed opinion than bail early. At least I’m having fun, even if I keep falling off the surfboard.
I hope you stick with it. It is a relatively hard-to-master language, IMHO. Java, or an GC language doesn’t force you to think about ownership the way Rust puts that front and center. The ownership model has a dramatic effect on the kind of datastructures you can and should design. And it effects the way in which you use and share data across the entire application.
I believe the tradeoff is worth it, because you can catch certain classes of memory safety bugs at compile time, which can lurk undetected for years in other codebases. These are the kind of bugs that cause big problems later.
I plan on it.
This might be the balance I’m struggling to find. A lot of my compiler errors are related to owneship and I keep having to come back to the docs to remind myself how it works.
I made a similar jump. It informed how I learned TypeScript and then made a whole lot more sense once I learned Haskell. Some tidbits that would’ve helped me follow.
Static typing, especially in something like TypeScript, can be thought of as an elaborate linter. At times it can be restrictive, but surprisingly often it’s leading you towards better code.
Strong static typing as in Rust’s case is like static typing but where your types are accurate to what will be there at runtime, so you’re probably going to need to do some validation or casting at your application boundaries. In Rust’s case also there’s no duck typing, you can’t just carry around data the compiler doesn’t know about at compile-time.
Traits are interfaces for shared abstractions across types. I really struggled with this until I saw typeclasses in Haskell. I think learning something relatively simple there like
Functorcan be instructive. It’s not entirely unlike extending prototypes in JavaScript, but here the compiler can use that information at compile-time to let you write functions which constrain input to support certain things, like for example anEqconstraint would mean “I’ll accept any input for which I can test equivalence”.There is always duck typing if you want it, in any language. The best way in rust IMO is trait objects, but you can even get Java-style dynamic typing if you want (with possible a single use of unsafe in the library providing it).