Putting “python” and “performance” in the same sentence just feels wrong. There are too many limitations in python runtime, and too many footguns, so it makes the whole optimization process risky and ineffective. If you have performance problems, the best approach is to rewrite the hot code into language supporting proper concurrency primitives like green threads or transactional memory.
Python extensions for hotspots (a common use case, it’s true) are typically written in C, C++, Fortran, or more recently Rust. None of these have green threads or transactional memory built-in.
In the context of the article, the question is how to parallelize an existing Python program, rather than how to optimize that program’s single-threaded performance. From that perspective, your answer isn’t complete, because neither green threads nor (software) transactional memory promise to parallelize.
I have personally beaten a Java program (vanilla Minecraft server) with a Python program on benchmarks; my primary technique was to choose better algorithms, and my secondary technique was to embrace PyPy. Choice of language is not as important as choice of runtime, and the article is reinforcing this point by treating the POSIX process/thread model as an exposed detail of the runtime. (For what it’s worth, in that case, I used threads for filesystem access and subprocesses for multicore parallelism; in particular, I used Ampoule.)
A self-hosted, open-source replacement for Discord. The pieces are all there: VOIP is a solved problem thanks to Mumble, text chat is an XMPP client, etc. All it really needs is some duct tape, and the barest of UI polish.
Photos and LightRoom both fail on the “let me put thousands of photos on a remote share and let any number of laptops/phones/tablets poke around, tagging, organizing, making smart albums, etc.”. I don’t care about RAW, I don’t care about anything more than basic editing, and would like to keep all the face recognition/etc. that the basic Photos app can do. If I could have it generate a “feed” or something I could point a smart TV or smart picture frame to and have it cycle through a smart album or something that’d be amazing.
This seems insane that Photos can’t do this. I tried something from Adobe too a while ago (can’t remember the name now) but it wasn’t up to the task either.
Did you ever look into digiKam? I use it a lot, from what you’re describing, it looks like it matches a lot of your “requirements”.
Some things that might fit your bill:
it can do face recognition for you (and learn and help you automate this stuff)
it doesn’t “import” your pics anywhere, it just works on wherever they are.
it has its database, but:
you can write most of the changes in the sidecar file, so that other programs can also use it.
the db is sqlite (by default) so you can even do something with tools like datasette e.g. query what’s the most frequent time of day or location your photos are taken, or - if you manage RAWs there as well - what focal lengths do you have the most or similar.
it can do some basic editing for you(for proper edits I still use external software)
you should be able to point it to a smart TV, I think, and even if not, there are many cool plugins for such stuff (as well as for e.g. backups to box, google, posts to twitter, facebook, regular html galleries and a lot of others.
The part that I like the most about it is that it is focused on the library functions: searching, tagging, filtering etc. (I also combine it with rapid photo downloader to get pics off my cameras, and darktable and/or gimp to do actual editing but that’s just me).
After using digiKam for a few years I only use it to import images from the camera. Persistent issues:
The DB is corrupted once in a while, and the application crashes on the regular.
The UI is 1990s. Why is the settings button not on the home screen? Why is there a splash screen (by default, you can turn this off but dammit, who actually wants that?)? Why is there no way to go from “WTF does this do?” to the actual documentation? Like, unless I spend every waking second editing photos, why would I know which “quality” setting to choose?
Darktable uses sidecar files, and although they refuse to believe the filesystem is important (so I have to keep re-importing the entire catalogue and run a service to clean out missing files nightly), at least it isn’t actually corrupted. And it’s much faster than digiKam, making it barely usable on a reasonably modern desktop.
Wow, we really have different experiences there. I don’t think I’ve ever seen the database get corrupt, for example, and with darktable I had that happen.
And the UI, some of it might seem “1990s”, but for me it’s just fine, and far, far better then Darktable for managing a library (not editing, I still use dt for that).
For importing, did you ever consider rapid photo downloader? For me this does noticeably better job then the other two.
I think Photoprism should be able to handle thousands of photos. However I’ve tried to import around 100 000 photos and it was almost impossible to manage that much in it.
Literally just set up my own PhotoPrism instance a week ago, and it currently has some 55k images and videos in it.
There are a few things that could be improved, e.g. allowing renaming of persons tagged through face detection, easier labeling of multiple images and better face detection for kids. All of these improvements already have issues on GitHub BTW.
One thing it doesn’t support is timeshifting of images with bad EXIF tags. For some reason I had a whole lot of photos that had GPS data but a wrong EXIF create date. Luckily exiftool was able to fix all bad timestamps. Here’s a oneliner that can fix the issue that I had:
exiftool -progress -r \
-if '$createdate eq "2002:12:08 12:00:00"' \
-if '$gpsdatetime' '-createdate<$gpsdatetime' \
-overwrite_original_in_place . |&
tee -a log.txt
Yea, I have a similar dream. For me, it would be an open source project with a single-binary self-hosting approach, which embeds a high performance web server for an interface and API similar to Google Photos. The storage is “just files” so the binary + photos can be backed up easily. Metadata stored in SQLite alongside the photos on the filesystem. Cool tricks like face recognition and autosizing, autocropping, etc. outsourced to other binaries with a scheme for storing “derived” images non-destructively, similar to what Picasa used to do. And then sync tooling that puts the same thing in a self-hosted cloud, with a VM + cache for 30 days of photos, with rest backed by Amazon S3, Google GCS, or Backblaze B2. Then maybe some tooling that can auto install this on a VM and secure it via Google Login, while supporting private photo sharing at stable URLs similar to Google Photos.
This would be a big project because, to be good, there’d also have to be great open source mobile apps on iOS and Android.
Some friends of mine prototyped the start of this project using redbean (Lua single binary web server) and got as far as the basics of photo serving from disk and having access to SQLite schema for metadata. It’s totally doable.
For the time being, I have been using Android’s support for SD card readers to sync all my photos from phones, DSLR, and mirrorless cams into Google Photos, and then keeping RAW storage separate on my Mac Mini and backed up using standard 3-2-1 backup approach. But, it’s cumbersome (requires manual processes), lossy (I lose backup of edits in GPhotos), proprietary, and I’m afraid for Google’s track record with longevity of services. It also saddens me that Google Photos doesn’t give me any way to post public photo albums at my domain (https://amontalenti.com) with stable URLs that I know I can make last forever, regardless of photo serving tech.
My theory for why something like this never got built is that Apple Photos and Google Photos are just so darn convenient, and non-phone photography relatively rare among consumers these days, so it just never fell into the indieweb and f/oss sweet spot of itch worth scratching. But I still have the itch and have for a long time.
It also seems like a lot of the backend functionality could be done by somehow modularizing the GNOME Shotwell codebase. For example to handle image metadata, color correction, cropping, and so forth.
When bazel works it’s a bliss and development cycle is fast. Caching is nice too.
However, when you need to make changes to your pipeline, and your pipeline breaks, it’s the worst kind of hell. Bazel uses starlark - so editor support is poor. You can’t really script in starlark, you’re forced to implement interop with bash/python/your favourite scripting language via cmd arguments or inputs/outputs. So when your script breaks you’re left with debugging of multiple layers of abstractions. I had to deal with starlark, layered in top of go extension on top of python scripts executing bash scripts on the bottom. Not fun. And then you have gazelle or whatever BUILD files generation machinery. Because bazel is meant to be declarative, you have to use those if you need to have bigger control over your project. So on our Haskell project we’re using gazelle to configure bazel to call Haskell Stack to generate cabal files, and in the end generate BUILD files from cabal files to use Haskell rules to run ghc. And this whole cake is very brittle. In the end the build takes a bit shorter than with just Haskell toolchain, but there’s so much overhead when changing dependencies, and it’s hard to avoid. And then we have simple bash targets to start database or do some automation, and for some reason bazel quite often redownloads and recompiles whole go toolchain just to run a bash script. That last bit is probably a misconfiguration on our side, but debugging that isn’t that easy in bazel, especially when it reinvents a bunch of abstractions.
My conclusion is pretty much the same as in the article - YMMV. Bazel solves a lot of problems, but the learning curve is steep and you are opening a new can of worms.
I have a bunch of issues with Bazel, but Starlark isn’t one of them. It’s basically a cut-down version of Python, so any Python support can help. As far as it not being Turing complete, that’s on purpose to make it decidable. By making it primitive recursive, Bazel can actually analyse the work that needs to be done. If it were more general, it couldn’t. It’s a feature, not a bug. The only thing that irks me is that sometimes sets would be nice to have.
I can agree with them, constantly needing to annotate code with types in Dhall when I just want 50 lines of template code to make my json outputs more reusable is annoying. That’s why I’m more keen towards Jsonnet.
yoctopuce seems to have some fine products, but holy hell are they pricey - and the calibration kits (coming from senseair) aren’t even available without contacting sales, so I guess as a private person you won’t normally get them
It looks like you have 2 options for calibration. You can either move the sensor outside and run the calibration procedure. That assumes that the outside air CO2 concentration is 400 ppm and resets the sensor accordingly. Or, you can buy one fo thoe calibration kits . The kits usually involve using nitrogen gas to eliminate all CO2 from the sensor, leading to it calibrating to a “true zero” value instead of a 400 ppm value.
The outside method works most of the case but it might be a little bit off in case the CO2 ppm concentration is higher in your area for whatever reason.
My vote goes to 1Password, for ease of use, built in security model (client side encryption), versatility in handling all kinds of data (notes, credit cards, etc) and reliability of the plugins to work with all websites and apps. Other password management apps that I’ve tried have frequently had problems with some websites. Sometimes 1Password still has edge cases where e.g. 2FA is not automatically filled in and you have to copy paste it manually. But I haven’t seen a better app yet.
My work used LastPass and I couldn’t have created a worst UI if I’d tried. There was no easy way to generate a new password. It took three clicks in non-obvious places to get to it.
I used LastPass for several years before switching to 1Password a year ago. Wish I had switched earlier. LastPass’s UI design needs a lot of work and over time actually got worse with various annoying small bugs.
Hard no to LastPass. I used it years ago, audited it one evening on a lark, found a few vulns, reported them, a couple got fixed, a couple got me told to fuck off.
When I previously used LastPass, there were some weird differences between the browser version and the desktop version - there were some things that each of them couldn’t do.
One oddity worth noting - I don’t use the desktop app with 1Password. I’ve found their browser extension, 1PasswordX, to be more stable (it also has the benefit of working on Linux).
I believe with the addition of HaveIBeenPwned integration on the LastPass security dashboard, they’re pretty much similar feature wise (though maybe 1Password can store 2FA tokens). I’ve used 1Password because it felt way less clunky than LastPass and it doesn’t require me to install a random binary on my Linux machines in order to access my passwords.
I switched to 1Password from LastPass a couple years ago and haven’t looked back.
LastPass got unusably slow for me after I had more than a few hundred entries in it. I don’t know if they’ve fixed their performance problems by now, but I can’t think of anything I miss.
Long time 1Password user here. It’s by far the best tool I’ve ever used. And I believe it goes beyond the application itself, as the support team is also great. Given a matter as sensible as all my credentials to login into several different services, having good support is mandatory IMO.
1Password here too. Excuse the cliché, but it just works. The cost is minimal for me — $4/mo, I think.
I’ve been slowly moving some 2FA to it, but it seems dependent on 1Password itself detecting that the site supports it vs. something like Authy where I can add any website or app to it.
I just switched to 1Password after 5-10 years on Lastpass. There’s some quirks, it’s not perfect, I generally prefer it to Lastpass.
The only thing Lastpass truly does better is signup form detection. Specifically I like the model Lastpass uses of detecting the form submission, 1Password wants you to add the password prior to signing up, which gets messy if you fail signing up for some reason.
1Password wants you to add the password prior to signing up, which gets messy if you fail signing up for some reason.
Oh yeah, this is a constant frustration of mine. ALso, whenever I opt to save thep assword, I seem to have a solid 4-5 seconds of waiting before I can do this. This seems to be 1Password X, FWIW. Back in the good old days of 1Password 6 or so when vaults were just local files, the 1P browser extension seemed to save forms after submission.
I’ve been able to get my whole family onto a secure password manager by consolidating on 1Password. I don’t think I would have been successful with any of the other options I’ve found.
Putting “python” and “performance” in the same sentence just feels wrong. There are too many limitations in python runtime, and too many footguns, so it makes the whole optimization process risky and ineffective. If you have performance problems, the best approach is to rewrite the hot code into language supporting proper concurrency primitives like green threads or transactional memory.
Python extensions for hotspots (a common use case, it’s true) are typically written in C, C++, Fortran, or more recently Rust. None of these have green threads or transactional memory built-in.
In the context of the article, the question is how to parallelize an existing Python program, rather than how to optimize that program’s single-threaded performance. From that perspective, your answer isn’t complete, because neither green threads nor (software) transactional memory promise to parallelize.
I have personally beaten a Java program (vanilla Minecraft server) with a Python program on benchmarks; my primary technique was to choose better algorithms, and my secondary technique was to embrace PyPy. Choice of language is not as important as choice of runtime, and the article is reinforcing this point by treating the POSIX process/thread model as an exposed detail of the runtime. (For what it’s worth, in that case, I used threads for filesystem access and subprocesses for multicore parallelism; in particular, I used Ampoule.)
Can’t wait for JS FFI
A self-hosted, open-source replacement for Discord. The pieces are all there: VOIP is a solved problem thanks to Mumble, text chat is an XMPP client, etc. All it really needs is some duct tape, and the barest of UI polish.
How about https://revolt.chat
matrix / element is getting there
Decent photo organizing software.
Photos and LightRoom both fail on the “let me put thousands of photos on a remote share and let any number of laptops/phones/tablets poke around, tagging, organizing, making smart albums, etc.”. I don’t care about RAW, I don’t care about anything more than basic editing, and would like to keep all the face recognition/etc. that the basic Photos app can do. If I could have it generate a “feed” or something I could point a smart TV or smart picture frame to and have it cycle through a smart album or something that’d be amazing.
This seems insane that Photos can’t do this. I tried something from Adobe too a while ago (can’t remember the name now) but it wasn’t up to the task either.
Did you ever look into digiKam? I use it a lot, from what you’re describing, it looks like it matches a lot of your “requirements”.
Some things that might fit your bill:
The part that I like the most about it is that it is focused on the library functions: searching, tagging, filtering etc. (I also combine it with rapid photo downloader to get pics off my cameras, and darktable and/or gimp to do actual editing but that’s just me).
After using digiKam for a few years I only use it to import images from the camera. Persistent issues:
Darktable uses sidecar files, and although they refuse to believe the filesystem is important (so I have to keep re-importing the entire catalogue and run a service to clean out missing files nightly), at least it isn’t actually corrupted. And it’s much faster than digiKam, making it barely usable on a reasonably modern desktop.
Wow, we really have different experiences there. I don’t think I’ve ever seen the database get corrupt, for example, and with darktable I had that happen.
And the UI, some of it might seem “1990s”, but for me it’s just fine, and far, far better then Darktable for managing a library (not editing, I still use dt for that).
For importing, did you ever consider rapid photo downloader? For me this does noticeably better job then the other two.
I think Photoprism should be able to handle thousands of photos. However I’ve tried to import around 100 000 photos and it was almost impossible to manage that much in it.
So far I’ve settled with plex for building playlist + digikam for organizing collection + my own photoframe software: https://github.com/carbolymer/plex-photo-frame
oh wow this looks very promising… thank you for the recommendation!
Literally just set up my own PhotoPrism instance a week ago, and it currently has some 55k images and videos in it.
There are a few things that could be improved, e.g. allowing renaming of persons tagged through face detection, easier labeling of multiple images and better face detection for kids. All of these improvements already have issues on GitHub BTW.
One thing it doesn’t support is timeshifting of images with bad EXIF tags. For some reason I had a whole lot of photos that had GPS data but a wrong EXIF create date. Luckily
exiftoolwas able to fix all bad timestamps. Here’s a oneliner that can fix the issue that I had:All in all I’m pretty satisfied with PhotoPrism!
Yea, I have a similar dream. For me, it would be an open source project with a single-binary self-hosting approach, which embeds a high performance web server for an interface and API similar to Google Photos. The storage is “just files” so the binary + photos can be backed up easily. Metadata stored in SQLite alongside the photos on the filesystem. Cool tricks like face recognition and autosizing, autocropping, etc. outsourced to other binaries with a scheme for storing “derived” images non-destructively, similar to what Picasa used to do. And then sync tooling that puts the same thing in a self-hosted cloud, with a VM + cache for 30 days of photos, with rest backed by Amazon S3, Google GCS, or Backblaze B2. Then maybe some tooling that can auto install this on a VM and secure it via Google Login, while supporting private photo sharing at stable URLs similar to Google Photos.
This would be a big project because, to be good, there’d also have to be great open source mobile apps on iOS and Android.
Some friends of mine prototyped the start of this project using redbean (Lua single binary web server) and got as far as the basics of photo serving from disk and having access to SQLite schema for metadata. It’s totally doable.
For the time being, I have been using Android’s support for SD card readers to sync all my photos from phones, DSLR, and mirrorless cams into Google Photos, and then keeping RAW storage separate on my Mac Mini and backed up using standard 3-2-1 backup approach. But, it’s cumbersome (requires manual processes), lossy (I lose backup of edits in GPhotos), proprietary, and I’m afraid for Google’s track record with longevity of services. It also saddens me that Google Photos doesn’t give me any way to post public photo albums at my domain (https://amontalenti.com) with stable URLs that I know I can make last forever, regardless of photo serving tech.
My theory for why something like this never got built is that Apple Photos and Google Photos are just so darn convenient, and non-phone photography relatively rare among consumers these days, so it just never fell into the indieweb and f/oss sweet spot of itch worth scratching. But I still have the itch and have for a long time.
It also seems like a lot of the backend functionality could be done by somehow modularizing the GNOME Shotwell codebase. For example to handle image metadata, color correction, cropping, and so forth.
I know it’s not software, but in case you didn’t know about this, if you have an Apple TV you can show slideshows on your TV.
When bazel works it’s a bliss and development cycle is fast. Caching is nice too.
However, when you need to make changes to your pipeline, and your pipeline breaks, it’s the worst kind of hell. Bazel uses starlark - so editor support is poor. You can’t really script in starlark, you’re forced to implement interop with bash/python/your favourite scripting language via cmd arguments or inputs/outputs. So when your script breaks you’re left with debugging of multiple layers of abstractions. I had to deal with starlark, layered in top of go extension on top of python scripts executing bash scripts on the bottom. Not fun. And then you have gazelle or whatever BUILD files generation machinery. Because bazel is meant to be declarative, you have to use those if you need to have bigger control over your project. So on our Haskell project we’re using gazelle to configure bazel to call Haskell Stack to generate cabal files, and in the end generate BUILD files from cabal files to use Haskell rules to run ghc. And this whole cake is very brittle. In the end the build takes a bit shorter than with just Haskell toolchain, but there’s so much overhead when changing dependencies, and it’s hard to avoid. And then we have simple bash targets to start database or do some automation, and for some reason bazel quite often redownloads and recompiles whole go toolchain just to run a bash script. That last bit is probably a misconfiguration on our side, but debugging that isn’t that easy in bazel, especially when it reinvents a bunch of abstractions.
My conclusion is pretty much the same as in the article - YMMV. Bazel solves a lot of problems, but the learning curve is steep and you are opening a new can of worms.
I have a bunch of issues with Bazel, but Starlark isn’t one of them. It’s basically a cut-down version of Python, so any Python support can help. As far as it not being Turing complete, that’s on purpose to make it decidable. By making it primitive recursive, Bazel can actually analyse the work that needs to be done. If it were more general, it couldn’t. It’s a feature, not a bug. The only thing that irks me is that sometimes sets would be nice to have.
I’m a little sad seeing 0/0 mentions of Dhall. After being ‘forced’ to use it with Spago, it became my prefered typed configuration language.
Check out the bottom of readme: https://github.com/tweag/nickel/
I can agree with them, constantly needing to annotate code with types in Dhall when I just want 50 lines of template code to make my json outputs more reusable is annoying. That’s why I’m more keen towards Jsonnet.
You didn’t mention it in your post, but in general CO2 sensors should be calibrated or you will be getting reading that are off. https://www.yoctopuce.com/EN/article/how-to-calibrate-a-co2-sensor
yoctopuce seems to have some fine products, but holy hell are they pricey - and the calibration kits (coming from senseair) aren’t even available without contacting sales, so I guess as a private person you won’t normally get them
It looks like you have 2 options for calibration. You can either move the sensor outside and run the calibration procedure. That assumes that the outside air CO2 concentration is 400 ppm and resets the sensor accordingly. Or, you can buy one fo thoe calibration kits . The kits usually involve using nitrogen gas to eliminate all CO2 from the sensor, leading to it calibrating to a “true zero” value instead of a 400 ppm value.
The outside method works most of the case but it might be a little bit off in case the CO2 ppm concentration is higher in your area for whatever reason.
My vote goes to 1Password, for ease of use, built in security model (client side encryption), versatility in handling all kinds of data (notes, credit cards, etc) and reliability of the plugins to work with all websites and apps. Other password management apps that I’ve tried have frequently had problems with some websites. Sometimes 1Password still has edge cases where e.g. 2FA is not automatically filled in and you have to copy paste it manually. But I haven’t seen a better app yet.
Yeah, me too. I ended up at 1Password after trying a lot of both offline and online systems.
Have you had a chance to compare it with LastPass?
My work used LastPass and I couldn’t have created a worst UI if I’d tried. There was no easy way to generate a new password. It took three clicks in non-obvious places to get to it.
I used LastPass for several years before switching to 1Password a year ago. Wish I had switched earlier. LastPass’s UI design needs a lot of work and over time actually got worse with various annoying small bugs.
Hard no to LastPass. I used it years ago, audited it one evening on a lark, found a few vulns, reported them, a couple got fixed, a couple got me told to fuck off.
And also, LastPass: Security Issues
When I previously used LastPass, there were some weird differences between the browser version and the desktop version - there were some things that each of them couldn’t do.
One oddity worth noting - I don’t use the desktop app with 1Password. I’ve found their browser extension, 1PasswordX, to be more stable (it also has the benefit of working on Linux).
I believe with the addition of HaveIBeenPwned integration on the LastPass security dashboard, they’re pretty much similar feature wise (though maybe 1Password can store 2FA tokens). I’ve used 1Password because it felt way less clunky than LastPass and it doesn’t require me to install a random binary on my Linux machines in order to access my passwords.
I switched to 1Password from LastPass a couple years ago and haven’t looked back.
LastPass got unusably slow for me after I had more than a few hundred entries in it. I don’t know if they’ve fixed their performance problems by now, but I can’t think of anything I miss.
Long time 1Password user here. It’s by far the best tool I’ve ever used. And I believe it goes beyond the application itself, as the support team is also great. Given a matter as sensible as all my credentials to login into several different services, having good support is mandatory IMO.
1Password here too. Excuse the cliché, but it just works. The cost is minimal for me — $4/mo, I think.
I’ve been slowly moving some 2FA to it, but it seems dependent on 1Password itself detecting that the site supports it vs. something like Authy where I can add any website or app to it.
I just switched to 1Password after 5-10 years on Lastpass. There’s some quirks, it’s not perfect, I generally prefer it to Lastpass.
The only thing Lastpass truly does better is signup form detection. Specifically I like the model Lastpass uses of detecting the form submission, 1Password wants you to add the password prior to signing up, which gets messy if you fail signing up for some reason.
Oh yeah, this is a constant frustration of mine. ALso, whenever I opt to save thep assword, I seem to have a solid 4-5 seconds of waiting before I can do this. This seems to be 1Password X, FWIW. Back in the good old days of 1Password 6 or so when vaults were just local files, the 1P browser extension seemed to save forms after submission.
I’ve been able to get my whole family onto a secure password manager by consolidating on 1Password. I don’t think I would have been successful with any of the other options I’ve found.