Threads for stavros

  1. 1

    One example is cryptography, by moving to rust implementation, the build time is now forever and takes gigabytes of disk.

    1. 7

      Like basically any package that involves compiled extensions, they provide pre-built binary .whl packages, so pip install cryptography should not involve having to compile anything on the target machine. Any extra time/space required by the compilation is thus incurred only by the people who build the packages.

      1. 2

        Unfortunately there are no ARM packages, so having to install anything that uses cryptography on a Raspberry Pi takes… well, I don’t know how long it takes, I only started it two months ago.

        1. 3

          There are aarch64 packages on PyPI: https://pypi.org/project/cryptography/#files

          One thing to try is upgrading pip before you install cryptography, old versions of pip won’t know about manylinux2014 for example.

          1. 1

            Ahh, I will try that next time, thank you! Just have been that.

        2. 1

          That’s exactly what I did to upgrade the py3-cryptography in alpine.

          1. 1

            For quite some time all Python .whl packages with compiled extensions were compiled with glibc (due to being the easiest way to define a lowest-common-denominator “Linux” compilation target), which meant they were not compatible with musl-based distros. There is now a platform tag on PyPI for packages built against musl, and the cryptography project now provides .whl pre-compiled packages using musl (as well as ones for glibc).

            1. 1

              That would be nice, however that also requires package maintainers to rebuild the whls. That feature is only added in September for cibw, so it might take a while to get fully adopted.

      1. 2

        Ah, my recommendation! I love this, get it and you won’t regret it if you’re at all into electronics.

        1. 1

          i did indeed find it via your recommendation (:

        1. 2

          Why not use Docker Swarm? It’s easy to setup, it’s compatible with Compose and it’s working fine with simple workload. You just need to deploy multiple stack.

          1. 1

            Does Swarm work with repositories (rather than published images)? I generally prefer that workflow as it’s simpler, but if you already build images, Swarm is a great choice.

          1. 2

            Can you please help me understand the use case?

            Is this so that if you have multiple docker-compose files that you want deployed to a single machine this makes that easier?

            When I saw the initial blurb I got excited thinking this might help me deploy docker-compose apps to different machines in my cluster, but I didn’t get a sense of that from the README?

            1. 2

              Yes, this is for deploying multiple Compose apps on a single machine. Is your use case for something more like Kubernetes?

              1. 3

                You might consider explicitly mentioning this fact in your project README so it’s clear to prospective users what you’re offering.

                Thanks for the response and for open sourcing your project!

                Kubernetes always felt incredibly heavyweight for my simple needs. I ran it here for a while but since I do zero clustering

                I have a couple of machines in my ‘homelab’ running ProxMox. Those machines host various VMs which themselves host Docker containers for the apps I want to run.

                If anything, perhaps something like Ansible would be a better choice, I just haven’t invested the time and energy to encode the various docker-compose files I have into Ansible playbooks.

                1. 2

                  I’ll definitely mention that, thanks! I think Harbormaster can still help you, what I didn’t like about the various other solutions was that there was no discoverability. None of them had a canonical list of apps that ran on a machine, whereas with Harbormaster you know what’s running where.

                  If you aren’t running the same app on N machines, you can just set up one Harbormaster config per machine and that’s it!

                  1. 2

                    Look into Nomad if you haven’t already.

                    1. 2

                      You’re easily the 3rd or fourth person to suggest that. I will absolutely get on it!

              1. 2

                If you did 10 keys (two hands), could you arrange things so that no prefix of a chord was also a chord? So that you could always unambiguously detect when an intended input event had occurred. With five keys, you probably don’t have enough bits for this.

                Alternately, you could split chords between “bottom-up chords” and “top-down chords”, depending on the order in which your fingers hit the keys. That way, you could probably even get an additional bit out of the five-key setup. Maybe top-down chords could be capital letters.

                1. 3

                  Hmm, that’s an intriguing idea, using the order of keys to tell if they should be capitals. Some keys are single-press, though, so I don’t know if it’d be worth the tradeoff. Josh suggested an accelerometer and turning the keyboard itself at an angle, but I don’t know how convenient that’d be either.

                1. 2

                  I once made a three-key keyboard where I first made an input filter for BeOS and created a log of every keystroke over a period of weeks that I then used to create a modified Huffman encoding based on the usage of the keys. It worked, but I can’t say it was very practical. Didn’t spend a lot of time trying to learn the alphabet, though, so maybe it could have been useful…

                  1. 2

                    That’s fantastic, why three and not two, though?

                    1. 2

                      More fingers/keys means shorter sequences!

                      In the data I generated I think the longest sequence was four or five keypresses to generate a character.

                  1. 6

                    Chorded keyboards are a thing, I have a https://www.gboards.ca/product/georgi, and apart from taking a bit of time to get used to it, it’s a nice way for input.

                    1. 4

                      Steno keyboards are a thing. Look at Plover if interested.

                      1. 4

                        Hahah, this is the keyboard that always breaks my autocompletion! I made a (full-size) keyboard called “George”, and whenever I try to cd into its QMK directory, autocompletion always annoyingly stops at “georg” because there’s “georgi” too. I had no idea it was chorded as well, thanks for the link!

                      1. 6

                        Believe it or not there are products that implement this: https://www.in10did.com/

                        My use case was for terminal commands and using my phone as an audio terminal (termux and espeak).

                        I recorded all my key presses on my laptop including non-printing commands like crtl, alt, esc, etc, built a key map based on weighted finger positions, flashed a new key map onto the thing and it nearly worked. Unfortunately anything more more complex than ed had too much state I needed to keep in mind when working, so I needed to look at the screen anyway.

                        Which brought me to the solution that actually works: https://en.wikipedia.org/wiki/MessagEase along with emacs.

                        I can enter all the weird symbols I want using software, and it’s kind of usable for entering text. Using that inside temux with emacs I have something that is good enough to edit files and run commands for any server that I could possibly want.

                        If I wanted to take notes though both are terrible. I’d look into stenography, but with how good speech-to-text has become I’d just dictate to an app with a red button.

                        1. 1

                          Do you have longer write-ups of your set up and/or videos of it running? I am extremely interested in an audio terminal setup.

                          1. 1

                            Unfortunately no.

                            The setup was always buggy as hell on regular bash+linux. I vaguely remember some sort of C monstrosity that dispatched espeak processes with each key press, and another one that killed whatever was running when a new key was pressed. I never managed to get anything that wasn’t a full text line to audio in termux.

                            What did work for audio terminal is running a shell in emacspeak. That’s how I got to the point that I realized that it was too much mental overhead for me to use a chorded keyboard along with an audio only for emacs interface, and the only programs that worked well were the oldest unix utilities that used to output to actual teletypes, never was the terseness of ed more appreciated.

                            Again that was on Linux only with the keyboard connected by bluetooth. Currently emacspeak does not compile for termux. If you know enough about emacs to compile it for termux that would help the project tremendously.

                            That said I am one of the few people to genuinely use ed from a shell in emacs which is something I’m both proud and ashamed of.

                            1. 1

                              I’m actually more interested in your keybindings and how your interactions with it when it did work. For the audio output part, it sounds like your relied on the underlying utility, like ed, being nice about it’s output. But what of the keyboard input part? Did that work well?

                              Currently emacspeak does not compile for termux.

                              I’m a bit baffled by what this means and why it’s needed. What do you mean by “compile for”? Also, how important is it to use termux versus another shell or terminal?

                              I’m make espeakui which has a Espeaker class that is IMO a better API for espeak. Would this help with the bash+linux approach? You can start, stop and pause speech easily without having to kill an espeak process. I use it to change speech rate mid sentence. It uses the same underlying speaklib library as espeak (but can also use espeak-ng with minimal change).

                              1. 1

                                The key chords worked as expected. All the smart stuff was done on board the keyboard’s micro controller with only regular key presses being sent out by bluetooth. So the computer couldn’t tell the difference between it and a regular keyboard. I don’t have the map any more and I was trying naively to use the same key bindings as I did on a regular keyboard. The amount of mental state I needed to keep track of was extremely high.

                                For my use case emacspeak is a better fit than trying to use a terminal and when I finally got it working on my main machine all the projects I’d been trying to look at became pointless.

                                I’m a bit baffled by what this means and why it’s needed. What do you mean by “compile for”? Also, how important is it to use termux versus another shell or terminal?

                                Termux is an android terminal emulator and Linux environment, my use case was to try and build an audio only work station on mobile so I could use the same setup where ever I was. I did not succeed, I did get to the point where I had emacspeak + terminal working on my desktop along with a chorded keyboard. The combination was a lot less usable than I thought it would be.

                                Ultimately I moved to an eink android tablet + keyboard + battery. The setup lasts me weeks between charges and it is only marginally more difficult to carry around than a regular laptop. Since emacs works under termux I have the same setup on all my machines for on the go editing. It is rather pleasant to work outdoors in the bright sunshine and for ‘serious’ work I ssh into my home network.

                                I’m make espeakui which has a Espeaker class that is IMO a better API for espeak. Would this help with the bash+linux approach?

                                From a cursory look, no. You need to spawn a process for every letter press and every word. Basically have a look at what emacspeak does and you’ll see something I’m 90% happy with for coding on.

                                That said the use case for that software is mostly covered by the following line of bash:

                                TMP=$RANDOM; xsel | tr '\n' ' ' | espeak-ng -p 100 -s 950 -v male7 -w /tmp/$TMP.wav; mplayer /tmp/$TMP.wav
                                

                                I will give yours a try later since I never did manage to get espeak to display the text being read currently, and never cared enough to look into how to solve the problem.

                                Not sure if that’s any help at all.

                                1. 1

                                  So key chords worked just as well as a keyboard? It wasn’t much slower to use one?

                                  I’ve only looked at emacspeaks briefly and that was a long time ago. Isn’t it basically a screen reader with emacs? Any significant difference between using that and, say, yasr or espeakup (other than being inside emacs)?

                                  The combination was a lot less usable than I thought it would be.

                                  Ah, that’s unfortunate. It sounds like you were mostly going the “regular” terminal and screen reader route. I want to know if there’s some design that would specifically be good for audio linux with a keyboard (or other input) with fewer keys.

                                  That said the use case for that software is mostly covered by the following line of bash:

                                  Yes, you can do that (with mplayer -af scaletempo) but for large webpages the wave file takes quite some time to generate and there’s significant audio distortion at >2x speed (which might be why you set your intial speed to 950?). I do still use the pipe-to-mplayer method on a phone though.

                                  But I was pointing at the library rather than the application. But it looks like you want a mostly completed screen reader, which indeed it isn’t. Its better suited for building “custom” audio applications.

                          2. 1

                            Huh, MessageEase looks interesting, thanks! Though I don’t know how much of a fit it is for non-keypad phones. Taking notes via speech recognition in a noisy environment would probably not be too great, and you’d need to summarize them later.

                            1. 2

                              Huh, MessageEase looks interesting, thanks! Though I don’t know how much of a fit it is for non-keypad phones.

                              It works vastly better on touch screens. Typing one symbol at a time is still much slower than swiftkey like swipes for words but for accuracy in typing it beats anything I’ve tried, including phones with physical keyboards and chorded keyboards.

                              Taking notes via speech recognition in a noisy environment would probably not be too great, and you’d need to summarize them later.

                              What you can do with audio engineering today is out of this world: https://www.youtube.com/watch?v=hrQ_2JhEKpY

                              The last time I played around with speech to text was when Mozilla released their audio thing way back when. Using it on a bunch of librevox recordings it averaged less than 1 error per thousand words.

                          1. 14

                            Add one more key and you could implement the Microwriter input method: https://en.wikipedia.org/wiki/Microwriter#Keyboard

                            The manufacturers claimed that most people could learn to use it in just a couple of hours. With some practice, it is possible to become a faster typist with the Microwriter than with a conventional keyboard, providing that what is being entered is just text. Typing is slowed if a substantial number of special characters have to be entered using the “shifting” mechanism.

                            The one of the coolest parts is it could be used completely detached from a computer:

                            At the top end of the unit is a 25 pin D-type connector providing an RS-232 port, an interface to an audio cassette player for saving and restoring files

                            1. 3

                              That’s very interesting, thanks! Looks like what’s old is new again.

                            1. 1

                              Just the other week I had found myself looking for a digital alarm clock that would automatically re-arm itself on weekdays, and found myself beyond disappointed with the options on the market. I’d been considering making my own, and seeing how simple it was to put this solution together, this might be the push I need to get started.

                                1. 1

                                  Are the files for the, like, circuit diagram and 3D printed box available? I don’t see them in the repo :)

                                  1. 1

                                    The circuit diagram isn’t there because it’s pretty trivial, you just connect the screen to I2C. I can send you the OnShape link for the box, if you like! (Or include it in the repo).

                              1. 2

                                Nice! I still have some nixie tubes laying around, which I also planned to use in an alarm clock that won’t alarm. I have had them for like a year or so. So, suddenly your alarm isn’t that futile ;)

                                Anyway, your post really makes me feel like making an alarm clock, especially since you 3d-printed your enclosure and I recently-ish got a 3d printer :)

                                1. 3

                                  I 100% encourage this, and would love to see your clock.

                                1. 10

                                  I really didn’t expect this kind of ending, but I like it :)

                                  I could create a web server where I could configure alarms, but I figured I’d rather get a rooster instead, so I did neither and just made an alarm clock that cannot alarm. This symbolizes the futility of human existence and its incessant search for meaning in a cold and unfeeling universe that’s ultimately profoundly devoid of such, and then you die.

                                  1. 9

                                    I’m glad you like it but what’s the point

                                  1. 3

                                    I think the site is kinda shady, won’t let me cancel my subscription. I hope I’ll remember to retry later on. Kinda sus, ngl.

                                    1. 4

                                      I can cancel it if you want. I really should write that code, it is pretty sus.

                                      1. 1

                                        Hahaha. I’ve never experienced lobste.rs equivalent of a standup comedy. I simply had to try and say thanks.

                                        So sad I couldn’t afford to pay the full price and rake in billions in dividends. :-)

                                        Do you write often?

                                        1. 3

                                          Not very often, usually on my site, and I wrote another IMGZ post recently (that lobste.rs didn’t like as it’s a parody of business), but my problem is that I don’t find inspiring topics to write about frequently. I most like writing when I can make fun of something I find ridiculous, and I guess there isn’t enough of that :P

                                          EDIT: By the way, the “cancel subscription” section should now be active for you, I just deployed that code a few minutes ago.

                                    1. 29

                                      Love the tone of the post. Thought it was really funny.

                                      As for the actual service itself, the clear disdain for Silicon Valley startup bs makes me want to use it for a small website that I will almost certainly never finish.

                                      Then again, I’m 51% sure I could build a very similar service for cheaper. Then again, $5/month is pretty cheap. Then again, this is a brand new SaaS, and may not be around in a year. Then again, I have no users and no website and no one relying on me.

                                      Hmmmm.

                                      1. 58

                                        Oh don’t worry, it will definitely be around in a year, We are committed to giving our customers the best experience possible. I guarantee that IMGZ will be up and running for a long, long time, or until someone gives me two bucks and I “amazing journey” the entire customer base.

                                        1. 8

                                          I “amazing journey” the entire customer base

                                          I lol’ed ^_^

                                        2. 5

                                          The CHEAPASS tier is actually $5 per year, or ~42 cents per month.

                                          1. 5

                                            Still overpriced, if you ask me.

                                            1. 1

                                              Oops, completely misread that

                                              1. 3

                                                I mean you called it “pretty cheap” so there’s no going back now, expect a 12x price hike

                                                1. 3

                                                  For a 12x price hike, I would hope to see some Serious Business features. Maybe you could provide a Satisfaction Guarantee. Or maybe Enterprise Support.

                                                  1. 4

                                                    I will add that to the backlog, which is a euphemism for “no”.

                                                2. 2

                                                  5€/mo is what the whole server costs stavros to run.

                                                  1. 2

                                                    So he’d need at least 12 users to recoup his costs.

                                                    I think that may be treading too far into the dreaded “at-scale” territory for stavros.

                                                    1. 2

                                                      Oh God and they’ll think they have the right to email me for stuff, what a hassle

                                            1. 4

                                              The “nine fives” gave me a long sweet laughter; thank you!

                                              1. 10

                                                Oh this reminds me I have to take the service down today if I’m going to meet the guarantee, it’s been up too long

                                              1. 4

                                                One thing I’ve read over and over is “don’t store your files in a database”. I assume there are caveats, and times when this does make sense, but could anyone care to make the case for why this is a good or a bad idea in this particular scenario?

                                                1. 16

                                                  In general: Storing files in the database generates a ton of IO load on the database server, which is usually already IO-bound. If your database is busy doing other stuff (unlike, say, IMGZ which doesn’t do anything else) that’s going to degrade performance.

                                                  On the flip-side, it’s terribly operationally convenient when you can backup a single database and actually have all your data, and having consistency between the data-store & the filesystem store is nice so you can’t refer to files that have been deleted / failed to upload / whatever.

                                                  1. 12

                                                    Generally, considerations for files are different than for other data. E.g. you almost never need to filter files by actual contents, or sum them, or do any of the other things databases are good at, you just want to be able to store them, retrieve them and delete them. If you saved everything in a database, it would be more expensive, just because of the type of guarantees that you need for data, which you don’t need for files.

                                                    That means you’d unnecessarily be paying all the costs that are associated with what we normally want to do for data, but not need any of the benefits.

                                                    1. 5

                                                      Sqlite did an analysis on this and found that for files under a certain size, it is actually faster to read them from a database (based on the database page size): http://0x0.st/iFUc.png

                                                      1. 4

                                                        It’s a bad idea in this scenario. But you’re not likely to care too much at very low scale.

                                                        One solid reason to care is that in PostgreSQL the maximum “row” size is 16KiB. If you go over that then “the next” 16KiB goes into what’s called a “TOAST” table, which has access characteristics similar to a singly linked list. (So, 1 request for a 64KiB row will cause 4 queries to the storage engine and so on.)

                                                        Other characteristics of the TOAST table is that it compresses on input, which can quickly saturate your CPU.

                                                        1. 2

                                                          Another way to say that is, “it will work, until it doesn’t.” Which is true of 100% of scaling problems. :-)

                                                          The nice thing about simple solutions is that they can be easier to adapt and extend later when needed.

                                                          1. 17

                                                            Scaling problems are for amateurs, if I get too popular for my architecture I’ll just disable the signup page.

                                                            1. 3

                                                              But that’s one customer too late - who are you going to evict to get out of scaling problem territory?

                                                              1. 14

                                                                The one with the most images, obviously.

                                                        2. 4

                                                          You basically have three options with Postgres: you can put the file contents in the row, you can use the large object facility, or you can write the file to disk somewhere and store the path.

                                                          Putting the file contents in the row is simple, and is the only option that gives you for-free the notion that deleting a row will get rid of the content of the file. It has the disadvantages that others have discussed, although I don’t think TOAST is so bad for performance.

                                                          The large object facility is basically file descriptors in the database. You have to take extra care to remove them when you’re done with them, and most ORMs have poor support for it. I have never seen the large object facility used in the wild, and it’s not a tool I would reach for personally.

                                                          The third option is probably the best. The filesystem will always be better at storing files than your layer on top of the filesystem. But you have integrity concerns here (deleting a row does not cause the corresponding file to disappear), and you have to back it up separately from the database.

                                                        1. 13

                                                          Stavros, what did you use to create the architecture diagram? It is fantastic and I shared it with all my friends. Both of them.

                                                          1. 13

                                                            Haha, thank you! Yes, I took the sample Cloudcraft diagram and customized it to the exacting specifications of IMGZ.

                                                            1. 5

                                                              It looks like the McMansion Hell of service architectures.

                                                              I love it.

                                                              1. 10

                                                                Why are you giving me ideas

                                                            2. 4

                                                              It looks like https://www.cloudcraft.co.

                                                            1. 8

                                                              I love your pricing page - those ‘our choice’ tags are such bullshit.

                                                              1. 4

                                                                What, like somehow $999,999.98 isn’t the best value for you?

                                                                1. 2

                                                                  What happens if someone seriously wants the $999k plan?

                                                                  1. 17

                                                                    I write an “amazing journey” post detailing how the service is seriously definitely never getting shut down for at least three days and retire on a beach while you deal with the rotting service.

                                                                    1. 1

                                                                      What startup is the phrase “amazing journey” in reference to?

                                                                      1. 15

                                                                        Many.

                                                                        1. 4

                                                                          Many! I think I’ve seen “incredible journey” used with that one email program you had to wait in line to get, one of the post-Flickr photo sites, Vine maybe?

                                                                      2. 1

                                                                        He throws a party… I mean… hires a lawyer and makes a Series-A announcement?

                                                                  1. 13

                                                                    BTW, a story of mine: When I was in high school, we had a class to teach us how to type on the keyboard efficiently. The classes consisted of using a training software, and the grades were given mostly by the score we achieved there. We could run the program on other computers in the school as well and do some lessons in the spare time.

                                                                    While never intentionally learning how to use the keyboard properly, over the years of coding and spending hours on IRC, I learned to type pretty fast, so I always was far ahead. However, some of my friends struggled with this and I was helping them. But this was boring.

                                                                    So naturally, I went ahead and wrote a simple program to type in the window automatically. During a break, I gathered a group of friends, took them to the computer and intended to demonstrate them this incredible time-and-effort-saving creation of mine. I opened one of the lessons, where we had to type a single sentence as many times as possible, fired up my program, typed the sentence there and pressed the “Start” button. It worked perfect, except… I made a typo.

                                                                    At around 800 bpm, it wrote the errorneous input all over again in a lesson, where it was particularly important to type as accurately as possible. Few seconds later, the lesson was over, because there were too many mistakes, and it threw me back to the previous lesson. But the expected input in that lesson was completely different, so it kicked me even faster. And it continued doing that for over and over again.

                                                                    Before I finally managed to kill the program, I lost about two weeks worth of progress. But the program got popular nonetheless as they understood it worked properly, I only entered an invalid input. Later on, an IT teacher asked me to demonstrate it for him. My memory is faint on that part, but I think nothing happened at all. I’m sure he would ban it if he knew how, but he wasn’t very good.

                                                                    1. 5

                                                                      Cool story… I love the irony that you built it to speed up your results but it caused the opposite to happen in the end… that’s probably a metaphor for about 50% of everything I do in my code :)

                                                                      1. 2

                                                                        It would have been great if you made it only type when the user pressed a key, so people could mash the keys as quickly as possible and have 100% correct output come out. Since you were just using it for training, though, your way makes sense too.

                                                                        1. 2

                                                                          Are you me?

                                                                          Did almost the exact same thing back in elementary/middle school.

                                                                          Loaded it up onto my purple 64mb JumpDrive that had cost me a small fortune so I could run it on any of the computers.

                                                                          1. 1

                                                                            Same here on Mavis Beacon. Except, the hackers weren’t impressed. So, we changed the challenge to seeing what numbers we could get doing that by hand on the actual keyboard. I can’t remember the number but it was insane.

                                                                        1. 3

                                                                          I’m a fan of https://usefathom.com. It’s very minimal, but I don’t have many requirements, so it suits my needs pretty well. I like that it’s a single Go binary, which I run on a $5/month Digital Ocean droplet.

                                                                          1. 7

                                                                            Please note that while this may currently work, it is no longer being maintained and is very different from the paid Fathom Analytics product. I built the initial open-source version of Fathom that you are running, but after I left the project moved to a centralized and closed-source model.

                                                                            1. 2

                                                                              I wasn’t even aware of this. Thanks for the update.

                                                                              1. 2

                                                                                Thanks for making that, I use Fathom for my site and like it. Too bad they closed it up.

                                                                                1. 1

                                                                                  The latest communication is that “We are going to be releasing a new version this year”, FYI

                                                                                  1. 2

                                                                                    We’ll see.