And then you’ll be surprised by the libc dependency. Or the library that’s a surprise transient dependency. Or the kernel API that some day will be changed exactly breaking what you expect.
Perhaps this statement is better: Everything that’s not Bedrock should be a saved artifact input to your build.
At the core of my complaints is the fact that distributing an application only as a Docker image is often evidence of a relatively immature project, or at least one without anyone who specializes in distribution. You have to expect a certain amount of friction in getting these sorts of things to work in a nonstandard environment.
This times a thousand. I have tried to deploy self-hosted apps that were either only distributed as a Docker image, or the Docker image was obviously the only way anyone sane would deploy the thing. Both times I insisted on avoiding Docker, because I really dislike Docker.
For the app that straight up only offered a Docker image, I cracked open the Dockerfile in order to just do what it did myself. What I saw in there made it immediately obvious that no one associated with the project had any clue whatsoever how software should be installed and organized on a production machine. It was just, don’t bother working with the system, just copy files all over the place, oh and if something works just try symlinking stuff together and crap like that. The entire thing smelled strongly of “we just kept trying stuff until it seemed to work”. It’s been years but IIRC, I ended up just not even bothering with the Dockerfile and just figuring out from first principles how the thing should be installed.
For the service where you could technically install it without Docker, but everyone definitely just used the Docker image, I got the thing running pretty quickly, but couldn’t actually get it configured. It felt like I was missing the magic config file incantation to get it to actually work properly in the way I was expecting to, and all the logging was totally useless to figure out why it wasn’t working. I guess I’m basically saying “they solved the works-on-my-machine problem with Docker and I recreated the problem” but… man, it feels like the software really should’ve been higher quality in the first place.
no one associated with the project had any clue whatsoever how software should be installed and organized on a production machine. It was just, don’t bother working with the system, just copy files all over the place, oh and if something works just try symlinking stuff together and crap like that.
That’s always been a problem, but at least with containers the damage is, well, contained. I look at upstream-provided packages (RPM, DEB, etc) with much more scrutiny, because they can actually break my system.
Can, but don’t. At least as long as you stick to the official repos. I agree you should favor AppImage et al if you want to source something from a random GitHub project. However there’s plenty of safeguards in place within Debian, Fedora, etc to ensure those packages are safe, even if they aren’t technologically constrained in the same way.
I agree you should favor AppImage et al if you want to source something from a random GitHub project.
I didn’t say that. Edit: to be a bit clearer. The risky bits of a package aren’t so much where files are copied, because RPM et al have mechanisms to prevent one package overwriting files already owned by another. The risk is in the active code: pre and post installation scripts and the application itself. From what I understand AppImage bundles the files for an app, but that’s not where the risk is; and it offers no sandboxing of active code. Re-reading your comment I see “et al” so AppImage was meant as an example of a class. Flatpak and Snap offer more in the way of sandboxing code that is executed. I need to update myself on the specifics of what they do (and don’t do).
However there’s plenty of safeguards in place within Debian, Fedora, etc to ensure those packages are safe
Within Debian/Fedora/etc, yes: but I’m talking about packages provided directly by upstreams.
Within Debian/Fedora/etc, yes: but I’m talking about packages provided directly by upstreams.
Regardless of which alternative, this was also my point. In other words, let’s focus on which packagers we should look at with more scrutiny rather than which packaging technology.
AppImage may have been a sub-optimal standard bearer but we agree the focus should be on executed code. AppImage eliminates the installation scripts that are executed as root and have the ability to really screw up your system. AppImage applications are amenable to sandboxed execution like the others but you’re probably right that most people aren’t using them that way. The sandboxing provided by flatpak and snap do provide some additional safeguards but considering those are (for the most part) running as my user that concerns my personal security more than the system as a whole.
On the other side, I’ll happily ignore the FHS when deploying code into a container. My python venv is /venv. My application is /app. My working directory… You get the picture.
This allows me to make it clear to anybody examining the image where the custom bits are, and my contained software doesn’t need to coexist with other software. The FHS is for systems, everything in a dir under / is for containers.
That said, it is still important to learn how this all works and why. Don’t randomly symlink. I heard it quipped that Groovy in Jenkins files is the first language to use only two characters: control C and control V. Faking your way through your ops stuff leads to brittle things that you are afraid to touch, and therefore won’t touch, and therefore will ossify and be harder to improve or iterate.
I got curious so I actually looked up the relevant Dockerfile. I apparently misremembered the symlinking, but I did find this gem:
RUN wget --no-check-certificate https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
RUN bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda
There was also unauthenticated S3 downloads that I see I fixed in another PR. Apparently I failed to notice the far worse unauthenticated shell script download.
I’ve resorted to spinning up VMs for things that require docker and reverse proxying them through my nginx. My main host has one big NFTables firewall and I just don’t want to deal with docker trying to stuff its iptable rules in there. But even if you have a service that is just a bunch of containers it’s not that easy because you still have to care about start, stop, auto-updates. And that might not be solved by just running Watchtower.
One case of “works on docker” I had was a java service that can’t boot unless it is placed in /home/<user> and has full access. Otherwise it will fail, and no one knows why springboot can’t work with that, throwing a big fat nested exception that boils down to java.lang.ClassNotFoundException for the code in the jar itself.
I missed this when it was first posted (here) but building a PCB has been on my bucket list. Is there any advice that the hardare hackers here on Lobsters would give to make this process less error prone for a first timer?
Double check all part footprints and orientations.
Have a target overall size in mind first.
Check multiple parts stores and track which you are using
Save your project in git or other VCS so you can undo experiments.
Silkscreen: all readable without rotating the board, and readable even with all parts populated. You want to know which component is where even after you solder it.
Add test points anywhere you are not certain about the signal.
Schematic design comes first. Get that part right and the PCB bits can be closer to an OCD Zen Garden.
don’t know what I’m doing but have built pcbs that worked.
Rule lists like this wouldn’t be any good if none of them felt at least a little bit controversial.
Rule #5 I disagree with. I hate APIs that return clear map structures with “key”s inside, but I cannot lookup by the key because it’s a list and not a map. Just use a map when the data fits into that pattern.
Rule #8. 404 is fine for an API because all the invalid endpoints can just return 5xx. If the client needs further information, you can return an error code that never changes along with the response body.
I think another way to say Rule 5 might be: return objects with well-defined schemas. Or: given a specific key, its corresponding value should always be the same type. So if you return {"user":"John"} in one situation, it’s not cool to return {"user":123} in a different situation.
Rule #5 I disagree with. I hate APIs that return clear map structures with “key”s inside, but I cannot lookup by the key because it’s a list and not a map. Just use a map when the data fits into that pattern.
I’ve heard that list returns also require breaking changes to make any improvements. It’s a “Probably Gonna Need It” to use a map.
Was there also something that broke if you returned a list instead of a map, or was that just a vague memory?
The hatred of C here is not reasonable. It seems that the author wanted to dunk on the language, but other languages would have known that it was possible to reach all statements in that pseudocode and thus I can’t imagine other languages helping.
My plan is to virtualize all of my development environments. So my work will be primarily over SSH, SSH tunnels, and perhaps tramp (in Emacs). I’m setting that up with bhyve, using the vm-bhyve scripts for convenient management.
This all runs on a server in my hallway cabinet - along with a brace of RPis offering utility stuff like OpenVPN, Pihole, and Home Assistant. Plus one Pi running FreeBSD to serve my Web and Gemini stuff.
My laptop will then only be for media, mail, and chat - at which point there’s no significant advantage to FreeBSD, I can just set up whatever cheap laptop with Mint and use it as a thin client. At the moment that’s a ThinkPad T470s with a docking station on my desk.
The point of double-entry bookkeeping isn’t the ledger (the diff) but the whole way credit balance accounts, the balance sheet and the accounting equation work together.
Yeah I think of the main principle as more that transactions always add up to 0. I.e. the conservation of money: Money is never created or destroyed, just moved between accounts. (Though revenue and expenses are both treated as bottomless and effectively “create” money in that sense).
It’s virtually impossible to create a future-proof way to solve this, given the (natural) difficulty of integrating with loads of different banks around the world, without doing some manual data entry. However, I should’ve clarified in the article that I do not recommend you enter data manually in TOML, but rather do some importing based on e.g. CSV statements. The advantage of TOML is that it is easy to read, edit and script around.
I do not recommend you enter data manually in TOML
Oh yeah, that makes a lot more sense hah. My reply was in reaction to thinking that’s what you were proposing. Automating it in another way and just using TOML as the backing store makes a lot more sense.
My initial impression was too that you offer to enter data in TOML and by the end of your article you would offer some convertor from TOML to ledger, hledger and/or beancount formats.
If you do not recommend entering data manually in TOML and if I gather it right, you argue for storing data in TOML, because there are tons of robust libraries to work with this human-readable and machine-friendly format. Hmm? The article title is a bit clickbait-y then (-;
I believe, some of the mentioned PTA software have plug-in systems, perhaps re-usable parser code, and some sort of export to some sort of CSV? How that doesn’t solve problems you are trying to solve? So far to me it seems that you add a superfluous layer in between the wild world of financial data formats and PTA. Although, you said that it was for your personal use, so I guess it’s totally okay (-:
There are plugin systems and reusable parser code, yes, but they often lock you into a certain programming language and I find that libraries for parsing common data formats like TOML often are of higher quality.
While I do agree with the author’s arguments against WAFs, I find the solutions quite disappointing as they require direct changes to the application, which isn’t always under your control.
I’m part of the ops team and in charge of securing the application access, not the application itself, and a WAF would let me do that (albeit in a clunky and painful way).
Then it seems especially difficult to get alignment between dev and ops, and I’m especially curious how others go about making that work!
I’m running one vendors application presently (mattermost). We have the source and a CI/CD pipeline; if some new exploit becomes apparent, I’m able to adjust the code myself, as well as escalating to the vendor, but as far as I can tell that’s a very rare state of affairs.
Of course the dev team is responsible for securing the application, and we’re working hard with them so that security gets taken into account from day one when they spin up a new microservice.
But life isn’t always that easy, and there are many legacy stuff that is either abandonware or so old that nobody dares changing anything to it, let alone refactor the code to bring better security to it.
Also no, they don’t carry the pager, and my point is that a WAF is a solution I can deploy on my own to increase the security of an application, and not wait to be breached to raise an issue to the dev team that will maybe fix it on the next sprint, because priorities. Chug
I’m guessing the WAFs you’re talking about are looking at the HTTP header (path, with, whatever) and maybe source IP address? That sounds almost more like a proxy with access controls. The article mostly complains about the WAFs which run a bunch of regexes on the request body. If you do make use of the body inspection features I’m curious about how.
The WAF I’m talking about is fairly advanced, and provides an “auto-learning” feature which will eat thousands of requests to build up an allow list based on the tokenization of the accesses (both URL, headers and body).
It also has access to the OWASP rulesets which features many regexes and tokens matches, etc…
And to be fair, I do agree that WAF rules are very complex, can be bypassed, and not self-sufficient to provide security for an application. But as an ops, I can set it up to increase the application security with no (or little) adherence with the dev team.
It’s a dozen small things that have all added up to a much better experience for me. I’ll pull out a few:
ZNC used to crash every few weeks on my little VPS. Soju runs on the same cheap machine with no such issues.
history playback works really well, if you have logs turned on you can just keep scrolling back in time, inside the client
with lots of channels, weechat would take ages to play back history, basically locking you out as it updated channel by channel
when you log in, all your servers and channels are immediately shown in the clients. With ZNC I had to create profiles for each server.
On a personal level, the ‘out of the box’ config for senpai is perfect for me. I spent a long time tweaking weechat.
More broadly, these programs all feel like they are part of an ecosystem, and work together really well. This makes a multi-device setup more manageable.
I’ve not yet found a downside to goguma or senpai, but there are some to the soju bouncer:
fiddly to setup non-tls connections
fiddly to setup networks which require commands to be sent upon connection
Traveling back home to Moscow from India. Comes with the usual question of what to do on a long flight, I cached all the dependencies needed for integrating MMTK into the Tvix evaluator, but I’ll probably just watch a movie and then fiddle with my Emacs config.
By the way, did you know that outputting to all sound devices simultaneously is literally trivial with Pipewire? On the flight here we watched a movie, with output to two different headphones, and it worked perfectly fine. The future is here.
I control Pipewire through its pulseaudio interface. Basically I just issued this command:
pactl load-module module-combine-sink
It then creates a sink that sends output to all other sinks, so you can select it and then control the individual other sinks as normal (volume/mute/etc.)
There is plenty of room for disagreement, but I think the notion here is that REST APIs have two types of resources fundamentally; collections and objects. A collection can hold zero or more objects, so it is (in English at least) natural and correct to for the name of a collection to be the plural form of the type of object which it contains.
Just because you have requested a specific object from a collection does not change the fact that the collection itself holds zero or more objects.
Pretty good advice overall, but I do want to spotlight something
Rule #2 DON’T add unnecessary path segments
Good advice, and yet the GOOD example…
# GOOD
GET /v3/application/listings/{listing_id}
commits the cardinal sin of REST API versioning… the entirely useless version segment! What’s weird is that the author demonstrates an understanding of why this is bad elsewhere:
URLs are resource identifiers, not representations. Adding representation information to the URL means there’s no canonical URL for a ‘thing’. Clients may have trouble uniquely identifying ‘things’ by URL.
Putting version information into the resource name also breaks this principle.
Tell me what is the distinction between these resources:
Nothing whatsoever. You are communicating something about the representation within the identitfier, which is a violation of concerns.
Why would you change a resource identifier? Let’s count the ways
Adding a new resource: just add it
Removing an old resource: HTTP already gives you the tools to handle this
Moving a resource: ditto
Changing the format of your unique identifier: okay that might be legitimate, but there’s a better way.
Stripe’s approach has been very influential to me. Using ISO8601 date stamps as version numbers, pinning API consumers based on first request, compatibility layer that translate older API schema to new, all good things. The only issue I have is that they chose to use a custom Stripe-Version header, and I disagree that this is right.
This is fundamentally about having multiple, incompatible representations of the resource. How do we choose between representations? Content Negotiation.
This abstract definition of a resource enables key features of the Web architecture. First, it provides generality by encompassing many sources of information without artificially distinguishing them by type or implementation. Second, it allows late binding of the reference to a representation, enabling content negotiation to take place based on characteristics of the request. Finally, it allows an author to reference the concept rather than some singular representation of that concept, thus removing the need to change all existing links whenever the representation changes (assuming the author used the right identifier).
Encoding the API version into the resource name means that the client becomes solely responsible for requesting the representation, there is no opportunity for negotiation between client and server.
# BAD
GET /v3/application/listings/{listing_id}
Accept: application/json
# GOOD
GET /application/listings/{listing_id}
Accept: application/vnd.company.my-api+json; version=2023-11-01
In short: use http headers instead of the url to encode the version. Is that really that much better though? I thought that http headers shouldn’t change semantics, and changing the versions is a semantics change to me. But then I guess you can argue that the version is not part of the semantics when talking about http entities.
Tbh, it seems fairly academic to me. I’d rather stick to putting the version into the url and forcing the clients to assume that the same $listing_id under v1 and v2 might be the same entity, but might actually be different entities, forcing the clients to read my release notes.
In short: use http headers instead of the url to encode the version.
No. Use the Accept header specifically to encode the version, because what you are doing is content negotiation, and HTTP designates that header for that purpose.
I thought that http headers shouldn’t change semantics, and changing the versions is a semantics change to me.
If you use path segments to version your API, you have no way to distinguish a syntactical change from a semantic one, because you’re putting the information in the wrong place.
A semantic change should be reflected in the resource identity. A syntactic change should be reflected in the representation. These are separate concerns and should not be conflated.
Tbh, it seems fairly academic to me. I’d rather stick to putting the version into the url and forcing the clients to assume that the same $listing_id under v1 and v2 might be the same entity, but might actually be different entities, forcing the clients to read my release notes.
This perfectly encapsulates the problems I am critiquing:
you have placed yourself entirely at the whim of the client implementations
your API design does not communicate intent and relies instead on out-of-band information to understand
there’s no orderly process of deprecation and data migration
This creates a situation that makes breaking changes very challenging to release, which in turn makes your API design accumulate undesirable warts, which makes integrating with your API unpleasant.
This is why people come to hate HTTP APIs like this.
(I object to your characterization of this as a scholastic exercise! I went to community college. My strong opinions have been formed by years of practical experience integrating with bad APIs.)
I guess the reason I don’t like to use headers is because they are rarely used for those things. The Accept-header also seems to be used only for transparent compression. Another thing I don’t like is that headers are often not captured by logging tools for various reasons - so in case of a bug, it’ll be hard to reproduce it without knowing the headers if they impact the response.
In other words, both solutions now seem unsatisfying to me.
How do you like to handle paths that change between versions?
EDIT - I guess I’m asking where you handle the routing for v2023-10-31 vs v2023-11-1. I could imagine you could handle it at the routing layer by checking the media type parameter and sending it to different servers entirely, or you could handle it in the application layer.
That depends on the implementation, and the reasons for your version change. If it’s all just happening in one app, you can handle it there. If you’re transitioning to a new codebase, you could introduce a reverse-proxy. Basically, your choices aren’t different here from the path segment, you just have more flexibility in introducing version updates to represent incompatible changes piecemeal without making a drastic change on the client side.
Negotiating between different representations of the same resource is content negotiation, and the HTTP way of doing that is the Accept header, not something else. Don’t reinvent the wheel.
I think it gets nuanced pretty quickly… Is it the same resource if what is returned has different structure? How different does it need to be such that you consider it a different resource?
Also, changing headers is harder (though not that hard) for clients. Often making your API easier to use for others trumps doing it correctly. Changing v1 to v2 in the URL (which I have to specify always) is easier than having to change na Accept header which you’re now making me set, and I otherwise might not have to.
You are confusing resource and representation. The URI is the resource. The HTTP body that gets returned is the representation. They are separate concerns. Choosing between represenations is content negotiation.
Changing the URI to indicate a change in representation is the wrong place according to HTTP. It’s like putting file extensions in your URL instead of using media types. You can do it, but you’re misusing the tools that the protocol has given you.
I don’t think I am confused, just poorly explaining my point of view. If a resource that represents a user switches from containing a user ID that is an int to one that’s a UUID, are those the same resource? I am arguing they are not.
I agree that if that resource is sent back as JSON or serialized protobuf, then it is the same resource in different representations. However, I do not think it is an abuse of the HTTP protocol to assign different URIs for a resource that has evolved/changed as the service has. Using a version number in the URI to indicate this change of the resource seems logical, intuitive, and valid with respect to the protocol.
The ID field is special in this regard, because it most likely indicates a change of URI, and that is why this would be considered a new resource, not the change of representation.
Let’s alter your example to exclude this ambiguity. Let’s say that you’re making a breaking change in the structure of the user data from
Is this the same resource? Clearly, yes. You are just changing the representation. This is when I would increment the API Version. Changes that are strictly additive don’t break a contract, but changing an existing field that a client may depend on does. The thing you are communicating in the Version is that the contract has changed.
The URI of this record should not change. The relationships of this User to other things has not changed, the identity of the User has not changed, merely your representation of its data. Needing to alter an unrelated property such as the URI is a violation of concerns. URIs should only change when the identity of the resource changes.
This is the principle of encapsulation. Things that are independent should not have to change in tandem. That means there is a conflation of responsibilities happening.
EDIT: Oh I’d do dispatch where it made sense. Probably in the application unless different versions are because the entire application has to change for some reason (new provider or something). That way all the possible answers to your Api call are in the same place.
Just was wondering whether it has the same issue as with ZFS encryption only supporting one key, and yes it does and it’s well documented over at the becachefs wishlist.
Just was wondering whether it has the same issue as with ZFS encryption only supporting one key
What’s the issue? ZFS supports per-dataset keys. One of the intended use cases was having per-user encrypted home directories that are decrypted only when the user’s passphrase is entered (or when they provide a PIN to a TPM, or similar).
Yeah, I feel like the solution here is to encrypt the key multiple times. You have the hdd encryption key, and then every time you want to add a new key you just encrypt the hdd encryption key with a new key.
I think this is how most HDD encryption works, since you don’t need to re-encrypt the data when you rotate keys, you just re-encrypt the initial key with your new key.
ZFS actually works like this: disk blocks are not encrypted with the key that you provide, they’re encrypted with a key that’s encrypted with the key that you provide (well, actually, with a key derived from a key that is encrypted with the one that you provide, I believe). Allowing a second key to be used to encrypt the dataset encryption key seems useful (especially when rotating keys: add the new key, check you got it right, delete the old key).
I’m not sure if this is possible with ZFS. I think the space for the master keys is fairly configurable, it’s only the key index for the dataset keys that reuses some other space in metadata.
At $work the workstations have one key that’s tied to local identity (a passphrase entered on the physical keyboard) and another that’s in escrow. The escrow keys are used to unlock based on corporate policy authentication - ie: remotely unlocking based on my corporate credentials. The escrow key access policies can of course be set up for whatever makes sense for the organization.
I think it’s using lvm2 & luks but I’ve never looked that closely.
Yes, fallback Keys or human key + automatic one etc can be crucial. Also switching keys becomes a slightly safer operation I’d wanted.
Also just wanted to share the link itself, it gives quite a good view on for what it’s ready to use at the first time of merging. Roadmap part is also interesting, since they talk about things like more efficient squashes alternative
Stream start. Hello. People using Macs in professional settings and schools. Porsche, Girls Who Code, medical environments, artists, etc…
Spooky graphics. Tim Cook at night to talk about the Mac. Laptops gained bigly. New chips in MacBook Pro.
HERE’S JOHNY! Spooky lab. Yes, we’ve got the best chips. Next-generation AS. Three chips at the same time. M3, M3 Pro, M3 Max. 3nm. EUV. Big GPU advancement. New µarch, “Dynamic Caching”. Non-static scheduling, memory dynamically allocated by hardware, improving average utilization. Dev-transparent. Mesh shading in hardware. Hardware ray-tracing. 2.5x render perf in pro apps? CPU cores. 30% faster P-cores, with same efficiency. 50% for E-cores. Same perf as M1 in half the power for both CPU and GPU. Still smoking x86, quarter of the power usage for same perf. 16-core, 15% faster ANE. AV1 decode support. M3 has 8-core CPU (4/4), 10-core GPU, 24 GB max. M2 Pro is 6/6, 18-core GPU, 36 GB max. M2 Max is 12/4, 40-core GPU, 128 GB of RAM.
HERE’S JOHN! To talk about products using these chips. MacBook Pro.
HERE’S…. Kate. To talk about the new MBP. 14” has M3 in base model. RIP 13”. M3 Pro models in 14”/16”. Two external displays on M3 Pro. M3 Max models. 2.5x Cinema 4D perf. Two ProRes engines on M3 Max. M3 Max can drive 4 external displays. Same perf on battery. 22 hours of battery life. 11x faster than fastest x86 MBP. Silent most of the time. 1000 sustained/1600 peak HDR nits on the display, just in case you didn’t know. SDR is 600 nits. New colour. Black? Space black. Dark aluminum. New chemistry that’s fingerprint resistant. 100% recycled apparently. Yes, it runs macOS 14. New show on Apple TV+, in case you have that. macOS 14 has higher quality screen sharing and game mode, in case you didn’t know that. MacBook Pros are good for people who do things, in case you didn’t know that. You can carry it around on battery doing stuff, yes.
HERE’S JOHN! Base model 14” (with M3, replacing 13”) is now 1599$. 16” is same price. M3 Max is later in November, everything else next week. 24” iMac gets M3. It has colours, unlike the other ones. M1->M3 is 2x faster. Yes, it curbstomps Intel. Same price. Order today, available next week.
Those performance improvements aren’t too shabby, though the downgrading of the M3 Pro is kinda odd. Went from 4 to 3 memory modules, meaning a 25% reduction in memory bandwidth, and went from a 8+4 CPU config to 6+6. Will definitely still be faster in many tasks, but some tasks will surely regress going from M2 Pro to M3 Pro.
on the space black, I was really hoping it would be a vibrant deep purple.
Widgets on the desktop seems like OSX Tiger’s “Dashboard” has been flattened onto the desktop instead of being an overlay.
Seems to be a “call out your normal apps for $thing” mandate. Software I’m sure I’d recognize if I were a pro audio editor, or gene sequencer. And XCode.
Huh, no Mac Pro update. But is the music supposed to be a bit halloweeny?
The various clips are already on youtube from the Apple account.
Widgets are the eternal idea. Every ten years, someone comes out with a new variation on them. They always have a small pocket of die hards, but most people ignore them, and then they go away again. Whether it’s Desk Accessories or Active Desktop or Dashboard or Tiles or whatever, they just never have any staying power.
Purely an anecdote, but I think the widgets feature on the iPhone will have some staying power, at least. Whenever I’m at a sporting event or concert or something I can’t help but glance at the phone screens of the people in front of me, and home screen widgets seem to be surprisingly popular. I think it’s mostly for aesthetic purposes (alongside the Shortcuts feature that lets you effectively change app icons), but it’s cool to see nonetheless.
Macs are a much more niche product and the customized desktop sharing posts on social media are somewhat less of a thing than iPhone home screen design tutorials, so I’m not sure the macOS version will catch on quite the same. Who knows, though. Maybe the idea has just been ahead of its time for the last 40 years and its real application is on phones, not desktops!
Shell scripts are best for programs that need to glue together a number of different programs together, or when you have no choice (environmental constraints).
If your code is mostly gluing together other programs, it may be best to stay in shell. Your logic however should get out of shell as quickly as possible. Can you turn your logic into another program? Can you “shell out” to python, or perl, or any other useful language for the nifty bits?
A good shell program is a lot like any other program. It’s all about your organization and discipline in authorship.
Yeah exactly, the fallacy is that it’s shell XOR something else. Basically ALL my shell scripts call Python scripts, and about half those Python scripts I wrote myself!
That’s how shell is designed to be used – factoring policy and mechanism.
The way I think of it is
policy / the “control plane” / shell
the values of command line flags (parameters)
auth parameters are policy, e.g. staging vs. prod
a set of machines is policy
parallelism is policy
retrying after failure is policy
mechanism / the “data plane” / Python / C++
e.g. how you physically copy from one system to another
how you check if a service is alive
A litmus test for this style of factoring: Don’t put file system paths or port numbers in Python scripts. Put them in shell scripts (or a config file that lives outside the application).
Unfortunately I think I never wrote a post about this, but I specifically mention the biggest misconception about shell XOR Python here:
What I will concede is that it’s harder to program in 2 languages at once (shell and Python) than to use one language.
Using multiple languages, and “factoring” them, is a skill. It takes practice and thought. But the benefit is having less code overall, and that code is more flexible and reusable – as coarse-grained processes.
(Though if you call os.system() from Python, now you’re back to 2 languages again :-) So I use shell as the main() and Python as “subroutines”, which can be concurrent concurrently.)
The other thing to conceded is that shell has some nearly fatal flaws like bad error handling.
By and large I think this post makes a reasonable point:
The instability most often comes from the inability (or difficulty in) adding sufficient test coverage. Yes, there are frameworks for bash script testing! I’ve rarely seen them effectively used.
I agree that the status quo for testing isn’t enough. But if that’s the only barrier, then shell is in a good place!
But I actually think a bigger barrier is the shell language itself. It’s powerful but many things about it actively inhibit learning (error messages and inconsistent syntax) … The saving grace is that it’s fast and predictable (if you pay close attention to details!)
Which is probably why these bills are heavily framed as being about “child safety”. No politician wants to give the impression that they’re against that.
Well yes, that is a given. It is a lawmaker’s prerogative to frame the laws they wish to pass in a way that maximizes the probability of them passing.
The “pro-encryption” camp (which I belong to, btw) needs better arguments than just shrugging their shoulders and saying “yes, encryption protects bad people too, but we believe the trade-off is worth it”. For many people who are not versed in the minutiae of online communication, the trade-off is not worth it. They want the bad guys to be caught and punished!
It’s very hard to argue against, because it relies on fundamental rights like privacy, which many people don’t care about in the first place, and is too nebulous to really nail down. Maybe you could argue something like “some pervert at Meta will also be able to see all your sextings”?
Most of the campaigning against CSAR (which was at least significantly amended) focused on the proposed solutions not being particularly effective, and the idea that client-side scanning would drown the already insufficient police forces in low-quality reports.
But EU directives aren’t really subject to the same media attention that laws on the national level are. The campaigning was aimed at convincing lawmakers, not the general public.
I think a lot of the resistance comes from an ambient sense that tech companies and governments are bad guys. This view is probably a lot more widespread than media would admit, but the taboo on making that argument no doubt has an effect.
In my bog standard infrastructure, I do containers on nomad with vault and consul in case I need to do a service mesh later. Single node nomad is, operationally at least, nicer than docker compose.
Sorry in advance for the long-winded context-setting, but it’s the only way I know how to answer this question.
There are a few important things to understand. First, even though HealthCare.gov was owned by CMS, there was no one single product owner within CMS. Multiple offices at CMS were responsible for different aspects of the site. The Office of Communications was responsible for the graphic design and management of the static content of the site, things like the homepage and the FAQ. The Center for Consumer Information and Insurance Oversight was responsible for most of the business logic and rules around eligibility, and management of health plans. The Office of Information Technology owned the hosting and things like release management. And so on. Each of these offices has their own ability to issue contracts and set of vendors they prefer to work with.
Second, HealthCare.gov was part of a federal health marketplace ecosystem. States integrated with their own marketplaces and their Medicaid populations. Something called the Digital Services Hub (DSH) connected HealthCare.gov to the states and to other federal agencies like IRS, DHS, and Social Security, for various database checks during signup. The DSH was its own separately procured and contracted project. An inspector general report I saw said there were over 60 contracts comprising HealthCare.gov. Lead contractors typically themselves subcontract out much of the work, increasing the number of companies involved considerably.
Then you have the request for proposals (RFP) process. RFPs have lists of requirements that bidding contractors will fulfill. Requirements come from the program offices who want something built. They try to anticipate everything needed in advance. This is the classic waterfall-style approach. I won’t belabor the reasons this tends not to work for software development. This kind of RFP rewards responses that state how they will go about dutifully completing the requirements they’ve been given. Responding to an RFP is usually a written assertion of your past performance and what you claim to be able to do. Something like a design challenge, where bidding vendors are given a task to prove their technical bona fides in a simulated context, while somewhat common now, was unheard of when HealthCare.gov was being built.
Now you have all these contractors and components, but they’re ostensibly for the same single thing. The government will then procure a kind of meta-contract, the systems integrator role, to manage the project of tying them all together. (CGI Federal, in our case.) They are not a “product owner”, a recognizable party accountable for performance or end-user experience. They are more like a general contractor managing subs.
In addition, CMS required that all software developed in-house conform to a specified systems design, called the “CMS reference architecture”. The reference architecture mandated things like: the specific number of tiers or layers, including down to the level of reverse proxies; having a firewall between each layer; communication between layers had to use a message queue (typically a Java program) instead of say a TCP socket; and so forth. They had used it extensively for most of the enterprise software that ran CMS, and had many vendors and internal stakeholders that were used to it.
Finally, government IT contracts tend to attract government IT contractors. The companies that bid on big RFPs like this are typically well-evolved to the contracting ecosystem. Even though it is ostensibly an open marketplace that anyone can bid on, the reality is that the federal goverment imposes a lot of constraints on businesses to be able to bid in the first place. Compliance, accreditation, clearance, accounting, are all part of it, as well as having strict requirements on rates and what you can pay your people. There’s also having access to certain “contracting vehicles”, or pre-chosen groups of companies that are the only ones allowed to bid on certain contracts. You tend to see the same businesses over and over again as a result. So when the time comes to do something different – eg., build a modern, retail-like web app that has a user experience more like consumer tech than traditional enterprise government services – the companies and talent you need that has that relevant experience probably aren’t in the procurement mix. And even if they were, if they walked in to a situation where the reference architecture was imposed on them and responsibility was fragmented across many teams, how likely would they be to do what they are good at?
tl;dr:
No single product owner within CMS; responsibilities spread across multiple offices
Over 60 contracts just for HealthCare.gov, not including subcontractors
Classic waterfall-style RFP process; a lot of it designed in advance and overly complicated
Systems integrator role for coordinating contractors, but not owning end-user experience
CMS mandated a rigid reference architecture
Government IT contracts favor specialized government contractors, not necessarily best-suited for the job
At any point did these people realize they kinda sucked, technically? Did they kinda suck?
I think they saw quickly from the kind of experience, useful advice, and sense of calm confidence we brought – based on knowing what a big transactional public-facing web app is supposed to look like – that they had basically designed the wrong thing (the enterprise software style vs a modern digital service) and had been proceeding on the fairly impossible task of executing from a flawed conception. We were able to help them get some quick early wins with relatively simple operational fixes, because we had a mental model of what an app that’s handling high throughput with low latency should be, and once monitoring was in place, going after the things that were the biggest variance from that model. For example, a simple early thing we did that had a big impact was configuring db connections to more quickly recycle themselves back into the pool; they had gone with a default timeout that kept the connection open long after the request had been served, and since the app wasn’t designed to stuff more queries into an already-open connection, it simply starved itself of available threads. Predictability, clearing this bottleneck let more demand flow more quickly through the system, revealing further pathologies. Rinse and repeat.
Were there poor performers there? Of course. No different than most organizations. But we met and worked with plenty of folks who knew their stuff. Most of them just didn’t have the specific experience needed to succeed. If you dropped me into a mission-critical firmware project with a tight timeline, I’d probably flail, too. For the most part, folks there were relieved to work with folks like us and get a little wind at their backs. Hard having to hear that you are failing at your job in the news every day.
Did anything end up happening to the folks that bungled the launch? Were there any actual monetary consequences?
You can easily research this, but I’ll just say that it’s actually very hard to meaningfully penalize a contractor for poor performance. It’s a pretty tightly regulated aspect of contracting, and if you do it can be vigorously protested. Most of the companies are still winning contracts today.
What sort of load were y’all looking at?
I would often joke on the rescue that for all HealthCare.gov’s notoriety, it probably wasn’t even a top-1k or even 10k site in terms of traffic. (It shouldn’t be this hard, IOW.) Rough order of mag, peak traffic was around 1,000 rps, IIRC.
What was the distribution of devices, if you remember?
You could navigate the site on your phone if you absolutely had to, but it was pretty early mobile-responsive layout days for government, and given the complexity of the application and the UI for picking a plan, most people had to use a desktop to enroll.
What were some of the most arcane pieces of tech involved in the project?
Great question. I remember being on a conference bridge late in the rescue, talking with someone managing an external dependency that we were seeing increasing latency from. You have to realize, this was the first time many government agencies where doing things we’d recognize as 3rd party API calls, exposing their internal dbs and systems for outside requests. This guy was a bit of a grizzled old mainframer, and he was game to try to figure it out with us. At one point he said something to the effect of, “I think we just need more MIPS!” As in million instructions per second. And I think he literally went to a closet somewhere, grabbed a board with additional compute and hot-swapped it into the rig. It did the trick.
In general I would say that the experience, having come from what was more typical in the consumer web dev world at the time - Python and Ruby frameworks, AWS EC2, 3-tier architecture, RDBMSes and memcached, etc. - was like a bizarro world of tech. There were vendors, products, and services that I had either never heard of, or were being put to odd uses. Mostly this was the enterprise legacy, but for example, code-generating large portions of the site from UML diagrams: I was aware that had been a thing in some contexts in, say, 2002, but, yeah, wow.
To me the biggest sin, and I mean this with no disrespect to the people there who I got to know and were lovely and good at what they did, was the choice of MarkLogic as the core database. Nobody I knew had heard of it, which is not necessarily what you want from what you hope is the most boring and easily-serviced part of your stack. This was a choice with huge ramifications up and down, from app design to hardware allocation. An under-engineered data model and a MySQL cluster could easily have served HealthCare.gov’s needs. (I know this, because we built that (s/MySQL/PostgreSQL) and it’s been powering HealthCare.gov since 2016.)
A few years after your story I was on an airplane sitting next to a manager at MarkLogic. He said the big selling point for their product is the fact it’s a NoSQL database “Like Mongo” that has gone through the rigorous testing and access controls necessary to allow it to be used on [government and military jobs or something to that effect] “like only Oracle before us”.
He’s probably referring to getting FedRAMP certified or similar, which can be a limiting factor on what technologies agencies can use. Agencies are still free to make other choices (making an “acceptable risk” decision).
In HealthCare.gov’s case, the question wasn’t what database can they use, but what database makes the most sense for the problem at hand and whether a document db was the right type of database for the application. I think there’s lots of evidence that, for data modeling, data integrity, and operational reasons, it wasn’t. But the procurement process led the technology choices, rather than the other way round.
I’m not Paul, but I recall hearing Paul talk about it one time when I was his employee, and one of the problems was MarkLogic, the XML database mentioned in the post. It just wasn’t set up to scale and became a huge bottleneck.
Paul also has a blog post complaining about rules engines.
Have you found something else as meaningful for you since then?
Yes, my work at Ad Hoc - we started the company soon after the rescue and we’ve been working on HealthCare.gov pretty much ever since. We’ve also expanded to work on other things at CMS like the Medicare plan finder, and to the Department of Veterans Affairs where we rebuilt va.gov and launched their flagship mobile app. And we have other customers like NASA and the Library of Congress. Nothing will be like the rescue because of how unique the circumstances, but starting and growing a company can be equally as intense. And now the meaning is found in not just rescuing something but building something the right way and being a good steward to it over time so that it can be boring and dependable.
Also what’s your favorite food?
A chicken shawarma I had at Max’s Kosher Café (now closed 😔) in Silver Spring, MD.
This is only slightly sarcastic. I’m hopeful with your successful streak you can target the EDI process itself, because it’s awful. I only worked with hospice claims, but CMS certainly did a lot to make things complicated over the years. I only think we kept up because I had a well-built system in Ruby.
It’s literally the worst thing in the world to debug and when you pair that with a magic black box VM that randomly slurps files from an NFS share, it gets dicey to make any changes at all.
For those not familiar, X12 is a standard for exchanging data that old-school industries like insurance and transportation use that well-predates modern niceties like JSON or XML.
X12 is a … to call it a serialization is generous, I would describe it more like a context-sensitive stream of transactions where parsing is not a simple matter of syntax but is dependent what kind of X12 message you are handling. They can be deeply nested, with variable delimiters, require lengthy specifications to understand, and it’s all complicated further by versioning.
On top of that, there is a whole set of X12-formatted document types that are used for specific applications. Relevant for our discussion, the 834, the Benefit Enrollment and Maintenance document, is used by the insurance industry to enroll, update, or terminate the coverage.
To give you a little flavor of what these are like, here is the beginning of a fake 834 X12 doc:
ISA*00* *00* *ZZ*CMSFFM *ZZ*54631 *131001*1800*^*00501*000000844*1*P*:~
GS*BE*NC0*11512NC0060024*20131001*1800*4975*X*005010X220A1~
ST*834*6869*005010X220A1~
BGN*00*6869*20131001*160730*ET***2~
QTY*TO*1~
QTY*DT*0~
N1*P5*Paul Smith*FI*123456789~
N1*IN*Blue Cross and Blue Shield of NC*FI*560894904~
INS*Y*18*021*EC*A***AC~
REF*0F*0000001001~
REF*17*0000001001~
REF*6O*NC00000003515~
DTP*356*D8*20140101~
NM1*IL*1*Smith*Paul****34*123456789~
PER*IP**TE*3125551212*AP*7735551212~
N3*123 FAKE ST~
N4*ANYPLACE*NC*272885636**CY*37157~
HealthCare.gov needed to generate 834 documents and send them to insurance carriers when people enrolled or changed coverage. Needless to say, this did not always go perfectly.
I personally managed to avoid having to deal with 834s until the final weeks of 2013. An issue came up and we need to do some analysis across a batch of 834s. Time was of the essence, and there was basically no in-house tooling I could use to help - it only generated 834s, not consume them. So naturally I whipped up a quick parser. It didn’t have to be a comprehensive, fully X12-compliant parser, it only needed enough functionality to extract the fields needed to do the analysis.
So I start asking around for anyone with 834 experience for help implementing my parser. They’re incredulous: it can’t be done, the standard is hundreds of pages long, besides you can’t see it (the standard), it costs hundreds of dollars for a license. (I emailed myself a copy of the PDF I found on someone’s computer.)
I wrote my parser in Python and had the whole project done in a few days. You can see a copy of it here. The main trick I used was to maintain a stack of segments (delimited portions of an X12 doc) and the in the main loop would branch to methods corresponding to the ID of the current segment, allowing context-sensitive decisions to be made about what additional segments to pull off the stack.
The lesson here is that knowing how to write a parser is a super power that will come in handy in a pinch, even if you’re not making a compiler.
just wanted to give you props. I’m a contractor myself and I’ve been on both sides of this (I have been in the gaggle of contractors and been one of the people they send in to try to fix things) and neither side is easy, and I’ve never come close to working on anything as high profile as this and even those smaller things were stressful. IME contracting is interesting in some ways (I have worked literally everywhere in the stack and with tons of different languages and environments which I wouldn’t have had the opportunity to do otherwise) and really obnoxious in other ways (legacy systems are always going to exist but getting all the contractors and civilian employees pulling in the same direction is a pain, especially without strong leadership from the government side)
Thanks, and back at ya. My post didn’t get into it, but it really was a problem of a lack of a single accountable product lead who was empowered to own the whole thing end-to-end. Combine that with vendors who were in a communications silo (we constantly heard from people who had never spoken to their counterparts other teams because the culture was not to talk directly except through the chain of command) and the result was, everyone off in their corner doing their own little thing, but no one with a view to the comprehensive whole.
Extremely. Aside from the performance-related problems that New Relic gave us a handle on, there were many logic bugs (lots of reasons for this: hurried development, lack of automated tests, missing functionality, poorly-understood and complicated business rules that were difficult to implement). This typically manifested in users “getting stuck” partway through somewhere, and they could neither proceed to enrollment or go back and try again.
For example, we heard early on about a large number of “lost souls” - people who had completed one level of proving your identity (by answering questions about your past ala a credit check), but due to a runtime error in a notification system, they never got the link to go to the next level (like uploading a proof of ID document). Fixing this would involve not just fixing the notification system, but figuring out which users were stuck (arbitrary db queries) and coming up with a process to either notify them (one-off out-of-band batch email) or wipe their state clean and hope they come back and try again. Or if they completed their application for the tax credit, they were supposed to get an official “determination of eligibility” email and a generated PDF (this was one of those “requirements”). But sometimes the email didn’t get sent or the PDF generator tipped over (common). Stuck.
The frontend was very chatty over Ajax, and we got a report that it was taking a long time for some people to get a response after they clicked on something. And we could correlate those users with huge latency spikes in the app on New Relic. Turns out for certain households, the app had a bug where it would continually append members of a household to a list over and over again without bound after every action. The state of the household would get marshalled back and forth over the wire and persisted to the db. Due to a lack of a database schema, nothing prevented this pathological behavior. We only noticed it because it was starting to impact user experience. We went looking in the data, and found households with hundreds of people in them. Any kind of bug report was just the tip of the iceberg of some larger system problem and eventual resolution.
I have a good story about user feedback from a member of Congress live on TV that led to an epic bugfix. I’ve been meaning to write it up for a long while. This anniversary is giving me the spur to do it. I’ll blog about it soon.
I have two young kids (6 and 3.5 years old) and have felt like that for the last 6 years at least. It does feel like it’s slowly getting less busy, so some space for hobbies and “useless” projects is opening up again.
I have a 5 and 3, and it feels like I’m slowly starting to get some time back as they get older. It helps when you can leave them in a room by themselves and trust them to not eat the paint chips anymore. 😂
I also have a young family and lament my lack of good free time. I go through phases. I’ll pour my free evening hours into playing games on my Switch for a month, and then I get the itch and spend a month working on a side project on my laptop. Both can happen while watching whatever with my wife on the couch, which is nice. It can be hard to accomplish a lot when I’ve only got an hour or two a night, but it’s still enough to scratch that itch and make me feel less trapped (by job and family and circumstance).
When you have young children, any activity that doesn’t contribute directly to your basic sustenance is time and energy stolen away from a better parent you could be for them. Especially so in these economically and politically turbulent times when you can’t set financial goals, so you have to aim to save as much as possible in order to reduce the chance of your worst-case outcomes. When you’re childless, there’s a wide range of options to weather through hard times, but you really can’t risk compromising on life’s necessities when you have children.
Absolutely. I find that thoughts of finances are keeping me awake now and then, while it never ever has in the past. And that’s with a handsome salary for a comfy dev job. I can’t imagine how tough life is with kids for people who earn more basic salaries.
We have half a dozen of these cluttering up our office and I wasn’t sure what to do with them
we run the website of our hackers youth club on one of those. From SD-card. The one from 2012 :-)
I’ll take some off your hands! I’ll pay shipping!
And then you’ll be surprised by the libc dependency. Or the library that’s a surprise transient dependency. Or the kernel API that some day will be changed exactly breaking what you expect.
Perhaps this statement is better: Everything that’s not Bedrock should be a saved artifact input to your build.
This is exactly what Nix[OS] does and is great at.
This times a thousand. I have tried to deploy self-hosted apps that were either only distributed as a Docker image, or the Docker image was obviously the only way anyone sane would deploy the thing. Both times I insisted on avoiding Docker, because I really dislike Docker.
For the app that straight up only offered a Docker image, I cracked open the
Dockerfilein order to just do what it did myself. What I saw in there made it immediately obvious that no one associated with the project had any clue whatsoever how software should be installed and organized on a production machine. It was just, don’t bother working with the system, just copy files all over the place, oh and if something works just try symlinking stuff together and crap like that. The entire thing smelled strongly of “we just kept trying stuff until it seemed to work”. It’s been years but IIRC, I ended up just not even bothering with theDockerfileand just figuring out from first principles how the thing should be installed.For the service where you could technically install it without Docker, but everyone definitely just used the Docker image, I got the thing running pretty quickly, but couldn’t actually get it configured. It felt like I was missing the magic config file incantation to get it to actually work properly in the way I was expecting to, and all the logging was totally useless to figure out why it wasn’t working. I guess I’m basically saying “they solved the works-on-my-machine problem with Docker and I recreated the problem” but… man, it feels like the software really should’ve been higher quality in the first place.
That’s always been a problem, but at least with containers the damage is, well, contained. I look at upstream-provided packages (RPM, DEB, etc) with much more scrutiny, because they can actually break my system.
Can, but don’t. At least as long as you stick to the official repos. I agree you should favor AppImage et al if you want to source something from a random GitHub project. However there’s plenty of safeguards in place within Debian, Fedora, etc to ensure those packages are safe, even if they aren’t technologically constrained in the same way.
I didn’t say that. Edit: to be a bit clearer. The risky bits of a package aren’t so much where files are copied, because RPM et al have mechanisms to prevent one package overwriting files already owned by another. The risk is in the active code: pre and post installation scripts and the application itself. From what I understand AppImage bundles the files for an app, but that’s not where the risk is; and it offers no sandboxing of active code. Re-reading your comment I see “et al” so AppImage was meant as an example of a class. Flatpak and Snap offer more in the way of sandboxing code that is executed. I need to update myself on the specifics of what they do (and don’t do).
Within Debian/Fedora/etc, yes: but I’m talking about packages provided directly by upstreams.
Regardless of which alternative, this was also my point. In other words, let’s focus on which packagers we should look at with more scrutiny rather than which packaging technology.
AppImage may have been a sub-optimal standard bearer but we agree the focus should be on executed code. AppImage eliminates the installation scripts that are executed as root and have the ability to really screw up your system. AppImage applications are amenable to sandboxed execution like the others but you’re probably right that most people aren’t using them that way. The sandboxing provided by flatpak and snap do provide some additional safeguards but considering those are (for the most part) running as my user that concerns my personal security more than the system as a whole.
On the other side, I’ll happily ignore the FHS when deploying code into a container. My python venv is /venv. My application is /app. My working directory… You get the picture.
This allows me to make it clear to anybody examining the image where the custom bits are, and my contained software doesn’t need to coexist with other software. The FHS is for systems, everything in a dir under / is for containers.
That said, it is still important to learn how this all works and why. Don’t randomly symlink. I heard it quipped that Groovy in Jenkins files is the first language to use only two characters: control C and control V. Faking your way through your ops stuff leads to brittle things that you are afraid to touch, and therefore won’t touch, and therefore will ossify and be harder to improve or iterate.
I got curious so I actually looked up the relevant
Dockerfile. I apparently misremembered the symlinking, but I did find this gem:There was also unauthenticated S3 downloads that I see I fixed in another PR. Apparently I failed to notice the far worse unauthenticated shell script download.
I’ve resorted to spinning up VMs for things that require docker and reverse proxying them through my nginx. My main host has one big NFTables firewall and I just don’t want to deal with docker trying to stuff its iptable rules in there. But even if you have a service that is just a bunch of containers it’s not that easy because you still have to care about start, stop, auto-updates. And that might not be solved by just running Watchtower.
One case of “works on docker” I had was a java service that can’t boot unless it is placed in
/home/<user>and has full access. Otherwise it will fail, and no one knows why springboot can’t work with that, throwing a big fat nested exception that boils down tojava.lang.ClassNotFoundExceptionfor the code in the jar itself.Another fun story was when I tried to setup mariadb without root in a custom user.
I missed this when it was first posted (here) but building a PCB has been on my bucket list. Is there any advice that the hardare hackers here on Lobsters would give to make this process less error prone for a first timer?
Use the DRC
double check your transistor orientations
Double check all part footprints and orientations.
Have a target overall size in mind first.
Check multiple parts stores and track which you are using
Save your project in git or other VCS so you can undo experiments.
Silkscreen: all readable without rotating the board, and readable even with all parts populated. You want to know which component is where even after you solder it.
Add test points anywhere you are not certain about the signal.
Schematic design comes first. Get that part right and the PCB bits can be closer to an OCD Zen Garden.
Rule lists like this wouldn’t be any good if none of them felt at least a little bit controversial.
Rule #5 I disagree with. I hate APIs that return clear map structures with “key”s inside, but I cannot lookup by the key because it’s a list and not a map. Just use a map when the data fits into that pattern.
Rule #8. 404 is fine for an API because all the invalid endpoints can just return 5xx. If the client needs further information, you can return an error code that never changes along with the response body.
I think another way to say Rule 5 might be: return objects with well-defined schemas. Or: given a specific key, its corresponding value should always be the same type. So if you return
{"user":"John"}in one situation, it’s not cool to return{"user":123}in a different situation.I’ve heard that list returns also require breaking changes to make any improvements. It’s a “Probably Gonna Need It” to use a map.
Was there also something that broke if you returned a list instead of a map, or was that just a vague memory?
The hatred of C here is not reasonable. It seems that the author wanted to dunk on the language, but other languages would have known that it was possible to reach all statements in that pseudocode and thus I can’t imagine other languages helping.
Some pretty big tech changes:
Why are you leaving all those previous tech choices on your laptop? Conforming to the crowd a bit more?
Inadvertently, yes.
My plan is to virtualize all of my development environments. So my work will be primarily over SSH, SSH tunnels, and perhaps tramp (in Emacs). I’m setting that up with bhyve, using the vm-bhyve scripts for convenient management.
This all runs on a server in my hallway cabinet - along with a brace of RPis offering utility stuff like OpenVPN, Pihole, and Home Assistant. Plus one Pi running FreeBSD to serve my Web and Gemini stuff.
My laptop will then only be for media, mail, and chat - at which point there’s no significant advantage to FreeBSD, I can just set up whatever cheap laptop with Mint and use it as a thin client. At the moment that’s a ThinkPad T470s with a docking station on my desk.
Is your server headless or do you run a desktop environment with it as well?
Headless, at least at the moment.
Me: Tracking my finances is hard, there’s lots of steps and it’s time consuming. I wonder if nerds have figured out a better way…
Article: Track your finances manually with a verbose data format!
Me: Maybe I’ll just check my bank account once a week like I do now…
Why concentrate on managing your finances when you can reinvent double-entry bookkeeping from first principles!
Every nerd who teaches themselves financial literacy builds their own double-entry bookkeeping tool.
The Lost Wisdom of the Ancient Renaissance Bankers.
I preferred Business Secrets of the Pharaohs.
Why is this me?
hah Maybe if I understood double-entry bookkeeping, a lot of these discussions would make more sense.
A git diff shows you what has been removed, and what has been added, and where. That’s double-entry bookkeeping
The point of double-entry bookkeeping isn’t the ledger (the diff) but the whole way credit balance accounts, the balance sheet and the accounting equation work together.
Yeah I think of the main principle as more that transactions always add up to 0. I.e. the conservation of money: Money is never created or destroyed, just moved between accounts. (Though revenue and expenses are both treated as bottomless and effectively “create” money in that sense).
I only regret I have but one upvote to give.
The second step is “and your diffs are every transaction, instead of every month”
Very interesting analogy, I wonder if you could use this to actually build an accounting system with git?
It’s virtually impossible to create a future-proof way to solve this, given the (natural) difficulty of integrating with loads of different banks around the world, without doing some manual data entry. However, I should’ve clarified in the article that I do not recommend you enter data manually in TOML, but rather do some importing based on e.g. CSV statements. The advantage of TOML is that it is easy to read, edit and script around.
Oh yeah, that makes a lot more sense hah. My reply was in reaction to thinking that’s what you were proposing. Automating it in another way and just using TOML as the backing store makes a lot more sense.
I totally get your confusion! I should’ve been a bit more clear about that
My initial impression was too that you offer to enter data in TOML and by the end of your article you would offer some convertor from TOML to ledger, hledger and/or beancount formats.
If you do not recommend entering data manually in TOML and if I gather it right, you argue for storing data in TOML, because there are tons of robust libraries to work with this human-readable and machine-friendly format. Hmm? The article title is a bit clickbait-y then (-;
I believe, some of the mentioned PTA software have plug-in systems, perhaps re-usable parser code, and some sort of export to some sort of CSV? How that doesn’t solve problems you are trying to solve? So far to me it seems that you add a superfluous layer in between the wild world of financial data formats and PTA. Although, you said that it was for your personal use, so I guess it’s totally okay (-:
There are plugin systems and reusable parser code, yes, but they often lock you into a certain programming language and I find that libraries for parsing common data formats like TOML often are of higher quality.
As for the title; yes, it’s clickbait
Mint getting shut down has been a bit of a nightmare scenario for me as that’s where everything I have is budgeted.
While I do agree with the author’s arguments against WAFs, I find the solutions quite disappointing as they require direct changes to the application, which isn’t always under your control.
I’m part of the ops team and in charge of securing the application access, not the application itself, and a WAF would let me do that (albeit in a clunky and painful way).
I’m quite curious about that arrangement!
Does the dev team carry the pager? If not, do you have difficulty getting them to fix issues that cause you to get paged?
Are the dev team also responsible for securing the application? If not, how do you get security issues fixed by the development team?
What if it’s a vendor’s application?
Then it seems especially difficult to get alignment between dev and ops, and I’m especially curious how others go about making that work!
I’m running one vendors application presently (mattermost). We have the source and a CI/CD pipeline; if some new exploit becomes apparent, I’m able to adjust the code myself, as well as escalating to the vendor, but as far as I can tell that’s a very rare state of affairs.
Of course the dev team is responsible for securing the application, and we’re working hard with them so that security gets taken into account from day one when they spin up a new microservice.
But life isn’t always that easy, and there are many legacy stuff that is either abandonware or so old that nobody dares changing anything to it, let alone refactor the code to bring better security to it.
Also no, they don’t carry the pager, and my point is that a WAF is a solution I can deploy on my own to increase the security of an application, and not wait to be breached to raise an issue to the dev team that will maybe fix it on the next sprint, because priorities. Chug
I’m guessing the WAFs you’re talking about are looking at the HTTP header (path, with, whatever) and maybe source IP address? That sounds almost more like a proxy with access controls. The article mostly complains about the WAFs which run a bunch of regexes on the request body. If you do make use of the body inspection features I’m curious about how.
(Also hi z3bra :)
The WAF I’m talking about is fairly advanced, and provides an “auto-learning” feature which will eat thousands of requests to build up an allow list based on the tokenization of the accesses (both URL, headers and body). It also has access to the OWASP rulesets which features many regexes and tokens matches, etc…
And to be fair, I do agree that WAF rules are very complex, can be bypassed, and not self-sufficient to provide security for an application. But as an ops, I can set it up to increase the application security with no (or little) adherence with the dev team.
Goguma, the soju bouncer and the senpai IRC have transformed my IRC life. Up until recently I’d been a very very long ZNC and weechat user.
These new tools have modernised my whole setup while still being quite unixy, and their configs and admin are much easier to manage.
Thank you for the list, I’m looking into making my own znc+weechat setup better and switching out the tools might be really good.
Goguma
Soju
Senpai
Looks like it won’t have the same backlog barfing issue I currently have when my weechat machine reconnects to the ZNC bouncer.
In what ways has your IRC life been transformed?
It’s a dozen small things that have all added up to a much better experience for me. I’ll pull out a few:
On a personal level, the ‘out of the box’ config for senpai is perfect for me. I spent a long time tweaking weechat.
More broadly, these programs all feel like they are part of an ecosystem, and work together really well. This makes a multi-device setup more manageable.
I’ve not yet found a downside to goguma or senpai, but there are some to the soju bouncer:
Soju was my entrance into the bouncer world and have been pretty happy with it.
Senpai also rocks
I believe it is the soju bouncer that comes with a paid SourceHut subscription
ZNC is hands down one of the most obtuse pieces of software I’ve ever tried to set up and use.
Traveling back home to Moscow from India. Comes with the usual question of what to do on a long flight, I cached all the dependencies needed for integrating MMTK into the Tvix evaluator, but I’ll probably just watch a movie and then fiddle with my Emacs config.
By the way, did you know that outputting to all sound devices simultaneously is literally trivial with Pipewire? On the flight here we watched a movie, with output to two different headphones, and it worked perfectly fine. The future is here.
What interface do you use to point pipe wire audio at different devices?
I control Pipewire through its pulseaudio interface. Basically I just issued this command:
It then creates a sink that sends output to all other sinks, so you can select it and then control the individual other sinks as normal (volume/mute/etc.)
I liked the advice except for rule 1. If you have an endpoint for all products, then sure, specific products should be under /products/:id
But if you can’t list all products, then sure, keep the name /product/:id
It’s more about consistency than the plural name, and consistency is a different rule in the list.
There is plenty of room for disagreement, but I think the notion here is that REST APIs have two types of resources fundamentally; collections and objects. A collection can hold zero or more objects, so it is (in English at least) natural and correct to for the name of a collection to be the plural form of the type of object which it contains.
Just because you have requested a specific object from a collection does not change the fact that the collection itself holds zero or more objects.
Pretty good advice overall, but I do want to spotlight something
Rule #2 DON’T add unnecessary path segmentsGood advice, and yet the GOOD example…
commits the cardinal sin of REST API versioning… the entirely useless version segment! What’s weird is that the author demonstrates an understanding of why this is bad elsewhere:
Putting version information into the resource name also breaks this principle.
Tell me what is the distinction between these resources:
Nothing whatsoever. You are communicating something about the representation within the identitfier, which is a violation of concerns.
Why would you change a resource identifier? Let’s count the ways
Stripe’s approach has been very influential to me. Using ISO8601 date stamps as version numbers, pinning API consumers based on first request, compatibility layer that translate older API schema to new, all good things. The only issue I have is that they chose to use a custom
Stripe-Versionheader, and I disagree that this is right.This is fundamentally about having multiple, incompatible representations of the resource. How do we choose between representations? Content Negotiation.
Encoding the API version into the resource name means that the client becomes solely responsible for requesting the representation, there is no opportunity for negotiation between client and server.
The best way to version an API is via media type parameter.
In short: use http headers instead of the url to encode the version. Is that really that much better though? I thought that http headers shouldn’t change semantics, and changing the versions is a semantics change to me. But then I guess you can argue that the version is not part of the semantics when talking about http entities.
Tbh, it seems fairly academic to me. I’d rather stick to putting the version into the url and forcing the clients to assume that the same $listing_id under v1 and v2 might be the same entity, but might actually be different entities, forcing the clients to read my release notes.
No. Use the
Acceptheader specifically to encode the version, because what you are doing is content negotiation, and HTTP designates that header for that purpose.If you use path segments to version your API, you have no way to distinguish a syntactical change from a semantic one, because you’re putting the information in the wrong place.
A semantic change should be reflected in the resource identity. A syntactic change should be reflected in the representation. These are separate concerns and should not be conflated.
This perfectly encapsulates the problems I am critiquing:
This creates a situation that makes breaking changes very challenging to release, which in turn makes your API design accumulate undesirable warts, which makes integrating with your API unpleasant.
This is why people come to hate HTTP APIs like this.
(I object to your characterization of this as a scholastic exercise! I went to community college. My strong opinions have been formed by years of practical experience integrating with bad APIs.)
Fair enough, you make a good point.
I guess the reason I don’t like to use headers is because they are rarely used for those things. The Accept-header also seems to be used only for transparent compression. Another thing I don’t like is that headers are often not captured by logging tools for various reasons - so in case of a bug, it’ll be hard to reproduce it without knowing the headers if they impact the response.
In other words, both solutions now seem unsatisfying to me.
How do you like to handle paths that change between versions?
EDIT - I guess I’m asking where you handle the routing for
v2023-10-31vsv2023-11-1. I could imagine you could handle it at the routing layer by checking the media type parameter and sending it to different servers entirely, or you could handle it in the application layer.That depends on the implementation, and the reasons for your version change. If it’s all just happening in one app, you can handle it there. If you’re transitioning to a new codebase, you could introduce a reverse-proxy. Basically, your choices aren’t different here from the path segment, you just have more flexibility in introducing version updates to represent incompatible changes piecemeal without making a drastic change on the client side.
You can use headers for that. X-API-VERSION
Negotiating between different representations of the same resource is content negotiation, and the HTTP way of doing that is the
Acceptheader, not something else. Don’t reinvent the wheel.I think it gets nuanced pretty quickly… Is it the same resource if what is returned has different structure? How different does it need to be such that you consider it a different resource?
Also, changing headers is harder (though not that hard) for clients. Often making your API easier to use for others trumps doing it correctly. Changing v1 to v2 in the URL (which I have to specify always) is easier than having to change na Accept header which you’re now making me set, and I otherwise might not have to.
You are confusing resource and representation. The URI is the resource. The HTTP body that gets returned is the representation. They are separate concerns. Choosing between represenations is content negotiation.
Changing the URI to indicate a change in representation is the wrong place according to HTTP. It’s like putting file extensions in your URL instead of using media types. You can do it, but you’re misusing the tools that the protocol has given you.
I don’t think I am confused, just poorly explaining my point of view. If a resource that represents a user switches from containing a user ID that is an int to one that’s a UUID, are those the same resource? I am arguing they are not.
I agree that if that resource is sent back as JSON or serialized protobuf, then it is the same resource in different representations. However, I do not think it is an abuse of the HTTP protocol to assign different URIs for a resource that has evolved/changed as the service has. Using a version number in the URI to indicate this change of the resource seems logical, intuitive, and valid with respect to the protocol.
The ID field is special in this regard, because it most likely indicates a change of URI, and that is why this would be considered a new resource, not the change of representation.
Let’s alter your example to exclude this ambiguity. Let’s say that you’re making a breaking change in the structure of the user data from
and now you want to represent it as
Is this the same resource? Clearly, yes. You are just changing the representation. This is when I would increment the API Version. Changes that are strictly additive don’t break a contract, but changing an existing field that a client may depend on does. The thing you are communicating in the Version is that the contract has changed.
The URI of this record should not change. The relationships of this User to other things has not changed, the identity of the User has not changed, merely your representation of its data. Needing to alter an unrelated property such as the URI is a violation of concerns. URIs should only change when the identity of the resource changes.
This is the principle of encapsulation. Things that are independent should not have to change in tandem. That means there is a conflation of responsibilities happening.
Add the new path?
EDIT: Oh I’d do dispatch where it made sense. Probably in the application unless different versions are because the entire application has to change for some reason (new provider or something). That way all the possible answers to your Api call are in the same place.
http 308 response?
Just was wondering whether it has the same issue as with ZFS encryption only supporting one key, and yes it does and it’s well documented over at the becachefs wishlist.
What’s the issue? ZFS supports per-dataset keys. One of the intended use cases was having per-user encrypted home directories that are decrypted only when the user’s passphrase is entered (or when they provide a PIN to a TPM, or similar).
I suspect it’s nice to have a set of keys, a backup set and a primary set. Of course you could just duplicate the same key to multiple places.
Yeah, I feel like the solution here is to encrypt the key multiple times. You have the hdd encryption key, and then every time you want to add a new key you just encrypt the hdd encryption key with a new key.
I think this is how most HDD encryption works, since you don’t need to re-encrypt the data when you rotate keys, you just re-encrypt the initial key with your new key.
And you can do that with ZFS, just creating a userspace utility for your scheme.
ZFS actually works like this: disk blocks are not encrypted with the key that you provide, they’re encrypted with a key that’s encrypted with the key that you provide (well, actually, with a key derived from a key that is encrypted with the one that you provide, I believe). Allowing a second key to be used to encrypt the dataset encryption key seems useful (especially when rotating keys: add the new key, check you got it right, delete the old key).
I’m not sure if this is possible with ZFS. I think the space for the master keys is fairly configurable, it’s only the key index for the dataset keys that reuses some other space in metadata.
At $work the workstations have one key that’s tied to local identity (a passphrase entered on the physical keyboard) and another that’s in escrow. The escrow keys are used to unlock based on corporate policy authentication - ie: remotely unlocking based on my corporate credentials. The escrow key access policies can of course be set up for whatever makes sense for the organization.
I think it’s using lvm2 & luks but I’ve never looked that closely.
Yes, fallback Keys or human key + automatic one etc can be crucial. Also switching keys becomes a slightly safer operation I’d wanted.
Also just wanted to share the link itself, it gives quite a good view on for what it’s ready to use at the first time of merging. Roadmap part is also interesting, since they talk about things like more efficient squashes alternative
Stream start. Hello. People using Macs in professional settings and schools. Porsche, Girls Who Code, medical environments, artists, etc…
Spooky graphics. Tim Cook at night to talk about the Mac. Laptops gained bigly. New chips in MacBook Pro.
HERE’S JOHNY! Spooky lab. Yes, we’ve got the best chips. Next-generation AS. Three chips at the same time. M3, M3 Pro, M3 Max. 3nm. EUV. Big GPU advancement. New µarch, “Dynamic Caching”. Non-static scheduling, memory dynamically allocated by hardware, improving average utilization. Dev-transparent. Mesh shading in hardware. Hardware ray-tracing. 2.5x render perf in pro apps? CPU cores. 30% faster P-cores, with same efficiency. 50% for E-cores. Same perf as M1 in half the power for both CPU and GPU. Still smoking x86, quarter of the power usage for same perf. 16-core, 15% faster ANE. AV1 decode support. M3 has 8-core CPU (4/4), 10-core GPU, 24 GB max. M2 Pro is 6/6, 18-core GPU, 36 GB max. M2 Max is 12/4, 40-core GPU, 128 GB of RAM.
HERE’S JOHN! To talk about products using these chips. MacBook Pro.
HERE’S…. Kate. To talk about the new MBP. 14” has M3 in base model. RIP 13”. M3 Pro models in 14”/16”. Two external displays on M3 Pro. M3 Max models. 2.5x Cinema 4D perf. Two ProRes engines on M3 Max. M3 Max can drive 4 external displays. Same perf on battery. 22 hours of battery life. 11x faster than fastest x86 MBP. Silent most of the time. 1000 sustained/1600 peak HDR nits on the display, just in case you didn’t know. SDR is 600 nits. New colour. Black? Space black. Dark aluminum. New chemistry that’s fingerprint resistant. 100% recycled apparently. Yes, it runs macOS 14. New show on Apple TV+, in case you have that. macOS 14 has higher quality screen sharing and game mode, in case you didn’t know that. MacBook Pros are good for people who do things, in case you didn’t know that. You can carry it around on battery doing stuff, yes.
HERE’S JOHN! Base model 14” (with M3, replacing 13”) is now 1599$. 16” is same price. M3 Max is later in November, everything else next week. 24” iMac gets M3. It has colours, unlike the other ones. M1->M3 is 2x faster. Yes, it curbstomps Intel. Same price. Order today, available next week.
HERE’S… Tim. That’s it?
Exactly my thoughts on this. I think they couldn’t get away with a blogpost on the new chips so they made a 30 minute halloween video.
Those performance improvements aren’t too shabby, though the downgrading of the M3 Pro is kinda odd. Went from 4 to 3 memory modules, meaning a 25% reduction in memory bandwidth, and went from a 8+4 CPU config to 6+6. Will definitely still be faster in many tasks, but some tasks will surely regress going from M2 Pro to M3 Pro.
on the space black, I was really hoping it would be a vibrant deep purple.
Widgets on the desktop seems like OSX Tiger’s “Dashboard” has been flattened onto the desktop instead of being an overlay.
Seems to be a “call out your normal apps for $thing” mandate. Software I’m sure I’d recognize if I were a pro audio editor, or gene sequencer. And XCode.
Huh, no Mac Pro update. But is the music supposed to be a bit halloweeny?
The various clips are already on youtube from the Apple account.
M3 Mac Pro presumably isn’t coming until after Mac Mini and Mac Studio updates.
Widgets are the eternal idea. Every ten years, someone comes out with a new variation on them. They always have a small pocket of die hards, but most people ignore them, and then they go away again. Whether it’s Desk Accessories or Active Desktop or Dashboard or Tiles or whatever, they just never have any staying power.
Purely an anecdote, but I think the widgets feature on the iPhone will have some staying power, at least. Whenever I’m at a sporting event or concert or something I can’t help but glance at the phone screens of the people in front of me, and home screen widgets seem to be surprisingly popular. I think it’s mostly for aesthetic purposes (alongside the Shortcuts feature that lets you effectively change app icons), but it’s cool to see nonetheless.
Macs are a much more niche product and the customized desktop sharing posts on social media are somewhat less of a thing than iPhone home screen design tutorials, so I’m not sure the macOS version will catch on quite the same. Who knows, though. Maybe the idea has just been ahead of its time for the last 40 years and its real application is on phones, not desktops!
Shell scripts are best for programs that need to glue together a number of different programs together, or when you have no choice (environmental constraints).
If your code is mostly gluing together other programs, it may be best to stay in shell. Your logic however should get out of shell as quickly as possible. Can you turn your logic into another program? Can you “shell out” to python, or perl, or any other useful language for the nifty bits?
A good shell program is a lot like any other program. It’s all about your organization and discipline in authorship.
Yeah exactly, the fallacy is that it’s shell XOR something else. Basically ALL my shell scripts call Python scripts, and about half those Python scripts I wrote myself!
That’s how shell is designed to be used – factoring policy and mechanism.
The way I think of it is
A litmus test for this style of factoring: Don’t put file system paths or port numbers in Python scripts. Put them in shell scripts (or a config file that lives outside the application).
Unfortunately I think I never wrote a post about this, but I specifically mention the biggest misconception about shell XOR Python here:
https://www.oilshell.org/blog/2021/01/philosophy-design.html#the-biggest-misconception-shell-xor-python
What I will concede is that it’s harder to program in 2 languages at once (shell and Python) than to use one language.
Using multiple languages, and “factoring” them, is a skill. It takes practice and thought. But the benefit is having less code overall, and that code is more flexible and reusable – as coarse-grained processes.
(Though if you call os.system() from Python, now you’re back to 2 languages again :-) So I use shell as the main() and Python as “subroutines”, which can be concurrent concurrently.)
The other thing to conceded is that shell has some nearly fatal flaws like bad error handling.
By and large I think this post makes a reasonable point:
I agree that the status quo for testing isn’t enough. But if that’s the only barrier, then shell is in a good place!
But I actually think a bigger barrier is the shell language itself. It’s powerful but many things about it actively inhibit learning (error messages and inconsistent syntax) … The saving grace is that it’s fast and predictable (if you pay close attention to details!)
Alright, they passed the law. Time to admit that we actually do know how to do secure key escrow, we just didn’t feel like it. 🙃
if people are serious about it, how about striking
There’s a good issue for Labour to campaign on, unless they believe that the bill is popular among voters.
Which is probably why these bills are heavily framed as being about “child safety”. No politician wants to give the impression that they’re against that.
Well yes, that is a given. It is a lawmaker’s prerogative to frame the laws they wish to pass in a way that maximizes the probability of them passing.
The “pro-encryption” camp (which I belong to, btw) needs better arguments than just shrugging their shoulders and saying “yes, encryption protects bad people too, but we believe the trade-off is worth it”. For many people who are not versed in the minutiae of online communication, the trade-off is not worth it. They want the bad guys to be caught and punished!
It’s very hard to argue against, because it relies on fundamental rights like privacy, which many people don’t care about in the first place, and is too nebulous to really nail down. Maybe you could argue something like “some pervert at Meta will also be able to see all your sextings”?
Most of the campaigning against CSAR (which was at least significantly amended) focused on the proposed solutions not being particularly effective, and the idea that client-side scanning would drown the already insufficient police forces in low-quality reports.
But EU directives aren’t really subject to the same media attention that laws on the national level are. The campaigning was aimed at convincing lawmakers, not the general public.
I think a lot of the resistance comes from an ambient sense that tech companies and governments are bad guys. This view is probably a lot more widespread than media would admit, but the taboo on making that argument no doubt has an effect.
Wait, we know how to do secure key escrow so only authorized parties can break in to encrypted traffic?
Is this just shamir secret sharing?
We don’t know how to give that key to the government and have the government never lose it to Edward Snowden or the GRU.
In my bog standard infrastructure, I do containers on nomad with vault and consul in case I need to do a service mesh later. Single node nomad is, operationally at least, nicer than docker compose.
This is my experience, it may not be yours.
AMA!
I’ve got a bunch of questions, but I guess a few things (appreciating that from a career standpoint you may not be able to answer):
(I can guess as to most of these, given the usual way of things, but figured I might as well ask.)
More on the tech side:
Thanks for posting here, and thanks for helping right the ship!
Sorry in advance for the long-winded context-setting, but it’s the only way I know how to answer this question.
There are a few important things to understand. First, even though HealthCare.gov was owned by CMS, there was no one single product owner within CMS. Multiple offices at CMS were responsible for different aspects of the site. The Office of Communications was responsible for the graphic design and management of the static content of the site, things like the homepage and the FAQ. The Center for Consumer Information and Insurance Oversight was responsible for most of the business logic and rules around eligibility, and management of health plans. The Office of Information Technology owned the hosting and things like release management. And so on. Each of these offices has their own ability to issue contracts and set of vendors they prefer to work with.
Second, HealthCare.gov was part of a federal health marketplace ecosystem. States integrated with their own marketplaces and their Medicaid populations. Something called the Digital Services Hub (DSH) connected HealthCare.gov to the states and to other federal agencies like IRS, DHS, and Social Security, for various database checks during signup. The DSH was its own separately procured and contracted project. An inspector general report I saw said there were over 60 contracts comprising HealthCare.gov. Lead contractors typically themselves subcontract out much of the work, increasing the number of companies involved considerably.
Then you have the request for proposals (RFP) process. RFPs have lists of requirements that bidding contractors will fulfill. Requirements come from the program offices who want something built. They try to anticipate everything needed in advance. This is the classic waterfall-style approach. I won’t belabor the reasons this tends not to work for software development. This kind of RFP rewards responses that state how they will go about dutifully completing the requirements they’ve been given. Responding to an RFP is usually a written assertion of your past performance and what you claim to be able to do. Something like a design challenge, where bidding vendors are given a task to prove their technical bona fides in a simulated context, while somewhat common now, was unheard of when HealthCare.gov was being built.
Now you have all these contractors and components, but they’re ostensibly for the same single thing. The government will then procure a kind of meta-contract, the systems integrator role, to manage the project of tying them all together. (CGI Federal, in our case.) They are not a “product owner”, a recognizable party accountable for performance or end-user experience. They are more like a general contractor managing subs.
In addition, CMS required that all software developed in-house conform to a specified systems design, called the “CMS reference architecture”. The reference architecture mandated things like: the specific number of tiers or layers, including down to the level of reverse proxies; having a firewall between each layer; communication between layers had to use a message queue (typically a Java program) instead of say a TCP socket; and so forth. They had used it extensively for most of the enterprise software that ran CMS, and had many vendors and internal stakeholders that were used to it.
Finally, government IT contracts tend to attract government IT contractors. The companies that bid on big RFPs like this are typically well-evolved to the contracting ecosystem. Even though it is ostensibly an open marketplace that anyone can bid on, the reality is that the federal goverment imposes a lot of constraints on businesses to be able to bid in the first place. Compliance, accreditation, clearance, accounting, are all part of it, as well as having strict requirements on rates and what you can pay your people. There’s also having access to certain “contracting vehicles”, or pre-chosen groups of companies that are the only ones allowed to bid on certain contracts. You tend to see the same businesses over and over again as a result. So when the time comes to do something different – eg., build a modern, retail-like web app that has a user experience more like consumer tech than traditional enterprise government services – the companies and talent you need that has that relevant experience probably aren’t in the procurement mix. And even if they were, if they walked in to a situation where the reference architecture was imposed on them and responsibility was fragmented across many teams, how likely would they be to do what they are good at?
tl;dr:
I think they saw quickly from the kind of experience, useful advice, and sense of calm confidence we brought – based on knowing what a big transactional public-facing web app is supposed to look like – that they had basically designed the wrong thing (the enterprise software style vs a modern digital service) and had been proceeding on the fairly impossible task of executing from a flawed conception. We were able to help them get some quick early wins with relatively simple operational fixes, because we had a mental model of what an app that’s handling high throughput with low latency should be, and once monitoring was in place, going after the things that were the biggest variance from that model. For example, a simple early thing we did that had a big impact was configuring db connections to more quickly recycle themselves back into the pool; they had gone with a default timeout that kept the connection open long after the request had been served, and since the app wasn’t designed to stuff more queries into an already-open connection, it simply starved itself of available threads. Predictability, clearing this bottleneck let more demand flow more quickly through the system, revealing further pathologies. Rinse and repeat.
Were there poor performers there? Of course. No different than most organizations. But we met and worked with plenty of folks who knew their stuff. Most of them just didn’t have the specific experience needed to succeed. If you dropped me into a mission-critical firmware project with a tight timeline, I’d probably flail, too. For the most part, folks there were relieved to work with folks like us and get a little wind at their backs. Hard having to hear that you are failing at your job in the news every day.
You can easily research this, but I’ll just say that it’s actually very hard to meaningfully penalize a contractor for poor performance. It’s a pretty tightly regulated aspect of contracting, and if you do it can be vigorously protested. Most of the companies are still winning contracts today.
I would often joke on the rescue that for all HealthCare.gov’s notoriety, it probably wasn’t even a top-1k or even 10k site in terms of traffic. (It shouldn’t be this hard, IOW.) Rough order of mag, peak traffic was around 1,000 rps, IIRC.
You could navigate the site on your phone if you absolutely had to, but it was pretty early mobile-responsive layout days for government, and given the complexity of the application and the UI for picking a plan, most people had to use a desktop to enroll.
Great question. I remember being on a conference bridge late in the rescue, talking with someone managing an external dependency that we were seeing increasing latency from. You have to realize, this was the first time many government agencies where doing things we’d recognize as 3rd party API calls, exposing their internal dbs and systems for outside requests. This guy was a bit of a grizzled old mainframer, and he was game to try to figure it out with us. At one point he said something to the effect of, “I think we just need more MIPS!” As in million instructions per second. And I think he literally went to a closet somewhere, grabbed a board with additional compute and hot-swapped it into the rig. It did the trick.
In general I would say that the experience, having come from what was more typical in the consumer web dev world at the time - Python and Ruby frameworks, AWS EC2, 3-tier architecture, RDBMSes and memcached, etc. - was like a bizarro world of tech. There were vendors, products, and services that I had either never heard of, or were being put to odd uses. Mostly this was the enterprise legacy, but for example, code-generating large portions of the site from UML diagrams: I was aware that had been a thing in some contexts in, say, 2002, but, yeah, wow.
To me the biggest sin, and I mean this with no disrespect to the people there who I got to know and were lovely and good at what they did, was the choice of MarkLogic as the core database. Nobody I knew had heard of it, which is not necessarily what you want from what you hope is the most boring and easily-serviced part of your stack. This was a choice with huge ramifications up and down, from app design to hardware allocation. An under-engineered data model and a MySQL cluster could easily have served HealthCare.gov’s needs. (I know this, because we built that (s/MySQL/PostgreSQL) and it’s been powering HealthCare.gov since 2016.)
A few years after your story I was on an airplane sitting next to a manager at MarkLogic. He said the big selling point for their product is the fact it’s a NoSQL database “Like Mongo” that has gone through the rigorous testing and access controls necessary to allow it to be used on [government and military jobs or something to that effect] “like only Oracle before us”.
He’s probably referring to getting FedRAMP certified or similar, which can be a limiting factor on what technologies agencies can use. Agencies are still free to make other choices (making an “acceptable risk” decision).
In HealthCare.gov’s case, the question wasn’t what database can they use, but what database makes the most sense for the problem at hand and whether a document db was the right type of database for the application. I think there’s lots of evidence that, for data modeling, data integrity, and operational reasons, it wasn’t. But the procurement process led the technology choices, rather than the other way round.
You laugh, but they advertise MUMPS like a modern NoSQL database suitable for greenfield.
I’m not Paul, but I recall hearing Paul talk about it one time when I was his employee, and one of the problems was MarkLogic, the XML database mentioned in the post. It just wasn’t set up to scale and became a huge bottleneck.
Paul also has a blog post complaining about rules engines.
See also Paul’s appearance on the Go Time podcast: https://changelog.com/gotime/262
I’m happy to see that you feel proud of your work. It’s an incredibly rare opportunity to be able to help so many people at such a large scale.
Have you found something else as meaningful for you since then?
Also what’s your favorite food?
Thank you!
Yes, my work at Ad Hoc - we started the company soon after the rescue and we’ve been working on HealthCare.gov pretty much ever since. We’ve also expanded to work on other things at CMS like the Medicare plan finder, and to the Department of Veterans Affairs where we rebuilt va.gov and launched their flagship mobile app. And we have other customers like NASA and the Library of Congress. Nothing will be like the rescue because of how unique the circumstances, but starting and growing a company can be equally as intense. And now the meaning is found in not just rescuing something but building something the right way and being a good steward to it over time so that it can be boring and dependable.
A chicken shawarma I had at Max’s Kosher Café (now closed 😔) in Silver Spring, MD.
Can you please kill X12. That is all.
This is only slightly sarcastic. I’m hopeful with your successful streak you can target the EDI process itself, because it’s awful. I only worked with hospice claims, but CMS certainly did a lot to make things complicated over the years. I only think we kept up because I had a well-built system in Ruby.
It’s literally the worst thing in the world to debug and when you pair that with a magic black box VM that randomly slurps files from an NFS share, it gets dicey to make any changes at all.
Seriously. ☠️
For those not familiar, X12 is a standard for exchanging data that old-school industries like insurance and transportation use that well-predates modern niceties like JSON or XML.
X12 is a … to call it a serialization is generous, I would describe it more like a context-sensitive stream of transactions where parsing is not a simple matter of syntax but is dependent what kind of X12 message you are handling. They can be deeply nested, with variable delimiters, require lengthy specifications to understand, and it’s all complicated further by versioning.
On top of that, there is a whole set of X12-formatted document types that are used for specific applications. Relevant for our discussion, the 834, the Benefit Enrollment and Maintenance document, is used by the insurance industry to enroll, update, or terminate the coverage.
To give you a little flavor of what these are like, here is the beginning of a fake 834 X12 doc:
HealthCare.gov needed to generate 834 documents and send them to insurance carriers when people enrolled or changed coverage. Needless to say, this did not always go perfectly.
I personally managed to avoid having to deal with 834s until the final weeks of 2013. An issue came up and we need to do some analysis across a batch of 834s. Time was of the essence, and there was basically no in-house tooling I could use to help - it only generated 834s, not consume them. So naturally I whipped up a quick parser. It didn’t have to be a comprehensive, fully X12-compliant parser, it only needed enough functionality to extract the fields needed to do the analysis.
So I start asking around for anyone with 834 experience for help implementing my parser. They’re incredulous: it can’t be done, the standard is hundreds of pages long, besides you can’t see it (the standard), it costs hundreds of dollars for a license. (I emailed myself a copy of the PDF I found on someone’s computer.)
I wrote my parser in Python and had the whole project done in a few days. You can see a copy of it here. The main trick I used was to maintain a stack of segments (delimited portions of an X12 doc) and the in the main loop would branch to methods corresponding to the ID of the current segment, allowing context-sensitive decisions to be made about what additional segments to pull off the stack.
The lesson here is that knowing how to write a parser is a super power that will come in handy in a pinch, even if you’re not making a compiler.
just wanted to give you props. I’m a contractor myself and I’ve been on both sides of this (I have been in the gaggle of contractors and been one of the people they send in to try to fix things) and neither side is easy, and I’ve never come close to working on anything as high profile as this and even those smaller things were stressful. IME contracting is interesting in some ways (I have worked literally everywhere in the stack and with tons of different languages and environments which I wouldn’t have had the opportunity to do otherwise) and really obnoxious in other ways (legacy systems are always going to exist but getting all the contractors and civilian employees pulling in the same direction is a pain, especially without strong leadership from the government side)
Thanks, and back at ya. My post didn’t get into it, but it really was a problem of a lack of a single accountable product lead who was empowered to own the whole thing end-to-end. Combine that with vendors who were in a communications silo (we constantly heard from people who had never spoken to their counterparts other teams because the culture was not to talk directly except through the chain of command) and the result was, everyone off in their corner doing their own little thing, but no one with a view to the comprehensive whole.
How helpful were non-technical metrics such as user feedback and bug reports to the process?
Extremely. Aside from the performance-related problems that New Relic gave us a handle on, there were many logic bugs (lots of reasons for this: hurried development, lack of automated tests, missing functionality, poorly-understood and complicated business rules that were difficult to implement). This typically manifested in users “getting stuck” partway through somewhere, and they could neither proceed to enrollment or go back and try again.
For example, we heard early on about a large number of “lost souls” - people who had completed one level of proving your identity (by answering questions about your past ala a credit check), but due to a runtime error in a notification system, they never got the link to go to the next level (like uploading a proof of ID document). Fixing this would involve not just fixing the notification system, but figuring out which users were stuck (arbitrary db queries) and coming up with a process to either notify them (one-off out-of-band batch email) or wipe their state clean and hope they come back and try again. Or if they completed their application for the tax credit, they were supposed to get an official “determination of eligibility” email and a generated PDF (this was one of those “requirements”). But sometimes the email didn’t get sent or the PDF generator tipped over (common). Stuck.
The frontend was very chatty over Ajax, and we got a report that it was taking a long time for some people to get a response after they clicked on something. And we could correlate those users with huge latency spikes in the app on New Relic. Turns out for certain households, the app had a bug where it would continually append members of a household to a list over and over again without bound after every action. The state of the household would get marshalled back and forth over the wire and persisted to the db. Due to a lack of a database schema, nothing prevented this pathological behavior. We only noticed it because it was starting to impact user experience. We went looking in the data, and found households with hundreds of people in them. Any kind of bug report was just the tip of the iceberg of some larger system problem and eventual resolution.
I have a good story about user feedback from a member of Congress live on TV that led to an epic bugfix. I’ve been meaning to write it up for a long while. This anniversary is giving me the spur to do it. I’ll blog about it soon.
I wish I had the time to noodle around like that. With a young family there’s no time (or energy) to work on side projects.
I have two young kids (6 and 3.5 years old) and have felt like that for the last 6 years at least. It does feel like it’s slowly getting less busy, so some space for hobbies and “useless” projects is opening up again.
They are getting to the age where they can explore spaces with you. Try taking up a hobby and including them!
I have a 5 and 3, and it feels like I’m slowly starting to get some time back as they get older. It helps when you can leave them in a room by themselves and trust them to not eat the paint chips anymore. 😂
I also have a young family and lament my lack of good free time. I go through phases. I’ll pour my free evening hours into playing games on my Switch for a month, and then I get the itch and spend a month working on a side project on my laptop. Both can happen while watching whatever with my wife on the couch, which is nice. It can be hard to accomplish a lot when I’ve only got an hour or two a night, but it’s still enough to scratch that itch and make me feel less trapped (by job and family and circumstance).
When you have young children, any activity that doesn’t contribute directly to your basic sustenance is time and energy stolen away from a better parent you could be for them. Especially so in these economically and politically turbulent times when you can’t set financial goals, so you have to aim to save as much as possible in order to reduce the chance of your worst-case outcomes. When you’re childless, there’s a wide range of options to weather through hard times, but you really can’t risk compromising on life’s necessities when you have children.
Absolutely. I find that thoughts of finances are keeping me awake now and then, while it never ever has in the past. And that’s with a handsome salary for a comfy dev job. I can’t imagine how tough life is with kids for people who earn more basic salaries.
Gives me anxiety just to think about that