-
pims
2 months ago
|
link
| on:
How not to structure your database-backed web applications: a study of performance bugs in the wild
Inefficient data accessing (ID) results in slow data transfers by either not getting enough data (resulting in the familiar N+1 selects problem), or getting too much data. This can be due to inefficient lazy loading (still prevalent in this study), or too-eager eager loading. Here’s an example of a patch in Lobsters that replaces 51 database queries with one:
-
There’s some really interesting theoretical work being done to compile template files into optimal queries.
In theory a templating language statically determine what data needs to be fetched ahead of time, but integrating it into an application framework is a significant job.
-
-
While I do find Pony interesting from a technical perspective, I’ll admit that publishing every point release changelog to lobsters is an extremely generous understanding of what some of us would like the “release” tag to showcase.
Major features? Sure. Security fixes, yes. But:
Pony 0.23.0 is primarily a bug fix release but does also include some additional changes. Upgrading is recommended. There is a single breaking change caused by the implementation of RFC 56.
is arguably less interesting.
-
-
as I write this, this story is hidden by 8, +10, -5 spam, which indicates to me that might be better to refrain unless there is a major milestone, or an interesting new feature that might spur some discussion.
Adding some additional commentary about why the breaking type change is interesting or useful might help; there are enough language implementation nerds who read lobsters regularly that it could generate some discussion, especially since the capability types are one of pony’s most compelling and novel features.
-
I wont be posting releases to lobsters anymore.
-
-
Perhaps. Either way. Not submitting them anymore.
-
-
-
-
-
I’m not qualified to make any judgment on the technical merits of several dependency management solutions, but as someone working primarily in Go, the churn of solutions is starting to have a real cognitive cost.
-
Some of the solutions suggested from a couple of the Go devs in that “thread” sound.. almost surreal to me.
My favorite one so far:
We’ve been discussing some sort of go release command that both makes releases/tagging easy, but also checks API compatibility (like the Go-internal go tool api checker I wrote for Go releases). It might also be able to query godoc.org and find callers of your package and run their tests against your new version too at pre-release time, before any tag is pushed. etc.
– https://github.com/golang/go/issues/24301#issuecomment-390788506With all the cloud providers starting to offer pay-by-the-second containers-as-a-service, I see no reason we couldn’t provide this as an open source tool that anybody can run and pay the $0.57 or $1.34 they need to to run a bazillion tests over a bunch of hosts for a few minutes. There’s not much Google secret sauce when it comes to running tests.
– https://github.com/golang/go/issues/24301#issuecomment-390790036That sounds… kind of crazy for anyone that isn’t Google scale or doesn’t have Google money.
Are the Go devs just /that/ divorced from the (non Google) reality that the rest of us live in?-
Kind of crazy, but not super crazy. As another example, consider Rust’s crater tool. When the Rust team are trying to evaluate the impact of fixing a syntax quirk or behavioural bug, they make a version of the Rust compiler with the change and a version without, and boot up crater to test every publicly available Rust package with both compiler versions to see if anything breaks that wasn’t already broken.
crater runs on Mozilla’s batch-job infrastructure (TaskCluster), and Mozilla is much, much smaller than Google scale. On the other hand, they’re still bigger than a lot of organisations, and I believe a crater run can take a few days to complete, so it’s going to be a lot more than “$1.34 … for a few minutes” on public cloud infrastructure.
-
I get the spirit of those responses; we’re getting to the point with cloud services were that kind of integration test suite could happen cheaply.
But it is not the answer to the problems that prompted those responses.
Dependency management is hard, and there isn’t a perfect solution, mvs is a cool approach and I’m curious how it shakes out in practice, but to OP’s point, I’m not sure I can do another switch like we’ve done up to now
Manual Vendoring ($GOPATH munging)
govendor
dep
vgo
whatever fixes the problems with vgo-
Agreed. I have a couple of projects that I have switched solutions at least 4 or 5 times already (manual GOPATH munging, godep, gpm, gb, dep), because each time it was either a somewhat commonly accepted solution, or seemed the least worst alternative (before there was any kind of community consensus).
-
-
-
I have yet to migrate a project between dependency managers.
The old ones work exactly as well as they always have.
-
-
-
It’s briefly mentioned in the article, but be very wary of the number of sockets in TIME_WAIT state. It’s bitten me more than I’d like admit.
-
-
The Linux kernel also has the following sysctl tunables which affect the behavior of sockets in TIME_WAIT (or more accurately, the kernel’s willingness to reuse them):
- net.ipv4.tcp_tw_reuse
- net.ipv4.tcp_tw_recycle (apparently removed in Linux 4.12+)
I have used the former without issue, but my understanding is they are best avoided as e.g., stateful packet filers will be confused.
-
-
-
Excellent talk about migrating Zuul to Netty: https://youtube.com/watch?v=2oXqbLhMS_A
-
-
While faster boot time is rarely a bad thing, for many use cases, with somewhat long running tasks, it doesn’t matter as much. See JVM.
-
-
This is a nice summary, thanks for sharing it. Combined with this tweet: https://twitter.com/kellabyte/status/996429414970703872
…I’m inclined to wonder how much time/bandwidth would be saved at larger sites if people cleaned these up, although I suspect that “size of HTTP headers” is not the worst bottleneck for most people.
-
-
I suspect the impact is minimal. It’s a few hundred bytes at worst, and the site is probably more affected by 3rd party adtech or unoptimized pictures.
-
Somewhat related, but even small changes to the request/response can have large impact on the bandwidth consumed.
From Stathat “This change should remove about 17 terabytes of useless data from the internet pipes each month” https://blog.stathat.com/2017/05/05/bandwidth.html
-
Optimized Images alone would most likely save a lot more since they can save a lot more too. A recent google blog loaded a 8MB GIF image to show a few second long animation in a 250x250 thumbnail. 2 minutes in ffmpeg reduced that to about 800KB.
Imagine if people did this on sites with more traffic than some random google product announcement blog…
-
-
-
-
I love Docker. I really do, but here’s the thing that we’ll regret about Docker, and we don’t need a massive article to explain it.
You are letting regular users do this:
echo 'echo "Hello." > /etc/motd' | docker run -i -v /etc:/etc -- bashHooray! I made a file as root! I don’t even need sudo or su any more!
Here’s hoping you’re not running the Docker HTTP server locally so that I can do that to you over HTTP in a coffee shop. Wait… You’re not, right?
While I’m at it, maybe I’ll flip the sticky bit on your shell or cat my system root over to another server!
This is great!
-
-
-
Obviously this assumes your user is in the docker group. I’m suggesting that - if we want to point fingers at Docker - the bigger issue is these kinds of things happening in places like development environments.
I’m much more worried about someone being able to SCP a developer’s root device because they are running the Docker API w/out auth than I am worried about Docker failing in production, for instance. People who run Docker in production generally know how it works. People running Docker on workstations generally have no idea, and will happily paste commands as they’re told.
-
-
The difference is specifically that it is surprising to most users I’ve worked with people on Docker projects with that when you launch a container, the process is run as your host system’s root user. Although this is obvious to someone who knows how Docker works, it’s not obvious to everyone and is dangerous.
Anyway, my original point was mostly sarcastic and apparently that didn’t come across well
-
-
-
-
Yeah. When you launch a container, it is running on the same kernel as the Linux host (no VM) as the root user. If you mount a directory as a volume, you essentially can access it as root. A lot of people don’t realize the permissions they are giving docker containers, and I personally believe that this is probably the most concerning issue with Docker right now.
We have all of our development on managed remote servers, because running local services in Docker is super dangerous if you don’t understand how Docker works internally imo.
-
-
-
-
My concerns are 100x more about workstations than servers. So many developers are running Docker and don’t even know what it does.
How many people just copy/paste
docker runcommands to get work done? This is a social exploit w/ Docker users, but I do believe that it’s still a valid exploit and I don’t think that invalidating it is helping anyone. -
-
To be clear, I’m not blaming docker. I feel like I very clearly stated that this is going to be something that people regret about using docker. I like and use docker a ton.
This issue is rooted in how people use it, not because of docker being completely able to allow avoiding this.
If you only judge the security of your devices by the idealist situation that you assume of people, you’ll never be secure. People will continue to impress with unexpected approaches to these things - even with clear guidelines explaining to the contrary.
Being an idealist is just not a reasonable way to approach this sort of issue. To be doubly clear (yes, redundant), I am talking about what actually happens - not what happens in the imagination world of theoreticals and idealism.
I have noticed enough people with this issue on their development machines that I honestly can’t believe that this comment attracted such a level of controversy. People with a reasonable amount of access to important information have issues like this, and it’s terrifying.
-
-
-
-
Using Nomad + Backplane strikes the right amount of simplicity and sophistication for me.
-
-
By using backplane I’m basically outsourcing the service discovery and routing to a single centralized service. This drastically simplify deploying simple http services from (potentially) different platforms: Heroku, Digital Ocean, AWS, etc. I can deploy without “releasing” new versions and slowly migrate traffic from one version to another. No security group, firewalls to configure thanks to the built-in reverse proxying. It’s a commercial centralized service, but that’s a trade off I’m willing to make for small scale projects.
-
-
-
I didn’t know the language supported adding type annotation to functions… that’s neat!
-
-
Solid post. I remember reading about a project that provided Redis clustering on top of RiakCore many years ago, before any native clustering implementation in Redis. A similar implementation would make for a great part-3 :)
-
this may interest you :)
-
-
-
Excellent. I toyed with the idea of building an iOS app that would send a push notification every morning with a random painting and short summary of the artist and context for the painting.
This app looks like a neat replacement until I roll up my sleeves and build it.
-
I’m a big fan of the Google Arts & Culture plugin for Chrome, which displays randomly-chosen art every day whenever you create a new tab. I’ve used it for years, with only a few repeats. I doubt it works on iOS, but it’s worth a shot.
-
-
One of the most powerful parts about streaming replication in PostgreSQL is that one can set the replication mode on a per transaction basis.I had no idea this was possible. It should be very useful when you have mix workloads on the same database.
-
Pretty good list of practical solution of the thorny problem of HTTP API evolution.
-
pims
3 months ago
|
link
| on:
Running FFmpeg on AWS Lambda for 1.9% the cost of AWS Elastic Transcoder
Exodus looks like a great piece of software. Makes me reconsider lambda for some low volume tasks.
-
This is scary. Because even at the time of install, after carefully reading all reviews, and inspecting the network tab everything looks okay. It’s only weeks/months later that the malicious code gets remotely loaded and executed.
Google needs to take a hard look at their Chrome Extension Store.
-
-
Meanwhile, in Firefox, you can just click the checkbox to disable automatic updates in the add-ons manager (either globally or for a particular add-on).
They also seem to have a pretty decent review process for add-ons; I don’t know whether they’d catch malicious updates, but when I submitted my add-on, I got actual technical feedback about the code (like a suggestion to replace
innerHTMLwithtextContenton a particular line).With how great Firefox Quantum is these days, maybe it’s time to consider switching?
-
A power user can always download the extension’s source code and install it separately in developer mode. Since that counts as a different “extension” than the one offered in the store, it won’t automatically update.
I’m probably in the minority here, but I think automatic updates as a default is a good idea. Ideally there would be a way to turn this off, but defaulting to automatic updates is better than defaulting to manual updates. The vast majority of updates fix real bugs and the vast majority of users won’t update manually.
-
I also think automatic updates as a default is a good idea. What I find odd is that you can’t disable auto-update on a per extension basis. That way I can only authorize official extensions to auto-update and then decide for every other extension. It would definitely reduce the risk of a popular extension being bought and re-uploaded with malicious behavior.
-
-
-
-
pims
4 months ago
|
link
| on:
mmproxy - Creative Linux routing to preserve client IP addresses in L7 proxies
That’s a clever hack.
-
I really like their approach to canary testing.
-
pims
4 months ago
|
link
| on:
How ipdata serves 25M API calls from 10 infinitely scalable global endpoints for $150 a month
Interesting architecture, but not impressed by the 25M API calls per month.
-
It’s always confused me when “queries per month” is used as a headline. That’s an indicator of the user base, not the technical infrastructure. On average it’s about 10 qps, which for this feature requires possibly a large Arduino. :) BTW, no idea why a web site owner would accept a paid dependency just to do a btree lookup “as a service”.
-
-
I’m surprised they didn’t get better support from AWS. In my experience, they’ve been pretty good at escalating to the right technical staff.
The technical analysis is by itself very interesting.