-
As someone who never used Rust I want to ask: does the section about crates imply that all third-party libraries are recompiled every time you rebuild the project?
-
Good question! They are not; dependencies are only built on the first compilation, and they are cached in subsequent compilations unless you explicitly
cleanthe cache.-
-
A Rust library includes the actual compiled functions like you’d expect, but it also contains a serialized copy of the compiler’s metadata about that library, giving function prototypes and data structure layouts and generics and so forth. That way, Rust can provide all the benefits of precompiled headers without the hassle of having to write things twice.
Of course, the downside is that Rust’s ABI effectively depends on accidental details of the compiler’s internal data structures and serialization system, which is why Rust is not getting a stable ABI any time soon.
-
-
-
-
I should’ve been more clear. Rust will not recompile third-party crates most of the time. It will if you run
cargo clean, if you change compile options (e.g., activate or deactivate LTO), or if you upgrade the compiler, but during regular development, it won’t happen too much. However, there is a build forcargo check, and a build forcargo test, and yet another build forcargo build, so you might end up still compiling your project three times.I mentioned keeping crates under control, because it takes our C.I. system at work ~20 minutes to build one of my projects. About 5 minutes is spent building the project a first time to run the unit tests, then another 10 minutes to compile the release build; the other 5 minutes is spent fetching, building, and uploading a Docker image for the application. The C.I. always starts from a clean slate, so I always pay the compilation price, and it slows me down if I test a container in a staging environment, realize there’s a bug, fix the bug, and repeat.
One way to make sure that your build doesn’t take longer than is needed to is be selective in your choice of third party crates (I have found that the quality of crates varies a lot) and making sure that a crate pays for itself. serde and rayon are two great libraries that I’m happy to include in my project; on the other hand, env_logger brings a few transitive libraries for coloring the log it generates. However, neither
journalctlnordocker container logsshow colors, so I am paying a cost without getting any benefit. -
-
Definitely, this is why MLton is doing it, it’s a whole program optimizing compiler. The compilation speed tradeoff is so severe that its users usually resort to using another SML implementation for actual development and debugging and only use MLton for release builds. If we can figure out how to make whole program optimization detect which already compiled bits can be reused between builds, that may make the idea more viable.
-
In last discussion, I argued for multi-staged process that improved developer productivity, esp keeping mind flowing. The final result is as optimized as possible. No wait times, though. You always have something to use.
-
-
Interesting you mentioned that; Chris Cannam has a build setup for this workflow: https://bitbucket.org/cannam/sml-buildscripts/
-
-
-
-
-
-
-
If both http:// and https:// is available, I think Chrome should redirect to the https:// page instead of complaining about the http:// counterpart.
-
There’s no guarantee that the site on port 80 is the site as the one on port 443. That’s why HTTPS Everywhere is a whitelist of sites where this is true.
-
-
-
For those wanting the rationale, this is in the same Pony article:
“From a practical perspective, having division as a partial function is awful. You end up with code littered with trys attempting to deal with the possibility of division by zero. Even if you had asserted that your denominator was not zero, you’d still need to protect against divide by zero because, at this time, the compiler can’t detect that value dependent typing. So, as of right now (ponyc v0.2), divide by zero in Pony does not result in error but rather 0.”
-
I’m going to be (when I have time) writing a longer and more detailed discussion of the issue.
-
Im sure many of us would find it interesting. I have a total, mental block on divide by zero given it’s always a bug in my field. This thread is refreshingly different. :)
-
I’ll post it on lobste.rs when its done and I’ve had several people review and give feedback.
-
Thanks!
-
-
-
-
-
-
-
The 46 lines of javascript inclusions, CSS and meta information at the top are not minimal.
-
On the contrary there’s nothing wrong with a few lines of Javascript, CSS or tags. For example, I include MathJAX javascript on my personal website. As far as I know, if you have Javascript disabled you just get the raw LaTeX tags instead of the rendered mathematics. That’s good, that’s progressive Javascript. JS isn’t inherently bad. Nor is there anything wrong with a few bytes of meta tags giving important semantic information about a website.
The JS in this website looks to me like it’s enabling some sort of Emoji thing, probably to get Emoji rendering properly in browsers and on operating systems where they can’t be rendered properly.
JS isn’t bad! Tracking is bad, ads are bad, bloat is bad.
-
-
-
Prob better to just modernize the existing C. net-tools shouldn’t have to depend on the installation of a python environment.
-
-
-
-
An Internet search tells me you still need a python environment to use shedskin and that it only supports python 2.4-2.6.
-
No, once you compile/transpile the Python application to C++, the resulting application does not need Python.
There is also a project named Grumpy, for compiling/transpiling Python to Go. Once that is done, Python is not needed for those executables either. Google used Grumpy to convert the Python code used to run YouTube, from Python to Go.
-
-
-
-
-
-
I assume using splice from Ruby would result in a similar increase in speed.
-
After writing Go for 5 years, I’d recommend Rust for C developers. It’s more complicated than Go for sure, but also has more to offer. The lack of garbage collection and support of generics are definitely a plus compared to Go.
Go is a better language for junior devs, but I wouldn’t call C programmers junior. They should be able to digest Rust’s complexity.
-
-
Rust can definitely get overly complex if the developers show no constraint (i.e. type golf), but the control afforded by manual memory management makes up for it, IMHO. Unless it’s a one-run project, performance will eventually matter, and fixing bad allocation practices after the fact is a lot harder than doing it right from the beginning.
-
Couldn’t they just start with a C-like subset of Rust adding from there to their arsenal what extra features they like? It’s what I was going to recommend to those trying it for safety-critical use since they likely know C.
-
-
I think C++ as a better C works because you still have libc besides the STL, etc. The Rust standard library uses generics, traits, etc. quite heavily and type parameters and lifetime parameters tend to percolate to downstream users.
Though I think a lot of value in Rust is in concepts that may initially add some complexity, such the borrow checker rules.
-
The problem with C++ is its complexity at the language level. I have little hope of teams of people porting various tools for static analysis, verification, and refactoring to it that C and Java already have. Certifying compilers either. C itself is a rough language but smaller. The massive bandwagon behind it caused lots of tooling to be built, esp FOSS. So, I now push for low-level stuff either safer C or something that ties into C’s ecosystem.
-
-
You could argue the same for C++ (start with C and add extra features). Complexity comes with the whole ecosystem from platform support (OS, arch), compiler complexity (and hence subtle difference in feature implementations) to the language itself (C++ templates, rust macros). It’s challenging to limit oneself to a very specific subset on a single person project, it’s exponentially harder for larger teams to agree on a subset and adhere to it. I guess I just want a safer C not a new C++ replacement which seems to be the target for newer languages (like D & Rust).
-
It’s challenging to limit oneself to a very specific subset on a single person project, it’s exponentially harder for larger teams to agree on a subset and adhere to it.
I see your overall point. It could be tricky. It would probably stay niche. I will note that, in the C and Java worlds, there’s tools that check source code for compliance with coding standards. That could work for a Rust subset as well.
“I guess I just want a safer C not a new C++ replacement which seems to be the target for newer languages (like D & Rust).”
I can’t remember if I asked you what you thought about Cyclone. So, I’m curious about that plus what you or other C programmers would change about such a proposal.
I was thinking something like it with Rust’s affine types and/or reference counting when borrow-checking sucks too much with performance acceptable. Also, unsafe stuff if necessary with the module prefixed with that like Wirth would do. Some kind of module system or linking types to avoid linker errors, too. Seemless use of existing C libraries. Then, an interpreter or REPL for the productivity boost. Extracts to C to use its optimizing and certifying compilers. I’m unsure of what I’d default with on error handling and concurrency. First round at error handling might be error codes since I saw a design for statically checking their correct usage.
-
I can’t remember if I asked you what you thought about Cyclone. So, I’m curious about that plus what you or other C programmers would change about such a proposal.
I looked at it in the past and it felt like a language built on top of C similar to what a checker tool with annotations would do. It felt geared too much towards research versus use and the site itself states:
Cyclone is no longer supported; the core research project has finished and the developers have moved on to other things. (Several of Cyclone’s ideas have made their way into Rust.) Cyclone’s code can be made to work with some effort, but it will not build out of the box on modern (64 bit) platforms).
However if I had to change Cyclone I would at least drop exceptions from it.
I am keeping an eye on zig and that’s closest to how I imagine a potentially successful C replacement - assuming it takes up enough community drive and gets some people developing interesting software with it.
That’s something Go had nailed down really well. The whole standard library (especially their crypto and http libs) being implemented from scratch in Go instead of being bindings were a strong value signal.
-
re dropping exceptions. Dropping exceptions makes sense. Is there another way of error handling that’s safer or better than C’s that you think might be adoptable in a new, C-like language?
re Zig. It’s an interesting language. I’m watching it at a distance for ideas.
re standard library of X in X. Yeah, I agree. I’ve been noticing that pattern with Myrddin, too. They’ve been doing a lot within the language despite how new it is.
-
Dropping exceptions makes sense. Is there another way of error handling that’s safer or better than C’s that you think might be adoptable in a new, C-like language?
Yes, I think Zig actually does that pretty well: https://andrewkelley.me/post/intro-to-zig.html#error-type
edit: snippet from the zig homepage:
A fresh take on error handling that resembles what well-written C error handling looks like, minus the boilerplate and verbosity.
-
Thanks for the link and tips!
-
-
-
-
-
-
-
-
-
-
Junior and senior developers also enjoy language features such as map, reduce, filter, and generics.
Those are great!
deterministic memory allocation, soft realtime, forced error checking, zero-cost abstractions, and (of course) memory safety.
Where are you finding juniors who care about this stuff? (no, really - I would like to know what kind of education got them there).
-
-
-
Map, reduce and filter are easily implemented in Go
How? Type safe versions of these, that is, without
interface{}and hacky codegen solutions?-
Here are typesafe examples for Map, Filter etc: https://gobyexample.com/collection-functions
Implementing one Map function per type is often good enough. There is some duplication of code, but the required functionality is present. There are many theoretical needs that don’t always show up in practice.
Also, using
go generate(which comes with the compiler), generic versions are achievable too. For example like this: https://github.com/kulshekhar/fungen-
When people say “type safe map/filter/reduce/fold” or “map, reduce, filter, and generics” they are generally referring to the ability to define those functions in a way that is polymorphic, type safe, transparently handled by the compiler and doesn’t sacrifice runtime overhead compared to their monomorphic analogs.
Whether you believe such facilities are useful or not is a completely different and orthogonal question. But no, they are certainly not achievable in Go and this is not a controversial claim. It is by design.
-
-
The implementation of generics in C++ also works by generating the code per required type.
But they are not really comparable. In C++, when a library defines a generic type or function, it will work with any conforming data type. Since the Go compiler does not know about generics, with
go generateone can only generate ‘monomorphized’ types for a set of predefined data types that are defined an upstream package. If you want different monomorphized types, you have to import the generic definitions and rungo generatefor your specific types.unless you consider code generation
By that definition, any language is a generic language, there’s always Bourne shell/make/sed for code generation ;).
-
-
That’s why I said “transparently handled by the compiler.” ;-)
-
-
Yes. And for the reasons mentioned by @iswrong.
My larger point here really isn’t a technicality. My point is that communication is hard and not everyone spells out every point is precise detail, but it’s usually possible to infer the meaning based on context.
-
-
I mean, I’ve written Go code nearly daily since before it was 1.0. I don’t need to argue with you about whether generics are “needed,” which is a pretty slimy way to phrase this.
Seems to me like you’re trying to pick a fight. I already said upthread that the description of generics is different from the desire for them.
-
-
-
-
-
-
-
-
-
-
-
-
-
Compiling a very large go project with a cold cache might take a minute (sub-second once the cache is warm).
Compiling a fairly small rust app with a warm cache has taken me over a minute (I think it’s a little better than that now).
-
-
-
-
-
I suppose I know why, but I hate that D is always left out of discussions like this.
-
-
-
Yeah, me too. I really love D. Its metaprogramming alone is worth it.
For example, here is a compile-time parser generator:
-
This is a good point. I had to edit out a part on that a language without major adoption is less suitable since it may not get the resources it needs to stay current on all platforms. You could have the perfect language but if somehow it failed to gain momentum, it turns into somewhat of a risk anyhow.
-
-
-
-
Both https://github.com/janestreet/async and https://github.com/ocsigen/lwt allow concurrent programming in OCaml. Parallelism is what you’re talking about, and I think there are plenty of domains where single process parallelism is not very important.
-
You are right. There is Multicore OCaml, though: https://github.com/ocamllabs/ocaml-multicore
-
-
-
-
As of last year it’s open source, joining LDC
with discussions on HackerNews and Reddit.
-
-
-
As one insignificant user of this language, please stop adding these tiny edge case syntax variations and do something about performance. But I am one small insignificant user …
-
This is exactly the attitude that leads to maintainers’ burn outs.
Do realize this:
- Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.
- Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.
- If you want something to happen in an open source project, volunteer to do it.
(None of this is aimed at you personally, I don’t know who you are. I’m dissecting an attitude that you’ve voiced, it’s just all too common.)
-
Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.
I agree with the general direction of your post, but Python is a product and it is marketed to people, through the foundation and advocacy. It’s not a commercial product (though, given the widespread industry usage, you could argue it somewhat is). It’s reasonable of users to form expectations.
Where it goes wrong is when individual users claim that this also means that they need to be consulted or their consultation will steer the project to the better. http://www.ftrain.com/wwic.html has an interesting investigation of that.
-
Where it goes wrong is when users claim that this also means that they need to be consulted or their consultation will steer the project to the better.
Wait, who is the product being built for, if not the user? You can say I am not a significant user, so my opinion is not important, as opposed to say Google which drove Python development for a while before they focused on other things, but as a collective, users’ opinions should matter. Otherwise, it’s just a hobby.
-
Sorry, I clarified the post: “individual users”. There must be a consultation process and some way of participation. RFCs or PEPs provide that.
Yet, what we regularly see is people claiming how the product would be a better place if we listened to them (that, one person we never met). Or, alternatively, people that just don’t want to accept a loss in a long-running debate.
I don’t know if that helps clarifying, it’s a topic for huge articles.
-
I often find what people end up focusing on - like this PEP - is bike shedding. It’s what folks can have an opinion on after not enough sleep and a zillion other things to do and not enough in depth knowledge. Heck I could have an opinion on it. As opposed to hard problems like performance where I would not know where to start, much less contribute any code, but which would actually help me and, I suspect, many other folks, who are with some sighing, migrating their code to Julia, or, like me, gnashing their teeth at the ugliness of Cython.
-
Yeah, it’s that kind of thing. I take a harsh, but well-structured opinion any time and those people are extremely important. What annoys me is people following a tweet-sized mantra to the end, very much showing along the path that they have not looked at what is all involved or who would benefit or not knowing when to let go off a debate.
-
-
-
-
-
Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.
Regrettably, a lot of languages and ecosystems suffer greatly from the incoherence that this sort of permissive attitude creates.
Software is just as much about what gets left out as what gets put in, and just because Jane Smith and John Doe have a pet feature they are excited about doesn’t mean they should automatically be embraced when there are more important things on fire.
-
the incoherence that this sort of permissive attitude creates
The Haskell community would’ve just thrown PEP 572 behind
{-# LANGUAGE Colonoscopy #-}and been done with it.Sure, this doesn’t get us out of jail free with regard to incoherence, but it kicks down the problem from the language to the projects that choose to opt-in.
-
I find it hard to see this as a good thing. For me, it mostly highlights why Haskell is a one-implementation language… er, 2 ^ 227 languages, if
ghc --supported-extensions | wc -lis to be taken literally. Of course, some of those extensions are much more popular than others, but it really slows down someone trying to learn “real world” Haskell by reading library code.-
Of course, some of those extensions are much more popular than others
Yeah, this is a pretty interesting question! I threw some plots together that might help explore it, but it’s not super conclusive. As with most things here, I think a lot of this boils down to personal preference. Have a look:
https://gist.github.com/atondwal/ee869b951b5cf9b6653f7deda0b7dbd8
-
-
-
Yes. Exactly this. One of the things I value about Python is its syntactic clarity. It is the most decidedly un-clever programming language I’ve yet to encounter.
It is that way at the expense of performance, syntactic compactness, and probably some powerful features that could make me levitate and fly through the air unaided if I learned them, but I build infrastructure and day in, day out, Python gets me there secure in the knowledge that I can pick up anyone’s code and at the VERY LEAST understand what the language is doing 99% of the time.
-
-
I find that “people working on what interests them” as opposed to taking a systematic survey of what use cases are most needed and prioritizing those is a hard problem in software projects, and I find it curious that people think this is not a problem to be solved for open source projects that are not single writer/single user hobby projects.
Python is interesting because it forms core infrastructure for many companies, so presumably they would be working on issues related to real use cases. Projects like numpy and Cython are examples of how people see an important need (performance) and go outside the official language to get something done.
“If you want something to happen in an open source project, volunteer to do it.” is also one of those hostile attitudes that I find curious. In a company with a paid product of course that attitude won’t fly, but I suspect that if an open source project had that attitude as a default, it would gradually lose users to a more responsive one.
As an example, I want to use this response from a library author as an example of a positive response that I value. This is a library I use often for a hobby. I raised an issue and the author put it in the backlog after understanding the use case. They may not get to it immediately. They may not get to it ever based on prioritization, but they listened and put it on the list.
Oddly enough, I see this kind of decent behavior more in the smaller projects (where I would not expect it) than in the larger ones. I think the larger ones with multiple vendors contributing turn into a “pay to play” situation. I don’t know if this is the ideal of open source, but it is an understandable outcome. I do wish the hostility would decrease though.
-
Performance has never been a priority for Python and this probably won’t change, because as you said, there are alternatives if you want Python’s syntax with performance. Also its interoperability with C is okeish and that means that the small niche of Python’s users that use it for performance critical operations that are not already supported by Numpy, Numba and so on, will always be free to go that extra mile to optimize their code without much trouble compared to stuff like JNI.
If you want raw performance, stick to C/C++ or Rust.
-
I also observe the same tendency of smaller projects being more responsive, but I think the issue is scale, not “pay to play”. Big projects get so much more issue reports but their “customer services” are not proportionally large, so I think big projects actually have less resource per issue.
-
-
-
-
-
Have you tried Pypy? Have you tried running your code through Cython?
Have you read any of the zillion and one articles on improving your Python’s performance?
If the answer to any of these is “no” then IMO you lose the right to kvetch about Python’s performance.
And if Python really isn’t performant enough for you, why not use a language that’s closer to the metal like Rust or Go or C/C++?
-
Yes to all of the above. But not understanding where all the personal hostility is coming from. Apparently having the opinion that “Should := be part of Python” is much less important than “Let’s put our energies towards getting rid of the GIL and creating a kickass implementation that rivals C++” raises hackles. I am amused, entertained but still puzzled at all the energy.
-
There was annoyance in my tone, and that’s because I’m a Python fan, and listening to people kvetch endlessly about how Python should be something it isn’t gets Ooooold when you’ve been listening to it for year upon year.
I’d argue that in order to achieve perf that rivals C++ Python would need to become something it’s not. I’d argue that if you need C++ perf you should use C++ or better Rust. Python operates at a very high level of abstraction which incurs some performance penalties. Full stop.
-
This is an interesting, and puzzling, attitude.
One of the fun things about Cython was watching how the C++ code generated approaches “bare metal” as you add more and more type hints. Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.
Haskell is a high level language that compiles down to very efficient code (barring laziness and thunks and so on).
Yes, I find this defense of the slowness of Python (not just you but by all commentators here) and the articulation that I, as one simple, humble user, should just shut up and go away kind of interesting.
I suspect that it is a biased sample, based on who visits this post after seeing the words “Guido van Rossum”
-
-
-
Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.
Isn’t that what mypy is attempting to do? I’ve not been following Python for years now, so really have no horse in this race. However, I will say that the number of people, and domains represented in the Python community is staggering. Evolving the language, while keeping everyone happy enough to continue investing in it is a pretty amazing endeavor.
I’ll also point out that Python has a process for suggesting improvements, and many of the core contributors are approachable. You might be better off expressing your (valid as far as I can see) concerns with them, but you might also approach this (if you care deeply about it) by taking on some of the work to improve performance yourself. There’s no better way to convince people that an idea is good, or valid than to show them results.
-
Not really. Mypy’s goal is to promote type safety as a way to increase program correctness and reduce complexity in large systems.
It doesn’t benefit performance at all near as I can tell, at least not in its current incarnation.
Cython DOES in fact do this, but the types you hint with there are C types.
-
Ah, I thought maybe MyPy actually could do some transformation of the code, based on it’s understanding, but it appears to describe itself as a “linter on steroids,” implying that it only looks at your code in a separate phase before you run it.
Typed Racket has some ability to optimize code, but it’s not nearly as sophisticated as other statically typed languages.
-
-
-
-
In no way did I imply that you should “shut up and go away”.
What I want is for people who make comments about Python’s speed to be aware of the alternatives, understand the trade-offs, and generally be mindful of what they’re asking for.
I may have made some false assumptions in your case, and for that I apologize. I should have known that this community generally attracts people who have more going on than is the norm (and the norm is unthinking end users posting WHY MY CODE SO SLOW?
-
Hey, no problem! I’m just amused at the whole tone of this set of threads set by the original response (not yours) to my comment, lecturing me on a variety of things. I had no idea that (and can’t fathom why) my brief comment regarding prioritization decisions of a project would be taken so personally and raise so much bile. What I’m saying is also not so controversial - big public projects have a tendency to veer into big arguments over little details while huge gaps in use cases remain. I saw this particular PEP argument as a hilarious illustration of this phenomenon in how Python is being run.
-
Thinking about this a little more - sometimes, when languages ‘evolve’ I feel like they forget themselves. What makes this language compelling for vast numbers of programmers? What’s the appeal?
In Python’s case, there are several, but two for sure are a super shallow learning curve, and its tendency towards ‘un-clever’ syntax.
I worry that by morphong into something else that’s more to your liking for performance reasons, those first two tenets will get lost in the shuffle, and Python will lose its appeal for the vast majority of us who are just fine with Python’s speed as is.
-
Yes, though we must also remember that as users of Python, invested in it as a user interface for our code ideas, we are resistant to any change. Languages may lose themselves, but changes are sometimes hugely for the better. And it can be hard to predict.
In Python’s 2.x period, what we now consider key features of the language, like list comprehensions and generator expressions and generators, were “evolved” over a base language that lacked those features altogether, and conservatives in the community were doubtful they’d get much use or have much positive impact on code. Likewise for the class/type system “unification” before that. Python has had a remarkable evolutionary approach over its long 3-decade life, and will continue to do so even post-GvR. That may be his true legacy.
-
-
-
-
-
-
-
-
-
-
He assumes that machine learning will not also change mathematics. This is baseless.
-
-
I might have missed an important point here, but couldn’t a more lightweight lambda / block syntax (not scoped) achieve approximately the same thing?
Something like:
z = {y=2}Where
{is for a block of code.Instead of:
z = (y := 2)Or are there fundamental differences?
-
It’s also developer-friendly because of its excellent wiki.
I learned Linux doing everything by hand on a Slackware system, then moved to Ubuntu after ~8 years when I realized I’d stopped learning new things. Then a couple years ago I realized I didn’t understand how a bunch of things worked anymore (systemd, pulseaudio, Xorg, more). I looked at various distros and went with Arch because its wiki had helped me almost every time I’d had an issue.
Speaking of distros, I’m currently learning Nix and NixOS. It’s very nice so far. If I can learn to build packages I’ll probably replace lobsters-ansible with it (the recent issues/PRs/commits tell a tale of my escalating frustration at design limitations). Maybe also my personal laptop: I can experiment first with using nix to try xmonad first because it’s mostly configured by editing + recompiling) and deal with python packaging, which has never worked for me, then move completely to NixOS if that goes well.
-
-
The Arch wiki actually seems to be the only good documentation for using the advanced functionality of newer freedesktop components like pulseaudio, or much older software like Xorg.
But I’ve noticed it’s documentation for enterprise software like ZFS is usually hot garbage. Not surprising given the community. The recommendations are frequently hokey nonsense: imaginary micro-optimizations or blatantly incorrect feature descriptions.
What do you find better about nix for making packages than, say, making an rpm or deb? I’ve found those package systems valuable for large scale application deployment. Capistrano has also been nice for smaller scale, with its ability to deploy directly from a repo and roll back deployments with a simple symlink swap. And integration libraries are usually small enough that I’m comfortable just importing the source into my project and customizing them, which relieves so many minor tooling frustrations overall.
Of course in the end the best deployment system is the one you’ll actually use, so if you’re excited about packaging and deploying with nix, and will thus devote more time and energy to getting it just right, then that’s de facto the best option.
-
What do you find better about nix for making packages than, say, making an rpm or deb?
I don’t, yet. The “If I can learn to build packages” sentence links to an issue I’ve filed. I was unable to learn how to do so from the official documentation. I’ve almost exclusively been working in languages (PHP, Python, Ruby, JavaScript) that rpm/deb have not had good support for, prompting those languages to each implement their own package management systems that interface poorly or not at all with system packaging.
I’ve used Capistrano, Chef, Puppet, and currently use Ansible for deployment. Capistrano and Ansible at least try to be small and don’t have a pretensions to being something other than an imperative scripting tool, but I’ve seen all of them break servers on deployment, let servers drift out of sync with the config, or fail to be able to produce new deployments that match the existing one. Nix/NixOS/NixOps approach the problem from a different direction; it looks like they started from what the idea of system configuration is instead of scripting the manual steps of maintaining one. Unfortunately nix replicates the misfeature of templating config files and providing its own config file on top of them instead of checking complete config files into a repo. Hopefully this won’t be too bad in practice, though it’s not a good sign that they implemented a programming language.
I appreciate your closing sentiment, but I’m not really trying to reach new heights of system configuration. I’m trying to avoid losing time to misconfiguration caused by services that fundamentally misunderstand the problem, leading to booby traps in common usage. I see almost all of my experience with packaging + deployment tools as a loss to be minimized in the hopes that they waste less time than hand-managing the global variables of public mutable state that is a running server.
-
Hmmm. I don’t think the problems you listed are 100% avoidable with any tool, just easier in some rather than others.
I like Puppet and Capistrano well enough. But I also think packaging a Rails application as a pre-built system package is definitely the way to go, with all gems installed and assets compiled at build time. That at least makes the app deployment reproducible, though it does nothing for things like database migrations.
-
-
What do you find better about nix for making packages than, say, making an rpm or deb?
Let me show you a minimal nix package:
pkgs.writeScriptBin "greeter" "echo Hello $1!"Et voila! You have a fine nix package of a utility called
greeterthat you can let other nix packages depend on, install to your environment as a user or make available innix-shell. Here’s a function that returns a package:greeting: pkgs.writeScriptBin "greeter" "echo ${greeting} $1!"What you have here is a lambda expression, that accepts something that you can splice into a string and returns a package! Nix packages in
nixpkgsare typically functions, and they offer an a great amount of customizability without much effort (for both the author and the user).At work, we build, package and deploy with nix (on the cloud and on premises), and we probably have ~1000 nix packages of our own. Nobody is counting though, since writing packages doesn’t feel like a thing you do with nix. Do you count the number of curly braces in your code, for instance? If you’re used to purely functional programming,
nixis very natural and expressive. So much so that you could actually write your application in the language if it’s IO system were designed for it.It also helps a lot that nix can seamlessly be installed on any Linux distro (and macOS) without getting in the way of its host.
-
-
-
If I can learn to build packages I’ll probably replace lobsters-ansible with it
Exactly. I don’t have much experience with Nix (none, actually). But in theory it seems like it can be a really nice OS-level replacement for tools like Ansible, SaltStack, etc.
-
This is exactly what NixOps does! See here.
-
Thanks for the video. I’ll watch it over the weekend!
Curious - are you also running NixOS on your personal machine(s)? I’ve been running Arch for a long time now but considering switching to Nix just because it makes so much more sense. But the Arch documentation and the amount of packages available (if you count the AUR in) is something that’s difficult to leave.
-
Yes, I’m using it on my personal machine :). I wouldn’t recommend switching to NixOS all at once, what worked for me was to install the Nix package manager, use it for package management and creating development environments, and then only switch once I was fully convinced that NixOS could do everything I wanted from my Ubuntu install. This took me about a year, even with me using it for everything at work. Another approach would be to get a separate laptop and put NixOS on that to see how you like it.
-
Interesting. I’ll try it out for some time on a VM to get a hang of it. Thanks for the info!
-
-
-
-
-
Even as a Ubuntu user, I’ve frequently found the detailed documentation on the Arch wiki really helpful.
-
I really want to use Nix but I tried installing it last month and it doesn’t seem to have great support for Wayland yet which is a deal breaker for me as I use multiple HiDPI screens and Wayland makes that experience much better. Anyone managed to get Nix working with Wayland?
-
Arch’s wiki explaining how to do everything piecemeal really seems strange given its philosophy is assuming their users should be able to meaningfully help fix whatever problems cause their system to self-destruct on upgrade. It’s obviously appreciated, but still…confusing, given how many Arch users I’ve met who know nothing about their system except what the wiki’s told them.
-
-
I’m happy to help if I can! I’m on the DevOps team at work, where use it extensively, and I did a presentation demonstrating usage at linux.conf.au this year. All my Linux laptops run NixOS and I’m very happy with it as an operating system. My configuration lives here.
-
-
I made a small change to the example to get it to build, and I’ve added it as a comment to your issue.
-
-
-
-
I’m often amused to remember that there are developers who want “bleeding edge” and rolling release and consider that to be “developer friendly”.
What I want more than anything else is to have my computer work the same day-to-day without fear of it breaking. When I do an update, I want to be to easy rollback if something breaks. When I do an update, I want it update the smallest possible number of components to get the change that I needed.
I’d be far more appreciative of articles like this if it was “why I like Arch” rather than “its developer friendly”. Its not “developer friendly”, its “you friendly”. I used Arch for a bit. Its a nightmare for someone like me, and… I’m a developer. That’s fine though. I don’t use it. For my Linux systems, I prefer point releases + long term support.
-
I’m a developer too. At my workplace, the Arch users just upgrade their packages, and have a stable system for a decade, with a small risk of having to configure or force-install a package every N years. The Ubuntu/Mint people do a distro-upgrade or reinstall the system every N years.
I’m not sure which is worse, but there is more cursing and complaints from the Ubuntu/Mint camp.
-
Maybe it’s just me (I like occasionally hacking on drivers and such), but I consider “developer friendly OS” to mean “friendly to developing the OS itself” (which is why I run FreeBSD -CURRENT).
Developing apps is not that special and doesn’t require much “friendliness”, IMO. Just having not-ancient versions of all common libraries/runtimes/etc. is enough.
That said, mainstream Linux distros like Debian/Ubuntu/Fedora do have an “unfriendliness” in terms of splitting headers into separate
-dev/-develpackages. What are they trying to do, save 10kb of disk space per shared library?! Artificially inflate package counts? This is infuriating: “What do you MEAN pkg-config can’t find this lib– OH DAMMIT I need the dev package! What, why is libwhatever-dev not found, oh I need to search because it’s called whatever01despacito-dev-69420.0.0002 WHO EVER THOUGHT THIS IS A GOOD IDEA” -
I haven’t used Arch but I have the same intuition. I run debian stable on my desktop and servers and have very rarely found myself wanting anything more cutting edge.
-
-
I’ll add to the chorus of folks with good experiences using Archlinux. I’ve been using it for almost 10 years now, and it’s been extremely solid. I use it at home, at work, on my laptop and on my various media PCs. My wife uses it now too. There is nothing in particular that has really made me want to switch to something else. NixOS has piqued my curiosity since I read their paper many years ago, but I’ve never given it a fair shake because Arch has been so good to me, and because I tend to be a “worse is better” kind of a guy, but only when necessary. ;-) I love me some
PKGBUILD!I’m not sure I’d sing praises for the Archlinux community. I do like their trial by fire approach to things. But my personal interactions with them haven’t been a shining beacon of friendliness, I can tell you that. I never really fit in with the Archlinux community, so I tend to avoid it. But I think that’s OK.
-
Arch Linux is (mostly) friendly towards experienced users that also strives towards keeping things simple. How can a distro communicate that it’s not for beginners, nor for maximalists (excluding large groups of users, on purpose) while also communicating that it is friendly?
-
I don’t know how to answer your question, but I will clarify my statement. I said that my personal interactions with people in the Archlinux community have not been what I’d call “friendly.” I’m talking about conversing with people, not the official documentation materials, which I’ve found do a great job of being friendly while simultaneously being a great example of the trial-by-fire approach to learning that really works.
To be fair, I think it is very hard to mix trial-by-fire learning (which I think is one of many great ways to learn) with friendliness in an online community. It’s too easy to slip. But that’s just my belief based on what I think I understand about humans.
Like I said, I just don’t fit in with that style of communication. But there are a lot of places that I don’t fit into. And I really do think that’s OK.
EDIT: Also to add a positive note, not all of my interactions with Archlinux community members have been bad. There have been (very) good ones too. I guess it’s just much easier to focus/remember the bad. :-/
-
I guess it’s just much easier to focus/remember the bad
This is a known psychological effect called Negativity Bias. In theory it protects you to from forgetting things that could harm you, and helps you notice potentially dangerous situations.
-
-
-
-
-
The USR1 signal is supported on Linux for making
ddreport the progress. Implementing something similar forcpand other commands, and then make ctrl-t send USR1 can’t be too hard to implement. Surely, it is not stopped by the Linux kernel itself?-
SIGUSR1 has an nasty disadvantage relative to SIGINFO: by default it kills the process receiving it if no handler is installed. 🙁 The behavior you really want is what SIGINFO has, which is defaulting to a no-op if no handler is installed.
- I don’t want to risk killing a long-running complicated pipeline that I was monitoring by accidentally sending SIGUSR1 to some process that doesn’t have a handler for it
- there’s always a brief period between process start and the call to signal() or sigaction() during which SIGUSR1 will be lethal
-
That’s interesting. The hacky solution would be to have a whitelist of processes that could receive SIGUSR1 when ctrl-t was pressed, and just ignore the possibility of someone pressing ctrl-t at the very start of a process.
A whitelist shouldn’t be too hard to maintain. The only tool I know of that handles SIGUSR1 is
dd.
-
On BSD it’s part of the TTY layer, where ^T is the default value of the STATUS special character. The line printed is actually generated by the kernel itself, before sending SIGINFO to the foreground process group. SIGINFO defaults to ignored, but an explicit handler can be installed to print some extra info.
I’m not sure how equivalent functionality could be done in userspace.
-
-
I have a small script and a tmux binding for linux to do this:
#!/bin/sh # tmux-signal pid [signal] - send signal to running processes in pids session # bind ^T run-shell -b "tmux-signal #{pane_pid} USR1" [ "$#" -lt 1 ] && return 1 sid=$(cut -d' ' -f6 "/proc/$1/stat") sig=$2 : ${sig:=USR1} ps -ho state,pid --sid "$sid" | \ while read state pid; do case "$state" in R) kill -s"$sig" "$pid" ;; esac done -
-
-
-
-
No, you don’t need C aliasing to obtain vector optimization for this sort of code. You can do it with standards-conforming code via memcpy(): https://godbolt.org/g/55pxUS
-
Wow, it’s actually completely optimizing out the memcpy()? While awesome, that’s the kind of optimization I hate to depend on. One little seemingly inconsequential nudge and the optimizer might not be able to prove that’s safe, and suddenly there’s an additional O(n) copy silently going on.
-
memset/memcpy get optimized out a lot, hence libraries making things like this: https://monocypher.org/manual/wipe
-
-
-
You can - and thanks for the illustration - but the memcpy is antethical to the C design paradigm in my always humble opinion. And my point was not that you needed aliasing to get the vector optimization, but that aliasing does not interfere with the vector optimization.
-
I’m sorry but the justifications for your opinion no longer hold. memcpy() is the only unambiguous and well-defined way to do this. It also works across all architectures and input pointer values without having to worry about crashes due to misaligned accesses, while your code doesn’t. Both gcc and clang are now able to optimize away memcpy() and auto vars. An opinion here is simply not relevant, invoking undefined behavior when it increases risk for no benefit is irrational.
-
Au contraire. As I showed, C standard does not need to graft on a clumsy and painful anti-alias mechanism and programmers don’t need to go though stupid contortions with allocation of buffers that disappear under optimization , because the compiler does not need it. My code does’t have alignment problems. The justification for pointer alias rules is false. The end.
-
-
In those cases, you need to use an alignment modifier or sizeof. No magic needed. There is a reason that both gcc and clang have been forced to support -fnostrict_alias and now both support may_alias. The memcpy trick is a stupid hack that can easily go wrong - e.g one is not guaranteed that the compiler will optimize away the buffer, and a large buffer could overflow stack. You’re solving a non-problem by introducing complexity and opacity.
-
In what world is memcpy() magic and alignment modifiers aren’t? memcpy() is an old standard library function, alignment modifiers are compiler-specific syntax extensions.
memcpy() isn’t a hack, it’s always well-defined while aliasing can never be well-defined in all cases. Promoting aliasing as a rule is like promoting using the equality operator between floats – it can never work in all cases, though it may be possible to define meaningful behavior in specific cases. Promoting aliasing as a rule is promoting the false idea that C is a thin layer above contemporary architectures, it isn’t. Struct memory is not necessarily the same as array memory, not every machine that C supports can deference an int32 inside of an int64, not every machine can deference an int32 at any offset. Do you want C to die with x86_64 or do you want C to live?
Optimizations don’t need to be guaranteed when the code isn’t even correct in the first place. First make sure your code is correct, then worry about optimizing. You talk about alignment modifiers but they are rarely used, and usually they are used after a bug has already occurred. Code should be correct first, and memcpy() is the rule we should be promoting since it is always correct. Optimizers can meticulously add aliasing for specific cases once a bottleneck has been demonstrated. You’re solving a non-problem by indulging in premature optimization.
-
-
The memcpy hack is a hack because the programmer is supposed to write a copy of A to B and then back to A and rely on the optimizer to skip the copy and delete the buffer. So unoptimized the code may fault on stack overflows for data structures that exist only to make the compiler writers happier. And with a novel architecture, if the programmer wants to take advantage of a new capability - say 512 bit simd instructions , she can wait until the compiler has added it to its toolset and be happy with how it is used.
As for this not working in all cases: Big deal. C is not supposed to hide those things. In fact, the compiler has no idea if the memory is device memory with restrictions on how it can be addressed or memory with a copy on write semantics or …. You want C to be Pascal or Java and then announce that making C look like Pascal or Java can only be solved at the expense of making C unusable for low level programming. Which programming communities are asking for such insulation? None. C works fine on many architectures. C programmers know the difference between portable and non-portable constructs. C compilers can take advantage of SIMD instructions without requiring C programmers to give up low level memory access - one of the key advantages of programming in C. Basically, people who don’t like C are trying to turn C into something else and are offended that few are grateful.
-
You aren’t writing a copy of a buffer back and forth. In your example, you are reducing an encoding of a buffer into a checksum. You are only copying one way, and that is for the sake of normalization. All SIMD code works that way, you always must copy into SIMD registers first before doing SIMD operations. In your example, the aliasing code doesn’t resemble SIMD code both syntactically and semantically as much the memcpy() code does and in fact requires a smarter compiler to transform.
The chance of overflowing the stack is remote, since stacks now automatically grow and structs tend to be < 512 bytes, but if that is a legitimate concern you can do what you already do to avoid that situation, either use a static buffer (jeopardizing reentrancy) or use malloc().
By liberally using aliasing, you are assuming a specific implementation or underlying architecture. My point is that in general you cannot assume arbitrary internal addresses of a struct can always be dereferenced as int32s, so in general that should not be practiced. In specific cases you can alias, but those are the exceptions not the rule.
-
All copies on some architectures reduce to: load into register, store from register. So what? That is why we have a high level language which can translate *x = *y efficiently. The pointer alias code directly shows programmer intent. The memcpy code does not. The “sake of normalization” is just another way of saying “in order to cooperate with the fiction that the inconsistency in the standard produces”.
In many contexts, stacks do NOT automatically grow.Again, C is not Java. OS code, drivers, embedded code, even many applications for large systems - all need control over stack size. Triggering stack growth may even turn out to be a security failure for encryption which is almost universally written in C because in C you can assure time invariance (or you could until the language lawyers decided to improve it). Your proposal that programmers not only use a buffer, but use a malloced buffer, in order to allow the optimizer (they hope) not to use it, is ridiculous and is a direct violation of the C model.
“3. C code can be non-portable. Although it strove to give programmers the opportunity to write truly portable programs, the Committee did not want to force programmers into writing portably, to preclude the use of C as a “high-level assembler;” the ability to write machine-specific code is one of the strengths of C. It is this principle which largely motivates drawing the distinction between strictly conforming program and conforming program.” ( http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2021.htm)
Give me an example of an architecture where a properly aligned structure where sizeof(struct x)%sizeof(int32) == 0 cannot be accessed by int32s ? Maybe the itanium, but I doubt it. Again: every major OS turns off strict alias in the compilers and they seem to work. Furthermore, the standard itself permits aliasing via char* (as another hack). In practice, more architectures have trouble addressing individual bytes than addressing int32s.
I’d really like to see more alias analysis optimization in C code (and more optimization from static analysis) but this poorly designed, badly thought through approach we have currently is not going to get us there. To solve any software engineering problem, you have to first understand the use cases instead of imposing some synthetic design.
Anyways off the airport. Later. vy
-
I’m willing to agree with you that the aliasing version more clearly shows intent in this specific case but then I ask, what do you do when the code aliases a struct that isn’t properly aligned? There are a lot of solutions but in the spirit of C, I think the right answer is that it is undefined.
So I think what you want is the standard to define one specific instance of previously undefined behavior. I think in this specific case, it’s fair to ask for locally aliasing an int32-aligned struct pointer to an int32 pointer to be explicitly defined by the standards committee. What I think you’re ignoring, however, is all the work the standards committee has already done to weigh the implications of defining behavior like that. At the very least, it’s not unlikely that there will be machines in the future where implementing the behavior you want will be non-trivial. Couple that with the burden of a more complex standard. So maybe the right answer to maximize global utility is to leave it undefined and to let optimization-focused coders use implementation-defined behavior when it matters but, as I’m arguing, use memcpy() by default. I tend to defer to the standards committees because I have read many of their feature proposals and accompanying rationales and they are usually pretty thorough and rarely miss things that I don’t miss.
Everybody arguing here loves C. You shouldn’t assume the standards committee is dumb or that anyone here wants C to be something it’s not. As much as you may think otherwise, I think C is good as it is and I don’t want it to be like other languages. I want C to be a maximally portable implementation language. We are all arguing in good faith and want the best for C, we just have different ideas about how that should happen.
-
what do you do when the code aliases a struct that isn’t properly aligned? There are a lot of solutions but in the spirit of C, I think the right answer is that it is undefined.
Implementation dependent.
Couple that with the burden of a more complex standard.
The current standard on when an lvalue works is complex and murky. Wg14 discussion on how it applies shows that it’s not even clear to them. The exception for char pointers was hurriedly added when they realized they had made memcpy impossible to implement. It seems as if malloc can’t be implemented in conforming c ( there is no method of changing storage type to reallocate it)
C would benefit from more clarity on many issues. I am very sympathetic to making pointer validity more transparent and well defined. I just think the current approach has failed and the c89 error has not been fixed but made worse. Also restrict has been fumbled away.
-
-
-
The chance of overflowing the stack is remote, since stacks now automatically grow and structs tend to be < 512 bytes, but if that is a legitimate concern you can
… just copy the ints out one at a time :) https://godbolt.org/g/g8s1vQ
The compiler largely sees this as a (legal) version of the OP’s code, so there’s basically zero chance it won’t be optimised in exactly the same way.
-
-
-
-
You don’t need a large buffer. You can memcpy the integers used for the calculation out one at a time, rather than memcpy’ing the entire struct at once.
Your designation of using memcpy as a “stupid hack” is pretty biased. The code you posted can go wrong, legitimately, because of course it invokes undefined behaviour, and is more of a hack than using memcpy is. You’ve made it clear that you think the aliasing rules should be changed (or shouldn’t exist) but this “evidence” you’ve given has clearly been debunked.
-
Funny use of “debunked”. You are using circular logic. My point was that this aliasing method is clearly amenable to optimization and vectorization - as seen. Therefore the argument for strict alias in the standard seems even weaker than it might. Your point seems to be that the standard makes aliasing undefined so aliasing is bad. Ok. I like your hack around the hack. The question is: why should C programmers have to jump through hoops to avoid triggering dangerous “optimizations”? The answer: because it’s in the standard, is not an answer.
-
Funny use of “debunked”. You are using circular logic. My point was that this aliasing method is clearly amenable to optimization and vectorization - as seen
You have shown a case where, if the strict aliasing rule did not exist, some code could [edit] still [/edit] be optimised and vectorised. That I agree with, though nobody claimed that the existence of the strict aliasing rule was necessary for all optimisation and vectorisation, so it’s not clear what you do think this proves. Your title says that the optimisation is BECAUSE of aliasing, which is demonstrably false. Hence, debunked. Why is that “funny”? And how is your logic any less circular then mine?
The question is: why should C programmers have to jump through hoops to avoid triggering dangerous “optimizations”?
Characterising optimisations as “dangerous” already implies that the code was correct before the optimisation was applied and that the optimisation can somehow make it incorrect. The logic you are using relies on the code (such as what you’ve posted) being correct - which it isn’t, according to the rules of the language (which, yes, are written in a standard). But why is using memcpy “jumping through hoops” whereas casting a pointer to a different type of pointer and then de-referencing it not? The answer is, as far as I can see, because you like doing the latter but you don’t like doing the former.
-
-
-
-
-
-
-
-
-
-
trousers
2 months ago
|
link
| on:
Copyleft Terms May Become Unenforceable in 11 Countries under CPTPP
If a specific class of licenses becomes unenforceable, what does that actually mean to software currently under that license?
Does the software, in that jurisdiction, simply behave as if it has no license?
Or would the copyleft /portion/ of the license simply no longer apply (eg. people would now get it without those restrictions)? -
-
I’ve had race conditions, problems with forgetting to close channels, issues with the verbosity of converting between types, stress with refactoring packages and import paths, vendoring issues and many other things, but
panic: runtime error: invalid memory address or nil pointer dereferenceis one that I can’t really remember ever having seen in Go.I just don’t use “nil” for anything but empty error values.
-
I had been vaguely aware of Copperhead OS but never looked into it or used it (I used Cyanogenmod before they imploded, and Lineage OS thereafter). I don’t know anything about the context for this other than the reddit and hacker news links here. Everything I’ve seen so far makes me feel inclined to be sympathetic to this Daniel Micay fellow, so I can’t help but wonder if there’s any information from his former business partner’s side of the story that would make me feel less sympathetic.
-
He’s a fellow Arch Linux Trusted User. He seemed like a pretty ok dude in my interactions.
-
I also chill in a few old irc channels with strncat post my major arch days, he has a lot of people in the open source community that respect his contributions. My bet is he’ll come out ahead of this if he can get untangled from the copperheados company.
-
-
Daniel Micay was a prolific Rust contributor. (In fact, he is still in the top 20 even if he has been inactive since 2015.) In his Rust work, I found him to be a straight person.
-
-
Before I had internet at home, there were internet connected computers at the local library. A random friendly person was chatting on IRC and taught me how to connect. I was about 15, and this was before ever having used the web, before Netscape Navigator and Altavista. Through IRC I found a channel for chatting about assembly, I think it was #asm, where I also met a friend, which I still have contact with. We were in the demoscene for years, but now we play in the same band, have “proper jobs” and kids.
I’ve had my share of silly arguments on IRC, but in general talked to and met loads of interesting and bright people I might otherwise never have met.
So, thank you IRC protocol designers, IRC server and client software developers, server administrators, moderators, lurkers, ops, bots, helpful people, bright and inquisitive people and not quite so bright and inquisitive people, for making IRC an experience worth remembering.
May you live at least another 30.