But a lot fewer people think holistically in terms of writing an application in “Python + C”.
This is actually my preferred development style now. For some recent work, I was using Python for quick iteration, and an inital prototyping, then I profiled it, found the slow sections in the code, and rewrote those in C++ called through the FFI. I initially wrote the project in C++, but after rewriting it in Python with a few utility functions written in C++ I found it much easier to maintain and debug. I rarely hit segfaults anymore :)
Unfortunately, it’s probably not especially common that C is the engineering language you want
I’ve been thinking about this a lot recently. My prefered combo would be Scheme (for wizardry) + SML (for engineering) + C (for speed/low-level control), but there doesn’t seem to be an easy way combine these languages currently. Sure, there are implementations for each of these languages that emit LLVM bitcode, so I guess you could make it all play nice at the LLVM bitcode level. However, a functional IR at a higher level (i.e. a typed lambda calculus IR that could be targeted by functional languages), would be extremely nice since things like types wouldn’t be lost. I’m working on a project for this right now, but I’m not sure how well it’ll turn out.
Another interesting combination is described in Out of the Tar Pit. The authors suggest a functional language paired with a relational database. I think this model is very interesting and plan to try it with game development (using a custom lightweight in-memory database) with the goal of being as performant as possible (which should be easy since it’s a very data-driven model). This idea isn’t really new - it’s partially what Rails does already, which is why it can scale to larger projects successfully. The database is an engineering tool that handles typing.
Finally, I really like Racket’s ability to combine languages together in the same project. You can have two different modules in one project, one with Typed Racket and one with regular Racket (i.e. dynamically typed), and the two languages will happily interop with each other. To be honest, I haven’t really had a reason to do this in any of my non-trival Racket projects yet, but from what I’ve experiment with, it seems to work well.
Terra Language seems like an interesting mixed language approach. High level Lua code and low level C-like Terra code can be mixed, with the Lua code being able to generate Terra for macro type purposes. One could write most of the system in Lua, dropping down to Terra for speed and mix and match as needed.
If you combine Python and C++, you might find Nim interesting. It’s a language with Python-like syntax that compiles to C with low-latency GC (several actually). You might still prefer C++ for something requiring performance advantages of its abstractions. Given all the Python-to-Go conversions, there’s likely a set of apps you’d write where Nim’s performance alone would be fine. From there, you might either use C++ for the rest or see what of its abstractions you can port over to Nim to retain benefits of Python-like style. The times I’ve seen Racket or Haskell used for things like C++ metaprogramming tell me there’s probably some value for Python/C++ programmers in doing C++ within Python-like language.
Just a brainstorming aloud here. :)
Edit to add: I wrote that comment reading your first paragraph. Then I see you bring up Racket. Maybe I wouldn’t have to sell you on benefits of mocking up a system language in a macro language after all.
In my experience with FFI (at least in Python, with ctypes or cffi), crossing the boundary between the languages is very costly. In libparsing, which uses a C core and a Python module, 90% of the time is spend wrapping match objects from C to Python. This is arguably an edge case given the fact that parsers do generate a ton of data (the match nodes and the AST).
Painfully true. It’s why both OpenVMS calling standard and Microsoft’s CLR were brilliant in standardizing that part across languages. Then, one can use the right tool for the job mixing them up as needed. That most hardware and compilers are optimized for C means its data types, calling conventions, etc are my default recommendation for this compatibility layer now regardless of what’s technically superior. If it’s possible: sometimes clean-slate stuff that’s not compatible actually gets adoption. Principle also applies to other massively-deployed runtimes like .NET, JVM, Apple Objective-C, or Android ART.
In some cases, a subset of a massively-deployed language becomes this layer like we’ve seen with Javascript. One I eyeballed for that recently was Go language due to all the articles griping about it. The solution would be some language fixing its shortcomings while totally plugging into its ecosystem with zero-cost integration. That might get a good response. Last popular one with low barrier-to-entry I looked into assuring was PHP. Turns out Quercus beat me to general concept by re-implementing it in Java with access to Java libraries. Still possible to redo something like it or Python in Nim supporting a mix of better-via-new-lang components and legacy-stuff-gradually-rewritten-or-enhanced.
JavaScript was indeed a promising candidate, with a highly optimized runtime, composite data types and prototypical inheritance. If only the steering committee of that language decided that instead of adding more features and make it look like Java or C#, they should focus on adding core primitives to make it faster (typing, pragma-like compiler hints), then we would have a pretty good candidate for a runtime on which to quickly build a wide range of languages. I’m sure that if we looked at the languages which are used as a base for other languages, JavaScript would come just after C, or maybe even first!
I never really measured the cost of jumping back and forth between Python and C, but I’m sure it’s horrible. It’s fine for me because I tend to have the C code do a significant amount of processing before returning back into Python. Python type conversions regularly surprise me with how expensive they are, but I guess that’s because Python is the only language I pay close attention to for performance issues (since the problems I tend to solve with it are computationally expensive).
And also if you try different runtimes like Py2, Py3 or PyPy you’ll see how things are handled differently. PyPy for instance has a tendency to reallocate strings, which forces you to copy string values from Python to C. I came to a point where I considered making the C library spit out a string representation of the matches to be parsed by Python. I did not do it, but I bet the performance might not be too different, maybe even better!
I really wish the “emacs thesis”[1] was more widely known. Scripting a native core sounds great in theory, but effectively decomposing the parts so that they can be scripted in a logical, fluent way, is very non trivial.
This is a really interesting article…
But it takes a turn at the end and ends up focusing too much on language choice as the boundary between wizardry and engineering.
Perhaps Rust will help, since it is a more attractive option compared to C. And perhaps we’ll see programming languages in the future developed as a pair: a statically-typed engineering language, and a wizarding scripting language, designed to work together.
I really don’t see it this way at all and really wish that we could evolve the discussion so that it can find a place within the context of a single project or even a single function. To use a crummy boxing metaphor: It’s pretty easy to “take the gloves off” of a statically typed language and to “put the gloves on” a dynamically typed one.
The metaprogramming-oriented languages with a general-purpose, high-performance core seem to be the most powerful here if looking what side to choose. They’re naturally highly productive. They can be constrained with either restriction of features (eg like Ada profiles) or DSL’s for specific things (eg Racket or sklogic’s mbase). In mbase, we see it where he uses DSL’s for most data-handling, embedding of Standard ML when he wants more safety/predictability, embedding of Prolog if it’s easy to express as logic, and (when DSL’s don’t fit) he drops into his LISP that everything integrates with. That combined with Design-by-Contract or other formal specification covers the common uses of static types with more on top.
I never heard of mbase before, you’re really good at finding gems like this! I feel the explosion of programming languages in the last 15 years hints at the need for more flexibility, both on syntax and semantics. I like the idea of C– as as low level (but not too low) foundation on which to build more specific/complex languages, and then LLVM arrived. I feel Python’s new typing module could be put to good use in a meta-programming approach.
If you read Hacker News stuff, keep an eye out for “sklogic” in any older threads you find about compilers, DSL’s, etc. He was really clever in how he handled that stuff. For instance, he’d avoid discussing which of two parsing concepts with opposing tradeoffs was best to work within: he’d just implement pieces of them both in a new DSL for parsing or something. His using SML within LISP was neat, too. He used it to win an argument with me over whether SML was better than LISP for compiler development by showing my OR should be an AND. ;) More pragmatic about tooling than lots of folks. He got banned for being an asshole in debates but lots of useful info scattered over threads worth collecting into one place. That would be time consuming, though.
I think his tool’s list of features by itself shows the power of the approach. So, I just link to it with a description. In most non-metaprogramming-heavy languages, those would be a mix of (language here) and/or external tools with some extra boilerplate or slow integrations via some pipeline.
The list of features was a bit daunting, but intriguing at the same time. I found the presence of a Prolog interpreted in mbase especially interesting, I can see how it could be used to experiment with type systems. I found some docs in a sibling repo https://combinatorylogic.github.io/mbase-docs/intro.html which I’m pasting for people who’d like to see more about mbase, as I find the readme a bit obscure about what mbase is in practice. I’m definitely going to check out how this impressive piece of engineering is built ;)
I can see how it could be used to experiment with type systems.
When I asked how a type system worked, he illustrated it by building a basic one in Prolog in a comment. Probably what he does. So, good call on on that.
“I’m definitely going to check out how this impressive piece of engineering is built ;)”
Hopefully, it’s as good as it appears. I do know he uses it to build commercial, static, analysis products. If not great build for you, it’s a great concept worth improving on. Reminded me of what Alan Kay et al do in STEPS with a metalanguage at bottom with lots of DSL’s on top. Only problem I had was it said it’s a .NET program. I’d rather it be a portable exe built on a simple tech with no chance of Microsoft’s lawyers laying a claim. But, hey, nothing’s perfect. :)
I’ve been thinking of type systems as a set of constraints, and experimenting with a prototype implementation to express them (along with capabilities) in Python – so that’s why it was a “logical” use for that ;)
Do you have any reference to STEPS in your treasure trove of CompSci articles? I’m a big fan of Alan Kay, but the historical material is hard to find (at least for me). I think .NET has been standardized and open sourced last year, at least part of it. As much as I dislike Microsoft, they did some good things technologically speaking in the last couple of years.
The STEPS work is here with a lot of other neat projects they did. STEPS aimed to create an OS in 20,000 loc that, like Smalltalk, allowed the user to see inside or modify the system. They released a series of reports called STEPS Toward Reinvention of Programming. On the site, the links are newest to oldest. So use find on “STEPS” to get to the one closest to the bottom arounnd 2008 or so. Then, work your way up through each report.
Yeah, the .NET runtime was open-sourced. The remaining risk is they patent uses of their tech, too. It’s possible they have leverage on some common tech. I know either they or someone else had patented cut and paste at some point. They also tried to destroy Linux over their patents and SCO’s IP but IBM stepped in saying they’d put a billion into advancing and/or defending it. That rare move of an Evilcorp 1 having a stake in protecting something good from Evilcorp 2 is probably why we didnt see how bad it couldve gotten for FOSS. I know vendors paid them over a billion in patent royalties for Android, though, just to keep it on market.
So, maybe there’s no risk. Maybe there is. I dont know. I advise we make our tools on tech by companies that are better than that or contractually promise no patent suits against FOSS. You can still get hit but at least your core dependency is safer. Used to recommend Wirth’s stuff cuz I couldnt imagine him or ETH suing over it. ;)
Lua occupies the mirror-image of C’s position here; it’s a scripting language that can be easily embedded into software written in any other “engineering” language you like.
This is actually my preferred development style now. For some recent work, I was using Python for quick iteration, and an inital prototyping, then I profiled it, found the slow sections in the code, and rewrote those in C++ called through the FFI. I initially wrote the project in C++, but after rewriting it in Python with a few utility functions written in C++ I found it much easier to maintain and debug. I rarely hit segfaults anymore :)
I’ve been thinking about this a lot recently. My prefered combo would be Scheme (for wizardry) + SML (for engineering) + C (for speed/low-level control), but there doesn’t seem to be an easy way combine these languages currently. Sure, there are implementations for each of these languages that emit LLVM bitcode, so I guess you could make it all play nice at the LLVM bitcode level. However, a functional IR at a higher level (i.e. a typed lambda calculus IR that could be targeted by functional languages), would be extremely nice since things like types wouldn’t be lost. I’m working on a project for this right now, but I’m not sure how well it’ll turn out.
Another interesting combination is described in Out of the Tar Pit. The authors suggest a functional language paired with a relational database. I think this model is very interesting and plan to try it with game development (using a custom lightweight in-memory database) with the goal of being as performant as possible (which should be easy since it’s a very data-driven model). This idea isn’t really new - it’s partially what Rails does already, which is why it can scale to larger projects successfully. The database is an engineering tool that handles typing.
Finally, I really like Racket’s ability to combine languages together in the same project. You can have two different modules in one project, one with Typed Racket and one with regular Racket (i.e. dynamically typed), and the two languages will happily interop with each other. To be honest, I haven’t really had a reason to do this in any of my non-trival Racket projects yet, but from what I’ve experiment with, it seems to work well.
Terra Language seems like an interesting mixed language approach. High level Lua code and low level C-like Terra code can be mixed, with the Lua code being able to generate Terra for macro type purposes. One could write most of the system in Lua, dropping down to Terra for speed and mix and match as needed.
If you combine Python and C++, you might find Nim interesting. It’s a language with Python-like syntax that compiles to C with low-latency GC (several actually). You might still prefer C++ for something requiring performance advantages of its abstractions. Given all the Python-to-Go conversions, there’s likely a set of apps you’d write where Nim’s performance alone would be fine. From there, you might either use C++ for the rest or see what of its abstractions you can port over to Nim to retain benefits of Python-like style. The times I’ve seen Racket or Haskell used for things like C++ metaprogramming tell me there’s probably some value for Python/C++ programmers in doing C++ within Python-like language.
Just a brainstorming aloud here. :)
Edit to add: I wrote that comment reading your first paragraph. Then I see you bring up Racket. Maybe I wouldn’t have to sell you on benefits of mocking up a system language in a macro language after all.
In my experience with FFI (at least in Python, with ctypes or cffi), crossing the boundary between the languages is very costly. In libparsing, which uses a C core and a Python module, 90% of the time is spend wrapping match objects from C to Python. This is arguably an edge case given the fact that parsers do generate a ton of data (the match nodes and the AST).
Painfully true. It’s why both OpenVMS calling standard and Microsoft’s CLR were brilliant in standardizing that part across languages. Then, one can use the right tool for the job mixing them up as needed. That most hardware and compilers are optimized for C means its data types, calling conventions, etc are my default recommendation for this compatibility layer now regardless of what’s technically superior. If it’s possible: sometimes clean-slate stuff that’s not compatible actually gets adoption. Principle also applies to other massively-deployed runtimes like .NET, JVM, Apple Objective-C, or Android ART.
In some cases, a subset of a massively-deployed language becomes this layer like we’ve seen with Javascript. One I eyeballed for that recently was Go language due to all the articles griping about it. The solution would be some language fixing its shortcomings while totally plugging into its ecosystem with zero-cost integration. That might get a good response. Last popular one with low barrier-to-entry I looked into assuring was PHP. Turns out Quercus beat me to general concept by re-implementing it in Java with access to Java libraries. Still possible to redo something like it or Python in Nim supporting a mix of better-via-new-lang components and legacy-stuff-gradually-rewritten-or-enhanced.
JavaScript was indeed a promising candidate, with a highly optimized runtime, composite data types and prototypical inheritance. If only the steering committee of that language decided that instead of adding more features and make it look like Java or C#, they should focus on adding core primitives to make it faster (typing, pragma-like compiler hints), then we would have a pretty good candidate for a runtime on which to quickly build a wide range of languages. I’m sure that if we looked at the languages which are used as a base for other languages, JavaScript would come just after C, or maybe even first!
I never really measured the cost of jumping back and forth between Python and C, but I’m sure it’s horrible. It’s fine for me because I tend to have the C code do a significant amount of processing before returning back into Python. Python type conversions regularly surprise me with how expensive they are, but I guess that’s because Python is the only language I pay close attention to for performance issues (since the problems I tend to solve with it are computationally expensive).
And also if you try different runtimes like Py2, Py3 or PyPy you’ll see how things are handled differently. PyPy for instance has a tendency to reallocate strings, which forces you to copy string values from Python to C. I came to a point where I considered making the C library spit out a string representation of the matches to be parsed by Python. I did not do it, but I bet the performance might not be too different, maybe even better!
I really wish the “emacs thesis”[1] was more widely known. Scripting a native core sounds great in theory, but effectively decomposing the parts so that they can be scripted in a logical, fluent way, is very non trivial.
[1] https://www.gnu.org/software/guile/manual/html_node/The-Emacs-Thesis.html
This is a really interesting article… But it takes a turn at the end and ends up focusing too much on language choice as the boundary between wizardry and engineering.
I really don’t see it this way at all and really wish that we could evolve the discussion so that it can find a place within the context of a single project or even a single function. To use a crummy boxing metaphor: It’s pretty easy to “take the gloves off” of a statically typed language and to “put the gloves on” a dynamically typed one.
The metaprogramming-oriented languages with a general-purpose, high-performance core seem to be the most powerful here if looking what side to choose. They’re naturally highly productive. They can be constrained with either restriction of features (eg like Ada profiles) or DSL’s for specific things (eg Racket or sklogic’s mbase). In mbase, we see it where he uses DSL’s for most data-handling, embedding of Standard ML when he wants more safety/predictability, embedding of Prolog if it’s easy to express as logic, and (when DSL’s don’t fit) he drops into his LISP that everything integrates with. That combined with Design-by-Contract or other formal specification covers the common uses of static types with more on top.
I never heard of mbase before, you’re really good at finding gems like this! I feel the explosion of programming languages in the last 15 years hints at the need for more flexibility, both on syntax and semantics. I like the idea of C– as as low level (but not too low) foundation on which to build more specific/complex languages, and then LLVM arrived. I feel Python’s new typing module could be put to good use in a meta-programming approach.
If you read Hacker News stuff, keep an eye out for “sklogic” in any older threads you find about compilers, DSL’s, etc. He was really clever in how he handled that stuff. For instance, he’d avoid discussing which of two parsing concepts with opposing tradeoffs was best to work within: he’d just implement pieces of them both in a new DSL for parsing or something. His using SML within LISP was neat, too. He used it to win an argument with me over whether SML was better than LISP for compiler development by showing my OR should be an AND. ;) More pragmatic about tooling than lots of folks. He got banned for being an asshole in debates but lots of useful info scattered over threads worth collecting into one place. That would be time consuming, though.
I think his tool’s list of features by itself shows the power of the approach. So, I just link to it with a description. In most non-metaprogramming-heavy languages, those would be a mix of (language here) and/or external tools with some extra boilerplate or slow integrations via some pipeline.
The list of features was a bit daunting, but intriguing at the same time. I found the presence of a Prolog interpreted in mbase especially interesting, I can see how it could be used to experiment with type systems. I found some docs in a sibling repo https://combinatorylogic.github.io/mbase-docs/intro.html which I’m pasting for people who’d like to see more about mbase, as I find the readme a bit obscure about what mbase is in practice. I’m definitely going to check out how this impressive piece of engineering is built ;)
When I asked how a type system worked, he illustrated it by building a basic one in Prolog in a comment. Probably what he does. So, good call on on that.
“I’m definitely going to check out how this impressive piece of engineering is built ;)”
Hopefully, it’s as good as it appears. I do know he uses it to build commercial, static, analysis products. If not great build for you, it’s a great concept worth improving on. Reminded me of what Alan Kay et al do in STEPS with a metalanguage at bottom with lots of DSL’s on top. Only problem I had was it said it’s a .NET program. I’d rather it be a portable exe built on a simple tech with no chance of Microsoft’s lawyers laying a claim. But, hey, nothing’s perfect. :)
I’ve been thinking of type systems as a set of constraints, and experimenting with a prototype implementation to express them (along with capabilities) in Python – so that’s why it was a “logical” use for that ;)
Do you have any reference to STEPS in your treasure trove of CompSci articles? I’m a big fan of Alan Kay, but the historical material is hard to find (at least for me). I think .NET has been standardized and open sourced last year, at least part of it. As much as I dislike Microsoft, they did some good things technologically speaking in the last couple of years.
The STEPS work is here with a lot of other neat projects they did. STEPS aimed to create an OS in 20,000 loc that, like Smalltalk, allowed the user to see inside or modify the system. They released a series of reports called STEPS Toward Reinvention of Programming. On the site, the links are newest to oldest. So use find on “STEPS” to get to the one closest to the bottom arounnd 2008 or so. Then, work your way up through each report.
Yeah, the .NET runtime was open-sourced. The remaining risk is they patent uses of their tech, too. It’s possible they have leverage on some common tech. I know either they or someone else had patented cut and paste at some point. They also tried to destroy Linux over their patents and SCO’s IP but IBM stepped in saying they’d put a billion into advancing and/or defending it. That rare move of an Evilcorp 1 having a stake in protecting something good from Evilcorp 2 is probably why we didnt see how bad it couldve gotten for FOSS. I know vendors paid them over a billion in patent royalties for Android, though, just to keep it on market.
So, maybe there’s no risk. Maybe there is. I dont know. I advise we make our tools on tech by companies that are better than that or contractually promise no patent suits against FOSS. You can still get hit but at least your core dependency is safer. Used to recommend Wirth’s stuff cuz I couldnt imagine him or ETH suing over it. ;)
Lua occupies the mirror-image of C’s position here; it’s a scripting language that can be easily embedded into software written in any other “engineering” language you like.