Thanks! Redbean has baked-in SSL support. I used MbedTLS and I love it, because it’s so much smaller and simpler than OpenSSL and BoringSSL which have huge dependencies like STL and Perl which would have needed to be ported beforehand. However its main drawback is it doesn’t have TLSv3 support. This makes me sad because it causes redbean to have a few extra milliseconds of connection latency if it’s used for self-contained public-facing serving. That’s something I intend to improve. However it’s not that big of an issue, since it’s very easy to use redbean in tandem with something like stunnel which can bolt on the SSL separately. But that only applies over long line internet connections. For local use cases (redbean’s primary and original intent) where CPU utilization is a greater factor than the speed of light, redbean’s SSL goes faster than any of the other stacks I’ve evaluated. For example, I haven’t posted SSL benchmarks on the website yet because wrk (Go) uses 2x as much CPU to establish ECDSA connections as redbean. So once it has the TLSv3 support, it’ll be a sight to behold.
Question: I get the “serve static files” part, and “I can do dynamic stuff with Lua” part. How does the sqlite part work? Can I literally write a full-fledged web app in lua which is, say, saving new user records using sqlite, and those are just getting written to the filesystem in the folder redbean is running in?
Indeed. Here’s the basic idea for how you’d implement a SQLite web service with redbean. https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c We’re in the process of adding features so that redbean can use its own .com file as the database. We have one function for just that, called StoreAsset() but right now SQLite is more stable.
redbean simply embeds sqlite with lua bindings thereby enabling you to handle them in any manner you want. It’s expected that the user knows SQLite and is aware of its caveats. For example, redbean is a forking web server and SQLite file descriptors should never cross fork() boundaries (if the database is mutable). This isn’t a problem if you’re using Lua Server Pages. However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis. This is nice since SQLite is significantly faster than other databases at doing this. Plus you get a nice process sandbox. So in case something goes wrong, like ASAN detects a bug, it’ll only impact one serving session. We’re also working on higher-level frameworks to abstract a lot of the handling aspects. Lastly I’ve also been focusing on creating single-file object store technology (see StoreAsset() and LoadAsset()) that should provide the fastest possible and most easily redistributable model of data storage under a forking model.
However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call sqlite3.open() from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis.
Sorry, but I’m not sure what that means. If each request gets its own SQLite descriptor in OnHttpRequest(), why would “database descriptor and prepared statements persist across requests”?
Because redbean supports HTTP keepalive pipelining and Lua variables are global unless you declare you function local. Forking happens when a TCP connection is received. That connection may send many HTTP messages. Each message for that connection is handled by the same process. If you’ve defined a hook like OnHttpRequest then that means any variables you use, unless you use local, will persist between messages so you can memoize operations on a per-session basis. When the TCP connection closes, it all gets garbage collected by the operating system when the process exits.
I see. Does that imply we can call sqlite3.open() from the global scope in Lua, since by that time we are already in a fork()ed child process for that TCP connection? You said we shouldn’t do that and should “instead create it lazily as requests are handled” so I am a little confused.
If it’s a foo.lua file in your redbean (i.e. a Lua Server Page) then you’re 100% OK. Those files are loaded and re-run on each request. There’s only one file that’s special and it’s .init.lua. That file is run before redbean calls bind() and no forks have been created yet. Anything you do in the global scope of that file, like import modules, will end up defining the main process “template” from which all forked processes are created. Therefore, if you create your SQLite object in the global context of that file, it needs to be strictly read-only. If it is, you can get like 800k RPS performance on read-only SQLite files. But any SQLite files you intend to mutate, it needs to happen after fork, otherwise corruption is possible. It also goes without saying, that you can have multiple SQLite file descriptors open simultaneously! So for example, if you built an Internet forum, all your years-old posts could be moved to immutable read-only SQLite files, for ultra-fast querying due to the pre-fork setup.
That’s awesome! I have an additional question: is it theoretically possible for redbean to support LuaJIT in the future? I’m not sure if the magic tricks you did with actually portable executables would interfere with JIT compilation/execution.
This is one of the coolest projects I’ve come across in the last few years, and I’ll never not upvote it. Even if it was posted every day.
I’m working on this data-science library, and one of my ideas is to wrap exported datasets including their metadata (properties, lineage information, annotations) and a generic html file, and redbean into a zip file. The html page would serve as a sort of ‘data-passport’, showing a preview of the data itself, the metadata, etc. So, basically every dataset would come with a cross-platform way of executing it and showing itself off in a browser if users so choose to do. I haven’t really thought that through entirely, esp. security considerations (making random files executable might not be the best idea to recommend to users), and how that would work having a server process running on an OS where the commandline is not used by most users (Windows) for example.
But even if I decide to not go forward with the idea, just the potential that I could blows my mind…
That’s exactly the kind of use case I intended to enable with redbean. It’d be great if every website with Jupyter Notebooks had a button to “download my notebook into a redbean”. It’s why with Cosmopolitan Libc we’ve been focusing on porting other languages too besides Lua, e.g. Python, LISP, JavaScript, etc. Regarding security, redbean is one of the few application servers that’s viable in an air-gapped environment. What could be more secure than that?
It’s why with Cosmopolitan Libc we’ve been focusing on porting other languages too besides Lua, e.g. Python, LISP, JavaScript, etc.
Oh, I wasn’t aware of this. Please stop, my imagination is running wild. I’m getting to an age where this sort of excitement is keeping me sleepless for nights on end…
Regarding security, redbean is one of the few application servers that’s viable in an airgapped environment. What could be more secure than that?
I don’t disagree with that. I meant more the practice of encouraging my less-then-aware users to just set the executable flag on any random (data or not) file people send them.
What is open source but posting files online containing code and encouraging others to download and execute them? If you want a snapshot of things to come, we managed to get Python 3.6 down to a 5.5mb Actually Portable Executable. That’s including all its standard library dependencies which are baked into the ZIP structure of the binary. It took quite a hacking marathon. https://justine.lol/actually-portable-python.png It’s going to do so much to help facilitate a sharing of things like scientific knowledge that can be reproduced and interacted with for years to come, rather than merely read like a PDF file, or being something that breaks the next time Ubuntu upgrades, or requires that people in other parts of the world with less Internet privilege download gigabytes of bundled stuff? The technology that underpins redbean is going to help address all those things. You can read more about it here: https://justine.lol/ape.html If anyone reading wants to help make this vision of eliminating the barriers to knowledge, please consider sponsoring me on GitHub.
What is open source but posting files online containing code and encouraging others to download and execute them?
Yeah, I guess you are right.
What’s the plan with Python packages? I reckon including pure-Python ones should be feasible, but esp. for scientific data numpy and pandas would probably be much more difficult to support. Having sqlite available will make for a good workaround in a lot of cases, though. Building a more portable version of https://datasette.io/ might be an interesting project idea…
Have you ever looked into Apache Arrow? Intuitively, it seems like it could be a good fit with readbean if you wanted redbean to venture into the data science space a bit. It being portable, and memory efficient, mmappable and all that.
Either way, I’ll definitely keep an eye out for the Python support, even if everything I would like to do right now would already be possible with Lua.
If anyone reading wants to help make this vision of eliminating the barriers to knowledge, please consider sponsoring me on GitHub.
Done, good on you for mentioning it, I wouldn’t have thought to check… If I end up using it in our project, I’m pretty sure I can get them to sponsor as well.
Porting Python packages is something we’ve been doing by hand so far. It’s largely been a community driven effort due to how native dependencies have effectively locked-in so many packages to specific operating systems. Some examples of things we’re doing, is I started writing a Python bindings generator for chibicc which is an embeddable ~300kb C compiler. It’s all part of the Cosmopolitan Libc repo. My intent here is that, when you just want the native code to work, and you don’t care about having the last inch of performance you’d get with GCC and Clang, then all you need to do is plop the .c files into the .com file using a zip tool, and the Python implementation will just magically build them and link them in automatically across platforms. That would be the pain free path. But if you want the full perf for things like scientific computing, then you just need to integrate it with the Cosmopolitan Libc makefile build.
This same concept would naturally also apply to generating Lua bindings in Redbean for any C code where you need extra performance. chibicc is great. Since half as fast as GCC is still 10x faster than scripting languages. Plus its performance will improve too over time as we add things like simple optimization passes.
Thank you for your work! Honestly, I can’t tell you how much I appreciate people like you spending what looks like a lot of energy, thoughtfulness (and probably love) on really original ideas. Things like this make my days brighter. I thought redbean was an outstanding idea, but the more I learn about Cosmopolitan libc itself,.. When I first heard about it I could not imagine how much further you’d be able to bend this.
chibicc does sound great, I really have to find the time to look into all of this. I don’t want to harp on about Python specifically, but now I’m really intrigued: do you think this could become a viable alternative to things like PyInstaller, nuitka, PyOxidizer in the future? Esp. for cases where performance is not of upmost importance, and even if it’d mean we can’t use every Python library under the sun. Being able to distribute a single-file, cross-platform Python command-line app would be absolutely fantastic.
Author of redbean here. Ask me anything.
YouTube presentation: https://www.youtube.com/watch?v=1ZTRb-2DZGs
Slides: https://docs.google.com/presentation/d/1WEtSyz5oyTTGe0Fi2NhhUX6EX7HAE-GZe3RNWHInTiE
What is the most cursed thing you have used redbean for or seen it used for?
This sounds amazingly useful. With what aspect of it are you least satisfied?
Thanks! Redbean has baked-in SSL support. I used MbedTLS and I love it, because it’s so much smaller and simpler than OpenSSL and BoringSSL which have huge dependencies like STL and Perl which would have needed to be ported beforehand. However its main drawback is it doesn’t have TLSv3 support. This makes me sad because it causes redbean to have a few extra milliseconds of connection latency if it’s used for self-contained public-facing serving. That’s something I intend to improve. However it’s not that big of an issue, since it’s very easy to use redbean in tandem with something like stunnel which can bolt on the SSL separately. But that only applies over long line internet connections. For local use cases (redbean’s primary and original intent) where CPU utilization is a greater factor than the speed of light, redbean’s SSL goes faster than any of the other stacks I’ve evaluated. For example, I haven’t posted SSL benchmarks on the website yet because wrk (Go) uses 2x as much CPU to establish ECDSA connections as redbean. So once it has the TLSv3 support, it’ll be a sight to behold.
I thought wrk was written in C! Is the code at https://github.com/wg/wrk/tree/master/src not the correct wrk?
This is so f’in cool.
Question: I get the “serve static files” part, and “I can do dynamic stuff with Lua” part. How does the sqlite part work? Can I literally write a full-fledged web app in lua which is, say, saving new user records using sqlite, and those are just getting written to the filesystem in the folder redbean is running in?
Indeed. Here’s the basic idea for how you’d implement a SQLite web service with redbean. https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c We’re in the process of adding features so that redbean can use its own .com file as the database. We have one function for just that, called StoreAsset() but right now SQLite is more stable.
It would be cool to pair redbean with Litestream to get streaming backups of the SQLite db.
I’m a huge fan of your work. Also fare’s, cool he invited you. Cool you are here.
Thank you! Lobsters in a nice community I regularly visit. I was so happy to see that someone linked to redbean.
How does it handle sqlite queries with database modification, e.g. INSERT / UPDATE / DELETE ?
redbean simply embeds sqlite with lua bindings thereby enabling you to handle them in any manner you want. It’s expected that the user knows SQLite and is aware of its caveats. For example, redbean is a forking web server and SQLite file descriptors should never cross fork() boundaries (if the database is mutable). This isn’t a problem if you’re using Lua Server Pages. However if you’re using the faster OnHttpRequest() handler then you need to make sure you don’t call
sqlite3.open()from the global scope and instead create it lazily as requests are handled. That means your database descriptor and prepared statements will persist across requests on a per-connection basis. This is nice since SQLite is significantly faster than other databases at doing this. Plus you get a nice process sandbox. So in case something goes wrong, like ASAN detects a bug, it’ll only impact one serving session. We’re also working on higher-level frameworks to abstract a lot of the handling aspects. Lastly I’ve also been focusing on creating single-file object store technology (see StoreAsset() and LoadAsset()) that should provide the fastest possible and most easily redistributable model of data storage under a forking model.Example https://gist.github.com/jart/f9b1a7ebe5c7ebea3abe8f98125c692c
Sorry, but I’m not sure what that means. If each request gets its own SQLite descriptor in
OnHttpRequest(), why would “database descriptor and prepared statements persist across requests”?Because redbean supports HTTP keepalive pipelining and Lua variables are global unless you declare you function local. Forking happens when a TCP connection is received. That connection may send many HTTP messages. Each message for that connection is handled by the same process. If you’ve defined a hook like
OnHttpRequestthen that means any variables you use, unless you use local, will persist between messages so you can memoize operations on a per-session basis. When the TCP connection closes, it all gets garbage collected by the operating system when the process exits.I see. Does that imply we can call
sqlite3.open()from the global scope in Lua, since by that time we are already in afork()ed child process for that TCP connection? You said we shouldn’t do that and should “instead create it lazily as requests are handled” so I am a little confused.If it’s a
foo.luafile in your redbean (i.e. a Lua Server Page) then you’re 100% OK. Those files are loaded and re-run on each request. There’s only one file that’s special and it’s.init.lua. That file is run before redbean callsbind()and no forks have been created yet. Anything you do in the global scope of that file, like import modules, will end up defining the main process “template” from which all forked processes are created. Therefore, if you create your SQLite object in the global context of that file, it needs to be strictly read-only. If it is, you can get like 800k RPS performance on read-only SQLite files. But any SQLite files you intend to mutate, it needs to happen after fork, otherwise corruption is possible. It also goes without saying, that you can have multiple SQLite file descriptors open simultaneously! So for example, if you built an Internet forum, all your years-old posts could be moved to immutable read-only SQLite files, for ultra-fast querying due to the pre-fork setup.That’s awesome! I have an additional question: is it theoretically possible for redbean to support LuaJIT in the future? I’m not sure if the magic tricks you did with actually portable executables would interfere with JIT compilation/execution.
It’s a work in progress. See https://github.com/ahgamut/LuaJIT-cosmo
This is one of the coolest projects I’ve come across in the last few years, and I’ll never not upvote it. Even if it was posted every day.
I’m working on this data-science library, and one of my ideas is to wrap exported datasets including their metadata (properties, lineage information, annotations) and a generic html file, and redbean into a zip file. The html page would serve as a sort of ‘data-passport’, showing a preview of the data itself, the metadata, etc. So, basically every dataset would come with a cross-platform way of executing it and showing itself off in a browser if users so choose to do. I haven’t really thought that through entirely, esp. security considerations (making random files executable might not be the best idea to recommend to users), and how that would work having a server process running on an OS where the commandline is not used by most users (Windows) for example.
But even if I decide to not go forward with the idea, just the potential that I could blows my mind…
That’s exactly the kind of use case I intended to enable with redbean. It’d be great if every website with Jupyter Notebooks had a button to “download my notebook into a redbean”. It’s why with Cosmopolitan Libc we’ve been focusing on porting other languages too besides Lua, e.g. Python, LISP, JavaScript, etc. Regarding security, redbean is one of the few application servers that’s viable in an air-gapped environment. What could be more secure than that?
Oh, I wasn’t aware of this. Please stop, my imagination is running wild. I’m getting to an age where this sort of excitement is keeping me sleepless for nights on end…
I don’t disagree with that. I meant more the practice of encouraging my less-then-aware users to just set the executable flag on any random (data or not) file people send them.
What is open source but posting files online containing code and encouraging others to download and execute them? If you want a snapshot of things to come, we managed to get Python 3.6 down to a 5.5mb Actually Portable Executable. That’s including all its standard library dependencies which are baked into the ZIP structure of the binary. It took quite a hacking marathon. https://justine.lol/actually-portable-python.png It’s going to do so much to help facilitate a sharing of things like scientific knowledge that can be reproduced and interacted with for years to come, rather than merely read like a PDF file, or being something that breaks the next time Ubuntu upgrades, or requires that people in other parts of the world with less Internet privilege download gigabytes of bundled stuff? The technology that underpins redbean is going to help address all those things. You can read more about it here: https://justine.lol/ape.html If anyone reading wants to help make this vision of eliminating the barriers to knowledge, please consider sponsoring me on GitHub.
Yeah, I guess you are right.
What’s the plan with Python packages? I reckon including pure-Python ones should be feasible, but esp. for scientific data numpy and pandas would probably be much more difficult to support. Having sqlite available will make for a good workaround in a lot of cases, though. Building a more portable version of https://datasette.io/ might be an interesting project idea…
Have you ever looked into Apache Arrow? Intuitively, it seems like it could be a good fit with readbean if you wanted redbean to venture into the data science space a bit. It being portable, and memory efficient, mmappable and all that.
Either way, I’ll definitely keep an eye out for the Python support, even if everything I would like to do right now would already be possible with Lua.
Done, good on you for mentioning it, I wouldn’t have thought to check… If I end up using it in our project, I’m pretty sure I can get them to sponsor as well.
Thank you for your support!
Porting Python packages is something we’ve been doing by hand so far. It’s largely been a community driven effort due to how native dependencies have effectively locked-in so many packages to specific operating systems. Some examples of things we’re doing, is I started writing a Python bindings generator for chibicc which is an embeddable ~300kb C compiler. It’s all part of the Cosmopolitan Libc repo. My intent here is that, when you just want the native code to work, and you don’t care about having the last inch of performance you’d get with GCC and Clang, then all you need to do is plop the .c files into the .com file using a zip tool, and the Python implementation will just magically build them and link them in automatically across platforms. That would be the pain free path. But if you want the full perf for things like scientific computing, then you just need to integrate it with the Cosmopolitan Libc makefile build.
This same concept would naturally also apply to generating Lua bindings in Redbean for any C code where you need extra performance. chibicc is great. Since half as fast as GCC is still 10x faster than scripting languages. Plus its performance will improve too over time as we add things like simple optimization passes.
Thank you for your work! Honestly, I can’t tell you how much I appreciate people like you spending what looks like a lot of energy, thoughtfulness (and probably love) on really original ideas. Things like this make my days brighter. I thought redbean was an outstanding idea, but the more I learn about Cosmopolitan libc itself,.. When I first heard about it I could not imagine how much further you’d be able to bend this.
chibicc does sound great, I really have to find the time to look into all of this. I don’t want to harp on about Python specifically, but now I’m really intrigued: do you think this could become a viable alternative to things like PyInstaller, nuitka, PyOxidizer in the future? Esp. for cases where performance is not of upmost importance, and even if it’d mean we can’t use every Python library under the sun. Being able to distribute a single-file, cross-platform Python command-line app would be absolutely fantastic.
So excited about the possibilities with Python support! Amazing!
I tried this on my M1 Mac & it didn’t work. Is that because the binary code is x86-only?
Yup, only works with x86 instruction set