1.  

    This is the first/only pure-JavaScript implementation of Parquet I could find. I say pure in that it can run in the browser alone. The bigger project, parquetjs, only works in Node.js.

    1. 2

      Hopefully getting desktop builds out for an IDE I’m working on: https://github.com/multiprocessio/datastation. That or cleaning up UI bugs in it.

      1. 5

        Working on my JVM programming language. I decided to take the plunge last week and it’s been a lot of fun. It’s partially an exercise in better understanding the JVM, but I’d like to get to the point where I can at least write a static site generator with it.

        The language itself is strict functional. It is most similar to Ocaml and Gluon, though it borrows from a bunch of different languages. I’m experimenting with making only a single way to declare something syntactically, e.g. a regular function and lambda are defined the same way. Similarly, modules are just a slightly specialized case of structs. It’ll be interesting to see how larger programs compose.

        1. 1

          Are you generating JVM bytecode or what else does the compiler generate?

          1. 2

            I’m generating bytecode, it’s surprisingly straightforward with the ASM library. I’m using ANTLR for parsing, the base was following this tutorial until part 10, at which point I got the hang of things: http://jakubdziworski.github.io/enkel/2016/03/16/enkel_3_hello_enkel.html

        1. 3

          Pretty surprising to see this kind of article (basically PL fanboying) on BlackBerry-(the cell phone maker)’s blog. Normally corporate blogs are more oriented toward teaching something specific or talking about experience whereas this is more of a love letter.

          As I write this, I have a serious Ada program consisting of 19 source code files containing 2,985 comment lines and 6,253 of Ada code open on my desktop. This forms part of a semi-formal verification suite of a new QNX development in progress at the moment. In the past I would have written this program in C or Python. Writing it in Ada has sometimes been infuriating because I turn on all compiler warnings and specify that warnings are to be treated as errors. However, Ada’s characteristics have saved me hours of debugging. Getting a clean compilation takes a little longer, but that is more than compensated by the reduced debugging effort.

          It’s possible that this “new QNX development” is a work project he’s using Ada on? Hard to tell.

          1. 10

            BB basically no longer makes phones and is entirely security software now; plus I think they still own QNX.

            1. 8

              Indeed. This is a post from a developer at QNX which in my mind is a whole different entity than Blackberry the cellphone maker.

              1. 1

                Wasn’t very clear to me from his linked bio:

                Chris Hobbs is the author of the definitive book on functional safety, Embedded Software Development for Safety-Critical Systems. He has advised some of the largest industrial control and automotive companies as they pursued safety-certification standards.

                But thanks for clarifying.

                1. 2

                  QNX is used in a lot of auto infotainment systems and I think they’re slowing absorbing onboard sensors and the like into the mix, thus the need for safety critical

          1. 1

            I’m curious. what is the use case that is not covered by other user process databases like sqlite3?

            1. 3

              It’s literally postgres. Sqlite and postgres aren’t the same thing. Postgres has many more datatypes and builtin support for cool json stuff. If you only wanted to use sqlite for testing you’d have to either avoid using some datatypes or use an orm that will polyfill missing types and functions.

              Edit: sqlite also doesn’t support altering tables. The list of differences just goes on and on. What’s the value in have test specific code? (I’ve definitely done this before too)

              1. 3

                Just pointing out that SQLite definitely has some JSON support https://sqlite.org/json1.html#the_json_function as well as ALTER TABLE support https://sqlite.org/lang_altertable.html

                1. 1

                  Yes JSON functions are available if you load the extension. But it’s not built in. Every db has lots of extensions. And even then if you’re using JSON in Postgres or MySQL you’re probably using the d->>‘x’->y syntax and that doesn’t exist in SQLite so you do need to have a transform layer before generating SQL if you want to support both.

                  And yes good callout; I think it’s just that you cannot change a column type in SQLite? It’s been a bit since I was doing work on it.

                  1. 3

                    For go at least, modernc.org/sqlite has the JSON support fully working out of the box.

                    1. 3

                      Gotcha, thanks.

                    2. 2

                      Yes JSON functions are available if you load the extension. But it’s not built in.

                      I think a lot of people build their distribution of sqlite3 from the amalgamation which does have JSON support built-in.

                      e.g. immediately to hand I have macOS Monterey (SQLite v3.35.3), Arch (v3.35.5), Cygwin (v3.34.0) all have the JSON support natively. On the other hand, my NAS (SQLite v3.8.4.3 from 2014) doesn’t have it.

                      (edit: has -> have)

                      1. 2

                        Ah cool, makes sense.

                  2. 2

                    Title says “in application or test”, I guess I should have been more specific, I didn’t mean the test part, that was obvious to me.

                  3. 1

                    automating the testing of postgres queries, stored procedures, etc, that are expressed in the source code of a Go program.

                  1. 7

                    The way I expected this to work from the README and name is that it wrapped the PostgreSQL C code to run the lot in-memory, or something along those lines.

                    But looking at the source this is not how it works at all: NewDatabase() fetches a binary from https://repo1.maven.org, extracts it to a temporary path, and runs pg_ctl start with exec.Command(). Nothing is “embedded” in any way I would use the word, and it’s not even easy to implement that in the current version as it expects to read stuff from the filesystem (rather than an io.Reader or whatnot). It’s also just an unmodified PostgreSQL as far as I can tell.

                    Useful for some things? I guess? Maybe? But very limited. I’d just run PostgreSQL.

                    Also, this was so obviously written by a Java developer.

                    1. 3

                      That’s kind of nasty. Why not just use a shell script that makes and runs a docker container?

                      1. 4

                        that would depend on docker, whereas the goal of this project is to depend only on the Go standard library.

                      2. 3

                        It is possible to run a PostgreSQL database in memory which is what I expected to find. I agree this doesn’t really fit the “embedded” idea at all.

                        1. 2

                          Any links to running a postgres database in memory for real?

                      1. 1

                        Someone mentioned this to me after posting: https://github.com/101arrowz/fflate. It’s about 1/6th the size of pako and up to 40% faster.

                        1. 2

                          Some of it reads like sarcasm or just being mean:

                          The 0.5 release is based on FreeBSD 12.2 and is progressing nicely. The release notes show a number of important fixes like “sudo su works now”, “fix wrong font sizes” and “fix menu and desktop on multi-monitor setups”.

                          But the rest of the article doesn’t have that tone.

                          1. 10

                            That’s just El Reg in general; they very much aim to be the British tabloid of the industry.

                            1. 1

                              Interesting. I’ve read them from time to time but never noticed anything before this article.

                              1. 2

                                Maybe we should start writing our changelog in order of importance rather than in chronological order.

                                1. 1

                                  I’m not sure what you mean: sudo su works now, thanks @crees is pretty far down on the page and not at the top of any section here. Unless you mean this article ignored the list on the release page and found a chronological order somewhere and used that.

                          1. 3

                            Doesn’t work on iPad (iOS 14). I get “a problem repeatedly occurred opening the page”. And the Back button is hijacked, sending me back to the same URL.

                            1. 1

                              Thanks! It is on my list to put up a banner on phones/tablets explaining that only desktop is intended to be supported.

                              1. 1

                                Do you have an About page to link?

                                1. 1

                                  The Github readme may have the most info at the moment https://github.com/multiprocessio/datastation.

                            1. 5

                              Seems like it doesn’t work at all in firefox. In the console: “Uncaught SyntaxError: private fields are not currently supported”. An interesting lesson in chrome monoculture I guess.

                              1. 1

                                Oops :) https://github.com/multiprocessio/datastation/pull/3 ctrl-shift-r should do it now. Thanks for the report!

                                1. 1

                                  Indeed, works better :-). It seems like it’s a notebook, but where you can mix data, SQL, http requests, etc.? Neat, I suppose.

                                  1. 1

                                    Yup, I wanted something easier to use than Jupyter notebooks. Long term goal is to have more first-class connectors for things like Drive/Dropbox, Excel files, JIRA API, Github/Gitlab API, Julia/R support, etc.

                                    The desire comes from my experience doing business analytics and having to either write everything from scratch every time in Python or using a SQL IDE and copying its result to Excel or writing a Python script to manipulate a CSV before loading in Excel. I just want a single place to do it all.

                              1. 2

                                An interesting thing about Jinja is that it actually compiles the templates into Python bytecode. I don’t think it’s well documented how they do it, but it allows them to use usual Python expressions and still have safety and speed.

                                1. 1

                                  I didn’t know that! But it make sense. I don’t know if it’s actually a law but from everything I’ve seen every tree-walking interpreting can be linearized into a bytecode compiler+interpreter and it tends to get a few order of magnitudes faster. Doing so for this library would be a good challenge for a second post.

                                1. 3

                                  Run make -j to compile using all cores.

                                  Noooo, don’t do this. Run make -j 16 (or however many threads you’ve got). Bare make -j will run with maximum parallelism, so eg if you have 200 source files, and 16 threads, it will start 200 compiler processes at once, and start time slicing them back and forth (and you may run out of ram and start swapping too).

                                  1. 2

                                    Ah! Thanks for that!

                                  1. 3

                                    The hassle with getting it to build felt like building up to comedy; “don’t be afraid of hacking, let’s dig in” and some six hours of googling-for-folklore later just giving up! And throughout, the cynical humor felt just around the corner ;)

                                    Liked it better this way, thanks, and looking at these codebases is encouraged :)

                                    1. 2

                                      There was a six hour period or so where I couldn’t find the right place to inject in the response, bunch of segfaults and or just unfilled out fields. I got super annoyed. 😂 By the time I had a better idea it was too far along for me to document all those mistakes. But I did call out this part took a few hours.

                                    1. 4

                                      I know the point of the post is to show how you could go about working on an unfamiliar code base, but I thought I would mention that Nginx already is able to returning a plain string in a response using the return statement: http://nginx.org/en/docs/http/ngx_http_rewrite_module.html#return.

                                      Perhaps it would be interesting as a follow up to see how it implements the functionality and if it uses some higher-level methods to do it?

                                      1. 1

                                        Oh nice, thanks! I should have remembered this. Yes that could be good to look into.

                                      1. 4

                                        Can I draw on it? I was only able to create squares, modify text, and move things around. Couldn’t figure out how to freeform draw.

                                        1. 1

                                          Oh good, I was worried a was the only one that couldn’t figure it out.

                                          I do have to say though, I could see using this to organize some flow charts and logic would be very very useful.

                                        1. 4

                                          I haven’t been able to complete a userland TCP/IP stack on raw sockets. Writing C or C++ is the easiest in one sense because for interacting with Linux APIs and for doing integer/bit twiddling more succinctly than it would be in Go or Rust. But I got caught 1) not writing unit tests for basic message transformations 2) basic dynamic memory management bugs and then 3) couldn’t figure out how to do state management for TCP.

                                          When I get back to it, unit testing all marshaling/unmarshaling will be an imperative and I’ll probably use boehm gc in C++ to make memory management easier on myself.

                                          1. 2

                                            Porting md5 sum, sudoku solver, and brainfuck implementations from C to Go for some micro benchmarks.

                                            1. 9

                                              This is about CPython builds specifically. The performance will change even more significantly when switching between Cython, PyPy, GraalVM, Jython, whatnot.

                                              1. 23

                                                How about, “Four books professional server-side Python developers should read (and a few they don’t need to)”.

                                                As a client-side C++ programmer, that list doesn’t do me much good, except for “High Performance Browser Networking” which is a killer read for anyone involved in HTTP-based APIs on either end of the wire. (And OK, I do write some utility scripts in Python.)

                                                On the other hand, “Design Patterns” is a stone classic.

                                                1. 9

                                                  The stone cold classic is SICP, to the point of cliche but no less valid for it. It doesn’t have the instant for-my-job practicality boost that a style guide like “90 Specific Ways to Write Better Python” does, but it’s in another ballpark of depth.

                                                  @eatonphil I am curious if you had criticisms of it or if you think its lessons are just not nuts-and-bolts enough for your list.

                                                  1. 5

                                                    It’s a perennial goal of mine to finish SICP. 😀 If someone is coming to this list and sees value in four top books, I can’t really recommend a book in the same category here that’s taken me years to finish. But for people who read a lot, of course you can give it a go.

                                                    I’m very much a PL nerd. As mentioned elsewhere, I’ve spent a lot of time programing in Standard ML (see ponyo on my github) and Scheme (see bsdscheme on my github). Forcing myself to spend years getting good at both has seriously improved my ability to write simple code.

                                                    But I also can’t really recommend that to most devs either because who knows if you’ll enjoy it. Doing what inspires you is the most important thing so you actually complete what it is you want to do.

                                                    1. 4

                                                      Scheme and the art of programming by George Springer is almost the same material as SICP but in Scheme. Honestly if you have worked for years in ML or Scheme you probably don’t need SICP since it is a introductory level text.

                                                      1. 2

                                                        I just picked this one up a few weeks back and have slowly been working through it. I’m surprised that I hadn’t run into it before then. It’s a textbook and it reads like a textbook, there’s no getting around that, but it’s accessible and seems to be a great intro to Scheme in general.

                                                        SICP is using Scheme too, isn’t it? I recall running across MIT being the recommended implementation, but I’ve had no problems with Guile, and there’s a language in Racket, too. I bounce between the Springer book, SICP, and Little Schemer when I’m not too busy. I’m in no rush, but I’ve found those three to be a great way to get acquainted with emacs/geiser & guile.

                                                      2. 3

                                                        I’m in the same boat with SICP and similar books with an academic feel. If it’s not project oriented, it’s usually too hard for me to focus on.

                                                    2. 3

                                                      I wish I could recommend books on other languages! But no book I’ve read on JavaScript (thinking JavaScript the Good Parts, How JavaScript Works), Go (there’s almost nothing here that’s not just reference), Java (Effective Java is massive and not all useful), or C++ (what, Meyers’ Effective C++?) is in the same category in terms of information density, being well-written, and still being applicable today. I’d be happy to take recommendations though for books not among these.

                                                      1. 6

                                                        Thinking Forth (1984) and Writing Solid Code (1993) are two books that radically changed how I approached programming. Yes, one is about Forth, and one is about C, but I’ve found them applicable to more than just those two languages.

                                                        Edit: added years first published.

                                                        1. 3

                                                          Fully agree on both and in particular that TF is applicable beyond Forth. Great book.

                                                          I wonder whether you might enjoy Elements of Programming and/or The Humane Interface (and here’s the bit I’ve quoted most often). Very different books, both of them insightful and hard. EoP is a C++ book that’s about C++ in the way TF is about Forth, THI is a UI book that uses phrases like “information-theorietic efficiency” instead of “at [famous company] we do this [this way]”.

                                                        2. 3

                                                          Personally, my qualm wouldn’t be about the language, it would be about the domain. I haven’t read the books, but those 4 titles really are web focused. I’m sure lots of those translate to other domains, but it’s clearly not aimed as the general purpose developer who doesn’t yet know which domain they will tackle. No, this is for backend web developers.

                                                          Thus, your title should really be closer to: Four books professional backend developers should read. That way it’s more focused to your actual target audience.

                                                          1. 3

                                                            My point is that it wasn’t intended to be backends focused. I haven’t read any really great books on JavaScript (but HPBN is clearly browser, not backend, focused). And I genuinely wish frontend developers I worked with knew more that’s covered in DDIA.

                                                            But I don’t pretend to speak for every domain of programming for sure.

                                                            1. 1

                                                              It’s easy to for us to get myopic about stuff outside our domain.

                                                              Note that even here you seem to be implicitly assuming that not-backend = in-browser. That’s leaving out (native) desktop, mobile and embedded development.

                                                              I suspect there aren’t actually any books that are super important for all serious programmers. The books I learned the most from are way out of date (K&R, the Smalltalk-80 Blue Book, Advanced Cryptography) or very domain specific (Unix Network Programming) or both (Inside Macintosh.)

                                                              1. 1

                                                                Yes I’m definitely missing domains but mobile isn’t one of them! HPBN devotes a ton of time to the effect of protocols on mobile battery life of all things. There’s a chapter or two on the behavior of mobile networks. Mobile developers are also responsible for monitoring and alerting on their production deploys. And they’ll also be heavy users of backend systems.

                                                                All this isn’t to say that I’ve got the list perfect. I hope I change my mind on it over time as I’ve read new books, learned new things, and expired different domains.

                                                          2. 3

                                                            I feel The Go Programming Language is quite good; certainly more than just a reference.

                                                            The Little Schemer is probably the best programming book I’ve read. While it’s quite limited in scope (functional programming in Scheme), it does explain it in such a way that’s very insightful and useful for other languages too, IMO. And the writing style is just fun!

                                                            1. 4

                                                              I feel The Go Programming Language is quite good; certainly more than just a reference.

                                                              It’s not bad. And certainly new Go developers would do well to use it as a style guide since writing “idiomatic” Go takes a bit. I’ll consider adding it to a future version of this post.

                                                              The Little Schemer is probably the best programming book I’ve read. While it’s quite limited in scope (functional programming in Scheme), it does explain it in such a way that’s very insightful and useful for other languages too, IMO. And the writing style is just fun!

                                                              In the category of niche languages there is no shortage of gems. :) I haven’t read The Little Schemer but I’m working through The Little Typer at the moment. I’ve got a post in mind for a rundown of 6-8 Lisp books alone. But I wouldn’t consider recommending any of these to developers en masse (even though I believe programming in Scheme or Standard ML for a while helps you write simpler code).

                                                              1. 3

                                                                I don’t think the value of The Little Schemer is in learning Scheme, but rather in learning recursion and functional programming, and Scheme is just the tool that explains these concepts. The reason I got it in the first place is because it came recommended to better understand JavaScript’s functional aspects.

                                                            2. 2

                                                              I highly recommend Eloquent JavaScript by Marijn Haverbeke. It has a great style of writing and starts from some really nice computing basics.

                                                              1. 2

                                                                Learn you a haskell has been transformative to me, for this reason

                                                                1. 1

                                                                  You Don’t Know JS was an excellent language-specific book back in ~2013 when I read it, and I think it’s been updated after that.

                                                                  I do prefer more general books these days, now that I know enough languages well enough. SICP is a famous one, but I think How to Design Programs might be a more approachable similar book. Nb. I’ve only worked about halfway through it.

                                                                  1. 1

                                                                    My copy of Effective Java by Joshua Bloch is 346 pages, are you perhaps confusing it with another? I found it useful, too. Some of it is a bit self-evident, but I opened my copy at a random page and read an item (46, pages 212-214, about for(:) compared to for(;;)) and still found it good.

                                                                    It’s not a book on java, though. It’s for people who already know java, and who write or maintain large java programs, say 50kloc or more.

                                                                  2. 3

                                                                    I dunno, I’m reading through Designing Data-Intensive Applications at the moment, and it feels pretty language agnostic. I’ve seen some snippets of bash, but that’s it. Definitely not Python-focused.

                                                                    1. 1

                                                                      Language-agnostic, but server-oriented. That’s why I wrote “server-side”. As a client (mostly mobile) developer, I’m not much concerned with terabyte-scale data or with distributed algorithms that only make sense for clusters in a data center (because they assume latency is zero and uptime is 99% and all peers are trusted.)

                                                                      I’m not dissing any of this, just pointing out the applicability is narrower than advertised :)

                                                                  1. 6

                                                                    They’re not all bad but give nowhere near as much return for the investment of your time.

                                                                    This is REALLY unfair.

                                                                    Reading a book like Structure and Interpretation of Computer Programs isn’t about delivering a quick hit productivity boost for your developer skills.

                                                                    It’s about cultivating a very deep understanding of computation and some of the patterns and tools that can elevate your craft to another level.

                                                                    This is an incredibly ham fisted description, but hopefully you’ll get the idea.

                                                                    1. 3

                                                                      For what it’s worth, as mentioned elsewhere here, I genuinely disagree.

                                                                      1. 2

                                                                        I totally get it, and think your point is very valid. SICP is a very high wall indeed for most of us to scale.