1. 1

    This looks like Visual Basic stuff I did back in 03

    1. 7

      The grammar of the questions asked in this article irks me. You can either say “Why development teams are adopting GraphQL”, as a statement, or you can say “Why are development teams adopting GraphQL?”, as a question. But what the author is doing is a strange mishmash of both. It would only be correct if it was meant as a quote with a follow-up: “Why development teams are adopting GraphQL? I have no idea!”.

      1. 5

        Or with an extra comma as an interjection:

        Why, development teams are adopting GraphQL? What a preposterous notion!

        1. 5

          I assume Tomek’s first language is not English.

        1. 4

          I just got Nextcloud setup on my raspberry pi for my family and I!

          1. 2

            What’s your experience with the calendaring application? I briefly used it a few years back and it paled in comparison to iCal and GCal…. I’m still trying to get away from Google

            1. 1

              I use Nextcloud calendar. I’m happy with it for basic personal usage, but where it really shines is the CalDav integration. Google I suppose can add extra features because they don’t support proper syncing via standards from what I remember. And even though I’m using an Android phone, I want my data portable.

          1. 3

            I’m hoping to finish a project at work I’m writing in C between today and Wednesday afternoon. My wife and I will be celebrating the US’s Independence Day by taking our eight-month-old puppy (pic taken at around six months) to Kent Island in the Chesapeak Bay. He absolutely loves the water! I cannot wait to take him camping and hiking next year.

            1. 2

              Adorable pupper! My little girl is eight-months as well, she’s going to her first water adventure this weekend. Have a safe and happy Fourth!

            1. 3
              Oddball/DSVA
              Moonlight
              • Upgrading an ancient Rails 3.2 application and redeploying on AWS as an exercise in making previous clients surprised and happy

              Personal
              • Maintain my weight-and-body-fat-loss rate by tracking calories and exercising discipline in diet
              • Visiting family and friends on vacation for US Independence Day
              • Finish Fall; Or Dodge in Hell

              Anyone experienced in SRE/Metrics/Monitoring have interesting reads for me?

              1. 25

                I give away Sidekiq and sell Sidekiq Enterprise. If you use Ruby/Rails, it’s the standard. https://sidekiq.org

                1. 3

                  it’s the standard

                  For a very good reason. Wonderful piece of software. Thanks @mperham!

                  1. 2

                    Woah thats awesome. I have been using sidekiq lots. Great bit of software.

                    1. 1

                      Hey @mperham!

                      Thanks for all of your code. I’ve used and loved several of your projects.

                      Out of curiosity what is the current status of Faktory? It sounded like an interesting project, but the rate of development looks like it kind of cratered last Dec.

                      1. 2

                        It’s under active development but summer has been slow due to family issues. Latest:

                        https://mastodon.xyz/@mperham/100583959557092421

                    1. 2

                      Oddball is hiring a Senior Full Stack Engineer (React, Rails) and a Devops engineer (AWS, Terraform). We work on cool projects like vets.gov and are partnered with AdHoc - the agency that salvaged the original healthcare.gov project. Fully remote, great compensation, US based only.

                      1. 1

                        Out of curiosity (I’ve always wanted to ask someone this), what’s the motivation for US-based only?

                        I’m an American living in Canada, and it’s a bit of a head-scratcher how often I see this. Having worked remotely for US companies in the past, I have trouble figuring out what it is that the companies adding this disclaimer are worried about.

                        1. 2

                          In this particular case the project we’re hiring for is under the umbrella of the US Gov’t, and any contractor/subcontractor is required to be located in the United States. (AFAIK).

                      1. 5

                        “missing” out of the box for composition and revision are tools for version control

                        There’s RCS and CVS in the base system for that.

                        One thing that I find somewhat unfortunate is that OpenBSD has a lot of great text editing tools, yet it’s missing any kind of typesetter (troff, TeX) in the base system.

                        1. 3

                          …editing because @xorhash had been kind enough to remind me of rcs(1) and cvs(1)…

                          OpenBSD’s base system doesn’t provide dictionary searches or spell check, either, but I’m fine with that. I’m grateful they provide X Window as part of the base system. Stuff like git, troff, aspell, diction, pandoc, and dictd I’m happy to install using the package mangler.

                          What I would love to know is why OpenBSD ports has the dict server but none of the dictionaries. If I want a dict daemon on my laptop so I can check definitions offline, I have to get the actual dictionary archives out of the FreeBSD port’s distfiles because ftp.dict.org is dead. While I can do that, I’d rather not have to. :)

                          1. 3

                            I second xorhash’s mention of RCS. (Though, I’m no BSD user.)

                            I heard somewhere that RCS was designed with your sort of use case in mind! Maybe it was this post (2009)?

                            It’s certainly an easily understood, unixy tool. Maybe I’ll try using it one day. ;)

                            1. 3

                              That’s an excellent introduction. Thanks.

                              However, RCS isn’t actually suited to my use case because I don’t use one file per novel. Instead, I write novels the way I code at my day job, with text distributed across various files in a directory tree. Yes, it’s probably overkill, but it beats paying a shitload of money for a Mac so I can use Scrivener or Ulysses.

                              My hierarchy currently looks somewhat like this:

                              $SERIES/
                                $TITLE/
                                  title
                                  dedication
                                  disclaimer
                                  acknowledgements
                                  $SUBPLOT1/
                                    01.scene
                                    02.scene
                                    01.revision01.sed
                                  $SUBPLOT2/
                                    01.scene
                              

                              When I’m ready to read what I’ve done as a whole, I’ll assemble the whole mess using cat and fmt. Likewise when I’m done with all revisions and am ready to submit to a publisher. At that point I’ll put everything together into a file like “submission01”, mark it up with with Markdown or reStructuredText (depending on whether I was pretentious enough to include footnotes), run it through pandoc and convert it to Word format (unless the publisher is hip enough to accept an OpenDocument Text file, and then edit the output in LibreOffice to suit the publisher’s house style.

                              You can’t manage something like this with RCS. CVS would be more appropriate, but as I mentioned in another comment I’m already familiar with git. I use it when tinkering with static site generators, build websies, and at my day job.

                              1. 4

                                I don’t know much about the BSDs but I use Scrivener on Debian via WINE, flawlessly! Just a note.

                                1. 3

                                  Apparently there’s an AppImage of the unfinished Linux version for people who don’t want to use WINE.

                                  Believe it or not, I’ve tried Scrivener. It’s not a bad app, but I don’t like that it stores everything in RTF files. When I’m drafting something, I’d rather work in plain text.

                                  Also, as @qznc noted, a tool like ed(1) is great if you have a tendency to go back and edit unfinished work. I have this tendency in spades.

                                2. 2

                                  I don’t see why you can’t use RCS.

                                  % ed test
                                  a
                                  this is a test of using
                                  RCS for version control.
                                  .
                                  w
                                  49
                                  !ci -l % 
                                  ci -l test
                                  test,v  <--  test
                                  enter description, terminated with single '.' or end of file:
                                  NOTE: This is NOT the log message!
                                  >> test check in
                                  >> .
                                  initial revision: 1.1
                                  done
                                  !
                                  ,n
                                  1	this is a test of using
                                  2	RCS for version control.
                                  a
                                  
                                  Now we add a new paragraph.
                                  .
                                  w
                                  78
                                  !ci -l %
                                  ci -l test
                                  test,v  <--  test
                                  new revision: 1.2; previous revision: 1.1
                                  enter log message, terminated with single '.' or end of file:
                                  >> new paragraph 
                                  >> .
                                  done
                                  !
                                  ,n
                                  1	this is a test of using
                                  2	RCS for version control.
                                  3	
                                  4	Now we add a new paragraph.
                                  q
                                  

                                  Compared to git, the only thing that’s missing is keeping track of contents that get moved from one file to another.

                                  1. 2

                                    RCS is one repository per file. That’s not what I want. I want one repository for the entire project. And I want the master repository to live on BitBucket (or some other provider I trust because I’m too lazy to self-host on a VPS). This lets me sync between multiple machines.

                                    This way, when I’m dead because somebody got upset about me typing in public and decided to beat me into the ground with my laptop, it’s possible that some other nerd who overdosed on JRPGs and Blue Öyster Cult albums as a kid might find it and take over. :)

                                    1. 1

                                      In the true spirit of unix, you use one tool for one purpose only. Just use a separate tool for syncing. scp(1) works. rsync(1) works better. unison(1) beats everything.

                                      You can’t really call RCS a ‘repository’. It is, after all, just one ‘,v’ file for the version history of a single file. You can setup rsync or unison to sync up ‘,v’ files exclusively, which essentially transforms rcs to a hand-rolled cvs.

                          1. 42

                            GitLab is really worth a look as an alternative. One big advantage of GitLab is that the core technology is open source. This means that anybody can run their own instance. If the company ends up moving in a direction that the community isn’t comfortable with, then it’s always possible to fork it.

                            There’s also a proposal to support federation between GitLab instances. With this approach there wouldn’t even be a need for a single central hub. One of the main advantages of Git is that it’s a decentralized system, and it’s somewhat ironic that GitHub constitutes a single point of failure.

                            1. 17

                              Federated GitLabs sound interesting. The thing I’ve always wanted though is a standardised way to send pull requests/equivalent to any provider, so that I can self-host with Gitea or whatever but easily contribute back and receive contributions.

                              1. 7

                                git has built-in pull requests They go to the project mailing list, people code review via normal inline replies Glorious

                                1. 27

                                  It’s really not glorious. It’s a severely inaccessible UX, with basically no affordances for tracking that review comments are resolved, for viewing different slices of commits from a patchset, or integrating with things like CI.

                                  1. 7

                                    I couldn’t tell if singpolyma was serious or not, but I agree, and I think GitHub and the like have made it clear what the majority of devs prefer. Even if it was good UX, if I self-host, setting up a mail server and getting people to participate that way isn’t exactly low-friction. Maybe it’s against the UNIX philosophy, but I’d like every part of the patchset/contribution lifecycle to be first-class concepts in git. If not in git core, then in a “blessed” extension, à la hub.

                                    1. 2

                                      You can sort of get a tracking UI via Patchwork. It’s… not great.

                                      1. 1

                                        The only one of those Github us better at is integration with CI. They also have an inaccessible UX (doesn’t even work on my mobile devices, can’t imagine if I had accessibility needs…), doesn’t track when review comments are resolved, and there’s no UX facility for viewing different slices, you have to know git stuff to know the links

                                      2. 3

                                        I’ve wondered about a server-side process (either listen on http, poll a mailbox, etc) that could parse the format generated by git request-pull, and create a new ‘merge request’ that can then be reviewed by collaborators.

                                        1. 2

                                          I always find funny that usually, the same people advocating that emails are a technology with many inherent flaws that cannot be fixed, are the same people that advocate using the built in fit feature using emails…

                                      3. 6

                                        Just re: running your own instance, gogs is pretty good too. I haven’t used it with a big team so I don’t know how it stacks up there, but I set it up on a VPS to replace a paid Github account for private repos, where it seems fast, lightweight and does everything I need just fine.

                                        1. 20

                                          Gitea is a better maintained Gogs fork. I run both Gogs on an internal server and Gitea on the Internet.

                                          1. 9

                                            Yeah, stuff like gogs works well for private instances. I do find the idea of having public federated GitLab instances pretty exciting as an alternative to GitHub for open source projects though. In theory this could work similarly to the way Mastodon works currently. Individuals and organizations could setup GitLab servers that would federate between each other. This could allow searching for repos across the federation, tagging issues across projects on different instances, and potentially fail over if instances mirror content. With this approach you wouldn’t be relying on a single provider to host everybody’s projects in one place.

                                          2. 1

                                            Has GitLab’s LFS support improved? I’ve been a huge fan of theirs for a long time, and I don’t really have an intense workflow so I wouldn’t notice edge cases, but I’ve heard there are some corners that are lacking in terms of performance.

                                            1. 4

                                              GitLab has first-class support for git-annex which I’ve used to great success

                                            1. 1

                                              I also use pass with keybase git.

                                            1. 2

                                              Wow the nostalgia factor here caught me by surprise. It was actually Easter Sunday in 2004?) when I first setup PHPBB and learned what MySQL was. Wow. This is my favorite prank thus far.

                                              Well done pushcx

                                              1. 3

                                                I’ve been remote for 3 years, and Monday I start my first office-based job (small office, open floor plan.)

                                                I’m excited to be around my colleagues and have the opportunity to discuss things in real-time. I’m not excited for what will be ~1hr commute per day. The freedom granted from working remote has been great - I can make my own lunch in my own kitchen. Hopefully I’ll transition back to a remote position after a few months of getting to know the team.

                                                As for tips I echo everything said in the other comments. It’s great to be out of bed and directly on to PC for work, but it’s even greater to wake up and have a breakfast routine, a separate office and time blocks for flow-state work.

                                                I found it’s easier to get into flow-state with people around because I’m less likely to be distracted if I’m in the office.

                                                1. 3

                                                  “It is very dangerous to run the latest version under sudo and I have a feeling it isn’t just me getting these results.”

                                                  It’s dangerous to run anything as root. FTFY :)

                                                  1. 2

                                                    No, it’s not.

                                                    1. 2

                                                      I disagree, it’s dangerous to run commands as root and not know what you’re doing.

                                                    1. 7

                                                      Great read. My favorite tl;dr;

                                                      By the way, NaN != NaN, so Aristotle was wrong about that whole “Law of Identity” thing.

                                                      1. 3

                                                        I’m working on a legacy Perl application (first time in this language, so check that goal off the list, too) for the ca$hflow.

                                                        Growing PB&Jealous Club into a nice little side hustle, broaching the subscriber milestones now. Woke up today (on my birthday nonetheless) to discover we’d been featured on the Cratejoy Editor’s Pick email list!

                                                        SaaS I’m building is starting to come into shape. Have spent two weeks writing documentation and mocking the API, sketching designs and deciding on features to build for a MVP. Next week or so starts code.

                                                        Reading: Working Effectively With Legacy Code (lol), Pragmatic Programmer, Clean Code. Listening to: Ready Player One, How To Win Friends and Influence People, The Subtle Art of Not Giving a Fuck, and 1Q84.

                                                        It’s also pick-and-pack week for PBJ which means manual entrepreneurship. :)

                                                        Trying to be a beacon of optimism in an ever pessimistic world. Scary things happening, and a lot of people feel powerless and disenfranchised. Be safe, be aware. You’re loved.

                                                        1. 2

                                                          This is resoundingly true for me as well. I spent a great deal of time pondering, which I’ve now learned was time squandered. Now I act, and I decided to leave my in-office QA position to branch out and freelance. I told myself I would work remotely no matter what, if it meant growing pains learning how to pick the right clients, etc.

                                                          Now my goal for 2017 is to get a Jr. Service Engineer position at Gitlab. I LOVE that software, I LOVE their team philosophies. Setting goals for “dream jobs” is step one, now working backwards to get there is the work needed to be done.

                                                          The work landscape is changing.

                                                          1. 15

                                                            The attacks don’t target all MongoDB databases, but only those left accessible via the Internet and without a password on the administrator account.

                                                            So this isn’t a Mongo-specific problem, this is a deployment issue.

                                                            It is very hard to believe that after this highly-mediatized rash of ransom attacks any database administrator won’t double-check to see if his MongoDB server is available online and if the admin account doesn’t use a strong password.

                                                            Ditto to this.

                                                            In a couple of weeks, it is reasonable to expect that all MongoDB servers exposed to the Internet will lose their data and have their content replaced with a ransom demand.

                                                            These “hackers” aren’t doing anything particularly intelligent, only targeting unsecured Mongo instances, so I don’t see where this statement is coming from. If everyone who publicly exposes Mongo to the Internet set the admin password, it sounds like the problem would be solved.

                                                            1. 28

                                                              This is focusing on Mongo mostly because “no password, bind all” is the default setting for any mongo deployment as it assumes you have a working firewall.

                                                              I have a web service (forum) that uses a “no password” mongo, but I only make it bind to a loopback address which I can probe from the outside using SSH tunnels if needed (and if anything happens, I still have daily backups). Even if I didn’t modify the settings to let it bind to that port, my firewall would have stopped the server from being publicly accessible.

                                                              1. 25

                                                                This is a deployment issue.

                                                                I thought this, so I read their Python Getting Started Guide - not a single mention of authentication.

                                                                You either have great documentation with big warning signs or safe secure defaults. Mongo currently has neither.

                                                                Personally, I would always advocate safe secure defaults, not everything can be solved with education.

                                                                1. 6

                                                                  I agree with you, I was mostly commenting on the sensationalism of the prose. This sort of “hack” is not all that advanced, it comes from misconfiguration. The default configuration should of course be more secure by default, there have been a number of articles written to this effect, cf. this one from Shodan ~1 year ago warning about this very issue with Mongo, and how easy it is to exfil/destroy publicly accessible instances.

                                                                  1. 6

                                                                    Absolutely, it’s the equivalent of a port scan. Some “hack”.

                                                                    It is very hard to believe that after this highly-mediatized rash of ransom attacks any database administrator won’t double-check

                                                                    I do take issue with this though as it makes some assumptions which from experience have never been true: a) that any given deployment will have a database administrator & 2) that said database administrator will be competent.

                                                                    To expect developers that have picked a database based on how easy it is to dump JSON in to have any clue about secure database deployment is asking way too much. And the only way to solve that is, as you say, sane defaults.

                                                                    1. 2

                                                                      To expect developers that have picked a database based on how easy it is to dump JSON in to have any clue about secure database deployment is asking way too much.

                                                                      maybe i’m missing sarcasm here. imho, one of the first things one has to do when using new software which is reachable from the network is to check how access can be restricted. regardless if developer or admin. if you use a new power tool which has the capability to maim yourself, you are also expected to take the common precautions.

                                                                      1. 4

                                                                        No sarcasm, it was aimed at new developers though. Expecting new developers to know the world == trouble. To be clear: the “imho” line is your (good) view, which you’ve probably garnered from years of experience and mistakes: a new developer would not have that world view yet.

                                                                        It causes very little extra pain to have some form of authentication by default. Then the user of said software has to learn about authentication from the get go, and expects that they have to handle it post deployment. It’s about creating the right intentions.

                                                                        1. 4

                                                                          Many of my younger colleagues simply don’t know how packets get from point A to point B, as well. So the idea that it could be insecure is surprising or something they just don’t consider.

                                                                          1. 2

                                                                            i’d like sane defaults for authentication too. it just feels wrong that the expectations for the knowledge of developers are that low :/

                                                                  2. 13

                                                                    I wouldn’t say it’s a deployment problem per se. I believe it’s more of a consequence of the industry valuing products that are “easy” above all else. Defense In Depth is a pretty standard security perspective and popular solutions such as Cassandra, Riak, MongoDB, and redis all prioritize making the default configuration very simple at the cost of security. But that’s what the people want.

                                                                    I’m not saying it’s ok to open your database up to the world but just that this is expected if you look at the incentives users are giving authors of databases these days.

                                                                    1. 9

                                                                      So this isn’t a Mongo-specific problem, this is a deployment issue.

                                                                      It is Mongo specific in that the default settings of MongoDB are brain dead and stupid with respect to security.

                                                                      It is very hard to believe that after this highly-mediatized rash of ransom attacks any database administrator won’t double-check to see if his MongoDB server is available online and if the admin account doesn’t use a strong password.

                                                                      Ditto to this.

                                                                      Again, this is somewhat Mongo specific because (a big IME here) MongoDB administrators are not usually at the same level as traditional DBAs. That’s why we’re seeing thousands of MongoDB instances compromised, and no mention of PostgreSQL, Oracle, MySQL, DB2, etc. Sure, this attack is possible with those databases, but they have more sane defaults, and their admins (again, IME) have a better idea of what they’re doing, so it’s not so much an issue there. Yes, now and then you’ll see some idiot leave his Oracle DB exposed, but you don’t see thousands and thousands of Oracle DBs exposed all at once for the same reason.

                                                                      These “hackers” aren’t doing anything particularly intelligent, only targeting unsecured Mongo instances, so I don’t see where this statement is coming from. If everyone who publicly exposes Mongo to the Internet set the admin password, it sounds like the problem would be solved.

                                                                      From the context it’s clear that “exposed to the Internet” in that statement is referring specifically to the MongoDB instances using the default setting of no password and no firewall.

                                                                      I agree with you that it’s really low hanging fruit as far as “hacking” goes. The MongoDB community should be really embarrassed about this.

                                                                      1. 5

                                                                        It seems it’s easy to hyper-inflate the impact or serious skill of a particular culprit behind acts like this. E.g., the Podesta phishing scandal & related events had the momentum of a US Presidential election behind their news cycle, even so it was highly over excitable in its attempt to paint the hacker a Mr. Robot Dark Army type person.

                                                                        The problem isn’t “oh shit hackers are dangerous” it’s “people should learn fundamental cybersecurity concepts before deploying anything with even remotely identifiable or important information.”

                                                                        Count me as a cynic, but if you don’t put a password on your internet-connected database administration account… then you can eat a plate of crow and stfu. Wake up tomorrow and start using better security practices. This is natural selection. We must expect and prepare for the worst, not just hope for the best.

                                                                      1. 19

                                                                        Let me see if I understand correctly: SQL is insecure because mainstream programming languages don’t have good interfaces to SQL databases?

                                                                        1. 15

                                                                          I read it as, SQL is insecure because bad developers will end up using it incorrectly.

                                                                          1. 18

                                                                            Bad programmers will write bad code using any tools, but that’s really besides the point. OP argues that SQL is insecure because:

                                                                            • Prepared statements are difficult to use correctly, which is an unarguable fact.
                                                                            • Not using prepared statements leads to SQL injection, which isn’t true.

                                                                            Raw SQL strings, prepared statements and ORMs are all interfaces between databases and programming languages. Unfortunately, none of them is perfect:

                                                                            • Raw SQL strings are insecure for obvious reasons.
                                                                            • Prepared statements are a chore to use and they don’t buy you that much security, because you’re still manually supplying strings.
                                                                            • ORMs are normally safer than either SQL strings or prepared statements, but they give up much of the expressive power of relational algebra.

                                                                            The real problem is types. Most programming languages don’t have sophisticated enough type systems to model the operations of relational algebra. (But some do!) In particular, nominal types don’t help. For example, if you have two classes Customer and Order, there is no type-level operation that can produce a third class corresponding to select * from Customer C join Order O on C.CustomerID = O.CustomerID. This is a very sorry state of affairs, and it absolutely isn’t SQL’s fault.

                                                                            1. 6

                                                                              Prepared statements are a chore to use and they don’t buy you that much security, because you’re still manually supplying strings.

                                                                              How so? The strings you do supply to prepared statements are incapable of changing the pre-prepared parse tree, which is the big insecurity of smashing random strings together.

                                                                              I agree I’d rather use an ORM with correct-by-design types, though.

                                                                              1. 4

                                                                                If a new maintainer of your software needs to add a field of a weird type to the query, will they learn how to add that flavor of field to the prepared statement, or will they interpolate a string in just this once?

                                                                                1. 1

                                                                                  I know what I would do with Go+Postgres: add a type conversion from string $1::json (etc) in the SQL, then marshal the data to string right before the query and give a string to the driver in Exec() or Query().

                                                                                  I’m actually not sure what the “correct” way to do that would be. One that popped out from the documentation is to implement sql/driver.Valuer on a local typedef or something like that. But that’s a massive pain in the behind and also depends on driver internals.

                                                                                2. 1

                                                                                  Nobody forces you to supply a string that is consistent with the database schema.

                                                                                  1. 1

                                                                                    That’s … not something you’re going to solve short of banning users and their dang input.

                                                                                    1. 9

                                                                                      That isn’t really true. Ur/Web rules out invalid queries at compile time. But this requires two things:

                                                                                      • The type checker must be aware of the database schema. This is the easy part.
                                                                                      • The type system must be capable of expressing the structure of arbitrary derived relations (in the relational model’s sense of “relation”). This requires row polymorphism at the very least.
                                                                                      1. 1

                                                                                        Oh you’re talking about the strings submitted for the prepared statements, not the user input filling in the ?s. I misunderstood and was talking about runtime input.

                                                                                        1. 2

                                                                                          Yes, I was primarily talking about the strings submitted for the prepared statements. However, even the user input filling in the ?s is often less statically checked than it could be. Will Java’s type checker complain if you attempt to suply an int where the database would expect a varchar? Ur/Web’s will.

                                                                                3. 3

                                                                                  How exactly can you argue in quantitative terms the difficulty of using prepared statements?

                                                                                  Isn’t the difficulty of a thing somewhat subjective?

                                                                                  This whole post seems like… satire

                                                                                  1. 11

                                                                                    Nobody guarantees that the result of preparing a statement will be meaningful according to the database schema. That’s the difficulty.

                                                                                    Contrast with Ur/Web, where the type-checker makes sure that your SQL statements make sense.

                                                                                    1. 6

                                                                                      You beat me to it. I was going to add Opa language, too, as it raises the bar vs common options. One could throw in memory-safe languages like Component Pascal or concurrency-safe languages like Eiffel or Rust. Like the web languages, these simply don’t allow specific classes of problems to occur unless the developer goes out of their way to make it happen. Always good to design languages to knock out entire classes of common problems without negative impact on usability if possible.

                                                                                  2. 2

                                                                                    The real problem is types. Most programming languages don’t have sophisticated enough type systems to model the operations of relational algebra.

                                                                                    Types, yes. Type systems, no.

                                                                                    Q has tables, and operations that work on tables. There’s no reason a lesser language like PHP couldn’t do this, it’s just that PHP programmers don’t do this.

                                                                                    Prepared statements are a chore to use and they don’t buy you that much security, because you’re still manually supplying strings.

                                                                                    If you move your authentication into the database (like with row-level security) then your web-layer can simply authenticate against the database and run the prepared queries like an RPC. The biggest problem I see people have with prepared statements is if they have inadequate tooling and don’t invest in it. (Migrations are a dumb and painful way to program, and while commercial offerings are much better, open source is very popular)

                                                                                    1. 3

                                                                                      My day job is to maintain a rather large ERP system. You know, the kind where the typical table has 40-50 fields and the typical primary key has 5 fields. The kind where people are afraid of altering existing tables, because who knows what queries might be affected, so they create another table with the same exact primary key, whose rows are intended to be in 1-to-1 correspondence with the original table, even though that will only make things harder in the long run and we know it.

                                                                                      This tremendous amount of pain is the price of the lack of coordination between language and database. If there were an automatic, convenient way to determine what parts of our application have to be changed in response to a given change in the database, I estimate that we could be twice as productive, while at the same time creating less technical debt. This is precisely the problem type systems solve.

                                                                                      1. 3

                                                                                        This tremendous amount of pain is the price of the lack of coordination between language and database.

                                                                                        I’m not disagreeing with that: Having the business logic in the same language as the database is another way to obtain that coordination, and it offers far more benefits:

                                                                                        A large amount of pain is had in synchronising the continuous single history of “the business database” with the many-branches of modern software development. Building directly on top of the database, and solving the problems that you need in order to do that eliminates pain that you never thought possible, like writing migrations or having to maintain test databases. A type system doesn’t help me get there.

                                                                                        I estimate that we could be twice as productive

                                                                                        Using the same language for your database and your application wins much more than 2x. I would say it wins 10x or even 100x.

                                                                                        The kind where people are afraid of altering existing tables, because who knows what queries might be affected, so they create another table with the same exact primary key

                                                                                        Really the goal should be to have the data in the correct shape. KDB is column-based, and column-based data stores are useful here because you don’t usually want to alter the existing table. You want to hang another column on there, or you want another rollup/index somewhere. That’s cheap (microseconds) in KDB.

                                                                                        Having the database contain your program also means you can easily to analytics on which queries touch which columns, which increases bravery significantly (and safely!).

                                                                                        My day job is to maintain a rather large ERP system.

                                                                                        I have a similar database, although in addition to those fat business data tables that is ingested from a bunch of Oracle/Siebel databases, it also contains very tall analytics data growing at a rate of around 300m web events per day and around 60k call records per day.

                                                                                        KDB also has the advantage of being quite a bit faster than other database engines, so it wouldn’t surprise me if I’m dealing with more data than you.

                                                                                        If you don’t know KDB/Q, you should look into it. Ur/web+postgresql is great, but it has nothing on commercial offerings.

                                                                                        1. 2

                                                                                          A large amount of pain is had in synchronising the continuous single history of “the business database” with the many-branches of modern software development.

                                                                                          Right. We need a notion of “time-evolving schema”, allowing new data to have a different structure from old data, while at the same time allowing queries to be meaningful across schema versions. As far as I know, that problem hasn’t been satisfactorily solved yet.

                                                                                          Building directly on top of the database, and solving the problems that you need in order to do that eliminates pain that you never thought possible, like writing migrations or having to maintain test databases. A type system doesn’t help me get there.

                                                                                          You piqued my curiosity. Let’s say you have a language where tables are first-class values. Altering the structure of a table amounts to changing its type. (As opposed to inserting, updating or deleting rows, which amounts to constructing a different value of the same type.) How do you validate that every part of your application that depends on this table is compatible with the new version, without type checking?

                                                                                          KDB is column-based, and column-based data stores are useful here because you don’t usually want to alter the existing table.

                                                                                          This is a physical implementation detail. I don’t want to worry about that.

                                                                                          KDB also has the advantage of being quite a bit faster than other database engines, so it wouldn’t surprise me if I’m dealing with more data than you.

                                                                                          I’m not too worried about the amount of data I need to process. I’m worried about the complexity of the logical constraints the data must satisfy in order to make sense. Logical errors can manifest themselves even with modest amounts of data.

                                                                                          If you don’t know KDB/Q, you should look into it.

                                                                                          I will.

                                                                                          1. 2

                                                                                            Right. We need a notion of “time-evolving schema”, allowing new data to have a different structure from old data, while at the same time allowing queries to be meaningful across schema versions. As far as I know, that problem hasn’t been satisfactorily solved yet.

                                                                                            Tooling can help a lot, though, and may be good enough. There is commercial tooling (like Control for Kx) which is basically an IDE for your database, complete with multi-user version control. It has the disadvantage of being an online tool, but it provides hints of what the correct solution might look like to me.

                                                                                            This is something I’ve been thinking about for a while.

                                                                                            Let’s say you have a language where tables are first-class values. Altering the structure of a table amounts to changing its type.

                                                                                            However adding a column doesn’t affect code that doesn’t use the column.

                                                                                            How do you validate that every part of your application that depends on this table is compatible with the new version, without type checking?

                                                                                            Static analysis remains possible without type systems provided you don’t learn column names from the network (and if you do, your type system would be incomplete anyway).

                                                                                            This is a physical implementation detail. I don’t want to worry about that.

                                                                                            I know you don’t, but removing abstraction is reduces program size (and therefore bugs), and increases program speed so much that I think it’s often worth thinking about the fact we are meat programming metal. Bugs mean fixes, which is programming we didn’t plan for, and slowness generates heat that harms the environment. And so on.

                                                                                            If you want to change the type of a column from an 64-bit unix-seconds to a 32-bit time and a 32-bit date (KDB has native date types, btw), you have to decide:

                                                                                            • Do you want to rewrite (potentially) multiple terabytes? What about the degradation in service? How long do you need both fields?
                                                                                            • Can you create an alias/helper for the other type so that accessing the variable containing the date will actually use the 64-bit value for old records, but the 32-bit value for new records? What’s the performance impact?
                                                                                            • Can you modify the existing code to support both versions?

                                                                                            And so on. These are real considerations that affect a real system. If we could only sit in our purely-software universe and have enough abstraction, we could make our decisions on what makes better software (asking for a date and getting a date is probably better than doing arithmetic on seconds – and what happens when the calendar changes, anyway) but someone has to solve them, and unfortunately a type system doesn’t actually solve these problems.

                                                                                            A type system only helps with the same part of the problem that tooling solves: Static analysis can find the code, and having a real table “type” means you just use a couple in-memory copy of some of the rows you the programmer believe are representative, which then form your tests for regression tracking.

                                                                                            However having views and a real table type (i.e. doing the database in your programming language) means (performance) testing is easier, there’s a migration path for the data, and you’ll have a good handle on what the real user-impact is.

                                                                                            I will.

                                                                                            Awesome. It is not easy to get into without a commercial need, but the #kq channel on freenode contains people willing to help answer questions. It’s not as high-volume as #ocaml so you might have to wait for the earth to turn and someone in the right timezone to wake up :)

                                                                                  3. 11

                                                                                    That’s a good statement of a important point. If the simplest, most obvious way to use a tool isn’t secure, we must consider the system fundamentally insecure because that’s what will happen in practice. The programmer’s UX of security concerns is vital.

                                                                                  4. 6

                                                                                    That seems mostly reasonable to me. Using SQL in PLs where the default way to use it is by passing in ordinary strings that contain code is indeed insecure. Imagine if mainstream PLs had us defining and calling functions by calling eval() on strings all over the place: I would expect that to lead to terrific quantities of horrid security problems too. I accuse that passing a string to sqlite3_exec() or mysql_query()or PQexec() is equally as scary as passing a string to eval() because RDBMS query languages are either powerful enough to execute arbitrary code or complicated enough to inevitably have bugs that can be leveraged into arbitrary execution.

                                                                                    I’ve seen an interesting alternative in one of C J Date’s older books, “An Introduction to Database Systems”. He has examples of relational queries embedded directly into a language that looks like PL/1, where the queries are actually fully parsed at compile time. I think they had all the niceties, like references to ordinary lexical scoped variables in the queries turning into code that does all the correct binding at runtime and everything.

                                                                                    I’m thinking that one could make a much safer language be just as convenient as doing broken string concatenation is in current PHP, by using quasiquoting, reader macros or just straight up embedding SQL’s entire grammar into the PL’s own grammar in an expression context. I’d identify “PHP with mysql_query() replaced by quasiquoting” as a safer PL than “current PHP”.

                                                                                    Another strategy for making SQL injection harder to write by accident that I’ve seen is in the postgresql-simple library for Haskell. The query execution functions accept a string-like type called Query for which there is an IsString instance, so you can switch on the OverloadedStrings pragma and write code like execute connection "INSERT INTO dogs VALUES (? ?);" (name, cuteness) ­— so the correct, parameterised-query pattern is easy and convenient to write. At the same time, the incorrect string-concatenation code is still possible but much less convenient, so you’re much less likely to write it it. While you can build Query objects from strings, the syntax to actually do that is longer and involves looking up more stuff than the syntax for putting parameters in your queries.

                                                                                    IIRC there are also quasiquoters that let you write that as something looking like [sql|INSERT INTO foo VALUES (${name}, ${cuteness});] as an expression and automatically turn that into the above parameterised-query.

                                                                                    In all of the above, anywhere I refer to “PHP” you may instead read “any PL in which you use SQL by passing an ordinary string to a function or method with query or execute in the name”, i.e. very nearly all of them. PHP only does slightly worse than average here because mysql_query() comes bundled with the runtime but you have to install an ORM on purpose, whereas plenty of other PLs come with neither SQL bindings nor an ORM so it’s almost equally difficult to install the ORM or the raw SQL binding.

                                                                                    1. 5

                                                                                      Imagine if mainstream PLs had us defining and calling functions by calling eval() on strings all over the place: I would expect that to lead to terrific quantities of horrid security problems too.

                                                                                      This gets to the heart of my position. Very well said.

                                                                                  1. 1

                                                                                    Congrats! This is a cool writeup….these posts always motivate me to be more productive

                                                                                    1. 3

                                                                                      Cryptonomicon I actually picked this up on an Audible credit as I wanted to read it for some time, but have had less success lately with reading a paper book than I’d like to admit. Plus audio books are fun!!! Great listen/read. 5/5 will continue to consume this genre of fiction.

                                                                                      Elon Musk: Tesla, etc… biography Great, fairly quick read. Inspiring and insightful but nothing especially astounding. Just a good read.

                                                                                      The Unix Administrator’s Handbook not so much read but cuddled with and learned from and grew with. A true friend, indeed.

                                                                                      Anansi Boys Mmmm.. Neil, at it again. Tore through it in a few afternoons. Highly recommended, just read it.

                                                                                      A Game of Thrones ASOIAF Yes, a long time coming. Lots of road-trip time to burn through this amazing fantasy novel, and give some context to the show.

                                                                                      2001: A Space Odyssey – also long overdue but a good read on a business trip in early 2016.

                                                                                      The Docker Book - Deep dive into Docker… I feel much more comfortable now that I have a fundamental and working knowledge! Great handbook.

                                                                                      Testing Rails Another toolkit-specific guidebook that I highly suggest for those trying to get a grasp on BDD/TDD with Rails or in general!

                                                                                      Beyond that, I started a new company and spent my first year entirely self-employed. It’s been a weird year, but personally I’ve probably grown more than ever.

                                                                                      1. 1

                                                                                        The ASOIAF links to Google. And congrats on your company!