1. 12

Let’s say I am writing a small thing, and I want to expose it as a simple JSON rpc. I value simplicity, so grpc and (I think) graphql are out.

If I want to define the API via language independent IDL, what are my choices? I know that JSON schema exists, but it doesn’t seem like a good IDL (not human readable, too much non-type based validation built-in, haven’t seen it actually being used in real world).

I kinda like the style LSP is defined in, via a bunch of TS interfaces, but that’s an informal subset.

Anything else to take a look at?

    1. 8

      Would OpenAPI fit the bill? You can generate both client and server code in tons of languages, and there are ready-to-use interactive sandboxes you can bundle with your servers.

      It does let you define a bunch of validation rules if you like, but you can just not include them if you don’t want them. It isn’t perfect (if you need to define a recursive data structure you will probably see some code generators crash or produce garbled output) but works pretty well if you’re defining straightforward structures. I find its YAML format reasonably easy to read.

    2. 5

      Honestly, graphql is perfectly reasonable. What type of simplicity do you think it violates?

      Avro has a a json representation and is intended to define rpc interfaces. It’s also a type based idl.

      1. 1

        For my case, I think graphql solves to much — I think I want a straightforward request/response RPC, I don’t need graphql capabilities for fetching data according to a specific query.

        This is kinda what protobuf does, but, impl wise, protobuf usually codgens a huge amount of ugly code which is not pleasant to work with (in addition to not being JSON). Not sure what’s the status with similar implementation complexity for graphql — didn’t get a chance to use it yet.

        1. 2

          Protobuf has a 1:1 json encoding. You could write your schema in Protobuf and then use the JSON encoding with POST requests or something to avoid all the shenanigans?

      2. 1

        Honestly, graphql is perfectly reasonable.

        Doesn’t it still violate HTTP, badly? IIRC, it returns error responses with status code 200. I thought that it sent mutations in GET requests too, but from a quick look it looks like I misremembered that (shame on me!).

        Regardless, REST is best.

        1. 1

          No, it returns errors inline in the body. There’s nothing non-restful about graphql. Indeed, the introspection and explicit schema, together with the ability to use it over a get request make it more restful than most fake rest rpc endpoints.

    3. 4

      This is a weird question because you seem to already know the answer — JSON Schema — but don’t seem to like it.

      I’ve used JSON Schema for “real world” things. It’s fine. I’m generally meh on the whole typed-RPC approach to web APIs, but if you’ve decided you’re going to do JSON-RPC, then JSON Schema is a tolerable way to do it.

      1. 1

        Yes, I believe both that JSON schema solves the problem I have, and that this solution, while technically working, is unsatisfactory. I don’t see a contradiction here: feels similar to, eg, inventing JSON while there’s already XML :)

        1. 3

          My takeaway would be less “I should look for an alternative to JSON Schema” and more “I should look for an alternative to doing this with an RPC protocol, because the things I dislike about JSON Schema are just a symptom”. But YMMV :)

    4. 4

      See if there’s a converter from types in your language of choice directly to a machine-readable schema language. If you think OpenAPI/JSONSchema is too wild to write by hand, see if you can generate it from your internal types. For example, Zod is a nice validador library for Typescript, and there’s a zod-to-openapi thing. I haven’t tried it, but that kind of pair could be what you’re looking for?

      I haven’t used gRPC/Protobuf much but it seems like the clear winner for internal distributed systems programming. There are tons of converters from Proto3 to X, so you can write a Proto3 and be reasonably sure you can target kinda whatever output. I think it’s super ugly though and has a bunch of peculiarities.

      Avro seems quite nice. It uses a nice JSON format for the IDL, has straightforward union and array types, fast JS implementation, but unfortunately not much ecosystem compared to Protobuf or OpenAPI.

      I agree that Typescript’s interface. My idle time project at work is fiddling a Typescript -> Protobuf converter.

    5. 4

      If I want to define the API via language independent IDL . . .

      Do you? Why?

      IDLs are schemas that define a formal protocol between communicating parties. They represent a dependency, typically a build-time dependency, coupling producers and consumers. That can bring benefits, definitely. But if I can’t call your API without fetching a specific file from you, somehow, and incorporating it into my client, that’s also a significant cost! Open networks are supposed to work explicitly without this layer of specificity. HTTP is all I need to talk to any website, right?

      HTTP+JSON is pretty language-agnostic. I can even do it at the shell. JSON-RPC is nowhere near as mature, or as supported. What does this extra layer get you? What risks does it actually reduce? And are those risks relevant for your use cases? Why do you think e.g. Stripe doesn’t require JSON-RPC for its API?

      IMO the only place that IDLs make sense are in closed software ecosystems — i.e. corporate environments — that have both experienced specific pains from loosely-defined JSON APIs, and can effectively mandate IDL usage across team boundaries. Anywhere else, I find it’s far more pain than pleasure.

      1. 3

        Heh, I actually was thinking about your’s similar comment in another thread when asking, thanks for elaborating!

        I think I agree with your reasoning, but somehow still come to the opposite conclusion.

        First, agree that the JSON-RPC is a useless layer of abstraction. I’ve had to use it twice, and both times the value was negative. In this question, I am asking not about the jsonrpc 2.0, but about an RPC which works over HTTP, encoding payloads in JSON. I do have an RPC, rather than REST, style in mind though.

        I also couldn’t agree more about the issue with build-time dependencies is worth solving. Build time deps is exactly the problem I see at my $dayjob. We have an JSON-over-HTTP RPC interface and the interface is defined “in code” – there’s a bunch of structs with #[derive(Serialize)] structs. And this leads people to thinking along the lines of “I need to use the API. The API is defined by this structs. Therefore, I must depend on the code implementing the API”. This wasn’t explicitly designed for, just a path of least resistance if you don’t define API explicitly, and your language has derives.

        That being said, I think there has to be some dependency between producer and consumers? Unless you go full HATEOAS, you somehow need to know which method to call and (in a typed language, for ergonomics) which shape the resulting JSON would have. For stripe, I need to fetch https://stripe.com/docs/api/pagination/search to figure out what’s available. And, again, there is https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json.

        And, if we need at least an informal doc for the API, I don’t see a lot of drawbacks in making it more formal and writing, say, literal typescript rather than free-form text, provided that the formalism is lightweight. The biggest practical obstacle there seems to be absence of such lightweight formalisms.

        So, the specific “why”s for me wanting an IDL in an open decentralized ecosystem are:

        • reifying “this is the boundary exposed to outside world” in some sort of the specific file, so that it is abundantly clear that, if you are changing this file, you might break API and external clients. You could do that in the “API is informally specified in code” scenario, but that requires discipline, and discipline is finite resource, running out especially quickly in larger teams.
        • providing the low-friction way to document and publish promises regarding the API to the outside world. Again, with some discipline, documenting API in api.md file would work, but it seems to me that doc-comments require less effort for upkeep than separate guides.
        • making sure that details about the language the implementation is written in don’t accidentally leak. Eg, APIs probably shouldn’t use integers larger than 32 bits because they won’t work in JavaScript, but, if my impl language is Rust, I might not realize that as native serialization libraries would make u64 just work. More generally, in Rust just slapping derive(Serialize) always works, and often little thought is given to the fact that the resulting JSON might be quite ugly to work with in any non-rust language (or just to look at).
        • Maaaaybe generating stub clients and servers from the spec.
        1. 3

          That being said, I think there has to be some dependency between producer and consumers?

          Yeah! When a client calls a server there is definitely a dependency there. And as you note, there has to be some kind of information shared between server in client in order for that call to succeed. I suppose my point is about the… nature? of that dependency. Or maybe the efficacy of how it’s modeled?

          At the end of the day, you’re just sending some bytes over a socket and getting some bytes back. An IDL acts as a sort of filter, that prevents a subset of those byte-sets from leaving the client, if they don’t satisfy some criteria which is assumed will be rejected by the server. Cool! That reduces a class of risk that would otherwise result in runtime errors.

          1. What is impact of that risk?
          2. What benefits do I get from preventing that risk?
          3. What costs do I incur by preventing that risk?

          I suppose I’m claiming that, outside of a narrow set of use cases, and certainly for general-purpose public APIs, the answers to these questions are: (1) quite low, (2) relatively few, and (3) very many. (Assuming you’re careful enough to not break existing consumers by modifying the behavior of existing endpoints, and etc. etc.)

          reifying “this is the boundary exposed to outside world” in some sort of the specific file, so that it is abundantly clear that, if you are changing this file, you might break API and external clients. You could do that in the “API is informally specified in code” scenario, but that requires discipline, and discipline is finite resource, running out especially quickly in larger teams.

          I get that desire! The problem is that the IDL is a model of reality, not the truth, and, like all models, it’s a fiction :) Which can be useful! But even if you publish an IDL, some dingus can still call your API directly, without satisfying the IDL’s requirements. That’s the nature of open ecosystems. And there’s no way for you to effectively mandate IDL consumption with plain ol’ HTTP APIs, because (among other reasons) HTTP is by construction an IDL-free protocol. So the IDL is in some sense an optimistic, er, optimization. It helps people who use it — but those people could just as easily read your API docs and make the requests correctly without the IDL, eh? Discipline is required to read the API docs, but also to use the IDL.

          . . . document and publish … API [details] . . . making sure that details about the language [ / ] implementation don’t accidentally leak . . .

          IDLs convert these nice-to-haves into requirements, true.

          generating stub clients and servers

          Of course there is value in this! But this means that client and server are not independent actors communicating over a semantically-agnostic transport layer, they are two entities coupled at the source layer. Does this model of your distributed system reflect reality?

          I dunno really, it’s all subjective. Do what you like :)

          1. 1

            Convinced! My actual problem was that for the thing I have in mind the client and the server are implemented in the same code base. I test them against each other and I know they are compatible, but I don’t actually know how the JSON on the wire looks. They might silently exchange xml for all I know :-)

            I thought to solve this problem with an IDL which would be close to on-the-wire format, but it’s probably easier to just write some “this string can deserialize” tests instead.

            I’d still prefer to use IDL here if there were an IDL geared towards “this are docs to describe what’s on the wire” rather than “these are types to restrict and validate what’s on the wire”, but it does seem there isn’t such descriptive thing at the moment.

            1. 1

              docs to describe what’s on the wire [vs] types to restrict and validate what’s on the wire

              Is there a difference here? I suppose, literally, “docs” wouldn’t involve any executable code at all, would just be for humans to read; but otherwise, the formalisms necessary for description and for parsing seem almost identical to me.

            2. 1

              an IDL which would be close to on-the-wire format

              (Just in case you missed it, this is what Preserves Schema does. You give a host-language-neutral grammar for JSON values (actually Preserves, but JSON is a subset, so one can stick to that). The tooling generates parsers, unparsers, and type definitions, in the various host languages.)

            3. 1

              If your client and server exist in the same source tree, then you don’t have the problems that IDLs solve :)

              edit: For the most part. I guess if you don’t control deployments, IDLs could help, but certainly aren’t required.

              1. 1

                They are the sever and a client — I need something to test with, but I don’t want that specific client to be the only way to drive thing.

                1. 1

                  They are the sever and a client — I need something to test with, but I don’t want that specific client to be the only way to drive thing.

                  I test them against each other and I know they are compatible, but I don’t actually know how the JSON on the wire looks. They might silently exchange xml for all I know :-)

                  It seems you have a problem :)

                  If the wire protocol isn’t specified or at least knowable independent of the client and server implementations, then it’s an encapsulated implementation detail of that specific client and server pair. If they both live in the same repo, and are reasonably tested, and you will never have other clients or servers speaking to them, then no problem! It’s a closed system. The wire data doesn’t matter, the only thing that matters is that client-server interactions succeed.

                  If all you want to do is test interactions without creating a full client or server component, which I agree is a good idea, you for sure don’t need an IDL to do it. You just tease apart the code which speaks this custom un-specified protocol from the business logic of the client and server.

                  package protocol
                  package server // imports protocol
                  package client // imports protocol
                  

                  Now you can write a mock server and/or mock client which also import protocol.

                  1. 2

                    This (extracted protocol with opaque wire format) is exactly the situation I am in. I kinda do want to move to the world where the protocol is knowable indepedently of the specfic client and server, hence I am seeking an IDL. And your comment helped me realize that just writing tests against specific wire format is a nice intermediate step towards making the protocol specification – this is an IDL-less way to actually see what the protocol looks like, and fix any weirdness. As in, after writing such tests, I changed JSON representations of some things because they didn’t make sense for an external protocol (it’s not stable/public yet, so it’s changable so far).

    6. 3

      I’m not sure it’s quite what you’re after, but maybe take a look at trpc. I’m using it happily at work, but if you’re not using Typescript everywhere then it might not be a good fit.

    7. 3

      Though I’m hesitant to suggest it given how it was received here last time it was posted, https://palantir.github.io/conjure/ might be worth a look — it was built as an IDL for JSON RPC. It’s spec format is more concise and more restrictive than OpenAPI (and as a result less flexible), and there’s decent support for Java, Typescript and Rust, and more limited support for Python.

    8. 3

      I created an IDL with plain typescript interfaces. The tool parses Typescript AST, converts exported interfaces and meta to JSON. From there I use simple ES6 template literals to generate code.

      I started creating my own DSL, but why not just use TypeScript. It’s simpler than gRPC or GraphSQL for generating code. I would never write OpenAPI (feels like J2EE for APIs).

      Typescript has mostly everything you need:

      export interface AddArg {
          a: number;
          b: number;
          // required is '?'
          c?: number;
      }
      
      export interface Math {
        // language specific instructions through tags
        /** @_go { ident: 'Add' } */
        add(arg: AddArg): number
      }
      

      I create strongly typed TypeScript requests in Next.js frontend and Go backend code from the JSON IDL.

      It’s not in a state where I want to share the repo yet. I’m iterating on how to process templates more intuitively, taking ideas from hygen.

      1. 1

        <3 exciting, from the cursory look, it does seem that the world is missing something like this, thanks!

    9. 3

      I wonder whether WebRPC or Twirp could be of interest to you.

    10. 2

      Excuse my newbieness, but what’s IDL? There’s just so many acronyms being thrown around here, it’s barely readable.

      1. 3

        An IDL is, as @fstamour says, and interface definition language. These are used to define interfaces in a way that is agnostic to the implementation language. Most ’80s object broker systems (COM, CORBA, and so on) had quite rich ones that allowed you to automatically map the abstractions that they provided into different languages. Many lightweight RPC mechanisms that evolved in the ‘90s in response to the perceived complexity of things like DCOM and CORBA also had them, though not all. As I recall, the W3C maintains one as well, for all of the DOM APIs, to provide the option for non-JavaScript languages in the browser to be able to take advantage of the same APIs (though since IE died, taking VBScript with it, and WebAssembly avoided direct access to JavaScript APIs, this hasn’t been very important).

        The idea behind an IDL is to provide a language-agnostic description of an interface. If your underlying RPC mechanism uses JSON then you implicitly have two languages that you need to support: JSON for the wire transport and whatever language you use to implement the endpoints and you want to be able to translate from foo(42) to something that generates some JSON that looks a bit like { method: "foo", args: [ 42 ] } (or, hopefully, something more sensible) and have the foo that’s exposed to the implementation language’s type checker (does foo always take one argument and does it have to be an integer?).

        Part of the problem with JSON here is that it doesn’t have a particularly rich set of primitive types. Everything is encoded as text, which has representations problems for binary floating point numbers, and JSON doesn’t have any mechanism for defining things like 32-bit integers other than as numbers. Worse, it can’t represent the entire range fo 64-bit integers at all.

        JSON Schema lets you represent integers by defining that they must be a multiple of 1 and fixed-width integers by expressing the minimum and maximum values, but that’s a validation constraint rather than something that’s expressed in the wire format. This means that, if you’re using JSON, you actually want to get three things out of your IDL:

        • A mapping into your implementation language(s).
        • A projection into JSON for your wire format.
        • A validator for the JSON that ensure that the values are all within the required ranges.

        OpenAPI is an IDL for web services. AutoRest can generate Java, C#, Python, and TypeScript definitions from OpenAPI definitions. There are other tools that will generate the IDL from annotations in the server-side implementation.

        1. 1

          Worse, it can’t represent the entire range fo 64-bit integers at all.

          Do not confuse JavaScript with JSON. There is no such limitation in JSON since it’s numbers are just strings.

          1. 2

            The JSON spec doesn’t define limits on the number type, but it does advise that

            Note that when such software is used, numbers that are integers and
            are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
            sense that implementations will agree exactly on their numeric
            values.
            
      2. 1

        Interface Definition Language

    11. 2

      This is in a similar spirit as what your describing: I might look at TRPC (https://TRPC.io) which is a TypeScript defined server + client approach.

      Otherwise, I would actually encourage taking a closer look at GraphQL, since the static analysis and documentation tooling for your clients are very good for the majority of use cases. Plus, there are a lot of resources to learn GraphQL and clients to choose from for different languages. And, they have an okay approach to unions/variant types that are a bit more approachable than straight up TS discriminated unions (which don’t have language level consistency) IMO.

    12. 1

      We had a similar requirement (document JSON output of a number of commands in build2) and came to the same conclusion regarding JSON Schema. We ended up documenting it as serialization of C++ structs and based on the user feedback this worked out well. If interested, the details are here (scroll to the JSON OUTPUT section): https://build2.org/stage/bpkg/doc/bpkg-common-options.xhtml

    13. 1

      https://awslabs.github.io/smithy/ is an interesting one

    14. 1

      You might find Preserves Schema interesting [1,2]. I’ve been using it (uh, and defining and implementing it :-) ) for multi-language interop in a few different settings including RPC. It’s still early days for it, though.

      [1] https://preserves.dev/preserves-schema.html
      [2] https://synit.org/book/guide/preserves.html#schemas

    15. 1

      I bet you could coax Dhall into generating JSON schema and/or OpenAPI files. That way you’d have the concision of a nice type system & functional language but compatibility with the OpenAPI ecosystem.