1. 28

    1. 12

      It’s frustrating that they describe the technologies they’re building on top of, and give a handwavy pitch of what it does (“conceptually similar to IPFS and Tor, but faster”) … but they don’t describe its functionality or feature set.

      The underpinnings look great. I’m using a lot of the same crypto & RPC components in my own vaguely-similar Tendril project. I just want to know what comes in between “Step 1. DHT” and “Step 3. Profit!”

      1. 5

        I spent a few days looking at it when it was first released and tried to summarise the main ideas in a post, which I will share in case it helps. https://thomask.sdf.org/blog/2023/08/15/a-few-notes-on-veilid.html

        1. 1

          Very useful overview, thanks!

        2. 1

          I appreciate your thoughts on this and wasn’t aware of this lack of encryption for keys. I’d imagine logic can be made to prevent a DHT record from being propagated (like only accepts changes for a set of keys I care to show? or some sort of blacklist like with BitTorrent)?

          I find communal storage to be more and more important outside of the West because of its durability and recentering the people instead of the cables as the thing that has to be stewards of the data.

        3. 1

          Thank you for the writeup!

          Personally I have a pretty low risk tolerance for [nasty/illegal media files] randomly existing on my PC or phone, even temporarily and in an encrypted form (since my device has the key).

          Yeah, that’s bad. I can see why they say this is unavoidable: even if you required E2E encryption for files, how do you prevent someone uploading an “encrypted” file that’s actually unencrypted? Although the only reason to do that would be to deliberately spread illegal/compromising content, and it wouldn’t be exactly obvious or easy to find.

          To some degree you accept this risk whenever you host an online service that accepts arbitrary data, but the risk feels elevated in the context of software that is explicitly designed to provide anonymous data sharing with onion routing.

          Yeah. I long since gave up on the domain of P2P social software with unlimited reach, for similar reasons. A lot of these problems go away if the only peers you exchange data with are members of the user’s direct social circle; although of course it exacerbates connectivity/discovery problems.

      2. 2

        This is a meta-problem in capability theory; we never really enumerated the various patterns that can be built with capabilities and published them in a single definitive way, so I have to link to janky old wikis or one-off blogs just to explain what’s possible.

        In short, a capability-aware network allows for delegation of arbitrary capabilities, up to the expressive power of the underlying cryptographic protocol.

        1. 2

          I’ve read a fair bit about capabilities. But they’re a very broad technology! I don’t get the impression Veilid is a general purpose capabilities platform like E or Spritely; the website says it’s focused on storing data for social networks.

    2. 7

      I was talking to someone about this and we both found it odd that there is no white paper and very little documentation for this. I realise this is early stages, but it seemed to me they went straight to implementation without formalising the protocol - I could very well be completely wrong about this though.

      The benefit of ironing out a protocol and formalising it in a paper is that it introduces academics, programmers, and a wider audience to the work, allowing them to scrutinise it for weaknesses before you implement it in code (or during early stages of implementation).

      It looks great based on the cryptographic primitives chosen, but other than that, the documentation and technical details on the protocol+networking itself seems pretty sparse for something claiming to be this influential+game changing to the anonymous/private/secure onion networking scene.

      This isn’t a dig at the creators of this project - I have high hopes for it and I hope it doesn’t fade into obscurity like so many of these kinds of protocols eventually do.

      1. 6

        I agree with wanting a white paper. Just from watching the launch at DEF CON and the little reading that I’ve done I collected a couple of surface level concerns that would likely be addressed or would be easier to infer how they could be addressed.

        That being said, I believe that the approach here was simply: cypherpunks write code. I think that approach also has some merit to it, but since that line was originally coined the world changed a whole lot, and the type of attacks one can mount at such a protocol has increased so much that code alone is not enough to convince the wider community that it’s a good idea to run with - yet.

        I also believe there’s a couple of assumptions built-in already that apply to power users in the western hemisphere, but that shouldn’t be universally applied. But regardless: I do want a protocol like that to exist, and I’m glad there’s some relatively reputable folks experimenting at least.

      2. 5

        it seemed to me they went straight to implementation without formalising the protocol

        Bravo to them! I’m of the opinion that too many projects nail down too many details in a formal spec too early. Planning it out in a formal spec certainly has its benefits, but it has the massive drawback of setting things in stone before you know if it’ll even work in practice.

        Much better to iterate, experiment, and evolve the system, figure out what works, and then draw up a spec only after you’ve figured out which of your ideas are good and which ones are bad.

        1. 3

          This is a pretty fair POV, actually. I never thought about it this way. I’m used to network protocols and cryptography in a more academic and formalised format. I see the appeal in doing it this way now.

          But I still hope they do put out a formal-ish whitepaper eventually.

      3. 3

        It’s using CapnProto, so that’s a lot of information about the network protocol. Above that it will be a high-level capability based RPC protocol; I haven’t found docs for that yet.

        1. 2

          They aren’t using CapnProto’s “native” RPC mechanics, they’re just using it for serialisation/deserialisation

    3. 1

      I had to ask bing AI bot to give me a code example how to connect to veilid and write and retrieve info. Was it all an hallucination? It looked nice and easy. I couldn’t find anywhere some basic examples on veilid’s website or GitHub. Anyone spot code and api examples? Please share a link. Project is promising at a glance. Have use for it as I rolled my own distributed / router system but for my specific project.

      1. 3

        It’s in a very very early stage at the moment. I’m sure better documentation is on the way.

        In fact, when I first heard of this, the documentation was pretty much nonexistent. It’s improved since then. Not in a usable manner yet for sure, but I have noticed progress.

        1. 1

          I see. That makes sense. I found this veilid chat app demo for python in a fit of rage. Just couldn’t believe there wasn’t any examples. The location wasn’t obvious.

          I autogenerated this possibly semi-working pseudo code from chat example code on how to write to and read from veilid’s DHT. The chat example was little too much stuff going on for me to parse:

          import logging
          import os
          import veilid
          LOG = logging.getLogger(__name__)
          KEY_TABLE = "veilid-demo"
          # Retrieve connection details from environment variables with default values.
          API_HOST = os.environ.get('VEILID_API_HOST')
          API_PORT = int(os.environ.get('VEILID_API_PORT'))
          def store_key(conn: veilid.json_api._JsonVeilidAPI, key: str, value: str):
              """Write a single key to the keystore."""
              tdb = conn.open_table_db(KEY_TABLE, 1)
              key_bytes = key.encode()
              value_bytes = value.encode()
              LOG.debug(f"Storing {key_bytes=}, {value_bytes=}")
              tdb.store(key_bytes, value_bytes)
          def load_key(conn: veilid.json_api._JsonVeilidAPI, key: str) -> str:
              """Read a single key from the keystore."""
              tdb = conn.open_table_db(KEY_TABLE, 1)
              key_bytes = key.encode()
              LOG.debug(f"Loading {key_bytes=}")
              value = tdb.load(key_bytes)
              LOG.debug(f"Got {value=}")
              return value.decode() if value is not None else None
          def main():
              # Connect to the Veilid API using environment variables
              conn = veilid.json_api_connect(API_HOST, API_PORT)
              while True:
                  choice = input("Choose: (w)rite, (r)ead, or (q)uit: ")
                  if choice == 'w':
                      key = input("Enter the key to write: ")
                      value = input("Enter the value to write: ")
                      store_key(conn, key, value)
                      print("Data stored successfully!")
                  elif choice == 'r':
                      key = input("Enter the key to read: ")
                      value = load_key(conn, key)
                      if value:
                          print(f"Value: {value}")
                          print("No data found for the given key.")
                  elif choice == 'q':
                      print("Invalid choice.")
          if __name__ == "__main__":
          1. 1

            I appreciate you doing that digging. I’ll have a look at this later. Looks surprisingly simple.

    4. 1

      This seems great similar to yggdrasal I also wonder why they don’t talk much about the underlying transport. You don’t need nat if it’s an IPv6 network. I do understand that they want to establish essentially a non-traceable overlay network, but the underlay is just as an important.

      1. 3

        “if it’s an IPv6 network” - well, sometimes it isn’t, so things like tailscale or veilid have to hole punch… because they want to be used by everyday devices :)

      2. 1

        Checked it out. There isn’t any real guides or documentation with practical examples on their GitHub or webpage. Veilid I see promised the ability to read and write objects across the network using content IDs of some sort, so that’s a big difference I suppose.