1. 6

    elasticsearch. I store 2k events per second, do string search on it and so on. Tuned everything to use as less ram, cpu and disk as possible, wrote my own log parser (logstash is very slow). I have two instances, one for indexing, and one for searching. I periodically move indices from the index node to the search node. Then I optimize the indices in the search node.

    That’s because for us it’s more important to store events fast, we don’t care if it takes more time to search. The search is for internal usage.

    To store such big data I create hourly indices of about 7 million events.

    Didn’t like kibana, tried to hack on it ecc. but was too limiting for my purposes. So wrote my own sql-to-elasticsearch language with a nice web interface. Once finished I hope my employer lets me publish it.

    We plan to store even more data in it.

    I’ve tried InfluxDB and it’s… don’t know, 100x slower than elasticsearch?

    1. 2

      I’ve seen your Nix posts - are you a Haskell user? If so, are you using Bloodhound?

      1. 2

        Not an haskell user, no. Bloodhound seems interesting, the demo does not work at the moment though :(

        1. 1

          Could you help me out here, what demo doesn’t work? Possibly I needed to update the docs and missed something.

          1. 2

            The one in the apache bloodhound homepage: http://goo.gl/9NqjB Cannot open it, error 502 proxy error. But I’m indeed not a good haskell user :P Now I understand what you mean, the haskell package, sorry.

            Edit: So it’s a very good idea to have a typed dsl for elasticsearch. The sql->elasticsearch is mostly for querying outside of a programming environment. E.g. I use it for plotting in the web interface.

            1. 1

              Phew, I was worried there for a moment! Apache Bloodhound ain’t me, sorry.

              This is me: https://github.com/bitemyapp/bloodhound/

              www.stackage.org/package/bloodhound

              So it’s a very good idea to have a typed dsl for elasticsearch.

              You’re telling me! Elasticsearch’s API drives me crazy.

              1. 2

                For documentation I think elasticsearch would benefit from something similar to http://api.highcharts.com/highcharts#navigation That is, display all the possible options, an abstract json that shows the whole structure.

                1. 1

                  I agree, although I’m rather fond of how Haskell’s data types are more expressive than JSON and can express things like sum types, which is really a way of saying “or”

      2. 1

        When did you try InfluxDB? @apg above claims it’s gotten a lot better since I tried it. I still have a hard time imagining it will have gotten that much better.

      1. 4

        I already left a comment on the post when this was first posted. You should read the comments, there’s a lot of them, and they’re good.

        1. 1

          I saw quite a few “just use Nix” comments. Nix seems very interesting, if a little unapproachable.

          Has any language decided to use Nix as part of its native build pipeline/recommended package manager?

          Anyone using Nix to build Go1 code?


          1. Go is hard enough to search for (kludge by searching for “golang”), but Nix is almost worse, because of all the usages of *nix on the internets.
          1. 5

            I came across this post earlier today about using Nix in place of virtualenv for setting up isolated Python environments. It’s also worth noting that there’s a project called Guix which swaps out the nix expression language for Guile scheme. There seems to be a fair amount of cross-pollination happening between Nix and Haskell’s Cabal, too, but a Haskeller would be able to speak to that better.

            1. 3

              I use Nix to build my pet Go projects. Go packaging in Nix is under work currently. You can already use it if you are confident with Nix. Example here: https://github.com/CiotoFlow/goagainst/blob/master/default.nix

              1. 2

                I’ve been considering using Nix in Dylan. One issue that we see is the lack of Windows support. There are a few other things that might be difficult as well, like establishing a good out-of-the-box experience for someone who doesn’t already have Nix installed.

                We’re in the early stages of thinking about this though and haven’t done any prototyping yet.

                Edit: I’d love to work with someone on this if anyone is interested in even talking about some of what is involved. I only know Nix from a high level. Chris Double packaged Dylan for Nix, so that part is already there.

                1. 3

                  Lack of Windows was a killer for us in Rust as well, as it’s a hard requirement.

                2. 2

                  I attempted this for Erlang, found it required to be way to hacky to pull it off and ditched it.

              1. 3

                Apart writing the usual nix pills, I’m trying to port systemd into initrd for nixos.

                1. 1

                  As usual I’m writing nix pills. In addition, I’m setting up a nix pastebin similar to sprunge.

                  1. 3

                    In particular, Nix pills: http://lethalman.blogspot.it/search/label/nixpills

                    It’s a series of in-depth advanced posts about the Nix deployment system.

                    1. 6

                      NixOS and Vala as usual for my free software projects. At work, I’m working on my second FUSE file system. We will probably open source the first one. It’s a star-topology (also other topologies available) replicated master-master active-active file system.

                      1. 4

                        What are you doing with NixOS? I’m a big fan and use it everywhere I can.

                        1. 1

                          I have a NixOS server, mostly for personal stuff or rebuilding packages instead of doing it on my desktop, and a couple of public services. About contributing, I’m maintaining gnome 3 and some other stuff.

                          Not yet using it as my primary desktop just because I’m still planning how to install nixos in a subdirectory /nixos of my existing debian desktop, and chainload grub. That way I don’t have to backup the data and redo everything in order to replace my debian installation.