Threads for jonb | Lobsters

1

Arrgh edited 3 years ago | link | on: Introducing Espresso - LinkedIn's hot new distributed document store

~~This blog post was made one year ago today. :)~~

AFAICT the source has not been released yet.
1. 4
  
  jonb 3 years ago | link
  
  This blog post was definately released today. I’ve just contacted the author to see what’s up with the date. :)
  1. 1
    
    Arrgh 3 years ago | link
    
    You’re right! That’s funny. :)

8

benbjohnson 4 years ago | link | on: What are you working on this week?

I’m wrapping up my LMDB port to Go this week. I just need to get deletion, tree rebalancing and page reclamation working. And a whole bunch of documentation. The library is pre-alpha but here’s the repo for anyone interested in poking around:

https://github.com/boltdb/bolt

I started using the testing/quick standard library which is a QuickCheck-style library for black box testing. It’s a cool library but I never hear anyone talking about it.
1. 2
  
  hedgehog 4 years ago | link
  
  Very cool. I haven’t used LMDB before, how would you compare it in practice with LevelDB?
  1. 3
    
    benbjohnson 4 years ago | link
    
    I used LevelDB before as a backing store in the past but I had some issues with it when using it from multiple threads using the C API. It worked pretty well but it’s approach is somewhat complicated. It has multiple levels of storage files that have to be compacted periodically so performance can be variable.
    
    LMDB uses a mmap’d B+tree that is updated in-place so there’s no compaction required. The mmap is read-only so data structures can be mapped directly to the underlying data so there’s no memcpy() required. (Writes occur using vectorized I/O on a regular file descriptor)
    
    Ultimately I like the LMDB approach because it’s simple and grokable. My implementation omits the niche features in LMDB and it’s currently 1500 LOC.
    1. 2
      
      hedgehog 4 years ago | link
      
      Thanks, that’s helpful. I’m using LevelDB in a project (on the topic of things we’re working on this week) but I’ve only read about LMDB. In the reading it’s hard to sort out its real merits from the author’s ranting but being able to use values directly from mapped memory sounds useful. One caveat to the LMDB benchmarks for anyone who’s following this and skims them is that they disable compression in LevelDB. This probably speeds it up for the memory workloads but will hurt if you’re using slower storage like AWS EBS volumes.
      1. 3
        
        jonb edited 4 years ago | link
        
        Also, take a look at BangDB.
        
        http://highscalability.com/blog/2012/11/29/performance-data-for-leveldb-berkley-db-and-bangdb-for-rando.html
        
        I’m currently trying to fight through integrating it into a C project (with an autotools flag to switch between LevelDB, BDB, and BangDB).
        
        2
        
        sirpengi 4 years ago | link
        
        Thanks for that! BangDB looks pretty awesome, and I’ll probably give it a try later. I’ve been looking at cross-platform kv store as a persistence layer for a lua project. My initial thought was leveldb, and while it’s supposed to support windows I couldn’t get it to compile at all. I ended up using UnQLite (and wrote luajit bindings for it), but I’m still open to anything with a decent license.
  2. 2
    
    vjoel 4 years ago | link
    
    https://symas.com/is-lmdb-a-leveldb-killer
    
    We state quite clearly that LMDB is read-optimized, not write-optimized. I wrote this for the OpenLDAP Project; LDAP workloads are traditionally 80-90% reads. Write performance was not the goal of this design, read performance is. We make no claims that LMDB is a silver bullet, good for every situation. It’s not meant to be – but it is still far better at many things than all of the other DBs out there that do claim to be good for everything.
    
    Disclaimer: I have no experience with LMDB. I maintain one of the ruby LevelDB wrappers: https://github.com/vjoel/ruby-leveldb-native.

2

csl edited 4 years ago | link | on: The Game of Distributed Systems Programming. Which Level Are You?

What level would it be to use CSP or temporal logic to model the distributed system? I know you can use model checkers to find flaws in the design, but they work by searching the program state space instead of doing proofs, and so can only randomly find errors.
1. 2
  
  jonb edited 4 years ago | link
  
  One of the papers related to Chubby (Paxos made live I think?) has a section about testing their implementation. At one point, they mention that their test suite was running for days before they found a rare bug in their implementation.
  
  Level 4 on this list should probably be something about testing your implementation. Level 5 should be the creation of new algorithms (and making liveness and safety proofs for them). Level 6 should be complete mastery (or some other level which I’m not aware of yet).
  
  To be really complete, something about using logs and basic database design should be included somewhere (system r, aries, etc).

4

dwrensha edited 4 years ago | link | on: The Game of Distributed Systems Programming. Which Level Are You?

See also Cap'n Proto RPC (http://kentonv.github.io/capnproto/rpc.html), which implements an elegant distributed object system. Cap'n Proto deals what this post calls “level 1” problems by providing promise pipelining; RPC actions are asynchronous and get reified as events and wrapped as promises, allowing the Cap'n Proto runtime to optimize away unnecessary network communication, and giving the programmer a simple, honest abstraction.

(Full disclosure: I’m a contributor on the Rust implementation of Cap'n Proto.)
1. 1
  
  jonb edited 4 years ago | link
  
  Since I want better control over epoll, I’m interesting in packing more than RPC. I originally went down the path of trying to use protobufs, then msgpack, then cap'n proto for a C app I’m building (not C++). I ended up running into tons of issues with stable C library support with all of them. Currently I’m using XDR (using rpcgen as a base) and it seems to be the best so far out of what I’ve tried. Another one that recently caught my eye is MQTT, but I’m not familiar with it yet.
  
  If you’re writing in C++ (or Rust perhaps?), you’ll probably have a different experience.

1

jonb edited 4 years ago | link | on: What are you working on this week?

A set of utilities for basic operations on really large filesystems (the largest we’ve tested it on so far is a 55PB single mount point lustre). 3-clause BSD license.

An older version of the file copy tool that will be included in this suite can be found at: https://github.com/hpc/dcp

Eventually (hopefully an alpha release by mid-February), the full set of tools will be available at: http://fileutils.io/ or http://filecopy.org/ or https://github.com/hpc/fileutils

2

jonb edited 4 years ago | link | on: Distributed Consensus: Beating Impossibility with Probability One

Does anyone know of any papers that extend Ben-Or beyond binary consensus?

I also found this, which seems to be a survey of randomized consensus algorithms:

http://www.cs.yale.edu/homes/aspnes/papers/randomized-consensus-survey.pdf

2

jonb edited 4 years ago | link | on: Parallelism And Concurrency Need Different Tools

I’ve mentioned this elsewhere in the past, but I still feel like there’s a special relationship between the Bakery algorithm and quorum-based consistency algorithms like Paxos – I just can’t put my finger on exactly what it is (something other than Lamport’s basic thought process). My (weak) hypothesis is that there exists a subtle formal connection between parallelism and concurrency that could be proven by linking the two somehow.

In other thoughts, non-determinism can be very beneficial in some cases, especially when it comes to self-stabilization (where it can increase performance by orders of magnitude in real-world systems).

6

kyle 4 years ago | link | on: Parallelism And Concurrency Need Different Tools

Since when has yosefk ever produced anything low-quality?

As might be surmised from my history, I’m a fan of Go (and not so much Haskell), but seeing the different approach was pretty enlightening. Particularly, the idea that concurrency is part of the problem while parallelism is part of the solution caught my attention. Something to think about, I guess.
1. 4
  
  drturtle edited 4 years ago | link
  
  I saw this link on the new page shortly after it was posted, it received (- 2 low-quality) almost immediately. Two of the links that were posted before it had received (- 2 low-quality) within a few minutes of being posted. Those two seemed to be good, on-topic links: ‘parallelism-concurrency-link and 'skyline-github-link.
  
  Before those two, there are three links to blogs posted by (I believe) the owner of those blogs under two different accounts. Once you get past the blog postings, the (- 2 low-quality) pattern starts again. The ‘julia-link immediately before the blogs received (- 2 low-quality). Then two more blog posts were submitted by (I believe) the same person operating two accounts. Moving earlier on the queue, the pattern continues.
  
  The ‘clojure-video-link, the 'parallel-api-link, and the 'Guido-talk-link (all good in my opinion) received the (- 2 low-quality) at about the same time as the earlier ones.
  
  I noticed this because I visited the new page a few times over the past several hours and saw a few well received links that I wanted to follow. I happened to check again around the time that the OP link was posted and noticed the (- 2 low-quality) on this link and all the others I mentioned.
  1. 2
    
    jonb edited 4 years ago | link
    
    A response from jcs on this:
    
    https://lobste.rs/s/fwxp3a/guido_is_giving_a_talk_on_pep_3156_at_linkedin_on_jan_23rd/comments/ohxsva
  2. 2
    
    kyle 4 years ago | link
    
    Yes, I’m fairly certain this is a pattern of malicious behaviour by a pair of accounts; jcs has been made aware and I’m sure he’ll get to the bottom of this.

3

jonb 4 years ago | link | on: Guido is giving a talk on PEP 3156 at Linkedin on Jan 23rd

The creator of Python is giving a talk on one of the most important features of Python related to scalability and it gets marked as “low quality”. I really don’t understand the moderation system here.
1. 7
  
  jcs 4 years ago | link
  
  Some newer users were stupidly downvoting articles on topics they didn’t like, rather than just filtering out those tags. I’ve discussed it with them.
  1. 2
    
    jonb 4 years ago | link
    
    Ah, thank you!

1

jonb edited 4 years ago | link | on: Parallel, distributed, and decentralized file treewalk with ftw(3) style API

To the folks who moderated this as “low quality”, please leave a comment explaining why.

This work was accepted into an IEEE conference (SC12), so saying that it’s “low quality” is confusing. It’s one of the fastest parallel tree walking algorithms currently known to mankind.

http://dl.acm.org/citation.cfm?id=2389114

Perhaps I’m misunderstanding what “low quality” means?

Edit: It looks like it’s just a mistake. A response from jcs on this complaint can be found at:

https://lobste.rs/s/fwxp3a/guido_is_giving_a_talk_on_pep_3156_at_linkedin_on_jan_23rd/comments/ohxsva

1

moses 4 years ago | link | on: New paper draft: “Joining Forces: Toward a Unified Account of LVars and CRDTs"

In the future, please link directly to the paper instead of the blog which links to the paper.
1. 4
  
  moses 4 years ago | link
  
  It would be useful to me to know why this was downvoted for incorrect / troll. I find it useful for the actual paper to be linked to, if the author just writes the abstract on the blog. I am aware that the blogger is the author of the paper, but the blog doesn’t seem to add anything.
  1. 3
    
    jonb 4 years ago | link
    
    I didn’t vote on your comments, but I found the blog to serve the purpose of letting folks know that the paper is a draft.
    
    Also, seeing a huge spike on their Google analytics page (which they probably wouldn’t be able to access on an edu hosted pdf) is probably some nice motivation to continue the work.

1

jonb 4 years ago | link | on: New paper draft: “Joining Forces: Toward a Unified Account of LVars and CRDTs"

There were a few talks at RICON West that might be of interest if you’re not intimately familiar with LVars and CRDTs (or perhaps even if you are):

http://basho.com/ricon-west-videos-distributed-systems-in-academia/

Basho’s Riak is a nice real-world example of CRDT usage outside of academia in general.