Every week, we like to check in with the Lobsters community and see what people have in store for the upcoming week. These threads are also a great place to encourage getting things done. Links to projects and self promotion are encouraged.
Every week, we like to check in with the Lobsters community and see what people have in store for the upcoming week. These threads are also a great place to encourage getting things done. Links to projects and self promotion are encouraged.
An MQTT-SN gateway in Scala and a MQTT client in Rust
Making a web scraping program which looks at the sites for all the high schools in the Bay Area scanning for physics job postings. I am also hoping to run this off a rasp pi so I don’t have to run my main computer all day. It’s a chance for me to exercise python, for beautiful soup, raspberry pi configuration, some database work and if it goes far enough a convenient GUI for looking at the results. I am new here, so it’s a pleasure to meet you all!
Working on a website to make it easy for people to Try Idris, a new and popular dependently typed language:
Screenshot
I’ve been in the sphere-of-influence of Idris for a while, but haven’t yet been pulled into it’s orbit, I look forward to hearing about your experience with it.
[Comment removed by author]
Releasing first Go based service into production!
Golang high five! Can you tell us a bit more about the service?
Its urgent alert service, sending emails, text and voice calls as fast as possible. Previously we were using ruby.
Last week, everything was working well for my Objective C / Dylan bridge, so this week, I’ve started documenting it will do some pre-release builds to share with people who want to experiment with it. I’m also starting to look at bridging some of the CoreFoundation types, although this will probably happen next week.
We’ve gotten some requests for Qt support, so I have been in touch with with the Qt Jambi team about extending their generator to 1) support multiple languages and 2) start to support Qt5. I’ll be working with them to pull their generator into a separate repository, create a new build system separate from Qt Jambi and then looking at adding multiple backends, starting with one for Dylan. (This is after evaluating other options including SWIG and the PyQt tool, sip.)
I also have contract work to do and I’m on a short holiday in Singapore.
Last week I released my Magic: The Gathering API side project. This weekend I finished adding full text search. I also improved the speed, so most queries should be under 100ms. I’m working on adding card prices next.
This week I’m also planning on rewriting volegtron, my baseball reddit bot, in Go. I just finished the first part, which is a Go package to read MLB gameday information. If you’re a baseball fan, it’s amazing what information MLB has on all past games.
I had already messaged him but for ease the repo is ‘volgeltron’.
Thanks!
The next step on the Go rewrite is to, for a given team, generate a game thread and a post game thread. This shouldn’t be too hard, as I’ll just be using the gameday data and a template to output some markdown.
Working on making the documentation for TokuMX (http://github.com/Tokutek/mongo) better, right now building a set of “getting started” guides for various configurations, and the best ways to migrate from various MongoDB installation types.
Progress: https://github.com/Tokutek/mongo/blob/master/README.md
I’ve been writing a Flask Tutorial building on the Backbone.js TodoMVC example: Flask TodoMVC. Trying to release a new article every week. This week I plan on integrating Flask Security for authentication.
I’ve been getting some file processing code updated to handle some eligibility files. The files that we’re processing have several hundred thousand records and the existing library was taking too long to read the file and persist it to the database. I figured this was a great time to rewrite the library a bit and use Go for its performant and concurrent properties. The file processor was originally written in python and estimated to take between 6-7 weeks to process. The new processor written in Go takes about 2-4 seconds to process the files and somewhere in the neighborhood of 20-30 minutes to persist all the data to the database. Fun times for sure.
In what seems like a neverending saga, more Jenkins configuration, more Ansible, more Devops, and a bit more social engineering in the IT department.
I’ve blown past the honeymoon phase with ansible, I’m starting to feel like the pieces it lacks compared to Chef are bigger than I first assessed. Chef’s ability to let you create what ansible would term a ‘module’ rapidly and easily is pretty killer. That said, I haven’t dug into the details of building Ansible modules, so my judgement will remain reserved.
On the math front, I’m still feeling less than confident, it’s stunning how much information leaks out of your brain over the course of only a few years. That said, I’m scheduling my GRE2’s this week whether I feel prepped or not, and may the FSM have mercy on me when the time comes. I think that – even if I get into some schools, I won’t be able to start for a while (finances are, I am increasingly convinced, the most difficult branch of mathematics), but it’ll be good to know whether folks’ll want me.
This is also the conclusion I’ve been coming to recently, except I only have previous experience of Puppet not Chef. Roles were a great step to reusability but I feel myself wanting to call parts of a role during a playbook rather than the whole thing.
For example take the example of Apache/nginx, you’ll want the core configs and packages in place and then you’ll want per-site configurations to be copied into place. Puppet handled this by having ‘methods’ on a module, which you call when you wanted to install a site, this allowed it to be done from multiple locations; Ansible however requires you to pass in a list to do things multiple times which means knowing all your sites at the time you want to make the call - and this leaves a nasty taste in my mouth as it sometimes ruins separation of concerns. You can include part of a role by simply including the task file, and that you can also pass parameters too - but then you lose all the advantages of using a role such as auto-loading variables & handlers from the correct files.
If what I’ve explained can be done in a nicer way then pray tell..
Exactly, w/ Chef, you define little re-usable resources which I suspect have some parity with Puppet’s “methods”, in particular, for an nginx recipe, I might have the default recipe install a blank nginx, and then have it ship with a few different LWRP (“Lightweight Resource Providers”) which give you new, custom resources for creating ‘sites’, so it might look like:
In particular, it makes each recipe not only function as a way to install some software, but also to give you tools/abstractions for configuring it in other recipes. With Ansible, the closest thing seems to be to write custom modules in some python-based framework. This in-and-of-itself isn’t terrible, but I gather that it essentially separates what I’d consider a single responsibility. That is to say, where Chef and Puppet treat a recipe/module as a sort of object with an external API (via LWRP’s or puppet-methods, resp.), and an initialization (chef’s default recipe, puppet’s equivalent). Ansible seems to treat these ideas as wholly separate. Initializing this ‘object’ is the job of the role, interacting with it is the job of a separate ‘custom module.’ This idea is emphasized by the fact that the role is written as a collection of yaml, and the module is written as python.
This becomes exceptionally frustrating when you want to add some abstraction to otherwise basic shell commands. In Chef, for instance, you can wrap the
jenkins-clitool really quite easily, giving a pleasant API for managing most of a jenkins install without having to remember the exact invocations of the tool. With Ansible, I’m stuck having to either write the shell, or write some generic python which ultimately executes shell.What would be ideal, instead, is to have some way of using ansible to define ansible modules. Most of these tools are essentially little declarative, mostly-functional languages anyway, so it’d be brilliant to be able to treat the low-level ansible tools as combinators like you can in Chef.
I will say this, I really like the
ansibleandansible-playbooktools, I found them quite simple to use compared to knife. I haven’t dug into puppet very much (I probably should), but I have watched a few videos about the “Marionette Collective” tool, which is puppet based, and looks pretty neat. I feel like that would give me at least that part in a perhaps-better setup. But like I said, I have to reserve most of my judgement of Ansible’s ability till after I dig into the module definition APIs.Thanks, it’s interesting to get the perspective on Chef having never used it.
I think my next foray into the world of provisioning tools will be with Salt if anyone has anything to say about that?
I separate those concerns by placing the common config in a
webserverrole, and creating another role for each site (e.g.,foo.comrole). Thewebserverrole makes sure all the necessary base bits are included, and the site role handles the site code and site configuration.Otherwise, even if you have a consolidated role, you can use ansible tags to specify/exclude certain tasks. In this method, you’d tag your common tasks with a
webserver_commontag, and your site specific tasks withfoo.com.That’s not quite the same use case, near as I can tell (though I’ve not used tags very heavily). In particular, it solves half the problem, but (perhaps) the unimportant half.
Indeed, I’ve separated several of my roles into more granular roles (some which might even be called “abstract”, in the sense that they aren’t really configuring a single piece of software, but several to fulfill some purpose). The real thing that I find myself missing is not the ability to do host-specific configuration (though I suspect I will need that in the relatively near future), it’s the extensibility.
When I have a LWRP that ships with a recipe in chef, it gives me an inverted control system for configuring that software later, and extensibly. For instance, I use chef to manage my personal server, which I use for a few different things, including a minecraft world, a mumble server, and some personal webservices I wrote to manage various batch jobs I need to run. I serve these on the same VM, and use nginx to correctly proxy between them. Whenever I need to add a new service, it’s easy to write a recipe which depends on the main nginx recipe to get those LWRP’s, configure the appropriate site-available/enabled files, and otherwise forget about the existence of nginx. Indeed, most of these services run as little Grape or Sinatra apps, so I have a recipe which has a LWRP for setting up the standard configuration of those services in an abstract way, and my individual recipes just use the LWRP it provides and install whatever packages are needed separately.
With ansible (and it sounds like you know, so if you could direct me towards a better nirvana, I’d appreciate it), it seems that the equivalent would be to build an nginx module, with a
siteaction which gets filled in, then to fully mimic the setup, have another ‘ruby_service’ (or otherwise better named) module which relies somehow on the former (I don’t know if there is a way to enforce dependencies between custom modules) to create the abstract service.I should reiterate, I don’t know that the latter methodology is better or worse, only that it is presently unfamiliar; and seems kind of clunky. One of the nice things about Chef (for many of it’s faults) is that it’s easy to extend the language (in a sense) to suit your particular needs. In the case of my little service cluster, I’m generally not looking for efficiency of running – most of the services remain off (indeed, I use knife to toggle them on, and they all default off after some period of time), rather, it was easy to build the system to make it easy to add new services rapidly, and with minimal boilerplate.
I should also say (since this could be construed as me being pretty down on ansible), that I really like a lot of what Ansible does. I mentioned before (I think) that it’s significantly nicer to use than knife for interactive use (e.g., for actually doing things to machines), and certainly a lot more straightforward in terms of features. It’s also nice that the primary language of configuration is exactly the same as the primary language of manipulation (that is,
ansible(the command) responds to all the same actions asansiblethe scripting tool. This compared to knife, which does technically, but only as a sort of afterthought). I guess my ideal tool would be one which measures between the wonderful simplicity ofansiblethe tool, but preserves the ability for DSL creation of chef. I’m mixed as to whether I’d prefer a DSL like chef for the scripting, to the simple YML of ansible. I will say that I like the former for making more complicated LWRPs, but the latter makes me think of ways not to need them, so it’s probably a wash.We have had preliminary talks with a large company (S&P 500 large) about our products. The CIO has invited us to send a video that shows off our products in more detail, and to put us in touch with somebody on his staff who deals with this area. So, my #1 goal for this week is to get this demo video recorded and sent out.
Sadly, it already should have gone out, I’m actually lagging a bit on this. Shame on me… :-(
Extending the support for Augeas in my product so you can set granular validation rules for the different kinds of values configuration files are allowed to contain.
In the previous version you could only whitelist Augeas nodes, i.e. say “I don’t care about differences here, but make sure the rest of the tree matches”
Working on some of my Vim plugins [1], cleaning my side project. Open sourced it two weeks ago, pretty excited about that. [1] http://github.com/jaxbot
Disentangling some hair. I’ve been sitting staring at a big scary Scala project for a few months, and now I’m diving in. I may also need to spend some time with sbt, so this first post here may well be my last. If I die, tell my wife … hello.
[Comment removed by author]
Wrapped up review application for two of my iOS apps and now back to the Node.js and CouchDb trying to move a new service toward MVP.
Last week, I wrote a PIC emulator in Go. Starting this week, I’m going to try and implement a PIC soft processor in Chisel. I expect this to take quite some time. Partially because I don’t know Scala yet and partially because designing a microprocessor takes a lot of time at any rate.
Whoa, Chisel looks really cool! If you need any help with Scala, feel free to ask. :)
I’m putting the finishing touches on a Python interface to DynamoDB that I open sourced recently.
I am nearing the end of a long project to resuscitate one of the first OO systems I wrote (badly). It archives a couple gigabytes of mailing list messages in a highly-readable way. This week I am replacing my homebrewed job queue and distributed id generation systems.
(You should put the date in the subject of these posts.)
Lots of work on storm topology for work.
Added the ability to get at rabbitmq message data to storm-rabbitmq today…
https://github.com/ppat/storm-rabbitmq/commit/8242de9ce85bb87794f336ebdffa3c7cc55f1d46
http://www.manning.com/sallen/
I used the free chapter of Storm Applied from Manning to introduce Storm to the rest of my team at Parse.ly. Nice work on that. I think you and your co-authors give a very solid introduction.
If you need any help, feel free to drop me a note on twitter @SeanTAllen
I’m still fulfilling small change requests to Project/Budgetmanagementsystems based on SAP PPM. And now I even have to fight with Adobe Lifecycle Designer. :(
At home I continue with Learn C The Hard Way and to keep myself hooked I blog about my progress.
Pushing a new documentation portal for my company’s API, which we built using Sphinx and Jinja2. Completing a migration of a large database (1.7TB of datas) to some new AWS boxes – their i2 instances.
On the side, might also do a little evaluation of lemur, a tool for submitting Hadoop jobs to local and Amazon EMR clusters using a Clojure DSL (example) for job specifications.