It’s Monday, which means it’s time for our weekly “What are you working on” thread! Please share links and tell us about your current project. Do you need feedback, proofreading, collaborators?
In the last week, I did a nice variety of things. While I’m still working on the next blog post about building a command shell, I took a moment to write a short post about Atom Shell and ES6 Harmony as I wanted to be able to use some ES6 features in my application running on Atom Shell.
I’ve been reading up on debuggers for fun, starting with some old work on something called Dalek, which was used data flow to help pick out what was happening from a series of events. It isn’t clear what happened to Dalek as the source was based on GDB and it sounds like they turned sources over to the FSF for integration into GDB back in 1989 or 1990 or so. There was also some interesting work in the Hotwire debugger. There’s plenty of other interesting things that I intend to read about shortly in this space. This includes the Causeway message-oriented distributed debugger.
I also helped housel out with some issues with the debug data being generated by the LLVM backend for Open Dylan. That seems to be progressing well.
I’m pretty excited about my “new” project … work is progressing well on it, and I hope to be able to start talking about it soon as well as talking about how it is going to be funded.
I’ve been making good headway at work where I’m using the heap / memory profiler that I wrote to track down some memory usage issues in one of their applications. Good fun, although it is a bit painful at times to understand all of what is going on.
Finally, I updated to Yosemite on my first gen (2012) retina Mac Book Pro. At first, I was having a lot of performance issues with it and things seemed sluggish. I realized this was because I’d set the resolution to something other than the default to get more screen space. Setting that back to the default made things really nice and smooth. At first, I didn’t really like some of the new UI stuff, but I noticed that at some point today, it just started working for me and it seems nice. I think part of it is resetting the resolution and now things look really crisp and clean. I am a bit jealous of that 5k display on the new iMac though.
Check out IRHydra2 for your v8 JS & c++ project, which was posted here in a presentation slide a few weeks/months ago: http://mrale.ph/irhydra/2/
Yeah, IRHydra2 is pretty cool. I posted the link here: https://lobste.rs/s/oiu0sm/irhydra_2_v8_and_dart_vm_ir_visualization
It won’t help me in this case as I don’t really care (in this case) what is happening to the JS code inside v8. My situation here is that I have a C++ library and I’ve wrapped it with JS bindings (via SWIG) to be able to run it inside of Atom Shell. It is looking like I’m going to have to do some work on SWIG’s JS support though to be able to do what I need.
This week should be pretty interesting.
I’m still playing with my idea for mitigating timing attacks with type systems, but I’m still not exactly how best to go forwards. I think I want to use Nat’s instead of my home-grown Byte type, but the refactoring involved in that got me quite sidetracked. The plus side of said side-tracking is that I got a couple pull requests in to the language itself, which I feel pretty good about, even if they’re minor, but on the other hand I made more progress on the language than on my own code. Also, after talking to @c7five when he stopped by my university Wednesday, I’m working on submitting this as a talk to Thotcon’s cfp (Side note: Thotcon is a super cool event, and if you do security and are in the area, I would highly recommend stopping by.)
College is definitely still a thing I need to figure out as well, and I think I’ve worked out my list of places to apply so far (Carnegie Mellon, Stanford, Minerva, Berkeley, University of Chicago, Northwestern, University of Illinois, NYU Polytechnic), but the amount of stuff that goes into applying to a good university or eight is occasionally somewhat overwhelming, and the whole process has been quite stressful lately and I would appreciate any advice on the subject you guys can give.
Also, this Saturday is when the merit badge day I organized is happening! I’m fairly excited, and I think I finally have a full staff and the facilities I need, but there’s still a ton of things that could go wrong which I am really hoping don’t, and it turns out to be surprisingly hard people to register in a way that’s both easy for them and useful for me. To fix some of that problem for whoever I get to stand in for me organizing this next year, I’m probably starting work on an online registration form and general website for next year once this year’s is done, but for now the first priority is definitely to make the 25th go as smoothly as possible.
I have some advice WRT college apps. I recently went through the grad school application process, which is somewhat similar (though not nearly as stressful). I went through the undergrad application process four years ago and was in a similar situation to you (ie. applying to a lot of different schools).
Lastly, I would be remiss if I didn’t try to plug my Alma Mater, Columbia University. When I was applying to colleges, my top choices were institutions with top engineering programs. However, I ended up going to Columbia because they gave the best financial aid of the ones that accepted me. In the end, I was glad I did. There’s a thriving tech community there, thanks to the Application Development Initiative. Being in NYC offers a distinct advantage when it comes to finding internships and post-grad jobs because there are a lot of big tech companies and smaller startups there. Columbia’s CS program might not be the best, but it’s certainly as good as or better than NYU Poly’s. If you choose your courses wisely you’ll get a very solid background in CS fundamentals and the practice of software engineering (assuming that’s what you want to do after graduation).
Thanks a ton for the advice! All three of those things are definitely things I plan to do. And I will definitely check out Columbia too, so thanks for the pointer there. Also, I was also checking out NYU Poly more for their security program than for CS, mostly because they run CSAW, which I think is a great program.
Hi - since you mentioned Berkeley, I think that means you can apply to other UC schools with basically a checkbox.
If I’m right about that, then I’d recommend you consider checking the box for UC San Diego also. I was a grad student there, but knew enough undergrads to know that it’s full of smart ones, and the CS department has a great faculty and lots of opportunities for undergrads to get involved with cool stuff if they want.
Best of luck!
I did not know that, thanks. I’ve actually seen some great papers out of UC San Diego, but for whatever reason I had always thought of it as more a graduate school, but now that I look at it it looks like an awesome undergraduate school. Thanks!
Glad to help.
This reminded me the CS department there is the kind of place where you can ask the facilities guy if it’s OK to hang a sign that says “Hack like a champion today” in the main stairwell, and he’ll run right out to get the permanent adhesive: https://www.flickr.com/photos/michaelmccracken/456978937/
It seems to have been taken in the intended spirit: http://www.youtube.com/watch?v=H-H-P79vh8M
Stand back everyone, I’m learning SharePoint! I had to fiddle with IIS auth…
This is my weekly update for Fire★. (Since apparently this is what I consistently work on…)
Last week I managed to separate the application API from the GUI into a backend and frontend. I modified over 3K lines of code using a lot of elbow grease (compiler driven development?). I also introduced a mailing list at firstname.lastname@example.org
This week I am going to have the new backend run on a separate thread from the frontend. So misbehaved applications won’t bring down the rest of the app. I am going to do this by providing an implementation of the frontend and backend that is asynchronous and uses messages to communicate between threads.
If I have time I am going to start writing some documentation for the project. This is the biggest piece lacking at the moment. If anyone is good at that sort of thing and would want to contribute, send me a message. Never mind, even if you are not good, you can contribute!
I fugging did it! Apps now run on background threads and don’t block UI. It is glorious.
Nice! Congrats. (I haven’t been following your progress, but it seems as though you accomplished some kind of feat, and speaking as someone who’s update this week would have otherwise just being “Everything sucks”, I suspect some adulation is not out of order).
firestr is pretty awesome. I shall be following your progress.
Last week, I declared Hython’s exception handling mostly done, which is very gratifying! I added support for re-raising exceptions, and punted on Python’s chained exception handling for now (raise from expressions). I’m not sure how to test for unhandled corner cases beyond what I currently have, but if anyone has any ideas, I’m game.
Now, it’s time to spend the same amount of love and care on either modules or inheritance. I’m leaning towards the latter at the moment, but we’ll see what happens this week. Also, I’ve been dragging my feet on writing the next article on Hython, so I should start taking notes on what that should be.
At $WORK I did a lot of work making a self-contained build environment based off of Packer and Vagrant. Both are great technologies, I’m semi-dreading ‘integration’ work I’ll be doing to get this going from Jenkins.
Last week, I based underscore-transducer on transducers.js, then again on transducers-js. As part of that exercise, I discovered that both libraries are working towards a common protocol, so I created transduce to work with either. It’s still very early, and I’ve only tested against transducers-js, but it shouldn’t be a problem to support both. Finally, I wrote Transducers Explained: Part 1.
This week, probably more of the same. I can’t seem to get enough of these things.
I’m not sure how much time I’ll have for it, but I’m currently fiddling with an implementation of LF (like Twelf): hlf. There’s a ton to do here, but my general plan is
I’ve gotten my post on quotient types out the door so I’m writing down some of my notes from a chat I had with Edward Kmett on lens theory. Profunctors over monoidal everywhere!
Are there any good primers on lens theory? I’ve read a good bit on using the the haskell lens library, but I haven’t seen much on lens theory and the construction of lenses and my google-fu tends to fail due to physics being all useful and stuff. (Not that I need more projects, but an idris lens library looks like it would be so cool to work with)
I’m not aware of anything at the moment sadly :( The #haskell-lens channel is supposed to be good so you could check that out.
This past weekend was HackTX, where I pretty much burned out bugstomping on Grimoire and trying to roll the next major version. That didn’t get done and now it’s time to go back to school and the “real world”.
Frankly it feels like I need to take a step back and reevaluate/find some passion again because right now I don’t feel like I’ve got a fun project going on in or out of school and that’s just hard to maintain.
I’m working on fixing gtk+ keyboard input on Windows. I had started on this awhile back, but ran into a couple issues that left me stumped at the time and I didn’t have time to spend on them because I had other bugs and projects to work on, so I kind of abandoned my patch until now. But now I’ve picked it up again and I think I’ve got it mostly figured out, and gtk+ will hopefully soon have much improved support for international keyboards on Windows.
School, school, school! Most of the way out of the woods on mid-terms. Just got my databases exam (loleasy) and a German oral exam (;_;).
Aside from that, working on topic modelling of some twitter data. Generated a plot of topic churn over time and the results look promising ^.^
Due to some family issues, I was unable to even begin my toy shell project last week. I only got a small amount of personal screen time and used that time to play with static linux distros.
I managed to get Moprheus built with relatively recent packages once I figured out the dark art that is musl-cross. I really think that suckless team is doing great work toward GNU-less Linux systems with their coreutils alternatives, sbase and ubase, along with their rewrites of heirloom utilities, hbase.
For work, I’m still (and probably will be for a while) updating loads of Puppet modules and servers to Puppet 3.
I am currently sat in the middle with my netbook - I have CRUX installed on it, but have replaced the initramfs/boot partition with musl’d sbase/ubase/smdev/mksh/cryptsetup. It also let me build a fun crypto setup: I boot to an encrypted partition which only has ubase/sbase/etc on it, whose /sbin/init runs integrity checks on my MBR and /boot, then cryptsetups my real partition and pivots into it.
I don’t want to jump to a full musl system because of package headaches, but the stuff I have moved over so far is working nicely.
That’s a great setup. Is your real / partition running dynamically linked suckless utils or are you just using coreutils? What’s the total size of your musl’d boot partition?
I think that without a solid upstream package build system it is very difficult to maintain a fully statically linked system. Despite that, I still love the idea and will continue playing with Morpheus and Sabotage.
I’ve been looking into Nix to take care of that, but I haven’t had a lot
of time to devote to it.
I’m using coreutils. Tried and tested. And/or I just enjoy having
/boot is 10MB and the pivot partition is 15MB. I could cut a few MB off
the latter by using sbase/ubase-box and deleting the man pages, but I
made the partition 4GB initially so there is no point. (I started with
CRUX on it because I didn’t know how to get to where I am now)
Nothing incredibly interesting unfortunately, but it’s the schlep that you inevitably have to grind through to build great products.
Currently writing a system to do automated imports of data files from S3, FTP, and HTTP-hosted file with validation. I normally spend a day or two every month handling ingestion jobs, and I’m pretty tired of throwing away time I could otherwise be spending more productively. With that in mind, the system has to be smart enough to
The goal is to push every line of these into Kafka as soon as available (per-minute polling) to help reduce the latency between production (other companies) and ingestion. Some of these flows are batch+real-time, but others are just batch or real-time. It’s simpler just to make the whole system streaming.
To make things more interesting, I’ve also added a web component for tracking current status and seeing in-progress imports.
The goal is to be done this week from design to finalization.
Working on setting up an instance of elastic search and so far it’s been going very very smoothly – of course the real challenge will go into seeing how well it can handle a lot of traffic. I’m using it on a table with about +800,000 rows in it that comes out to about 22mb as a file. I’ve tested in on about 1/20 of the data and searches load up onto the client within 110ms-10ms. Yay!
I’ll have to eventually do the same thing in sphinx, because the lead developer I’m working with only want to work with sphinx, and even though I have elasticsearch working already, he’d like to see it in sphinx to compare. So, this should be an interesting assignment…
If anybody has any thoughts on real time indexing, elastic search, and sphinx, please PM me and give me your thoughts!
What kind of data/traffic are you planning on indexing with elasticsearch or sphinx? I’ve used elasticsearch with Logstash before, and never had too many problems (though, I didn’t have an insane amount of logs). I’ve heard that Dreamhost uses logstash/elasticsearch to index about 10,000 log events per second, so I imagine the elasticsearch part scales very well depending on how you set it up!
I’ve heard differently. One of the senior devs that I work with told me he used Elastic Search for a much bigger project where he had it running on multiple nodes (3 or so) and he ran into huge syncing issues – which eventually lead to abandoning the project. The conclusion at my company came down to that elastic search isn’t stable enough for production use. I’m just experimenting with it – hence why I have to eventually reproduce everything with sphinx.
It might work out though since I really only need one node for a relatively small table. We’re using it to index the user names of the members of our site – which comes out to a little over ~800,000 (and on its way up) so that when a user is typing in a input box, it shows a drop down of a search of the members that match whatevers been typed into the input box currently.
I’ve used ES in two previous projects, one small (a few hundred thousand docs) and one larger (single-digit millions). The former was done on a single node and eventually moved to heroku, the latter is more recent. It’s internally focused and only runs 3 nodes, but the response time is great and I haven’t run into too much syncing. That said, it’s not being used as a read/write store in either setup. Just a write-once-read-many setup where the search index is built in one big batch. For the smaller instance, this took about 5 seconds, the larger takes a few minutes, which is good enough. I suspect at very large scales you’d have to do incremental updates, but I'ven’t got there. Overall I’ve never had massive syncing issues running using that strategy on a few nodes, it also makes reasoning about where data lives canonically a lot easier.
Just some food for thought, good luck with your experiments.
@aphyr’s excellent “Call Me Maybe” post on Elasticsearch is worth seeing. It explains some of the data loss behavior that your coworkers might have seen.
However, depending upon the kind of logs you’re searching, it may not matter. If this is your primary data store for the data you care the most about, that might be a bummer. If this is the journal for data where the source of truth is another data store, it might not matter so much. Also, if these are warnings and errors you’re logging from your application, it also might not matter that much.
Thank’s for that. I’ll shall read that during my lunch break.
That was such a heavy blog post… those barbie gifs don’t do much justice to the extent and depth of the content in there. So comcast paid him to play around with elastic search, huh…
Deploying an AD back-end on Azure for M-Files QMS using Powershell scripting and Azure Automation. I think the only time I have ever been this excited for an AD deploy was before I had ever done/knew anything about AD.
@Work: Getting into docker & coaching the team on how to use it for service management for an upcoming project.
@Home: Continuing work on my first Ionic project - impromptu. Got it up and running reasonably well on Android. Now trying to sort out the iOS packaging issue, as I’m a Linux user without any easily accessible Mac. Currently looking into MacInCloud and similar.
I’m gonna put in 2-3 days on my Go compiler. I’m not done with the parser yet (some constructs are pretty tricky to get right), but I’m gonna move a head into semantic analysis. Probably going to implement alpha-renaming and initial symbol table work.
Holy Crap I’ve done nothing recently.
So, this week I’m going back to get some real work done on my Idris to Erlang project, I had a really good meeting with Edwin today, which means I should be able to make some real progress in the next few days. I hope to have a working (if buggy) compiler by next Monday, and then go from there to looking at some concurrency problems.
I also have a lot of other school work, and I should be a lot further through PhD applications than I am right now. At least I got the GRE done last week, so I can work hard on everything else.
At my job I’m working on a Python interface for querying our MongoDB logging infrastructure. The goal is to avoid everybody having to learn MongoDB, as well as hiding the low level details about how things are tied together inside MongoDB. It’s going pretty well and I’ve finally decided on a design and implementation I really like and that works really well.
Outside of work, I got sidetracked from the Text Algorithms book I mentioned a few weeks ago, and have been reading An Introduction to Functional Programming Through Lambda Calculus and The Recursive Universe instead.
As far as coding outside of work, I made a continuous version of Conway’s Game Of Life and some experiments with 3D versions, though the results haven’t turned out very good so far. Not sure why I’ve been so interested in cellular automata lately.