The weekly thread to discuss what you have done recently and are working on this week.
Be descriptive, and don’t hesitate to ask for help!
The weekly thread to discuss what you have done recently and are working on this week.
Be descriptive, and don’t hesitate to ask for help!
I’m going to be at Strange Loop & PWLConf this week!
I did more work on theft this weekend: I’m making progress on adding multi-core support (what’s faster than generating and running tests on 1 core? doing it on 40 cores in parallel!). I got theft running property tests on its own concurrent planner/scheduler, which found a couple subtle bugs by generating arbitrary event interleavings.
Work: Writing code to compile a huge graph data structure to generated C code. While I can’t say anything about what it represents, I seem to have discovered some pathological cases in clang related to large switch/case bodies. In particular, changing the formatting was the difference between building in 3m45sec and 7h20m. When my main work is done, I want to write a script that generates similarly structured (but IP-free) C code and post a bug report for clang.
What on earth? I thought I knew how compilers worked.
The parser is often the slowest pass - it’s the only one that traverses every single byte of the input.
I’d love to hear more about what happened here.
Parsing isn’t the problem – while it will be clearer when I get around to writing that generator script (which may not be for a bit), the general idea is this:
Slow version: There’s a switch-case statement, which has about 100,000 cases. Each of the case bodies is braced (i.e., has no variables that escape the scope), calls a function with an ID and a bool that sets or clears one or more flag bits on a struct, and a subset of the case bodies end in a goto that jumps to an earlier case body. (The rest end in a break; there are no cycles or fall-throughs.) The generated .c file is about 125 MB. This version took over 7 hours to compile (on a new MBP laptop).
Medium-fast version: There are no gotos, each case body calls a function named “case_X” for each case body that it would previously goto, walking up the chain of earlier case bodies (i.e., there is duplication). There are forward references to every function (static) generated at the beginning of the file, the functions all appear after the huge switch-case function. This version took about 75 minutes to compile, and is about 300 MB. (Parsing isn’t the problem!)
Fast version: Same as the medium fast version, except the switch-case body is broken up into several functions, and there’s a wrapper that says
if (id < 10000) { return switch_fun_lt_10000(id); } else if (id < 20000) { return switch_fun_lt_20000(id); } else ...up to 100,000. (This could be broken up into a binary tree, but cache locality probably makes up for it. I’ll benchmark later.) I shouldn’t need to do that, but this version only takes 3m45sec to compile. Reducing the number of cases in the switch statement helps quite a bit.Something I haven’t tried yet is just indexing into a numeric array of function pointers; it should be functionally identical. I started using switch-case because it’s more declarative, and should have the same result.
There seems to be something super-linear (
> O(n)) in the handling of switch-case (probably some sort of analysis, checking everything against everything else), and it gets noticeably worse as the number of case labels increases past 10,000 or so. Adding gotos between them adds more burden to the analysis. It should be able to rule out interactions between them, though.That’s fascinating. I look forward to the bug thread when they find the cause. :)
Heading to StrangeLoop. Come say hi!
starting to crank up the rigor of my lock-free algorithm testing on sled by making a tool for deterministically exploring various interleavings. right now it’s just a python gdb script that spits out a seed when it finds interesting invariant violations, but I’m exploring the possibility of integrating more sophisticated schedule exploration based on BPOR, and maybe eventually something that works with z3 or something for MCR. It’s fun to be able to have deterministic testing for lock-free algorithms at last! Exhaustive testing is coming soon, but will really only be practical for trivial stuff until we can prune the executed schedules down a bit.
Working on a cli network configuration manager for OpenBSD. So far it can enable/disable/start/stop/restart interfaces as well as create/destroy and switch between locations. Locations are currently limited to just one wap config for wireless interfaces. I’m tidying up the man page later today.
I have some older code that will connect to the strongest wap from a group of configurations. I’ll integrate that next.
It’s not a replacement for ifconfig(8)…it’s written in pure shell (using no commands outside of shell, /bin and /sbin), modeled after rcctl(8). I think I can get automated wap connecting working at boot time. Keeping it in pure shell (or c) is the only way to ensure it will work at boot time.
I’m using it on my my laptop to switch between work and home. I should have a preview ready for general consumption pretty soon.
Are you aware of nsh? http://www.nmedia.net/nsh/
No, I don’t recall hearing anything about it before.
nsh looks nice, but it requires changing your system from default. It replaces the functions of netstart(8) and some parts of rc(8). You have to delete things like /etc/hostname.* and /etc/mygate and give nsh management of some networking daemons.
My project is less ambitious. It’s modeled after rcctl(8) and works with netstart(8). Other than creating symlinks to manage hostname.if(5) configuration files, it’s bog-standard OpenBSD.
Still, very cool stuff.
I like the sound of it. Please post a link on the board when you decide to release it.
Thanks. I’ll definitely post it here, hopefully in a couple of days.
I’ll also post it to http://github.com/akpoff.
Initial version posted on github, and article submitted to lobste.rs.
https://github.com/akpoff/netctl
https://lobste.rs/s/di8j1j/netctl_cli_network_location_manager
Work is the same as last week.
At not work, I’m still mostly focusing on figuring out NixOS. Right now the version if Rust/Cargo is quite old, so I’m trying to learn as much as I can to see if I can help with packaging the latest. I’ve decided that I’m going to stick with NixOS until at least the end of October to give it a fair chance, and not just drop it because it’s unfimilar to me.
Trying to finish up a little utility called linkview that is similar to urlview but parses only HTML documents to display more context by showing link text or image alt/title tags in the menu instead of just the URL. Also, how to suck less at chess.
Profiling a website running Drupal. Attempting to install Prometheus to help with various profiling metrics…
Attempt to finish a sync of 2 sites, site migrations woes.
Try to stay above water.
Good luck. From someone who’s in a team that attempted to use Prometheus recently, do everything The Prometheus Way.
Rewriting my (small) initial attempt at a side-project (the equivalent of gitignore.io, but for Brewfiles) from Go to Ruby because there’s a readily available DSL, and the Homebrew ecosystem is Ruby. Then, revisiting the design, and hoping to launch something on the weekend.
Small geohash library for scala. There are a few options but all are ugly. I want to do something nicely packaged, maybe with some indexing included.
Thinking about security/privacy as it relates to HDD and SSD firmware. They are essentially independent computers complete with processing power and persistence.
Search for transparent and auditable storage devices. I did find the OpenSSD project, which seeks to develop open-source SSD firmware. The Cosmos platform has a controller implemented in FPGA, whereas the Jasmine platform uses the Indilinx Barefoot™ SSD controller (some kind of ARM core). I didn’t find anything for HDDs. Edit: Looks like OpenSSD has a brand new site: http://www.openssd.io/
Given untrusted storage devices, mitigate the damage they can do. The Libreboot project recommends connecting storage devices over USB rather than SATA, to protect against direct memory access from the SATA controller.
Getting ready to open source the product we’ve been working on for 18 months.
I made my first hire today!
New job. Full time W2 for the first time in about 10 years. Learning the systems, planning, and whatnot.
Meeting up with my colleagues from all over Europe and the Americas in Lisbon, Portugal! My first time here, but we already have a full program so I don’t expect to be able to see many sights. Still, really looking forward to meet up in person again. (Very few of us—none in the tech team—share an office normally.)
I’m starting my week by learning Erlang (via LearnYouSomeErlang) and reporting security bugs to RabbitMQ clients. Depending on how the testing goes, I may have 3-5 CVEs come out of it.
Carrying on, I’m planning on implementing a message bus like FedMsg for NixOS events. A single message bus to receive events about PRs being merged, commits, issues, channel statuses, etc.
The last one in particular (channel status) is to replace the nasty collection of scripts powering https://channels.nix.gsc.io and the corresponding IRC bot (written with echo,sleep, cat, sed, and telnet) with a more proper implementation that can also broadcast git events, and is more usable to consumers of this information than the polling I’m currently asking people to do.
Launching my first project since joining my new gig; getting org and JIRA final bits together; finally carving some time out to get into Avro. At home, I’m not going to try living on NixOS any longer – there simply isn’t anything usefully comparable to Lightroom, so it’s back to OS X for me.
Working on Helmspoint, which helps people deploy ML models on the web. Just upload the model and the architecture, and it takes care of provisioning the servers and the API.
Got chain-able background jobs working with map and and_then semantics. That way, I have composable jobs that can be written in isolation from each other. And I can use composable jobs to write processes that have many steps with side effects.
I’ll be working on integration with Kubernetes and figure out whether to put it on AWS or GCloud.
More development on bee2, a little system of mine for running Docker apps with a VPN/jumpbox type set.
https://github.com/sumdog/bee2
Just added a bunch of functionality and am working on a new blog post. I have a feeling this tool is going to be really specific to myself, but I hope the work I’ve done can help other people build something similar for themselves.
I’ve been working on a REPL based graphics library using Common Lisp and OpenGL. In some ways it’s similar to CEPL, but less ambitious.
My goal is to make it a quick and easy way to throw 2D and 3D graphics on the screen without re-implementing a bunch of boilerplate OpenGL and GLFW.
Twiddling my thumbs waiting for a replacement fan for my aging 17” MBP and pondering what to buy as a daily use machine, as this one is now a business risk given its “we won’t even take your money to fix it” status with Apple/authorised repairers.
Work: Mostly worrying about crypto and key management across our microservice deployments.
Non-work: I’ve got a research library app on the go…very slowly, because aside from Cocoa which I don’t want to use for this project, I find that other frameworks don’t supply PDF viewers and I’ve done my own. I’m using Poppler with Qt5 bindings to get page images on the screen, but there’s a lot to be done to make it nice and I have to learn a lot about the Qt APIs on the way.
Also non-work: I’ve been thinking a lot about some, well, disappointing features of modern dev, most recently triggered by the node.js/windows 3.1 article. The problem I have is a cycle of get sad->think about what needs fixing->discover I want to boil the oceans->get overwhelmed->stay sad. To that end, my work on this “project” this week is an org-mode document that will let me both scream into the void, and try to prioritise anything I want to do about it.
Work 1: Getting our custom install ISO finalized with the bizarre preseed control file. Avoiding any interaction is the goal for manufacturing. Get some stats on our various kinds of boxes.
Work 2: Update our web control; see if I can make the move from QWebKit to QWebEngine. Don’t want to use Electron, as I hate hate hate npm.
Non-Work: Learn nim