This is the weekly thread to discuss what you have done recently and are working on this week.
Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.
This is the weekly thread to discuss what you have done recently and are working on this week.
Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.
Writing more of my current book, OOP the easy way. Handily, I’ve inserted an ordering constraint between it and another project of the form “I will not buy the kit I need to progress that project until it has been paid for by the book”, which means I get to focus on the book and not flit between lots of non-progressing things.
Some more mapping out data paths for the legacy^Wheritage app I’ve been doing that on at work: having completely scoped out a port of the customer-facing side, I need to build an understanding of what happens on the back office side and decide whether we want to keep any of it or replace parts with workflows in our CRM.
Teaching someone a dance that they’ll be performing in a competition in about three weeks. Choir practice. Going to a music session to meet members of a band I may be asked to join.
Continuing learning Go and rewriting my IRCd, originally a self-contained C project. The sucky part, which is rewriting the event loop on top of a different model and ensuring the entire thing including client connections shuts down correctly, seems to be almost over.
I might or might not be surprised as to how well it translates, e.g. TLS autodetection and abstracting over the two possible transports was not an issue. On the other hand, Golang cannot interrupt libc functions like getaddrinfo() or getnameinfo() by pthread_cancel()ling a dedicated worker thread, which is what I did in my C library. Should I bother suggesting it at golang-nuts@? I’m not sure if core devs would have a positive attitude towards the idea.
Provisioning my new apartment.
Working on a compiled implementation of Javascript - https://github.com/timruffles/js-to-c. Started as an answer to SICP exercise 5.52 (https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book-Z-H-35.html#%_thm_5.52)
This is cool! I’ve been thinking of doing something like this for my Python compiler:
http://www.oilshell.org/blog/2018/03/04.html
That is, instead of compiling to bytecode, compile to C.
Have you tackled garbage collection yet? I suppose it is no harder to do in C code than interpreted bytecode, but I haven’t really gotten into the details.
Thanks! Will have to take a look at your Python work.
Re GC I’ve sketched out a plan to do something similar to the SICP stop and copy GC:
Actually I have a more immediate question besides GC: Are you handling JavaScript exceptions, and if so, how?
I was thinking of compiling down a control flow graph and then the generated C code would just have “goto” / jumps instead of while/for/if/etc.
As an alternative, maybe there is some way to hack exceptions with
setlongjmp()?About GC, I was researching the stop-and-copy strategy, as well as mark-and-sweep. It seems like almost every language does mark-and-sweep, but the “compaction” property of stop-and-copy appealed to me. I think I saw that the
esshell used stop-and-copy (although the project iself is basically defunct).One thing I wasn’t sure about was moving pointers. If you want to have a C API, then C code might hold pointers to JS/Python objects, and then you don’t want to move those objects. I guess this can be solved by another layer of indirection (e.g. immovable stubs).
But I haven’t gotten very far with this – just brainstorming!
Spookily, yes, it’s the other big topic on my mind! I’ve already written up my thoughts on
setjmpor the more explicit goto strategy you identified. I’m going forsetjmpbecause it felt like everything you can do in JS can throw, so the explicit route would be lots of code.Re the C API, yes that would certainly make stop’n’copy awkward!
Isn’t yours a scripting language for quick-and-dirty commands like BASH? If so, can’t you just do reference counting or something? I don’t see them running long enough for things like compaction to matter. I mean, maybe you’ll do it for fun or to feed into a future project. I just think simpler stuff might be a better default if it’s short-running scripts.
Actually shell does not require any GC or ref counting whatsoever, because strings are just copied everywhere! Same with awk. I didn’t realize this before implementing a shell!
But this is for OPy – the subset of Python that Oil is implemented in. The whole parser is written in Python, and I want to compile to either C or my own bytecode, and in either case you might reasonably want GC.
One reason is that parsers creates lots of tiny objects. Although, it might be reasonable to just hack some special case “arena” support into the VM, and forget about GC, which is a big implementation effort.
I think that the “ephemeral process” model of shell makes GC and memory management somewhat unnecessary. You can just allocate and throw the whole process away. But for the shell itself, there are two places where memory management matters:
If those 2 use cases can be handled with manually allocated arenas and not GC, maybe I don’t need GC. But the problem with arenas is that I want to detect invalid references if you make a “free” error. I’m not sure how to do that, at least without slowing down common cases.
“Actually shell does not require any GC or ref counting whatsoever, because strings are just copied everywhere! “
Seems like that offers opportunity for efficiency improvement in a system where they’d pass pointers to strings instead of copy them. Probably what the capability-based systems did or could’ve done.
“ Although, it might be reasonable to just hack some special case “arena” support into the VM”
That’s what I was thinking when you said “ephemeral.” In networking and games, I’ve seen people use memory pools that let you fast allocate the first N values, let you get more in chunks if you need, and release it all at once when done. Really easy to implement. For determining size, you might write a little profile that runs a lot of bash scripts through it measuring typical and worst-case uses. I know you already have a bunch for compatibility testing. Then, just pick starting and allocation size that fits most of that well.
What you think?
Yes that’s definitely in line with how I think about things. If you have very good test coverage it opens up a lot of possibilities for feedback-based / data-driven optimization.
I need to get the compiler/VM working first, which will inevitably be without memory reclamation, since that is mostly a separate concern. And then I will see how I can fit in arenas or GC.
It’s a little complicated because I’m not sure if the language will have the concept of stack and heap. Python really doesn’t have that – logically everything is on the heap and garbage collected.
So it’s not clear you want to allocate stack variables on the arena meant for the syntax tree.
This sort of relates to the “generational hypothesis” as far as I understand. Most loca” variables are expected to be in the young generations, but this is not a hard-and-fast guarantee. It’s just a heuristic.
But if OVM doesn’t have GC then there might be “garbage” from local vars?. I think that OPy will be more value-based than Python, but it is constrained by the fact that it is valid Python!
In other words, the first cut might be really rough and waste a lot of memory :-/
Trying to learn and hopefully improve PyD, a tool to interoperate both ways between D and Python. It seems to work but it’s fiddly and the docs are lacking. This is my first experience interfacing with the CPython implementation, so perhaps all of these
PyObjectstructs look familiar to anyone else who has tried to help Python go native.I’m building a gym inside of my apartment and started working on a little CLI app to track my lifts.
https://i.imgur.com/9cm8O9l.gif
Going to get the stats overview scene done tonight and hopefully also some method of switching scenes.
All the parts are here; it’s finally time to set up the Talos II server!
Since POWER9 is so new, there are only a few OSes to choose from: Debian/Ubuntu, RedHat/Fedora/CentOS, and maybe a couple others. No BSDs yet to my knowledge.
First priorities for the server are setting up a ZFS array, and getting KVM working with guests.
Need to make stabilisation of a few checkers for https://github.com/go-critic/go-critic
Have an idea for hyped blogpost(Go+Wasm), want to make a service not a tutorial at the end.
Hopefully will continue working on my personal SaaS :(
I’m still working on Dinit, my init system / service manager. I’m trying to keep up momentum and get to the point where I’d be happy releasing a “1.0” version, but there’s quite a bit to do yet - and then there’s the problem of filling in the functionality gaps compared to eg Systemd (which I plan mainly to do with external programs rather than pushing more and more functionality into Dinit itself). I added a bunch of tests recently and even did some fuzzing of the control protocol, which I was surprised to find did not reveal any bugs (this was almost disappointing - it was the first time I’d tried fuzz-testing anything, I was hoping for a useful result! But it’s hard to be unhappy about it).
Currently I’m trying to add some distribution-friendly functionality: the ability to enable or disable services simply via the command line (prior to now it requires hand-editing the service descriptions), so that’s what I’ll be working on this week.
I was at a wedding last week, so this week is going to be a grindstone. I’m pairing with my technical reviewer to grind through a few more chapters of Practical TLA+ and then immediately flipping into more work on the 3 day corporate workshop. If anybody wants to prod their bosses about hosting a workshop feel free :p
Almost done setting up the business accounting, too. I’ve decided I’ll celebrate by making my first purchase (after bills) a subscription to the ACM digital library. I couldn’t justify it as a personal purchase but it’ll be really helpful for research so YOLO.
I nearly got the F# library XPlot converted to multitargeted net45/netstandard2.0, when I ran into the darndest problem. It builds fine in Visual Studio, but the build script fails in the build target on not finding the Newtonsoft.Json dependency.
I’m sure I’ll figure it out this week. Here’s the branch, if anyone is interested. https://github.com/jackfoxy/XPlot/tree/magicmode
Working on a couple of things for Merit.
Source
I’m working on my book, Practical Elm for a Busy Developer.
Maybe https://korban.net/elm/book ? :)
Yes, thanks! Clearly time for a break.
Lots of things, because I’m terrible at concentrating on one thing, but in the next couple days I will probably continue working on audittool (I’m thinking of changing the name to depaudit except that dep-audit already exists…).
audittool is basically a way to manage audits of (parts of) your npm dependency tree. So I can say e.g. I want to audit modules a, b and c and everything that they load for stupid behavior, because those modules are loaded as root before I drop privileges so they’re extra important. Or, I want to audit everything included by
index.jsto make sure that they have decent test suites so I know I can rely on the code. audittool will compute a dependency graph and then help you track exactly what you’ve checked for each module version, so when you upgrade (indirect) dependencies you can say, “help me audit only the new parts of this dependency tree for xyz property,” and the tool will know exactly what work you’ve already done and what still needs to be done.Side note, I sat down to write this last night and I was like “this actually doesn’t seem too bad; I bet I could bang out an MVP in a few hours.” Ended up sinking 4 hours into this bug, ragequitting^W giving up and going to bed, then spending 2 more hours on it today before filing that bug report. I’m pretty proud I finally found it, actually.
I’m settling into my apartment, and also working away at building out another erlang web site based on this idea, though I’m currently not sure what to name the thing.
At work, currently on sprint, which has gone well this time around.
Trying to get up to speed with Kubernetes and the whole ecosystem around it. Got a small cluster running with Kubespray, Rook for storage management and an Nginx Ingress controller. Certificate management, Network Policies, RBAC, Monitoring and so on still need a lot of work and learning on my side. I hope to get a MySQL Galera cluster and Wordpress to run with OpenID connect via Keycloak by the end of the week.
Besides that I should start to apply for a new job :)
Redhat has a podcast series called PodCTL if you are interested.
Its largely produced by the team behind OpenShift & there has been somewhat of a lull in production & production quality in recent months.
- Work on Congressional outreach tool:
- eBooks as private podcast RSS feed:
WorkI’ve gotten most of the hard part out of the way. APIs to the Census and Propublica API are done. At this point I just need to finish the front end, and stop bikeshedding on the design.
This is mainly meant as an application to learn go. Very early stages.
Back from three weeks on vacation in Scotland, so I guess probably spending the early part of the week figuring out what’s going on and picking up some threads of work again. The vacation was nice.
The builders finished renovating our house yesterday. So now I have 36 sq.m (~400 sq.ft) of new lounge/dining room to put lights, power, furniture in. And the builders are back in ~1 week to add the carport extension, which will mean more lights and power…. so plenty of DIY stuff going on.
With the builders no longer running hammer-drills at 7am I can also get in more full work “days” (which aren’t really during the day) with
$CLIENT1so, that’ll mean progressing with the Bamboo Models migration mostly.Writing a bunch of Terraform modules to provision a secure Nomad + Consul cluster on Hetzner cloud. Turns out that Terraform is fairly limited and cumbersome in some places. Really cool idea, though!
Finally writing a proper Ansible deployment chain for Rails and Clojure apps
Work: Last week of work, so lots of different things and (trying to) finish what I started. Hopefully finishing some provisioning scripts written from scratch in Python 3.6. it’s the first time I’m really using the language and it’s a bit strange. I’m still not sure if I like it or not.
Home: We bought a new car today and I’ve been looking at canbus hacking to enable features. So far I found I will have to create a custom adaptor for it, but I’ll be able to enable Android auto for free. That feature wasn’t even an option we could choose so I have zero issues enabling it. It seems there are a lot of hackers using Renault cars so I hope to be able to enable more goodies. It will take more than a month for it to arrive so I have a lot of time to research.