Today I’m writing perl scripts to parse wikipedia entries on house and senate membership, with the ultimate goal of joining this data to coronavirus fatality rates for age and sex, so that I can simulate how the virus might shift legislative power.
the ultimate goal of joining this data to coronavirus fatality rates for age and sex, so that I can simulate how the virus might shift legislative power.
I’m confused. Are you trying to see how if a congressional district’s constituencies suddenly dies off from the disease how the number of eligible voters might change?
No, just the reps themselves. Doing a deep dive on voting pattern changes based on voters dying would also be interesting, but a little beyond my current scope.
I expect the result will be that power shifts not very much to either side, maybe mode of 2 dead congressmen? But I want to actually run the numbers.
Yeah, perl is our main squeeze. A bunch of us are from Amazon back when Amazon was a perl shop. It’s a terrible language for programming in the large, but for text munging there’s really nothing better, even today.
Picked up Animal Crossing, which is going to occupy a good chunk of time.
I bought the game this weekend, too, and I’m still not sure what the hype is. I enjoy Stardew Valley immensely, and AC just seems really clunky UI wise. I understand this is a slow-burn of a game.
That’s the trick: It’s almost as much of a meditation tool as a game in many ways. I keep catching myself wanting to play more (especially under our current circumstances) only to take a step back, breath, realize that the game time-locked content to the real world so that I wouldn’t rush through it, and put it down for a few hours or for the day.
I also thoroughly enjoyed Stardew Valley but ultimately found the rush to increase my productivity every single day exhausting. Right after the big “end game” event occurred, I dropped it and moved on. I already have to hyper-optimize and perform capitalism enough in my real life.
tl;dr: Stardrew Valley is a farming sim while Animal Crossing is a casual rural life sim.
That clarifies things. Thanks! I’ll try to think of the game, as you said, as a casual, rural life simulator, a game focused on relaxation instead of a productivity.
Forecasting an increase in the number of jobseekers, I’m revisiting a bespoke job board software that’s been pretty neglected and under-monetized to see how I can improve it in a way that makes it more attractive and profitable.
The closing of well, everything, is driving some points that my non-profit Meta Mesh Wireless Communities has tried to make: that people who rely on public computers at libraries and Internet cafes, etc. are at risk in the event that they lose access to those public facilities. We’d accounted for temporary inaccessibility in the form of short-term illness, during which the individual would be likely unhealthy enough to have the desire to conduct computing tasks. We had not accounted for closures related to a pandemic. So, our constituency is very impacted by the closure of non-essential businesses, isolated from electronic resources that the government is pushing in this dark time. We have much work to do to bridge that digital divide and much funding to raise once medical needs are addressed.
For work stuff I’m hooking up a solution I built to ensure all messages make it through our pipeline. Traditional distributed tracing really doesn’t work with our batching requirements and get expensive since we need a real receipt for every single event that passes through. Excited to see it happen because the simplicity of the system is something I’m proud of.
For personal stuff I’m building a lightweight stream database/log abstraction called Remits which stands for “Remote Iterator Server”. The concept is that you can push Messages to an append only log, but can only query them back out via Iterators. An Iterator is a map, filter, or reduce function you define in Lua that iterates over either a Log or another Iterator.
Iterators can be composed and optionally persisted. So the data can be transformed at query time, or persisted and just read from disk. Iterators allow you to start at an arbitrary message, so it can be used as a persisted message queue or similar.
I’m only a few commits in, and there isn’t much to see yet. I’m excited by the power and simplicity of the idea though. Any feedback on the concept or possible use cases to think through would be super appreciated so that it grows in a useful/healthy way.
Ohh, I finally have something to share! I’ve been learning Golang and wrote my first “real” program in it, a small little program that reports back on which browser extensions are installed on a computer via a JSON blob to a other Golang program which sticks it into a Postgres server. I’ve really liked learning Go and have just started on a new program accessing some C libraries.
Golang is great. I first learned C and then transitioned to Go. The difference is night and day. There are many things I need to do with strings in C that is a breeze in Go.
At work, setting up some token-based authentication for a couple of our services. I’ve also been experimenting with “Change harvesting”, which seems to be a lot of working in a very timeboxed manner. The timeboxing definitely takes some getting used to, I was able to do it in the morning yesterday, but it was hard to want to do it after lunch. It is a technique I plan on giving a shot for the rest of the week.
For fun, the Slay the Spire devs just released The Watcher character on the Switch, so that will probably be the next 40 hours of gaming for me. Her stance changing and cards that scale on retain allows you to build some pretty nuts combos, if you can survive the time to set them up. My first run with her had nowhere near enough defense cards, but I make it to the 3rd boss on the second run.
making a web-based multiplayer anagrams game, my friends and i miss getting together to play it live using elixir + phoenix channels for the backend, which i’m learning as i go but which is really nice to use so far.
I’m working on a multiplayer online Monopoly game. I’m trying to model the game as a series of events/transactions, like RollDice and MovePlayer, that can be accepted or rejected by the central game server and then streamed to the other players. Since using only the event log to validate moves is slow, there will probably have to be some kind of game state cache. Is there any prior art for modeling this sort of thing that I can reference? Maybe I am not using the right terms.
I hate to say it like this, but… blockchain? For board games? Ugh.
The technique of keeping a full or partial history is pretty widely used outside of the blockchain world.
For example, it’s often part of Event Sourcing and/or Command-Query Responsibility Segregation. The “game state cache” you describe is is sometimes called a “projection” of the history in event sourcing. However these are complex techniques that are oriented towards scaling systems. You might be able to crib some useful terminology from them but they will add more complexity than value for you.
Another, likely more relevant, place it comes up is CRDTs and related synchronization techniques (such as operational transform). These techniques track history so that they can automatically merge conflicting changes (for some value of “merge” and “automatic”). They can handle peer-to-peer communication, divergent history, and intermittent connectivity — but your game probably doesn’t care about those.
Most likely what you want is an audit log that stores move history. You can track the current board state to validate moves as players send them, append valid ones to the audit log, and have clients stream moves from the log in order (e.g., with server-sent events for a web client). The server can synthesize events that involve randomness, like dice rolls.
These are excellent resources, thank you! I think you’re right that CRDTs are not needed, since the server has to do turn-based synchronization anyway.
Regarding scaling the system: I don’t imagine that a monopoly game will get more than 10,000 moves deep or so, but if each query about the game state (whose turn is next, can this person build a house right now, what is the rent for X property) has to read back the whole audit log it could still be slower than necessary.
Or are you suggesting maintaining the cache as program state and having an audit log as a “backup”, so that state queries are always O(1)?
You stream the event straight to everybody else in the game. Their clients then validate the move, and if in agreement, pass. If not, then your server can do the validation as the final referee. Validation is not expensive if other people do it most of the time.
Integrating and polishing a small C library that solves the “threads and fork(2)” problem into a larger project. I’ve spent a fair amount of time writing unit tests for this library, but in practice there are bugs that you can’t easily notice with a couple of small artificial tests, you know. It should run on any modern POSIX system and macOS, BSDs, Linux all have their own subtleties and slight differences that can’t be easily explained in manpages of already obscure APIs like recvmsg(2) or even (to a lesser extent I think) signal handling. So I’m still fixing a few issues.
Writing about programming. Also programming. As part of one chapter, wrote a fun little app that uses Cecil to start walking the syntax and semantic tree in .NET/CIL. I love little two-day projects like that.
Personal: Continue working on our project VC4ALL to enable people to stay in touch with videocalls using Jitsi in the Netherlands. We have setup one Jitsi server ourselves (thanks to an anonymous sponsor!) and four organizations/people donated their own instance. Also washing hands and staying at home, isolating ourselves as much as we can.
Work: The usual but instead of just me working from home, everyone is here. It’s a challenge, but we’ll manage ;)
Doing research and coming to term with the reality that Covid-19 will kill millions and cause a huge recession. From the many people that I have talked to, they don’t seem to be aware of the likely trajectory in the 1-12 months timeline.
Median time from first symptoms to death is 14 days. For anybody infected, the next 14 days might be their last, and most of it will be unconscious.
After a while rewriting it, finally switched over my IRC bot from the Go version I’ve been running for almost 7 years to a Rust version I’ve been using to learn the language. There’s still a lot to clean up, but it’s passable.
Additionally, I’d like to jump back in and clean up gitdir a bit more if I can find the time after work.
Whenever I’m trying to learn a new language, I generally try and implement an IRC bot. It involves text parsing, network IO, potentially DB access and API calls. A pretty broad set of technologies. Whenever I find a language I enjoy using more than the previous version I start seriously looking at taking a subset of the features in the previous bot and porting them to the version in the new language.
My main complaint in that article was around async not being as easy to use as Go… which is still true (complete reversals don’t often happen in a month) but a large portion of the bot doesn’t need async. I think after a little more learning and experimenting, it became clearer how async fits in and how to use it effectively which made me enjoy it more.
Also, I stopped trying to use external libraries to talk to APIs. I cannot begin to describe how much this helped - removing one library which doesn’t seem to get updates any more and replacing it with a custom solution cleaned up the code a ton and dropped about 100 external dependencies. Because of serde, the hardest part of writing API clients for my use cases has been writing the data types for responses. I’m about half way through planning a blog post around API clients in rust as a result of that work.
Also considering picking up an almost-abandoned project again, that enhances Github with better code reading and review experience using language servers: https://i.imgur.com/S74WawL.gif
I’m wrapping up a 30 minute Rust presentation for my company, to be given during lunch on Friday. Honestly I can’t wait until this is all over with because I can’t stand the attention.
After that, I’ll be working on some UI stuff for the open source game my friend and I have been working on in our free time. We started in 2016 and I’m very much hoping it’ll be ready this year.
Probably Angular stuff for work. I’m growing more and more frustrated with it (especially after doing a commercial side-project in Rust a few weeks ago); I feel like taking a break from web development for a while.
Learning how to use Talon Voice effectively. In the meanwhile this is teaching me a lot about intellij power features (especially navigation shortcuts and live templates)
Today I’m writing perl scripts to parse wikipedia entries on house and senate membership, with the ultimate goal of joining this data to coronavirus fatality rates for age and sex, so that I can simulate how the virus might shift legislative power.
Not sure if helpful, but you may be able to get that data from https://www.wikidata.org in a less brittle way
I’m confused. Are you trying to see how if a congressional district’s constituencies suddenly dies off from the disease how the number of eligible voters might change?
No, just the reps themselves. Doing a deep dive on voting pattern changes based on voters dying would also be interesting, but a little beyond my current scope.
I expect the result will be that power shifts not very much to either side, maybe mode of 2 dead congressmen? But I want to actually run the numbers.
That’s definitely interesting! Will you be posting your results anywhere?
Yup, that’s the plan! It will be a blog post here some time this week: https://www.dolthub.com/blog/
That sounds fascinating. Do you use a lot of Perl at dolthub for data mangling like this?
Yeah, perl is our main squeeze. A bunch of us are from Amazon back when Amazon was a perl shop. It’s a terrible language for programming in the large, but for text munging there’s really nothing better, even today.
Picked up Animal Crossing, which is going to occupy a good chunk of time.
Also finally starting with a new client at work and that will account for much of the rest.
Aspiring to work out more at home since we have the equipment.
Squeezing in some Rust and WebAssembly prototype work between all that.
I bought the game this weekend, too, and I’m still not sure what the hype is. I enjoy Stardew Valley immensely, and AC just seems really clunky UI wise. I understand this is a slow-burn of a game.
That’s the trick: It’s almost as much of a meditation tool as a game in many ways. I keep catching myself wanting to play more (especially under our current circumstances) only to take a step back, breath, realize that the game time-locked content to the real world so that I wouldn’t rush through it, and put it down for a few hours or for the day.
I also thoroughly enjoyed Stardew Valley but ultimately found the rush to increase my productivity every single day exhausting. Right after the big “end game” event occurred, I dropped it and moved on. I already have to hyper-optimize and perform capitalism enough in my real life.
tl;dr: Stardrew Valley is a farming sim while Animal Crossing is a casual rural life sim.
I hope that explanation helps!
That clarifies things. Thanks! I’ll try to think of the game, as you said, as a casual, rural life simulator, a game focused on relaxation instead of a productivity.
Forecasting an increase in the number of jobseekers, I’m revisiting a bespoke job board software that’s been pretty neglected and under-monetized to see how I can improve it in a way that makes it more attractive and profitable.
The closing of well, everything, is driving some points that my non-profit Meta Mesh Wireless Communities has tried to make: that people who rely on public computers at libraries and Internet cafes, etc. are at risk in the event that they lose access to those public facilities. We’d accounted for temporary inaccessibility in the form of short-term illness, during which the individual would be likely unhealthy enough to have the desire to conduct computing tasks. We had not accounted for closures related to a pandemic. So, our constituency is very impacted by the closure of non-essential businesses, isolated from electronic resources that the government is pushing in this dark time. We have much work to do to bridge that digital divide and much funding to raise once medical needs are addressed.
For work stuff I’m hooking up a solution I built to ensure all messages make it through our pipeline. Traditional distributed tracing really doesn’t work with our batching requirements and get expensive since we need a real receipt for every single event that passes through. Excited to see it happen because the simplicity of the system is something I’m proud of.
For personal stuff I’m building a lightweight stream database/log abstraction called Remits which stands for “Remote Iterator Server”. The concept is that you can push Messages to an append only log, but can only query them back out via Iterators. An Iterator is a map, filter, or reduce function you define in Lua that iterates over either a Log or another Iterator.
Iterators can be composed and optionally persisted. So the data can be transformed at query time, or persisted and just read from disk. Iterators allow you to start at an arbitrary message, so it can be used as a persisted message queue or similar.
I’m only a few commits in, and there isn’t much to see yet. I’m excited by the power and simplicity of the idea though. Any feedback on the concept or possible use cases to think through would be super appreciated so that it grows in a useful/healthy way.
Ohh, I finally have something to share! I’ve been learning Golang and wrote my first “real” program in it, a small little program that reports back on which browser extensions are installed on a computer via a JSON blob to a other Golang program which sticks it into a Postgres server. I’ve really liked learning Go and have just started on a new program accessing some C libraries.
Golang is great. I first learned C and then transitioned to Go. The difference is night and day. There are many things I need to do with strings in C that is a breeze in Go.
Curious, how are you retrieving the list of extensions, and from what browsers? Do you have the code on Github?
First I grab a list of the user’s profiles from:
HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList
From there, I check for extensions in Chrome, Edge and Firefox by examining their %APPDATA% folders.
I don’t have any public code for it, but if I do put some I’m I’ll let you know.
At work, setting up some token-based authentication for a couple of our services. I’ve also been experimenting with “Change harvesting”, which seems to be a lot of working in a very timeboxed manner. The timeboxing definitely takes some getting used to, I was able to do it in the morning yesterday, but it was hard to want to do it after lunch. It is a technique I plan on giving a shot for the rest of the week.
For fun, the Slay the Spire devs just released The Watcher character on the Switch, so that will probably be the next 40 hours of gaming for me. Her stance changing and cards that scale on retain allows you to build some pretty nuts combos, if you can survive the time to set them up. My first run with her had nowhere near enough defense cards, but I make it to the 3rd boss on the second run.
NZ is just about to go into Covid 19 lockdown….
Sigh. Embedded developers have tonnes of shit and cables to move and set up to work from home.
making a web-based multiplayer anagrams game, my friends and i miss getting together to play it live using elixir + phoenix channels for the backend, which i’m learning as i go but which is really nice to use so far.
Will you be publishing the source? I know some people who might enjoy that too.
yes, definitely
I’m working on a multiplayer online Monopoly game. I’m trying to model the game as a series of events/transactions, like
RollDiceandMovePlayer, that can be accepted or rejected by the central game server and then streamed to the other players. Since using only the event log to validate moves is slow, there will probably have to be some kind of game state cache. Is there any prior art for modeling this sort of thing that I can reference? Maybe I am not using the right terms.I hate to say it like this, but… blockchain? For board games? Ugh.
The technique of keeping a full or partial history is pretty widely used outside of the blockchain world.
For example, it’s often part of Event Sourcing and/or Command-Query Responsibility Segregation. The “game state cache” you describe is is sometimes called a “projection” of the history in event sourcing. However these are complex techniques that are oriented towards scaling systems. You might be able to crib some useful terminology from them but they will add more complexity than value for you.
Another, likely more relevant, place it comes up is CRDTs and related synchronization techniques (such as operational transform). These techniques track history so that they can automatically merge conflicting changes (for some value of “merge” and “automatic”). They can handle peer-to-peer communication, divergent history, and intermittent connectivity — but your game probably doesn’t care about those.
Most likely what you want is an audit log that stores move history. You can track the current board state to validate moves as players send them, append valid ones to the audit log, and have clients stream moves from the log in order (e.g., with server-sent events for a web client). The server can synthesize events that involve randomness, like dice rolls.
These are excellent resources, thank you! I think you’re right that CRDTs are not needed, since the server has to do turn-based synchronization anyway.
Regarding scaling the system: I don’t imagine that a monopoly game will get more than 10,000 moves deep or so, but if each query about the game state (whose turn is next, can this person build a house right now, what is the rent for X property) has to read back the whole audit log it could still be slower than necessary.
Or are you suggesting maintaining the cache as program state and having an audit log as a “backup”, so that state queries are always O(1)?
The latter, definitely! Late-game Monopoly is slow enough on its own, no need to make the implementation slow.
You stream the event straight to everybody else in the game. Their clients then validate the move, and if in agreement, pass. If not, then your server can do the validation as the final referee. Validation is not expensive if other people do it most of the time.
I think having consensus is definitely overkill and perhaps not what we want here. The server can just be rule-keeper.
Integrating and polishing a small C library that solves the “threads and fork(2)” problem into a larger project. I’ve spent a fair amount of time writing unit tests for this library, but in practice there are bugs that you can’t easily notice with a couple of small artificial tests, you know. It should run on any modern POSIX system and macOS, BSDs, Linux all have their own subtleties and slight differences that can’t be easily explained in manpages of already obscure APIs like recvmsg(2) or even (to a lesser extent I think) signal handling. So I’m still fixing a few issues.
Working on a rust bot for this very fun challenge. Currently in the top 30, but competition is fierce!
I’m working on becoming a better writer. With all of this free time due to Covid-19 I can write more often, and spend longer reading other articles.
Writing about programming. Also programming. As part of one chapter, wrote a fun little app that uses Cecil to start walking the syntax and semantic tree in .NET/CIL. I love little two-day projects like that.
Personal: Continue working on our project VC4ALL to enable people to stay in touch with videocalls using Jitsi in the Netherlands. We have setup one Jitsi server ourselves (thanks to an anonymous sponsor!) and four organizations/people donated their own instance. Also washing hands and staying at home, isolating ourselves as much as we can.
Work: The usual but instead of just me working from home, everyone is here. It’s a challenge, but we’ll manage ;)
Doing research and coming to term with the reality that Covid-19 will kill millions and cause a huge recession. From the many people that I have talked to, they don’t seem to be aware of the likely trajectory in the 1-12 months timeline.
Median time from first symptoms to death is 14 days. For anybody infected, the next 14 days might be their last, and most of it will be unconscious.
After a while rewriting it, finally switched over my IRC bot from the Go version I’ve been running for almost 7 years to a Rust version I’ve been using to learn the language. There’s still a lot to clean up, but it’s passable.
Additionally, I’d like to jump back in and clean up gitdir a bit more if I can find the time after work.
Why switch to Rust?
Whenever I’m trying to learn a new language, I generally try and implement an IRC bot. It involves text parsing, network IO, potentially DB access and API calls. A pretty broad set of technologies. Whenever I find a language I enjoy using more than the previous version I start seriously looking at taking a subset of the features in the previous bot and porting them to the version in the new language.
I had a post a few weeks ago that sort of outlines some of the strengths of Rust over Go (and vice versa) at https://coded.io/2020/02/early-impressions-of-rust-from-a-go-programmer/.
My main complaint in that article was around async not being as easy to use as Go… which is still true (complete reversals don’t often happen in a month) but a large portion of the bot doesn’t need async. I think after a little more learning and experimenting, it became clearer how async fits in and how to use it effectively which made me enjoy it more.
Also, I stopped trying to use external libraries to talk to APIs. I cannot begin to describe how much this helped - removing one library which doesn’t seem to get updates any more and replacing it with a custom solution cleaned up the code a ton and dropped about 100 external dependencies. Because of serde, the hardest part of writing API clients for my use cases has been writing the data types for responses. I’m about half way through planning a blog post around API clients in rust as a result of that work.
Started playing around with Revery and libgit2, building something that will maybe become a native git client that’s sort of Magit-like: https://gfycat.com/measlynecessarybasenji
Also considering picking up an almost-abandoned project again, that enhances Github with better code reading and review experience using language servers: https://i.imgur.com/S74WawL.gif
Starting to read the HTTP 1.1 spec and try to port Hyper h11 (I/O free implementation) in Zig.
Hope the spec will be less complex than the one for HTTP2 :P
I’m working on a livecoding environment for building face filters (in a way that’s not owned by facebook)
feeling very very depressed and anxious, coding is a nice escape.
I’m wrapping up a 30 minute Rust presentation for my company, to be given during lunch on Friday. Honestly I can’t wait until this is all over with because I can’t stand the attention.
After that, I’ll be working on some UI stuff for the open source game my friend and I have been working on in our free time. We started in 2016 and I’m very much hoping it’ll be ready this year.
Probably Angular stuff for work. I’m growing more and more frustrated with it (especially after doing a commercial side-project in Rust a few weeks ago); I feel like taking a break from web development for a while.
Learning how to use Talon Voice effectively. In the meanwhile this is teaching me a lot about intellij power features (especially navigation shortcuts and live templates)