1. 9

This is the weekly thread to discuss what you have done recently and are working on this week.

Please be descriptive and don’t hesitate to champion your accomplishments or ask for help, advice or other guidance.

  1. 7

    Been a while since I posted in one of these threads :)

    I burnt out and stopped working last summer, and now it’s looking like I should probably start again. If anyone needs low latency C++ or graphics programmers, hit me up!

    In the meantime, I’ve been working on a game with a friend. We forked from an existing game (itself a fork of Quake 2) because we thought it would be easier to start from a complete game, but the engine was in a very bad state so most of my work has been spent modernising it and cleaning things up. So far I’ve:

    • Deleted nearly 200k lines of code, or about 60% of the codebase
    • Set up CI for everything, including prebuilt third party libs which I store in the engine repo and statically link
    • Replaced cmake with a far simpler Lua/ninja build system. The build system is in the repo so you can clone and build on any pc with no fuss
    • Added support for easy ASAN builds
    • Reduced compile times by about 80%
    • Converted the codebase from C to C++11. I don’t like most of C++ and I’m certainly not going full C++ best practices, but things like operator overloading/constexpr/safe array count/sane strings/simple templates are nice to have
    • Upgraded the renderer from OpenGL 1 and extensions (for real) to OpenGL 3 with a modern GL loader
    • Downgraded the renderer to GL2.1 plus most of the extensions from GL3 because some DX10 level Intel GPUs claim to not support GL3
    • Added profiling infrastructure based on microprofile
    • Deleted the UI, which was a librocket (buggy slow CSS2/XHTML library) and angelscript (meh scripting language) abomination, and replaced it with imgui. -25k LOC of engine code and removed a gigantic third party lib
    • Rewritten the sound system. 11k LOC to 400 LOC and dropped some deps. New sound system has fancy HRTF directional audio which everyone loves. It had a nasty bug which took a long time to figure out (and two lines to fix): if you remove-swap elements from an array sometimes you need to special case removing the last element, which gets swapped with itself :(
    • Added support for glTF. The old model formats (MD3/IQM) are difficult to work with, especially IQM because absolutely nothing supports it, so glTF support means we can actually get content into the game now. Weapon models are still in MD3 and are hard to convert. MD3 doesn’t have skeletal animation so they are split into a “hand” model, which is empty except for one animated joint and is used by the engine to play firing animations and stuff, the actual weapon model. Some guns also have a “barrel” model for any moving parts, and the joints for those animations are defined by hand in a text file. It’s a big mess and very difficult for us to work with! glTF has lots of annoying things and is difficult to implement but it’s widely supported by tools and the only other realistic option is FBX which sucks more. cgltf is really good.
    • Wrote a new memory allocation system. I did a tracking allocator (basically std::unordered_map< void *, AllocInfo >) and a temporary allocator. The temporary allocator allocates two big blocks of memory at startup, which I use then as memory arenas on odd/even frames. So I can allocate temp stuff and it will automatically get freed at the end of the next frame. Simple to add ASAN support to it too
    • Replaced the animation system. The old one was terribly complicated so I just deleted it, the new one is very simple: Classify model pose for the given frame (running/jumping/etc) -> allocate memory for pose on the frame arena -> sample animation into temp memory -> blend multiple poses together as needed (e.g. walking and also holding a gun) -> apply pose adjustments for looking up/down and leaning -> send pose to the renderer -> pose is automatically freed.
    • Tons of random gameplay tweaks and fixes

    And over the next few months I hope to:

    • Drop MD3 support and use glTF for everything
    • Replace the text renderer. The current one is a lazy atlas generator, so whenever you use a new font size or a new glyph it has to go render it and upload it to the GPU. The implementation is also a huge mess. I’m in the middle of writing a new text renderer based on signed distance fields, so we only need one atlas and can do nice effects like adding borders. It’s been hard because SDF was a meme technology some years ago so most implementations and info you can find online is just complete garbage and msdfgen is really difficult to use. But I got my test renderer working and now I’m working on properly integrating it with the engine
    • Replace the whole renderer. It’s a huge mess and very difficult to work with. I nearly gave up on the whole project like 3 days in when I saw how bad it is. It’s been slowing us down since the beginning, but I’m nearly at the point where I can delete it and start fresh
    • New fancy pants rendering techniques like teammate outlines and decals that don’t need you to clip polys against the world and uploading things to the GPU and drawing them without ridiculous boilerplate
    • New asset system. The current one is very complicated and slow and supports things like streaming (i.e. random hitching) and reference counting, which is all very pointless because the assets folder is 50MB. I want to rewrite it to load everything at startup in < 1 second, and then hotload things when they change on disk to make development easier. I also want to refer to assets by hash rather than by name, so we can use compile time string hashing and simplify some server->client communication (right now the server sends a big list of asset paths it plans to use then sends indices into that array, easier to just send hashes)
    • Modernise the engine’s standard library. typedef vec_t vec4_t[4]; and 500 macros to operate on it is not cool
    • Steam and/or EGS release :)

    It’s a lot of work and everything takes longer than I would like it to, but what I’ve done so far has been the result of however many years of learning and working on engines for fun. It’s been much easier to stay motivated working on an actual game rather than one of my toy engines which will never become a game, and what I’ve done so far has mostly been very high quality. After burning out a year ago I’m relieved and proud to see that I still got it!

    1. 1

      Your summer project sounds like it would burn me out ;) On a more serious note: I’ve never been burnt out but imagine it being pretty shitty. I’m happy you’ve (almost?) recovered.

      Sounds like a cool project. Looking forward to see more from it!

    2. 6

      After a fair bit of work, I just announced the Call for Proposals for EmacsConf 2019 :)

      1. 6

        After more than 3 years, the sled Rust embedded database is preparing for beta :] This week will be spent cleaning up a few API warts, documentation, optimizing some paths that have gotten slower over the course of a few refactors, and intensely burning in on a 96-core monster machine that Packet + ARM have donated time on for testing!

        1. 6

          Going to try rewrite https://github.com/flipxfx/sVim for new Safari so I can use it on macOS Catalina.

          Safari is unusable for me without that extension.

          1. 1

            Not to add any pressure, but I look forward to the result. ;)

          2. 4

            Not-work: Going to work on my regex engine that compiles to JVM bytecodes. I’ve let this project stall a bit, but I think there’s really only a moderate amount of work remaining before it’s useful. I was really excited about this project, but just let myself run out of steam and start working on other things.

            Work: I’ll be working with a consulting architect to offload my knowledge of our system’s very strange persistence layer, so that their team can create a replacement. On the one hand, it’s disappointing: I’ve spent a while figuring out that very strange persistence layer, and had hoped to remove it myself. On the other hand, this way it’ll get done, which probably would’ve never happened otherwise.

            1. 3

              Werk: Haskell bindings for the “new” W3C Webdriver protocol. Using sweeeet generic deriving shenanigans that mean I only have to write the type signature for an endpoint and the compiler does the rest. With >50 endpoints, still a bunch of work to do.

              1. 3

                A program that will parse music [files], and check for voice leading “errors” (as per classical rules of harmony).

                1. 3

                  I’m working my way through Nate Silver’s The Signal and the Noise, which I picked up at a book sale some years ago. It’s sort of fascinating to read a book so concerned with political prediction, published shortly before the 2012 election, today – but I get the impression that the theoretical meat will end up being a combination of material that has, since publication, been more widely disseminated by Tetlock & other numbers-focused public intellectuals mixed with bayesian reasoning.

                  1. 3

                    I’m at SIGGRAPH this week! If any other graphics lobsters (or geo-ducks) want to grab a coffee or something you should message me.

                    1. 3

                      Outside of client work, I’ll be focusing on:

                      • ActivityDesk, a Rust/Qt client for the social Web. Think of KDE’s Choqok or Pidgin.
                      • Limye, a Webmention storage service. A bit of a long-term storage service but also a place where I can expermint with things like Websockets and GraphQL for querying for my Webmentions.
                      • Lwa, a personal take on a (social reader)[https://indieweb.org/reader]. It’ll have a bit of inspiration from Tumblr, a service that I think had the best take on social reading and interactions on the Web.

                      I’m also trying to gain partnership on Twitch at https://twitch.tv/jackyalcine, it’d help out a lot if y’all followed me and watched me work on some Rust code live!

                      1. 3

                        I’m working through the second half of Crafting Interpreters, or at least what’s already done of that. I’m doing it in Rust though, as a learning exercise, and I’m also aiming to build a slightly different language than the book does, but it’s very interesting so far

                        1. 3

                          This week I’m headed to BronyCon. I’m hoping to meet up with a bunch of friends I know from the internet.

                          1. 5

                            The visual styling of that site is awesome.

                          2. 2

                            Work: Typescript annotating and Typedoc documentation. I am beginning to learn I love to give “programmer support” and write documentation as one form of this. I could actually imagine a future of me just doing this as a job of its own.

                            Life: Apex, PUBG, UT2k4…Maybe some programming. Programming in my freetime is beginning to feel “dumb” unless I am testing something very niche and I know there are few if any implementations. This is coming from someone who did freetime programming for hours on end (like many who are reading this now, probably). I much more enjoy learning music more, working out (5/7 days a week), seeing myself improve in general. Cracking open some maths books to refresh my knowledge is great even…but programming, just feels dumb. Usually what I need already exists, or can be forked and modified - which I much prefer.

                            1. 2

                              Trying to finish porting my photo sharing app from UIKit to SwiftUI.

                              1. 2

                                I’m working my way through Nate Silver’s The Signal and the Noise, which I picked up at a book sale some years ago. It’s sort of fascinating to read a book so concerned with political prediction, published shortly before the 2012 election, today – but I get the impression that the theoretical meat will end up being a combination of material that has, since publication, been more widely disseminated by Tetlock & other numbers-focused public intellectuals mixed with bayesian reasoning.

                                1. 2

                                  Work: I’m working on getting our runtime’s IO stack up and running! I got in-memory IO (on strings, etc) working last week and now it’s time to wrap some syscalls.

                                  1. 2

                                    A hashids implementation in Julia, learning some socket programming in C and writing a patch for CoreDNS in Go

                                    1. 2

                                      Starting week 1 of 2 at my new employers offices getting set up with their projects and meeting a few of the people in the company before I go remote again.

                                      1. 1

                                        For $COMPANY:

                                        For $CLIENT:

                                        • Aiming to get a HA setup sendy done
                                        • Finally deploying the final change that drops all local cron based workers in favour of Qless triggered job queues, to the production environment
                                        • Working on the setup for a ‘web/app-server’ role, so that all “layers” (LB, DB/KV, and now WEB/APP) are running on at least two servers each

                                        For $HOME:

                                        • Installing 5 water filters and 120m of PVC piping, plus hopefully more sprinklers.
                                        1. 1

                                          Hopefully finishing our Colorado Trail bike ride. It’s been, by far, the most challenging riding I’ve ever done. We’re taking a half rest day in Lake City today, and I’m writing this from a coffee shop there. Next up is ~35 miles and 6k feet of climbing on rough single track followed by a forest service road descent into Silverton. After that there’s one last push into Durango.

                                          In the evenings I’ve been reading Olaf Stapledon’s “Star Maker”. I’m about half way through, and enjoying it, even though I don’t usually like “space” sci-fi.

                                          1. 1

                                            Lots of work, I’m still getting up to speed at my new job. In the spare time I can allocate to coding for fun I am doing rustlings, thanks to someone here at lobste.rs :)

                                            1. 1

                                              Migrating a bunch of services to Scaleway ARM64 VPSes. Porting Firefox 68 to ppc64, which is going to be great since Skia is still completely un-endian-safe. (Anyone with gfx knowledge want to help? If you used to do stuff for PS3 or XB360, you might even know this stuff already!) If there’s any time this week left after those two things, binging my favourite TV show.

                                              1. 1
                                                • Offsite with my engineering team, where we’re all meeting up face to face for a couple of days. Sharing stuff, seeing each other and generally hanging out in meatspace. (We’re all remote around the UK usually.)
                                                • Visiting head office to meet some new people in the business
                                                • Hopefully lining up some people to do some work in the garden laying patio slabs and generally tidying up some borders. I’d rather pay someone skilled to do a job once than attempt it myself and take 2-3 times the time for a worse end result.
                                                1. 1

                                                  I’m deploying a series of Frankenbuilds to production. It goes against just about everything I have into our development environment and continuous integration system, but one of our software developers was recently head-hunted by ZipRecruiter (they told him to only give 2 days notice) and left me in an untenable situation with an un-reproducible build for our main web site (dependency on an accessory in-memory database outside of the main build process, with a corresponding service running code that doesn’t exist in the repository).

                                                  The only rule in play now is the primary directive (“Git ‘er done”). Now, I need to fix some things (including blocking an APT) but can only do so through hot patches of individual classes and jar files, as long as they don’t break API and bytecode compatibility. Its starting to feel like Apollo 13 mission control over here. Development and Production are now 23 commits removed from each other, and “all is well”, so to speak, even though it is completely outside our build cycle.

                                                  I have a new rule now: Only one person has access to Production.