1. 20

    That was an amazing read and I am in awe of byuu’s skill and dedication.

    1. 35

      Thank you so much ^-^

      1. 12

        Man this community is so cool because authors regularly pop up in the comments about their own software.

        People like you are hard to come by though, you’re awesome.

        1. 8

          Thank you so much for your work, I love your take on it as your life’s work. These games will still be fun to play 100 years from now.

          I remember reading a year ago about some improvements you (or others?) made in upscaling bsnes to HD displays and still need to update my RasPi to pull in a recent version with those changes. My brain hurts every time I see my kid playing Mario Kart with the terrible shimmering!

          https://arstechnica.com/gaming/2019/04/hd-emulation-mod-makes-mode-7-snes-games-look-like-new/

          1. 1

            It doesn’t look like RetroPie has pulled in the latest bsnes. In fact, development of RetroPie seems to have halted a year ago… :-( https://github.com/libretro/bsnes-libretro

            1. 2

              https://github.com/libretro/bsnes-libretro

              That fork is not intended to be current, and even changed its library name to bsnes2014. Upstream bsnes supports libretro by itself now, which is probably the reason that fork is semi-abandoned.

              development of RetroPie seems to have halted a year ago

              RetroPie/EmulationStation last commit was yesterday, RetroPie-Setup two days ago??

              1. 2

                Thanks for the pointers but commits don’t matter if they aren’t shipping to users. I’m referencing https://retropie.org.uk/news/ where the latest release announcement was 9 months ago and the forum also doesn’t seem to have threads from 2020.

                I’m going through the steps to install Lakka instead, which appears to have modern bsnes support baked in.

                1. 1

                  Retropie has an update menu in the UI, so these commits are making it to users who run updates.

      1. 19

        I always get pushback when I encourage people to try their hand at crypto for fun and to learn how it works. Just don’t deploy it into production, make sure you pass the massive test vector lists available for everything, and always assume you are leaking side-channel information, and everything’s fine. Elliptic curves are really fun to work with when you have 512-bit integer types available, and ChaCha20 is very simple in practive.

        Daniel J. Bernstein started out now knowing a thing about cryptography either. Imagine if he’d listened to everyone’s advice.

        1. 5

          “Don’t roll your own” has become one of those memes that people repeat without qualification or context :(

          1. 3

            I mean, sure you could qualify it with “unless it is never actually used” or “unless you are an expert and know how to deal with your mistakes” but the first is sort of implicit in a tree-in-the-forest kind of way and the second applies only to very few people. I think it’s a good meme.

        1. 8

          Emulation is a fascinating rabbit hole insofar as perfection is basically impossible.

          You could spend your entire life on a single machine and never get it perfect. But you could get it to the point of being indistinguishable to a human observer fairly rapidly (say 5-10 years.) So that’s what we do.

          I suspect that in another 10-30 years, we’ll increasingly just be scanning (at least older) system chips in and mapping it all out, Visual 6502-style. If that code is then optimized and refined and boiled down to readily-available FPGA cores and the like, it stands the potential to render things we’ve spent our whole lives reverse engineering and emulating redundant.

          But I’m okay with that. In fact I hope it happens that way. Black box reverse engineering is immensely draining and challenging, and less popular systems often are heavily neglected, while the more popular systems receive so much attention that there’s just endless reinventing of the wheel that is not a useful allocation of rare talent. Preserving history is what matters the most here.

          1. 3

            I have to acknowledge the massive differential in experience to qualify my thoughts here. I am talking more to the audience than to you, who knows and has experienced all I’m about to say.

            Emulation today is largely adversarial. The systems being emulated depend on being opaque. Opacity allows for security through obscurity, which does not solve the DRM Problem but allows for it to be deferred long enough to prevent pirates from releasing same-day cracks and to preserve face. However, the systems being emulated are also sold on a market by a business for a putative profit. Horizontal business alliances are therefore desirable, since horizontal integrations both improve economies of scale and improve the quality of the supply chain. These same alliances also lower the prices of individual parts in the supply chain, commoditizing them and leading to a breakdown of the adversarial condition due to wide availability of the individual parts needed to reconstitute ad-hoc versions of the original system.

            To cut to the point, we are only fighting with console manufacturers and PC game publishers up to the point that they dictate the hardware used to play the game. As hardware becomes more generic and affordable, publishers lose control over the hardware. Reverse-engineering becomes more symbolic, climbing towers of abstraction, giving us simpler and more powerful tools for emulation, cheating, homebrew, TAS, etc.

            Your point about perfection is very sharp. I wonder whether we may find a relaxed notion of correctness which we can use instead. We often do not care about the precise nature of the internal state of an emulation; we only care about its ability to reproduce specific effects. Similarly, we do not care about the structure of its evaluation, only that it is fast enough to execute a certain number of steps per second. Perhaps there may come a day when we have enough symbolic insight to archive games by abstract specification, rather than by executable bytes alone. In that day, we might expect that emulators never have to be written from scratch, but composed from structured modules.

            (It bothers me that I do not have a good ready link to a description of the DRM Problem. It is the obvious one: DRM producers give consumers a key and a lock, and then attempt to make access conditional when the consumer can just(ly) unlock the produced content once and for all.)

            1. 1

              I suspect that in another 10-30 years, we’ll increasingly just be scanning (at least older) system chips in and mapping it all out, Visual 6502-style. If that code is then optimized and refined and boiled down to readily-available FPGA cores and the like, it stands the potential to render things we’ve spent our whole lives reverse engineering and emulating redundant.

              From a practical I-just-want-to-run-this-software-correctly point of view, this forecast looks very plausible and even desirable. There is, however, a sort of side effect of all those reverse engineering efforts that we should preserve: clear documentation for posterity. To quote MAME:

              MAME’s purpose is to preserve decades of software history. […] This is achieved by documenting the hardware and how it functions. The source code to MAME serves as this documentation. The fact that the software is usable serves primarily to validate the accuracy of the documentation (how else can you prove that you have recreated the hardware faithfully?).

              Scan of decapped chips will help us preserve old hardware/software, but not understand it.

              But to rephrase what you said: is understanding every single detail of a weird forgotten IC worth a lifetime of hard studies?

              (In the humanities, where retrospection is common, the answer would be a clear yes. In the hard sciences, where discovery is the main focus, the answers is probably not so clear.)

              1. 2

                There is a case to be made for both understanding the logical behavior of systems alongside adequately preserving the experience of using them.

                From a cultural perspective, emulators are important because they allow us to continue experiencing and sharing software that would otherwise be difficult to experience without legacy hardware (which will inevitably decay at some point). They stand to make our media more accessible and timeless. This was the main focus of the blog post, though you guys do raise some great points about the next facet of emulation:

                From a scientific perspective, preserving the behavior and implementation of these systems is important because it allows us to deeply understand the design and operation of that system, and having accurate documentation of that matters - because again, those systems will decay eventually. And if we want to build a truly 1:1 functional replica, we’re gonna need documentation. We literally do not know how to make Damascus steel, because the methods of creating it were not successfully preserved - we merely know that it existed.

                But I do still believe that these can be achieved separately. They definitely benefit each other mutually, and it is in our best interests to pursue both whenever possible. But ultimately I do wonder if the experience of our media might be just a little bit more important in the grand scheme of human history, as opposed to technical documentation. What do you think is more important - the specifications of the printing press that was used to print and publish Shakespeare’s works, or the content of the works themselves?

                I realize this is deeply subjective and there isn’t really a correct answer. At the end of the day, it still is in our best overall interests to preserve everything as extensively as we can.

            1. 3

              The benefits of switching to CSD for dialogs is rather clear: reduces wasted space and duplicate window titles

              The new titles take 60 pixels of height compared to 20 before. Why was the window title repeated on the form before? That could have been removed instead of making the title bars so much taller.

              I think I have to just give in and start moving windows with alt+click. I always liked the title bar for rearranging windows, but you just can’t get away from people trying to put things into them these days (web browsers especially.)

              1. 3

                This particular XFWM4 theme might interest you ;)

                And, to my knowledge, you can disable CSDs by global environment variables completely, just the opt-in thing might be annoying.

              1. 10

                Well, to begin with, don’t use Google Chrome. And do yourself a better favor and get off all Google products as reasonably possible.

                Secondly, for a legal side, this sure seems like AI-automated libel.It would be nice if, say, the EFF could take a shot at them for libel.

                1. 2

                  Firefox also uses Google’s safe browsing to block sites though, and probably so do a lot of other browsers.

                  1. 2

                    Wikipedia says that the Google Safe Browsing API is used by Safari, Firefox, Vivaldi, and GNOME Web in addition to Chrome. That doesn’t leave a lot of alternative browsers.

                    1. 3

                      I’m not sure, but I believe at least Firefox uses it but is a lot less aggressive with how it deals with different classes of “threats” reported by the GSB API.

                      Firefox will block things that are clearly malware, but it will not block niche unsigned binaries etc.

                    2. 1

                      Libel is definitely a thought that crossed my mind when claiming my site contained “harmful content.” But their lawyers are infinitely more expensive than mine =/

                    1. 8

                      Google’s Safe Browsing technology, which in an effort to combat malware, flags perfectly safe new releases of software as “harmful content” until they have been downloaded a secretive number of times (it is well in excess of 1,000 times from personal experience.)

                      Then Google must know what files people are downloading, even when it’s not downloaded from their servers. It is so strange to me that this is supposed to be normal.

                      1. 15

                        Let me repeat:

                        Most people don’t even know Chrome is reading their entire hard drive, thanks to software_reporter_tool.exe

                        My only question is, do Lobsters not generally know about this ? Because I didn’t until, I did.

                        [0] https://news.ycombinator.com/item?id=19653881

                        [1] https://imgur.com/QtcSXY9

                        1. 10

                          At least some of us assume any statement of the form “google does [evil thing here]” to be overblown to the point of untruth, unless accompanied by accurate evidence. There’s just so much bullshit floating around.

                          I see a bunch of DLL file names in that screenshot. What was the DLL loading path set to, I wonder? IIRC that’s the %PATH% environment variable in Windows, and when I last used Windows, applications had an awful having of adding their internal directories to the system-global %PATH%. I see an assertion that Steam has nothing to do with Chrome, but I don’t see anything about what directories are and aren’t included in %PATH%.

                          1. 2

                            FWIW, since XP SP2 Windows has not used PATH for dll searching unless a certain system setting is changed in the registry for legacy reasons.

                            Even if it was changed, that wouldn’t account for unique to an app dll names, the executables, and the chm. For the dll loader to hit all those would imply the executable from Chrome is attempting to load them all for usage.

                            1. 1

                              Or that something else is attempting to load things from that directory into Chrome (and other processes), and Chrome notices. Does Steam try to do such things, I wonder?

                          2. 1

                            Interesting. That would mean that once Chrome becomes less popular browser, Safe Browsing becomes useless, as it will flag more and more stuff as malware.

                            1. 1

                              AIUI that API is fed by Googlebot more than by Chrome. Chrome does no analysis, its use of the API is read-only. The analysis and assessment are based on data fetched by Googlebot.

                              1. 1

                                My understanding from the message I replied to is that having Google Chrome installed means software_reporter_tool.exe and Chrome would send the information to Google about downloaded files. My interpretation was that once the use of Chrome drops, the number of downloads per file will be quite inaccurate, and if used as a metric of validity, would cause many more legitimate files to be flagged as malware, rendering the whole service useless. If Googlebot does it, then it shouldn’t be a problem, but if Google uses number of downloads as a metric, it can’t come from Googlebot.

                                1. 1

                                  Oh, I see. Sorry for the misunderstanding.

                                  I agree that the download count must really be a count of API lookups. But if Chrome’s usage declines, then the browsers most people are likely to switch to are Firefox and Safari, and they also use the same API.

                            2. 1

                              I’m using Chrome (v. 79.0.3945.88) on Windows 10 Enterprise and I cannot find an instance of “software_reporter_tool.exe” in Resource Monitor (as per the screenshot).

                              Given the intense scrutiny Google is under, especially with regards to privacy concerns, I would imagine this would be more widely reported if it was actually an issue.

                            3. 2

                              Not exactly; IIRC it was something like google knows which domains host malware by looking for themselves, and then your browser sends the hash of the domain to google to figure out if it’s probably on the list before sending the domain.

                              1. 3

                                They know somehow. Their Googlebot crawler is the one locating the ZIP archives, opening them up and scanning them, and seeing an EXE. Instead of running it through a scanner, they just treat it as dangerous right away.

                                They couldn’t tell you that a file was “uncommonly downloaded” unless they’re keeping a counter that increments each time it’s downloaded, which means Safe Browsing sends your download history to Google.

                                My guess is they record a hash, and when users download files, it submits it to some online database to get information about that file. But since Google is also logging where the files were from for Search Console, they know what file you are downloading.

                                My domain has never in its history hosted anything harmful, so it’s not related to that.

                                1. 6

                                  You don’t need to guess, the safe browsing APIs are documented. Strictly speaking, what Google counts isn’t downloads, how the number of times Chrome runs through the lines of code just prior to the first attempt at downloading something.

                                  Google also provides another API to do the same job, which sends less data to Google at the cost of having to download a largish file often. Anyone who uses this API will generally not be counted. I don’t know anything about how much the other API is used; “download large file to end-user device often” sounds like a bad tradeoff to my ears.

                                  1. 1

                                    Wikipedia says the download-a-largish-file API is the API used by Chrome, Firefox and Safari. It supports differential updating, so the update size should be proportional to the number of suspicious URLs added or removed… which might still be awkwardly large, I have no idea.

                              2. 2

                                Google doesn’t know exactly what files people are downloading. As I understand it, browsers download a database of abbreviated hashes of bad URLs; most URLs a browser visits are not in the database, so they won’t be sent to Google at all.

                                If a URL’s abbreviated hash does appear in the database, the browser asks Google for all the full hashes matching the problematic abbreviated hash… and also some randomly-generated abbreviated hashes, so it’s not obvious to Google which abbreviated hash the user actually visited. Once the browser gets the full hashes of problematic URLs, it can compare them with the full hash of the URL it’s loading to find out whether the URL is unsafe.

                                1. 1

                                  Isn’t this from users enabling opt-in telemetry in Chrome?

                                1. 12

                                  Incidentally, archive.codeplex.com (still owned by Microsoft!) has been marked as containing harmful programs by Google Safe Browsing. As in, all of it. This is mildly entertaining to me. If inactive/archived code repositories are now getting flagged, how come code.google.com/archive isn’t?

                                  And finally, I am also providing my binaries on my Discord server in a special #releases channel so that there’s a method of obtaining the binaries outside of web browsers where pages and files can be blocked.

                                  Infosec Twitter has been trying to convince Discord to actually scan executables for malware. I wonder if this won’t end up with Discord going down the code signing route, too.

                                  1. 12

                                    Infosec Twitter has been trying to convince Discord to actually scan executables for malware. I wonder if this won’t end up with Discord going down the code signing route, too.

                                    Article author here: I think that’s a great idea to scan binaries for malware. Google’s Safe Browsing flags binaries as “harmful content” without scanning them at all, solely on the basis that it hasn’t seen those particular binaries ‘much’ before.

                                    If it were to run a Virus Total scan like this on my file before flagging it, it would have seen the file was safe in 70 of 70 different scanners. If it had considered that my domain was 14 years old and never once hosted anything harmful, that would have also been great.

                                    Unfortunately, Safe Browsing is a shoot-first, don’t let alone ever contact you to ask questions approach =/

                                  1. 6

                                    It would be interesting to know if this is an issue for developers of software that does not skirt trademark infringement[1] (which I believe that OP’s software does)

                                    [1] I am not making a moral judgement here, it’s just that this space has a lot of shady downloads associated with it, so it’s natural that Google’s malware detection would find false positives here.

                                    1. 6

                                      Hmm, I can imagine that might be an issue for bsnes, which after all is one letter away from a registered trademark, but I’m pretty sure this post was motivated by the latest release of higan, which (as far as I know) is not any kind of trademark. And even though bsnes is similar to an existing trademark, it’s not a software trademark. If it were called “Adobie Photoshape” or something, then I can see a risk of confusion, but for bsnes…

                                      Besides which, we already have trademark law to handle trademark infringement, do we really need Google inventing their own secret laws that they can enforce?

                                      1. 4

                                        The author seems to be suggesting this is common, but this is the first I’ve heard of it being a problem. I’m not going to say Google’s monopoly isn’t bad in many ways, just not convinced yet that Google is flagging all sites with downloads and this is a widespread problem.

                                        1. 4

                                          Dozens of emulators across dozens of systems use (letter)(system-name) naming convention, and nothing has ever come of it. I believe in good faith that the name is fair use, but if it’s an issue, I’ll be fine with changing the name.

                                          The software flagged was “higan v107”, and to my knowledge that is not a trademarked term for either computer software or any video game systems that higan supports.

                                          I have heard from a few developers now who have had similar issues to me on my Twitter feed and on the orange Y site, so I am confident it’s not related.

                                        1. 1

                                          I did this exact thing (F24 key and all) with a Visual Basic script I wrote at my previous job for the same reason.

                                          Apologies for not being helpful and posting it here, I didn’t keep any files from that time. But mentioning it here because it’s an effective way to suppress the screen saver even if you can’t run downloaded executable files.

                                          1. 44

                                            Google search results have got worse and worse over the years. It used to be that the first result of my search was nearly always what I wanted. Now Google insists on trying to be clever, and often the MOST IMPORTANT keyword in the search isn’t even there at all.

                                            1. 16

                                              A million times this. Google seems far less useful today than it did 10 years ago. Most of the time, I get search results for what Google thinks I’m trying to search for based on popular searches, rather than what I am actually searching for. Basically if your search query is fairly uncommon, Google won’t show you any relevant results, period.

                                              There is a big gaping vacuum in the market for a search engine specifically focused on technical users looking for technical content. Who wants to start a company with me?

                                              1. 6

                                                Isn’t that … almost literally what DuckDuckGo is?

                                                1. 4

                                                  No, DuckDuckGo pulls from Bing, and both tend to change what you searched for to what it thinks you want instead. Even the old trick of +wanted_keyword -unwanted_keyword does not guarantee it will honor your request (they are treated as ‘suggestions’ instead of rules now), but it does help a lot.

                                                2. 3

                                                  I’m sure it started with the demise of: https://www.google.com/bsd

                                                  1. 3

                                                    google.com/linux was literally my first contact with Google. I was attending a local Linux user group (are those still a thing?) in the city I grew up in back in South America, and someone told us we should check that out next time we were looking for Linux resources. I remember the quality and breadth of the results was mind blowing, and I immediately stopped using any other search engines. Never thought I would end up working for them about 15 years later, heh.

                                                3. 3

                                                  Catering to the lowest common denominator rather than to people who actually know how to structure search queries.

                                                1. 3

                                                  The one real concern I have about it is Mozilla moving to default all US users to Cloudflare DNS exclusively. Even if you trust a publicly traded company that you aren’t paying with your privacy, that is a huge amount of power to give them and presently it doesn’t seem like it will be adequately conveyed to users that Firefox will start giving this one external company the entirety of their DNS history in the near future.

                                                  We do not build resilient systems by centralizing. I’d like to see a dialog upon upgrading to select which DoH provider to use, if any, and links to each provider’s privacy policies presented to the user.

                                                  1. 2

                                                    I did not find it hard to set up my own DoH server at home. I already run my own DNS at home, and it was relatively straightforward to write a CGI script to run under Apache to handle DoH requests (since I’m resigned to Firefox using DoH if I want it to or not).

                                                    1. 2

                                                      Realistically, what percentage of users do you think will do that?

                                                      1. 1

                                                        Zero. But what’s stopping someone from making a simple executable (say in fasionable Go) that can run on a home system that accepts a DoH request and uses regular DNS reolution and releasing it so those that can’t, can?

                                                  1. 1

                                                    My old ISP, Wide Open West, performed a similar HTTP-level (not DNS) hijacking to 307 redirect HTTP requests to their messages such as planned outages, etc. A particularly fun “glitch” on their part made it not register that I had acknowledged the notification and so it kept doing it to me on any HTTP site I went to. This and Comcast are what finally drove me to using only strict HTTPS, no exceptions, for my own websites, and to start using the HTTPS everywhere extension. We should not have to force HTTPS as the only option for static content (though it is a fine default), yet here we are.

                                                    1. 5

                                                      Since when do you have to love a tool to use it?

                                                      1. 6

                                                        It’s not a requirement, but it’s nice. I used to love Firefox back in the early Phoenix days. These days, I dread updates to any browser because I know it will come with advertising-related changes I don’t want and/or removal of features that I do.

                                                      1. 10

                                                        Hello, article author here. If you folks have any questions or suggestions, I can respond here.

                                                        I just started byuu.net and this is the first original piece. I want to turn the domain into a resource for emulator developers, and cover the more nuanced, deeper-dive areas that often get omitted from existing ‘how to write an emulator’ tutorials. Things like thread schedulers, priority event queues, chiptune anti-aliasing, and all of the various components that go into simulating another device’s display (color emulation is just the first of many parts to this.)

                                                        If anyone’s interested, I’ll have an RSS feed for new content up in about a week, which will be at byuu.net/feed once it’s live.

                                                        Thanks for reading!