1. 22
  1. 15

    It’s the 30 year anniversary of CHI’88 (May 15–19, 1988), where Jack Callahan, Ben Shneiderman, Mark Weiser and I (Don Hopkins) presented our paper “An Empirical Comparison of Pie vs. Linear Menus”. We found pie menus to be about 15% faster and with a significantly lower error rate than linear menus!

    So I’ve written up a 30 year retrospective:

    This article will discuss the history of what’s happened with pie menus over the last 30 years (and more), present both good and bad examples, including ideas half baked, experiments performed, problems discovered, solutions attempted, alternatives explored, progress made, software freed, products shipped, as well as setbacks and impediments to their widespread adoption.

    Here is the main article, and some other related articles:

    Pie Menus: A 30 Year Retrospective. By Don Hopkins, Ground Up Software, May 15, 2018. Take a Look and Feel Free!


    This is the paper we presented 30 years ago at CHI’88:

    An Empirical Comparison of Pie vs. Linear Menus. Jack Callahan, Don Hopkins, Mark Weiser () and Ben Shneiderman. Computer Science Department University of Maryland College Park, Maryland 20742 () Computer Science Laboratory, Xerox PARC, Palo Alto, Calif. 94303. Presented at ACM CHI’88 Conference, Washington DC, 1988.


    Open Sourcing SimCity. Excerpt from page 289–293 of “Play Design”, a dissertation submitted in partial satisfaction of the requirements for the degree of Doctor in Philosophy in Computer Science by Chaim Gingold.


    Recommendation Letter for Krystian Samp’s Thesis: The Design and Evaluation of Graphical Radial Menus. I am writing this letter to enthusiastically recommend that you consider Krystian Samp’s thesis, “The Design and Evaluation of Graphical Radial Menus”, for the ACM Doctoral Dissertation Award.


    Constructionist Educational Open Source SimCity. Illustrated and edited transcript of the YouTube video playlist: HAR 2009: Lightning talks Friday. Videos of the talk at the end.


    How to Choose with Pie Menus — March 1988.


    BAYCHI October Meeting Report: Natural Selection: The Evolution of Pie Menus, October 13, 1998.


    The Sims Pie Menus. The Sims, Pie Menus, Edith Editing, and SimAntics Visual Programming Demo.


    The Design and Implementation of Pie Menus. They’re Fast, Easy, and Self-Revealing. Originally published in Dr. Dobb’s Journal, Dec. 1991.


    Gesture Space.


    Empowered Pie Menu Performance at CHI’90, and Other Weird Stuff. A live performance of pie menus, the PSIBER Space Deck and the Pseudo Scientific Visualizer at the CHI’90 Empowered show. And other weird stuff inspired by Craig Hubley’s sound advice and vision that it’s possible to empower every user to play around and be an artist with their computer.


    OLPC Sugar Pie Menu Discussion Excerpts from the discussion on the OLPC Sugar developer discussion list about pie menus for PyGTK and OLPC Sugar.


    Designing to Facilitate Browsing: A Look Back at the Hyperties Workstation Browser. By Ben Shneiderman, Catherine Plaisant, Rodrigo Botafogo, Don Hopkins, William Weiland.


    Pie Menu FUD and Misconceptions. Dispelling the fear, uncertainty, doubt and misconceptions about pie menus.


    The Shape of PSIBER Space: PostScript Interactive Bug Eradication Routines — October 1989. Written by Don Hopkins, October 1989. University of Maryland Human-Computer Interaction Lab, Computer Science Department, College Park, Maryland 20742.


    The Amazing Shneiderman. Sung to the tune of “Spiderman”, with apologies to Paul Francis Webster and Robert “Bob” Harris, and with respect to Ben Shneiderman.


    And finally this has absolutely nothing to do with pie menus, except for the shape of a pizza pie:

    The Story of Sun Microsystems PizzaTool. How I accidentally ordered my first pizza over the internet.


    1. 2

      Also, a more modern pie demo was tried within quicksilver by nick jitkoff many years ago: https://www.youtube.com/watch?v=d4LkTstvUL4&feature=youtu.be&t=18m17s

      (Shameless plug and not a pie menu, I spent a chunk of time experimenting with a gesture/zooming palette menu of sorts with https://thimblemac.com - the idea was to find something that felt a bit more scalable than a pie menu layouts but I arguably went too far in the steps to trigger the gesture for it. )

      1. 1

        Wow, that is really nice! Does it somehow magically integrate itself with existing unmodified unsuspecting tools like Photoshop and Sketchup? Could it be applied to many other applications too? I’ll read your blog post about making thimble.

        But before I do, and speaking of magic, I’d love to introduce you to Morgan Dixon’s brilliant and important work on Prefab: The Pixel-Based Reverse Engineering Toolkit.

        I wrote a summary and links to his work in relation to an idea I have called “aQuery – like jQuery for Accessibility” on Hacker News. I’d love to figure out a way to work on this stuff so many people can easily use it!


        Here are some ideas and discussion about aQuery – like jQuery for accessibility – which would be useful for implementing this kind of stuff out of of existing window systems and desktop applications.


        Morgan Dixon’s work is truly breathtaking and eye opening, and I would love for that to be a core part of a scriptable hybrid Screen Scraping / Accessibility API approach.

        Screen scraping techniques are very powerful, but have limitations. Accessibility APIs are very powerful, but have different limitations. But using both approaches together, screencasting and re-composing visual elements, and tightly integrating it with JavaScript, enables a much wider and interesting range of possibilities.

        Think of it like augmented reality for virtualizing desktop user interfaces. The beauty of Morgan’s Prefab is how it works across different platforms and web browsers, over virtual desktops, and how it can control, sample, measure, modify, augment and recompose guis of existing unmodified applications, even dynamic language translation, so they’re much more accessible and easier to use!

        James Landay replies:


        This is right up the alley of UW CSE grad student Morgan Dixon. You might want to also look at his papers.

        Don emails Morgan Dixon:

        Morgan, your work is brilliant, and it really impresses me how far you’ve gone with it, how well it works, and how many things you can do with it!

        I checked out your web site and videos, and they provoked a lot of thought so I have lots of questions and comments.

        I really like the UI Customization stuff, and also the sideviews!

        Combining your work with everything you can do with native accessibility APIs, in an HTML/JavaScript based, user-customizable, scriptable, cross platform user interface builder like (but transcending) HyperCard would be awesome!

        I would like to discuss how we could integrate Prefab with a Javascriptable, extensible API like aQuery, so you could write “selectors” that used prefab’s pattern recognition techniques, bind those to JavaScript event handlers, and write high level widgets on top of that in JavaScript, and implement the graphical overlays and gui enhancements in HTML/Canvas/etc like I’ve done with Slate and the WebView overlay.

        Users could literally drag controls out of live applications, plug them together into their own “stacks”, configure and train and graphically customize them, and hook them together with other desktop apps, web apps and services!

        For example, I’d like to make a direct manipulation pie menu editor, that let you just drag controls out of apps and drop them into your own pie menus, that you can inject into any application, or use in your own guis. If you dragged a slider out of an app into the slice of a pie menu, it could rotate it around to the slice direction, so that the distance you moved from the menu center controlled the slider!

        While I’m at it, here’s some stuff I’m writing about the jQuery Pie Menus. http://donhopkins.com/mediawiki/index.php/JQuery_Pie_Menus

        Web Site: Morgan Dixon’s Home Page.


        Web Site: Prefab: The Pixel-Based Reverse Engineering Toolkit.


        Video: Prefab: What if We Could Modify Any Interface? Target aware pointing techniques, bubble cursor, sticky icons, adding advanced behaviors to existing interfaces, independent of the tools used to implement those interfaces, platform agnostic enhancements, same Prefab code works on Windows and Mac, and across remote desktops, widget state awareness, widget transition tracking, side views, parameter preview spectrums for multi-parameter space exploration, prefab implements parameter spectrum preview interfaces for both unmodified Gimp and Photoshop:


        PDF: A General-Purpose Target-Aware Pointing Enhancement Using Pixel-Level Analysis of Graphical Interfaces. Morgan Dixon, James Fogarty, and Jacob O. Wobbrock. (2012). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’12. ACM, New York, NY, 3167-3176. 23%.


        Video: Content and Hierarchy in Prefab: What if anybody could modify any interface? Reverse engineering guis from their pixels, addresses hierarchy and content, identifying hierarchical tree structure, recognizing text, stencil based tutorials, adaptive gui visualization, ephemeral adaptation technique for arbitrary desktop interfaces, dynamic interface language translation, UI customization, re-rendering widgets, Skype favorite widgets tab:


        PDF: Content and Hierarchy in Pixel-Based Methods for Reverse-Engineering Interface Structure. Morgan Dixon, Daniel Leventhal, and James Fogarty. (2011). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’11. ACM, New York, NY, 969-978. 26%.


        Video: Sliding Widgets, States, and Styles in Prefab. Adapting desktop interfaces for touch screen use, with sliding widgets, slow fine tuned pointing with magnification, simulating rollover to reveal tooltips: https://www.youtube.com/watch?v=8LMSYI4i7wk

        Video: A General-Purpose Bubble Cursor. A general purpose target aware pointing enhancement, target editor:


        PDF: Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. Morgan Dixon and James Fogarty. (2010). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’10. ACM, New York, NY, 1525-1534. 22%


        PDF: Prefab: What if Every GUI Were Open-Source? Morgan Dixon and James Fogarty. (2010). Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’10. ACM, New York, NY, 851-854.


        Morgan Dixon’s Research Statement:


        Community-Driven Interface Tools

        Today, most interfaces are designed by teams of people who are collocated and highly skilled. Moreover, any changes to an interface are implemented by the original developers and designers who own the source code. In contrast, I envision a future where distributed online communities rapidly construct and improve interfaces. Similar to the Wikipedia editing process, I hope to explore new interface design tools that fully democratize the design of interfaces. Wikipedia provides static content, and so people can collectively author articles using a very basic Wiki editor. However, community-driven interface tools will require a combination of sophisticated programming-by-demonstration techniques, crowdsourcing and social systems, interaction design, software engineering strategies, and interactive machine learning.

        agumonkey writes:

        Small temporary answer while I unwind all the linked content, Dixon’s target aware pointing is already missing so much. I wonder how on earth nobody in smartphone land thought to implement something similar. I’m already hooked :)

        Don replies:

        It’s missing from many contexts where it would be very useful, including mobile. It’s related in many ways to those mobile gui, web browser and desktop app testing harnesses. It could be implemented as a smart scriptable “double buffered” VNC server (for maximum efficiency and native Accessibility API access) or client (for maximum flexibility but less efficient).

        The way jQuery widgets can encapsulate native and browser specific widgets with a platform agnostic api, you could develop high level aQuery widgets like “video player” that knew how to control and adapt many different video player apps across different platforms (youtube or vimeo in browser, vlc on windows or mac desktop, quicktime on mac, windows media player on windows, etc). Then you can build much higher level apps out of widgets like that.

        Target aware pointing is one of many great techniques he shows can be layered on top of existing interfaces, without modifying them.

        I’d like to integrating all those capabilities plus the native Accessibility API of each platform into a JavaScript engine, and write jQuery-like selectors for recognizing patterns of pixels and widgets, creating aQuery widgets that tracked input, drew overlays, implemented text to speech and voice control interfaces, etc.

        His research statement sums up where it’s leading: Imagine wikipedia for sharing gui mods!

        Berkeley Systems (the flying toaster screen saver company) made one of the first screen readers for the Mac in 1989 and Windows in 1994.


        Richard Potter, Ben Shneiderman and Ben Benderson wrote a paper called Pixel Data Access for End-User Programming and graphical Macros, that references a lot of earlier work.


        1. 1

          The integrations with Sketch/SketchUp was done via AXAPI, using https://github.com/robrix/Haxcessibility with some small extensions to scan the top menu. So yes, for those apps there’s no modification, the setup so far’s been done by curated plist/icon assets contained within my app, but in theory this could be made to be user-editable & leverage icons contained within the app bundles.

          Initially I started with Illustrator via an API (and dropped it due to API bugginess). And while the website still shows Photoshop support I dropped it because it used another buggy network-based api. The lesson learned is that on the desktop accessibility hacks proved to be much easier, but not entirely perfect - occasionally there’s still times where AXAPI seemingly decides not to cooperate and rejects requests until switching away from the window and back.

          Off the top of my head, a couple other apps in the space of leveraging macos AXAPI would be shortcat ( https://shortcatapp.com/ ), bettertouchtool (I thiink it does some axapi traversing: https://folivora.ai/ ), and ghostnote ( https://www.ghostnoteapp.com/ )

          Prefab: Wow, took a look at https://www.youtube.com/watch?v=w4S5ZtnaUKE and got enthused enough to make a quick montage of their work ( https://twitter.com/vivekgani/status/997727951583035393 ). Have definately had some similar thoughts but never thought to dig through HCI papers to see existing work in the space.

          aQuery: Will have to look into this idea more, sounds doable though I’d be curious about whether it could also be structured allow for subscribing to state changes (e.g. a button going from enabled to disabled). I find myself still debating whether to go down a more ‘pure’ path towards a whole environment where things are designed to be altered (dynamicland, smalltalk - https://youtu.be/AnrlSqtpOkw?t=10m41s )

          1. 2

            Wow, this is wonderful stuff and pieces of the puzzle I never heard about!

            Another puzzle piece that I’m interested in, that’s useful for live performance, video manipulation, screen scraping, image processing, pattern matching and vision based approaches like Prefab, is the ability to efficiently send images between apps running on the same system, via the GPU, without any performance penalty of copying between GPU and system memory (or even GPU and GPU memory unnecessarily)!

            Integrating uncooperative apps and libraries with virtual webcams and virtual framebuffers is also very useful, and can bridge many gaps between existing monolithic apps you can’t otherwise modify or extend with plugins.

            There is a library for the Mac called Syphon, and a similar one for Windows called Spout, that’s used by the VJ community and other people to efficiently integrate many programs like Max/MSP/Jitter, Pure Data, Quartz Composer, Processing, Unity3D, AfterEffects, VirtualDJ, FreeFrameGL, VLC, Java, Cinder, Vuo, Plask, Open Frameworks, Omnidome, Little Projection Mapping Tool, and even CEF (Chrome Embedding Framework), etc.


            Syphon is an open source Mac OS X technology that allows applications to share frames - full frame rate video or stills - with one another in realtime. Now you can leverage the expressive power of a plethora of tools to mix, mash, edit, sample, texture-map, synthesize, and present your imagery using the best tool for each part of the job. Syphon gives you flexibility to break out of single-app solutions and mix creative applications to suit your needs.




            Syphon is a Mac OS X technology to allow applications to share video and still images with one another in realtime, instantly.

            It’s quite useful to have a web browser in the mix, so this is an essential ingredient:


            Syphon is built on top of the macOS “IOSurface” API:


            Web browsers like Chrome and Safari that break the browser up into separate heavy weight processes for a user interface process and multiple rendering processes also use IOSurface to efficiently share images in the GPU between processes.

            iOS also supports IOSurface, and that’s how WKWebView works (which runs Safari in another hidden background process and embeds the browser in the foreground iOS app process). But you can’t multitask on IOS or run Max and AfterEffects, so it doesn’t make much sense for Syphon to support iOS. Syphon only makes sense on desktop computers.

            IOSurface currently supports both OpenGL and Metal. But unfortunately Syphon is totally OpenGL based. So sadly you can’t use it with Unity3D if you need to use the Metal device drivers (which are vastly preferable to OpenGL).


            IOSurface is neat. A shared bitmap that can cross between programs, and it’s got a relatively easy API including two super critical functions named IOSurfaceLock and IOSurfaceUnlock. I mean, if you’re sharing the data across process boundaries then you’ll need to lock things so that the two apps don’t step on each other’s toes. But of course if you’re not sharing it across processes, then you can ignore those locks, right? Right?

            And then a single file came back, named IOSurface2D.mm, which was some obscure sample code from Apple that I had received at one point a number of years ago.

            As I said, the documentation is thin and examples are rare! But the Safari and Chrome web browser source code, bug reports and discussion group is a great source of cutting-edge information, because internal Apple developers are working on it who have access to secret information that’s not public, and know how to make it perform well.

            I think it’s even possible for an OpenGL app and a Metal app to communicate with each other via IOSurface. (But Syphon doesn’t currently support that.)

            The web browsers have moved on to using a newer higher level but mostly undocumented API, that I believe is built on top of IOSurface, called CARemoteLayerClient / CARemoteLayerServer.



            Here are some comments on supporting it in Chrome:


            Safari must also be using something like IOSurfaces to make pixel data available across processes, so I’m guessing similar performance must be achievable in Chrome.

            Safari uses private SPI to share surfaces, which has different behaviors than IOSurface, so things that Safari can do aren’t necessarily feasible in other browsers (which is why Invalidate Core Animation exists in the first place).

            I’m not sure what advantages of using CARemoteLayerClient/Server are over IOSurface, or what “SPI” means exactly, but I think it has to do with this header file:


            At any rate, there are currently two problems with Syphon, which is otherwise very wonderful but getting long in the tooth:

            1. it’s not a cross platform API so it doesn’t support Windows. (But there is Spout for that.)

            2. it depends on OpenGL and doesn’t support Metal. Ideally it would support both.

            3. perhaps there is a good reason to do the same thing as the web browsers are now doing and use CARemoteLayerClient/Server instead of IOSurface, but I don’t know the issues involved or why they switched, and the documentation is sparse.

            On Windows, there’s a similar library called Spout that supports many different applications and libraries like Unity3D, OBS (Open Broadcaster Software), FreeframeGL, Java, Processing, Max/MSP/Jitter, Ableton Live, Virtual Webcam, OpenFrameworks, Cinder, etc:



            Senders and receivers include FreeframeGL plugins, a Java interface for Processing, Jitter externals for Max/Msp, VIZZable modules for Ableton Live, and a Virtual Webcam as a universal receiver. There is also example code for creating your own applications with openFrameworks and Cinder. Now these applications running on Windows can share video with each other in a similar way to Syphon for OSX.

            For compatible graphics hardware, OpenGL textures are shared by way of DirectX using the NVIDIA DirectX/Opengl interop extension. If hardware is not compatible, SPOUT provides a backup by way of CPU memory.

            Ideally I’d love to have one glorious cross platform API that supports whatever transport (or transports) work best on each platform, and even transparently supports less efficient APIs like simply sending images over shared system memory, uncompress or compressed over the network, WebRTC, etc, and also support whatever graphics APIs you need to integrate any particular existing app.

            Then the same library would let you integrate apps using different graphics APIs on the same system super-efficiently, and even Macs and Windows systems on the same network to communicate as efficiently as possible by streaming video.

            1. 2

              Huh, I think I’m starting to get your vision - you want the hacks/piping work built to fling the existing surfaces of ‘personal computing’ to facilitate the other forms of computing (social, presentative, etc.) - one use i’ve also been thinking of has been ‘protected sharing’ - e.g. screencasts or recording where the user selects areas and text is magically blurred out.

              FWIW, iOS is probably vastly different now but I remember that Adam Bell ( https://twitter.com/b3ll ) sorta played in the space of getting app framebuffers a while ago when jailbreaking was still easy - https://youtu.be/Ox09rWJTuCA?t=18m26s .

    2. 6

      No mention of Neverwinter Nights? As a game dev I am disappointed. That game had nested pie menus for almost all in game interaction and it worked great. A lot of people complained and said they didn’t like them but it was mostly just unfamiliarity. Once you got used to them they worked great.

      1. 2

        I think an important goal is to not only give users the tools to create their own pie menus, but design tools that support and motivate user to intuitively understand Fitts’s law, and how to design good pie menus for themselves.

        A recent game that lets players create their own radial menus, in a way that deeply improves game play, is Monster Hunter: World!

        Awesome Example: Monster Hunter: World — Radial Menu Guide Monster Hunter: World is a wonderful example of a game that enables and motivates players to create their own pie menus, that shows how important customizable user defined pie menus are in games and tools.

        Want access to all your items, ammo and gestures at your fingertips? Here’s a quick guide on the Radial Menu.

        With a multitude of items available, it can be challenging to find the exact one you need in the heat of the battle. Thankfully, we’ve got you covered. Here’s a guide on radial menus, and how to use them: The radial menu allows you to use items at a flick of the right stick. There are four menus offering access to eight items each, and you can fully customize them, all to your heart’s content. Radial menus are not just limited to item use, however. You can use them to craft items, shoot SOS flares, and even use communication features such as stickers and gestures.


      2. 6

        Somebody raised an interesting point on reddit about patent trolls:

        BobTheSCV> Neverwinter Nights also implemented them, and they worked very well. I had just assumed it was patent trolls or something that kept them from being widely adopted.

        I replied:

        You are absolutely correct about the patent trolls!

        Bill Buxton at Alias and his marketing team spread a bunch of inaccurate FUD about their “marking menu patent”, which I accidentally discovered and tried to correct and get him to stop doing decades ago, but he refused, and continued to spread FUD.

        So Alias kept advertising their “patented marking menus” for DECADES, purposefully and successfully discouraging their competition 3D Studio Max, AND many other developers of free and proprietary apps as collateral damage, from adopting them.

        When I asked Buxton about the “marking menu patent” before it was granted, he lied point blank to me that there was no “marking menu patent”, so I couldn’t prove to Kinetix that it was OK to use them, or contact the patent office and inform them about the mistakes in their claims about prior art, and the fact that the “overflow” technique they were claiming in the patent was obvious.

        The whole story is here:

        Pie Menu FUD and Misconceptions: Dispelling the fear, uncertainty, doubt and misconceptions about pie menus.


        Some excerpts:

        There is a financial and institutional incentive to be lazy about researching and less than honest in reporting and describing prior art, in the hopes that it will slip by the patent examiners, which it very often does.

        Unfortunately they were able to successfully deceive the patent reviewers, even though the patent references the Dr. Dobb’s Journal article which clearly describes how pie menu selection and mouse ahead work, contradicting the incorrect claims in the patent. It’s sad that this kind of deception and patent trolling is all too common in the industry, and it causes so many problems.

        Even today, long after the patent has expired, Autodesk marketing brochures continue to spread FUD to scare other people away from using marking menus, by bragging that “Patented marking menus let you use context-sensitive gestures to select commands.”

        A snapshot of Alias’s claim about “Patented marking menus” from one of their brochures that they are still distributing, even years after their bad patent has expired:


        “Marking Menus: Quickly select commands without looking away from the design. Patented marking menus let you use context-sensitive gestures to select commands.”


        The Long Tail Consequences of Bad Patents and FUD

        I attended the computer game developer’s conference in the late 90’s, while I was working at Maxis on The Sims. Since we were using 3D Studio Max, I stopped by the Kinetix booth on the trade show floor, and asked them for some advice integrating my existing ActiveX pie menus into their 3D editing tool.

        They told me that Alias had “marking menus” which were like pie menus, and that Kinetix’s customers had been requesting that feature, but since Alias had patented marking menus, they were afraid to use pie menus or anything resembling them for fear of being sued for patent infringement.

        I told them that sounded like bullshit since there was plenty of prior art, so Alias couldn’t get a legitimate patent on “marking menus”.

        The guy from Kinetix told me that if I didn’t believe him, I should walk across the aisle and ask the people at the Alias booth. So I did.

        When I asked one of the Alias sales people if their “marking menus” were patented, he immediately blurted out “of course they are!” So I showed him the ActiveX pie menus on my laptop, and told him that I needed to get in touch with their legal department because they had patented something that I had been working on for many years, and had used in several published products, including The Sims, and I didn’t want them to sue me or EA for patent infringement.

        When I tried to pin down the Alias marketing representative about what exactly it was that Alias had patented, he started weaseling and changing his story several times. He finally told me that Bill Buxton was the one who invented marking menus, that he was the one behind the patent, that he was the senior user interface researcher at SGI/Alias, and that I should talk to him. He never mentioned Gordon Kurtenbach, only Bill Buxton.

        So I called Bill Buxton at Alias, who stonewalled and claimed that there was no patent on marking menus (which turned out to be false, because he was just being coy for legal reasons). He said he felt insulted that I would think he would patent something that we both knew very well was covered by prior art.

        At the time I didn’t know the term, but that’s what we now call “gaslighting”: https://en.wikipedia.org/wiki/Gaslighting

        Gee, who do we all know who lies and then tries to turn it all around to blame the person who they bullied, and then tries to play the victim themselves? https://en.wikipedia.org/wiki/Donald_Trump

        Gordon Kurtenbach, who did the work and got the patent that Alias marketing people were bragging about in Bill Buxton’s name agrees:

        Gordon> Don, I read and understand your sequence of events. Thanks. It sounds like it was super frustrating, to put it mildly. Also, I know, having read dozens of patents, that patents are the most obtuse and maddening things to read. And yes, the patent lawyers will make the claims as broad as the patent office will allow. So you were right to be concerned. Clearly, marketing is marketing, and love to say in-precise things like “patented marking menus”.

        Gordon> At the time Bill or I could have said to you “off the record, its ok, just don’t use the radial/linear combo”. I think this was what Bill was trying to say when he said “there’s no patent on marking menus”. That was factually true. However, given that Max was the main rival, we didn’t want to do them any favors. So those were the circumstances that lead to those events.

        What’s ironic is that Autodesk now owns both Alias and 3D Studio Max. Gordon confirmed that Alias’s FUD did indeed discourage Kinetix from implementing marking menus or pie menus, which were not actually covered by the patent:

        Gordon> After Autodesk acquired Alias, I talked to the manager who was interested in getting pie menus in Max. Yes, he said he that the Alias patents discouraged them from implementing pie menus but they didn’t understand the patents in any detail. Had you at the time said “as long we don’t use the overflows we are not infringing” that would have been fine. I remember at the time thinking “they never read the patent claims”.

        Don> The 3D Studio Max developers heard about the Alias marking menu patent from Alias marketing long before I heard of it from them on the trade show floor.

        Don> The reason I didn’t know the patent only covered overflows was that I had never seen the patent, of course. And when I asked Buxton about it, he lied to me that “there is no marking menu patent”. He was trying to be coy by pretending he didn’t understand which patent I was talking about, but his intent was to deceive and obfuscate in order to do as much harm to Kinetix 3D Studio Max users as possible, and unfortunately he succeeded at his unethical goal.

        What’s even worse is that in Buxton’s zeal to attack 3D Studio Max users, he also attacked users of free software tools like Blender.

        The Alias Marking Menu Patent Discouraged the Open Source Blender Community from Using Pie Menus for Decades

        Here is another example that of how that long term marketing FUD succeeded in holding back progress: the Blender community was discussing when the marking menu patent would expire, in anticipation of when they might finally be able to use marking menus in blender (even though it has always been fine to use pie menus).


        As the following discussion shows, there is a lot of purposefully sewn confusion and misunderstanding about the difference between marking menus and pie menus, and what exactly is patented, because of the inconsistent and inaccurate definitions and mistakes in the papers and patents and Alias’s marketing FUD:

        “Hi. In a recently closed topic regarding pie menus, LiquidApe said that marking menus are a patent of Autodesk, a patent that would expire shortly. The question is: When ? When could marking menus be usable in Blender ? I couldn’t find any info on internet, mabie some of you know.”

        The good news: Decades late due to patents and FUD, pie menus have finally come to 3D Studio Max just recently (January 2018)!

        Radially - Pie menu editor for 3ds Max: https://www.youtube.com/watch?v=sjLYmobb8vI

        1. 5

          The best usages of pie menus I’ve seen are invariably in video and computer games, usually because they actually care about UX and efficiency. Three really brilliant examples I’ve seen (one of which is mentioned in the article):

          • The Sims action menus
          • CS:GO/ CS:CZ
          • Natural Selection mod for Half-Life (interesting radial/tree compromise)
          • Mass Effect
          • Fallout 4 (vastly worse than the old New Vegas style, but easy to use)
          1. 1

            I like how Mass Effect also puts the keypress into the menu, too. So, you can use it immediately in an intuitive way plus quickly learn the shortcut buttons. An extra boost comes with the buttons being different colors.

          2. 4

            If these PIE menus are so awesome, why can’t I just use them?

            • In Android, there as an patch on (now discontinued) Paranoid Android fork, later respawned as SlimPIE but retored due to problematic maintenance in newer Android. There are some overlay-like applications for that though, but they all suffer from minor problems and feel like extraordinary additions not integrated well with the rest of OS.

            • Using such thing as context menu in GTK or Qt could be probably possible (on X.org at least, with XShape ext.) but there are too many places where people assumed that menus are in form of rectangle (both in 3rd party apps and library itself) that you would probably need to rewrite too many LOCs

            • Not even saying anything about WinNT, as they moved scrollbars rendering code into kernelspace because “it’s faster lol”

            • I don’t know too much about MacOS GUI rendering mechanics, but they already did a great improvement over standard UIs with that global menu thing, which is also pretty hard to reimplement in any other environment without breaking stuff

            I’ll be really glad to see the software world migrating to more ergonomic UI/UX concept instead of flattening the world for fun and profit (I mean, reducing amount of skilled graphicians), even when they use some hybrid concepts intead of full PIE, for example “trees” or regular menus expanding from root PIE items. This particular concept has been well presented in Sword Art Online anime and also kinda predicted the UX paradigms for holographic/floating interfaces:

            concept 1 concept 2 actual screencap from the series

            1. 5

              Like most widget types that are poorly supported or unsupported in popular GUI toolkits, I see pie menus almost exclusively in games – since people writing games have the expectation that they’ll need to essentially roll their own GUI toolkit in order to make it sufficiently themable anyhow.

              It’s an unfortunate state of affairs: widget functionality, once you’ve got past the initial learning curve, is proportional to the degree to which the widget behavior matches the user’s internal mental model of the task, and so expressive UI elements have the ability to make real work much more efficient; limiting the use of good GUIs to video game menus means UI design is limited in its capacity to make anything but our leisure time easier.

              (As a side note: SAO is a bad example of UI design in anime – the titular game is, on many levels, not professional enough to make it to market, and the way the menus are laid out is no exception. Geoff Thew explains this in detail in one of his video essays. The way SAO uses pie menus is sort of a shallow copy of how pie menus have been used in games for the past decade, as understood by someone who has never played a video game. A good example of an interesting use of pie menus in a real game is the conversation system in Mass Effect – where segments have their size changed in order to make certain responses more likely, and where pie areas correspond to emotions or tactics.)

              1. 4

                Great points!

                I wrote about “Ersatz Pie Menus” in this additional article, “Pie Menu FUD and Misconceptions: Dispelling the fear, uncertainty, doubt and misconceptions about pie menus.”

                Ersatz Pie Menus

                Richard Stallman likes to classify an Emacs-like text editor that totally misses the point of Emacs by not having an extension language as an “Ersatz Emacs”.

                In the same sense, there are many “Ersatz Pie Menus” that may look like pie menus on the surface, but don’t actually track or feel like pie menus, or benefit from all of their advantages, because they aren’t carefully designed and implemented to optimize for Fitts’s Law by being based purely on the direction between stroke endpoints instead of the entire path, minimizing the distance to the targets, and maximizing the size of the targets.

                Microsoft Surface Dial: Someone on Hacker News asked me: Any thoughts on Microsoft’s Surface Dial radial menu?

                Good question — glad you asked! (No, really! ;) Turning a dial is a totally different gesture than making directional strokes, so they are different beasts, and a dial lacks the advantages pie menus derive from exploiting Fitts’s Law. […]

                Beautiful but Ersatz Pie Menu Example – the graphics are wonderful but the tracking is all wrong: http://pmg.softwaretailoring.net/

                Turning Is Not Like Stroking: In terms of “Big O Notation”, pull down menus, click wheels, and carousel selection is linear O(n), while with a pie menu you only have to perform one short directional gesture to select any item, so selection is constant O(1) (with a small constant, the inner inactive radius of the hole in the middle, which you can make larger if you’re a spaz).

                Yucky Pie Menus Recipes

                Bedazzling and Confusing Graphics and Animations […]

                Rectangular Label Targets Instead of Wedge Shaped Slice Targets […]

                Triggering Items and Submenus on Cursor Motion Distance Instead of Clicking […]

                Not Starting Pie Menus Centered on the Cursor […]

                Improperly Handling Screen Edges […]

                Improperly Handling Mouse-Ahead Display Preemption and Quick Gestures on Busy Computers […]

                Yummy Pie Menu Recipes

                I’m certainly not saying that pie menus should never be graphically slick or have lots of cool animations. Just that they should be thoughtfully designed and purposefully easy to use first, so they deeply benefit users from Fitts’s Law, instead of just trying to impress users with shallow useless surface features.

                Spectacular Example: Simon Schneegans’ Gnome-Pie, the slick application launcher for Linux

                I can’t understate how much I like this. Not only is it slick, beautiful, and elegantly animated, but it’s properly well designed in all the important ways that make it Fitts’s Law Friendly and easy to use, and totally deeply customizable by normal users! It’s a spectacularly useful tour-de-force that Linux desktop users can personalize to their heart’s content.

                Gnome-Pie — Simon Schneegans

                Homepage of Gnome-Pie, the slick application launcher for Linux. simmesimme.github.io

                Gnome-Pie is a slick application launcher which I’m creating for Linux. It’s eye candy and pretty fun to work with. It offers multiple ways to improve your desktop experience.

                Check out the project’s homepage @ http://gnome-pie.simonschneegans.de



                1. 2

                  I saw. You had a lot of interesting stuff on your Medium account.

                  (Are you likely to post more on HyperTIES? As a Xanadu-er & someone interested in the history of pre-web hypertext systems, I find it interesting, since it has a pretty distinct look & feel and seems like it might have more interesting UI ideas to copy. The ‘pop-out’ mechanism for linked areas in images interested me when I saw it mentioned.)

                  1. 3

                    One of the good ideas was that every article had a brief definition, which it would show to you the first time you clicked a link, without leaving where you were. That’s a feature I wish was universally supported by the web, so you didn’t have to leave your current page to find out something about the link before following it.

                    You could then click again (or click on the definition), or pop up a pie menu to open the link in the current or the other window. Also you could turn pages and navigate with the pie menus, swiping in obvious directions, like with an iPad, but having the pie menu to provide a visual affordance of which gestures are available (“self revealing” gestures).

                    Here are a couple of articles about HyperTIES that I haven’t moved to Medium yet:

                    Designing to Facilitate Browsing: A Look Back at the Hyperties Workstation Browser. By Ben Shneiderman, Catherine Plaisant, Rodrigo Botafogo, Don Hopkins, William Weiland. http://www.donhopkins.com/drupal/node/102

                    Abstract: Since browsing hypertext can present a formidable cognitive challenge, user interface design plays a major role in determining acceptability. In the Unix workstation version of Hyperties, a research-oriented prototype, we focussed on design features that facilitate browsing. We first give a general overview of Hyperties and its markup language. Customizable documents can be generated by the conditional text feature that enables dynamic and selective display of text and graphics. In addition we present:

                    an innovative solution to link identification: pop-out graphical buttons of arbitrary shape.

                    application of pie menus to permit low cognitive load actions that reduce the distraction of common actions, such as page turning or window selection.

                    multiple window selection strategies that reduce clutter and housekeeping effort. We preferred piles-of-tiles, in which standard-sized windows were arranged in a consistent pattern on the display and actions could be done rapidly, allowing users to concentrate on the contents.

                    Pie menus to permit low cognitive load actions: To avoid distraction of common operations such as page turning or window selection, pie menus were used to provide gestural input. This rapid technique avoids the annoyance of moving the mouse or the cursor to stationary menu items at the top or bottom of the screen.

                    HyperTIES Hypermedia Browser and Emacs Authoring Tool for NeWS. http://www.donhopkins.com/drupal/node/101

                    That has a screen dump, an architectural diagram, a list of interesting features, data structures, and links to the C, Forth, PostScript, Emacs MockLisp, and HyperTIES markup language source code!

                    Here’s a demo of HyperTIES and the pop-out embedded menus:

                    HCIL Demo - HyperTIES Browsing: Demo of NeWS based HyperTIES authoring tool, by Don Hopkins, at the University of Maryland Human Computer Interaction Lab.


                    A funny story about the demo that has the photo of the three Sun founders whose heads puff up when you point at them:

                    When you point at a head, it would swell up, and you pressed the button, it would shrink back down again until you released the button again.

                    HyperTIES had a feature that you could click or press and hold on the page background, and it would blink or highlight ALL of the links on the page, either by inverting the brightness of text buttons, or by popping up all the cookie-cut-out picture targets (we called them “embedded menus”) at the same time, which could be quite dramatic with the three Sun founders!

                    Kind of like what they call “Big Head Mode” these days! https://www.giantbomb.com/big-head-mode/3015-403/

                    I had a Sun workstation set up on the show floor at Educom in October 1988, and I was giving a rotating demo of NeWS, pie menus, Emacs, and HyperTIES to anyone who happened to walk by. (That was when Steve Jobs came by, saw the demo, and jumped up and down shouting “That sucks! That sucks! Wow, that’s neat. That sucks!”)

                    The best part of the demo was when I demonstrated popping up all the heads of the Sun founders at once, by holding the optical mouse up to my mouth, and blowing and sucking into the mouse while secretly pressing and releasing the button, so it looked like I was inflating their heads!

                    One other weird guy hung around through a couple demos, and by the time I got back around to the Emacs demo, he finally said “Hey, I used to use Emacs on ITS!” I said “Wow cool! So did I! What’s was your user name?” and he said “WNJ”.

                    It turns out that I had been giving an Emacs demo to Bill Joy all that time, then popping his head up and down by blowing and sucking into a Sun optical mouse, without even recognizing him, because he had shaved his beard!

                    He really blindsided me with that comment about using Emacs, because I always thought he was more if a vi guy. ;)

                    1. 2

                      Accidentally stumbled upon the xanadu-esque side of this conversation. Just to throw in a thought, it’s been on my mind to attempt applying a high level of polish to the federated wiki project such that it facilitated a way to zoom in on reading one thing at a time (a zoom of sorts). So..a ‘big head mode’ of sorts to let you focus on consuming (or editing an article when you needed it then zoom back out to see the connections.

                      The other crazy thought on my mind is seeing if there’s a way to bend existing oss text editors (particularly atom.io) to facilitate more freeform things like ‘code bubbles’ or liquidtext.

              2. 3

                I just added this example to the article, which you may have missed (since it wasn’t there until a few minutes ago):

                Spectacular Example: Simon Schneegans’ Gnome-Pie, the slick application launcher for Linux

                I can’t understate how much I like this. Not only is it slick, beautiful, and elegantly animated, but it’s properly well designed in all the important ways that make it Fitts’s Law Friendly and easy to use, and totally deeply customizable by normal users! It’s a spectacularly useful tour-de-force that Linux desktop users can personalize to their heart’s content.

                Gnome-Pie - Simon Schneegans

                Homepage of Gnome-Pie, the slick application launcher for Linux. simmesimme.github.io

                Gnome-Pie is a slick application launcher which I’m creating for Linux. It’s eye candy and pretty fun to work with. It offers multiple ways to improve your desktop experience.

                Check out the project’s homepage @ http://gnome-pie.simonschneegans.de


                If that doesn’t blow your mind, check this out – there are so many great things about it:


                1. 2

                  I used a great pie interface on Android for a while, though I cannot remember what it was called and am failing to find screenshots online. It helped considerably with one-handed operation.

                2. 2

                  How does this compare to other menus if its keyboard driven (both menu opening and item selection).

                  What about compared to some regexp search through the menu items?

                  1. 4

                    One good thing that my horrible ActiveX pie menus did handle nicely was keyboard navigation, shown in the following demo.

                    Four and eight item pie menus work very well with the numeric keypad and arrow keys, as well as joysticks and d-pads. You can also move to the next/previous item around the circle with tab / shift-tab, and type the name of an item to jump to it directly, hit escape to cancel the whole menu tree, backspace to peel back one submenu layer, etc, just like normal menu keyboard navigation.

                    It’s great to be able to navigate the same memorable menu trees with many kinds of input devices!

                    Terrible Example: ActiveX Pie Menus

                    This demo of an old OLE/ActiveX implementation of pie menus in Internet Explorer shows the weakness and complexity of the traditional desktop GUI MFC based approach to creating and editing pie menus with typical property sheets, scrolling lists and tree editors. It also has a weird way of paging between large numbers of items in pages of eight or fewer items each, which I was never very happy with. And programming complex user interfaces with XML/C++/ActiveX/OLE/MFC/Win32 is hell on earth.


                    1. 1

                      Did you do any experiments specifically with joystick input? Because mouse drivers are intended to produce actual cursor position deltas, even trackpoint-style mice produce more information than is used in a pie menu (and while warping can make up for overshooting, it can be confusing for the user unless done carefully). On the other hand, while Mass Effect never nested their pie menus, they essentially made use of one of the two controller sticks as a pure direction-input device with good radial resolution (better than a d-pad, which has only four directions & thus can defeat the point of a pie menu if there are more than 4 options).

                      Using a dual-stick video game controller for a pair of pie menus that are statically placed means simultaneous navigation of two menus can be fast. Using fans instead of pies and orienting the fan such that the fan center is the center of the previous selection and the fan’s options are arrayed perpendicular to the swipe direction used to select it could make it possible to navigate and back out of many layers of menu pretty easily without extra selection gestures (the one complaint I had about the pie menus in The Sims 1)

                      1. 3

                        No, I haven’t done any joystick experiments myself.

                        Before he started spreading FUD about marking menus, Bill Buxton wrote an excellent paper on input devices in 1983, called “Lexical and Pragmatic Considerations of Input Structures”, which categorizes input devices and how their unique properties map to the problem domain, and discusses a lot of important concepts like pragmatics, chunking and closure, device independence, taxonomy of devices, and the nulling problem:


                        lexical: issues having to do with spelling of tokens (i.e., the ordering of lexemes and the nature of the alphabet used - symbolic or iconic, for example).

                        pragmatic: issues of gesture, space and devices.

                        Figure 1: Taxonomy of Input Devices. https://www.billbuxton.com/lexical1.gif

                        Continuous manual input devices are categorized. The first order categorization is property sensed (rows) and number of dimensions (columns). Subrows distinguish between devices that have a mechanical intermediary (such as a stylus) between the hand and the sensing mechanism (indicated by “M”), and those which are touch sensitive (indicated by “T”). Subcolumns distinguish devices that use comparable motor control for their operation.

                        Since you mentioned fans, I should point out that Simon Schneegans, the same guy who did the excellent Gnome-Pie, also did these amazing spoke-like and fan-out pies for his bachelor’s thesis:

                        The Coral-Menu: https://vimeo.com/51072812

                        The Trace-Menu: https://vimeo.com/51073078

                        Those are great on so many levels!

                        1. 2

                          Thanks! A lot of food for thought here

                          1. 1

                            You’re welcome! I’m glad you’re hungry for it. ;)

                            Enjoy some pizza: https://medium.com/@donhopkins/the-story-of-sun-microsystems-pizzatool-2a7992b4c797

                            And some more pie too: https://www.youtube.com/watch?v=Xj1LFYSO2Kw

                      2. 1


                        Ignoring the thing its built on, that’s an interesting demo. I like that expert gesture-like mode. How well does it works in the end? Too bad this video doesn’t show keyboard or joystick use (or I missed it having skipped most of the editor portion).

                        1. 2

                          It was very snappy because it was directly calling the Win32 API to manipulate windows and handle events (well, going through some layers of OLE/MFC, but not as squishy as a web browser). So mouse-ahead was quite reliable since it wasn’t polling for events and letting some of them slip through the cracks when the computer slows down (i.e. always), like some systems do.

                          That was the great thing about NeWS: it had perfect, air tight, synchronous event handling, and never dropped one event on the floor, or sent a keystroke to the wrong window.

                          Even my modern MacBook Pro always gets overheated, slows down, and when I open a new tab I have to stand back, be patient, and wait for the vehicle to stop moving completely before embarking with the mouse and keyboard.

                          On Medium, I keep typing cmd-f to search before the page has settled down, and the first few characters of my search string that I type after cmd-f get inserted into the document somewhere at random, then the rest go into the search field! So I have to type cmd-f, sit back, take a breath, wait for a while, then type what I want to search for. Emacs on a 300 baud terminal was better than that!

                          But I really hurt for an easy to use drawing API like cairo or canvas (i.e. I’m an old PostScript programmer, and it’s a huge pain in the ass carefully selecting resources in and out of the win32 GDI context absolutely perfectly every time without making the slightest mistake or else GDI memory would overflow and Windows would freeze up and the reset button would not work so you had to remove the laptop’s battery to reboot).

                          But then again it was even worse programming on a Mac without any memory protection whatsoever! Those old Macs tend to freeze up instantly whenever you made the slightest programming mistake, instead of slowly grinding to a halt while glitching out the screen like Windows. To their credit, the PowerPC Macs could crash really really really quickly! You instantly knew when to reach for the “programmer’s switch”.

                    2. 1

                      This kicks ass: The foot menu!


                      1. 1

                        Imagine how safe democracy would be if only voting machines used pie menus: