1. 15

I am aiming to compile a framework for testing and testing various input methods. This is the first post of the blog about the project. The github repo contains additional resources that hopefully will turn out to something of a book or wiki on the subject.

I really find this subject interesting.

  • inspire people to think in new ways, UI-wise
  • good for systems with special requriments
  • not to forget for people with accessability issues

I hope you find this subject as inieresting as I do

Any feedback is greatly appriciated and the harsher the better

  1.  

  2. 5

    You should look into plover, I was surprised to see that you had the notion of chorded keyboards but didn’t include the most widely adopted chorded keyboard style. While as a programmer you might want to add a few words to the dictionary that encapsulate the keywords from the language you use it should otherwise be friction free. If you’re doing onehanded or you have some kind of constraint that makes standard steographic keyboards less useful I would still say that it may be useful to bootstrap off of the existing standard.

    1. 1

      Thank you so much I’ll look into it!

    2. 4

      Here’s one alternative that I’ve seen: MessagEase.

      Basically, it’s an onscreen keyboard with a compact layout where the most common characters are taps, and less common characters are directional swipes. Compared to a standard onscreen keyboard, I find it more precise and more expressive. It’s lacking, unfortunately, in predictive capabilities, which take away a lot of the speed advantages it would otherwise have.

      I don’t use it all the time because if I do, it gives me RSI in my right thumb, which is something that’s probably not considered much when comparing input methods.

      1. 1

        Oh, sweet! This actually mirrors one the idea I had for using gamepad for writing text. So basically what I did was presenting a tree with the symbols sorted by probability from a ngram style prediction engine. Quite simple actually. Selection or moving was done either by left or right stick. Or having diagonal movements (South West, South East, North West, North East) for one and “straight” (N,E,S,W) for the other.

        I never quite figured out how to translate it to touch. I come from a very button oriented perspective so I don’t really think in a touch oriented fashion.

        I did some tests using this my mathematical predictions turned out to be true. It gets lower movements per character than using a one axis input on a virtual keyboard. I guess that would translate to finger movements on a screen keyboard again.

        Fitt’s law tells us that the chance of error goes up the further the finger have to move. A larger object is of course easier to hit.

        What I really would like to do is to get a group of test subjects and a high speed camera and do some statistical analysis on the matter!!

        1. 1

          RSI and related injuries are of interest. Posture really plays a part. From my research on gamepads I found the following. Whereas while coding on a laptop I naturally lean forward after a while, as if there where a gravity pulling me closer. Due to my arms having to be moved forward and the laptop is way beneath my head.

          While practicing for hours with a gamepad. I noticed that I naturally moved the gamepad towards my heart and leant back. So things like these really plays a part.

          Also gamepad text entry would enable working from the coach which is an important milestone!!

          There is nothing I like more than to find out that someone else having implemented an idea that I have been considering. It proves that I am not crazy and saves me the time from implementing it.

          This are my findings from a structurally similar method however for gamepads. That is leveraging the same statistical properties of english language:

          Results from my tree model contra classic keyboard using gamepad: https://www.bitchute.com/video/aKl7jUrtcOt8/ http://eruditenow.com/theory.html

          A somewhat silly feel to the project but the data is still valuable I think. So I do believe that their theory makes sense!!

        2. 2

          I’ve wanted to dual-wield wireless chorded keyboards for some time but never found a project that had everything I wanted. I don’t mind keyboards, I think they are a reasonable input mechanism, chorded keyboards moreso. I find touchscreen keyboards very painful and there are some chorded keyboard apps like http://asetniop.com/ but I never stuck it.

          I think a Morse keyboard could be fun, maybe easier to type on a touchscreen while walking too?

          1. 1

            I don’t mind keyboards either, I’ve put so much time into it. Yes touchscreen keyboards are hellish. Prediction makes it better. David MacKay from Dasher project points out that the prediction part is a context switch. More so the screen only fit like three or four words so it’s quite limited.

            Also from a more human perspective autocomplete doesn’t allow you to play with spelling for fun and doesn’t handle quick language changes or even dialects in many cases.

            Chorded doesn’t really play nicely with touch as your keys are on the other side.

            I am glad that you mentioned Morse! I will do a write up on it because it is a really interesting method due to a lot of mathematical properties and so on. Along with it I’ll try to implement a HTML5 javascript toy to play around with. Maybe it can inspire something good. Don’t underestimate morse it’s really clever!

            If you want to get updates I guess you can use this http://tbf-rnd.life/feed/ of course I’ll post it here as well due to the nothing short of fantastic community. Do you have any other prefeered way to follow content?

          2. 2

            This comment will be pretty tangential, but you inspired me to try something.

            I read the chorded keyboard inspired by guitar idea with my old playstation controller sitting on my desk. Realizing I can use the left hand to select a key map and the right hand to execute one, I wrote a quick program to take input from my controller and turn it into key events, with a little gui to tell what the buttons would do before pressing them.

            It is actually somewhat usable! With the gui telling me what the keys are, I feel like I can actually use it even without practice… though it is quite slow, I used to be a very slow typist on a standard keyboard too, so maybe some practice can improve that.

            So the basic idea I went with is the d-pad (or the left analog stick, on the older playstation controllers these are the same, and the analog stick is indeed fairly easier to use) selects a key-page. Holding left displays four options, holding up displays another 4 options, etc., around an 8-point circle. The L1 and L2 buttons shift these pages up.

            The d-pad gives 56 options (8 depressed positions plus one null position, all times 4, because there are circle, square, triangle, cross on the right hand to press). With L1 and L2 combinations added, we have a whopping 224 selections for your right thumb to do.

            Then the right thumb buttons correspond to the key events. Just like guitar, left hand selects the sound/key, right hand actually makes the sound/strikes the key.

            (the rest of this I haven’t implemented in my proof of concept yet, I might change my mind after actually trying them)

            And, of course, there are also R1 and R2 buttons, which I will make correspond to ctrl and shift, respectively. Now, if I was doing this from scratch, I’d probably do something different with them, but for utility today, mapping it to existing keyboard concepts is useful.

            Then the right analog stick can move the mouse cursor, and the L3 and R3 pushes of the stick correspond to mouse clicks.

            I can actually see myself using this just because I can rest my wrists while using the game controller in ways I can’t with keyboards and mouses I’ve tried.

            the source code to my sample: http://arsdnet.net/jsinput.d it is written in the D programming language and depends on three of my library files (color.d, simpledisplay.d, joystick.d from here https://github.com/adamdruppe/arsd) and might only work on my computer with my esoteric linux and my old controller anyway, so you might not be able to play with it.. but still there it is.

            The gui in my implementation is super minimal, but I could imagine it showing like a circle of circles indicating all the options you have available to point the stick toward or something like that but meh it is already good enough for me.

            1. 1

              It’s so nice to be able to inspire someone! I’d love to take this further, if you have the time. Let’s strike while the iron is hot! Unfortunately I only have an Android tablet to work with right now and while I do have a usb gamepad controller I can see from termux I can not access it while the no-brand device is not rooted.

              Would it be to much for you to ask of a quick screencast of your project. If you don’t want to publish it publicly ask for my e-mail via PM.

              I’d argue for using HTML5+javascript to create a demo. Javascript gamepad support is really good and it gives you platform independence. Using web tech also allows for ease of demonstration in the age of goldfish-like attentionspans.

              So what I am doing software wise is to try to create a framework for experimentation on input methods. Providing the following:

              • os integration, not only keyevents also OS functions, shell access etc
              • text prediction
              • statistical analysis, side by side analysis on performance
              • configuration
              • shell commands along with arguments could be provided as alphabet to provide a quick and actually useful way to control a computer!!!

              The individual parts would be tied together by a simple JSON/websocket protocol modeled after javascript events. Providing least common denominator access and giving browser access to a daemon controlling the OS. Also a cloud server with access to a huge shared public n-gram dataset could provide predictions in combination with a local personalized dataset.

              I do see a lot of potential in your idea to be honest. I have as a matter of fact been working quite a lot with gamepads in particular (See links below). There are applications

              • better posture, you naturally lean towards the keyboard laptop screen
              • code in your coach in front of 60” TV

              What my earlier attempt was was to create a type fast game doing a side by side comparison of four different algorithms.

              My sigma experiment constructed as a typing speed game to get data on performance. That is actions/inputs per character/word along with wpm score.

              Does side by side comparison of four algorithms

              • linear (worst case linear search)
              • keyboard (reference keyboard implementation)
              • tree (somewhat similar to MessagEase mentioned earlier at list concerning theoretical efficiency)
              • quad (my initial idea, turned out a disappointment apart from being useful for entering math formulas using latex roosevelt.eruditenow.com/bootstrap.html check youtube video)

              Try it out: http://sigma.eruditenow.com Theory: http://sigma.eruditenow.com/theory

              Videos screencasts etcetera https://www.bitchute.com/channel/eruditenow/

              most watched; keyboard v.s. most succesfull method: https://www.bitchute.com/video/aKl7jUrtcOt8/ (positive outcome)

              Introduction video: https://www.bitchute.com/video/FklGhgQGbuDk/

              Video on gamepad support: https://www.bitchute.com/video/jqM0f6r06Kwx/

              The linear worst case model actually had some good scores when put in dynamic prediction mode. Courtesy of having a ngram prediction engine: https://www.bitchute.com/video/DH97lqy3TH0Z/

              of course there is a github https://github.com/richard-jansson/sigma

              1. 1

                Yeah, I have it right now being a somewhat passable keyboard+mouse substitute, but it is awfully slow. I realize though that it could also just put whole words (or other sequences, like “) {\n\t” might be useful when programming!) on keys too.

                Or, if it is a dynamic ui, it could provide autocomplete option trees on buttons. This would require cooperation with applications though… so not as easy to do in 45 minutes of hacky code lol.

                But I am hesitant to make it too dynamic because one of the things I like about a regular keyboard is everything is in the same space. You can get to know it and type without the gui from muscle memory. Consistent spacial layout is important for that. But maybe some combinations would be dynamic and then the regular keys remain the same to fallback on.

                edit: the reason I used D here is 1) d rox and i can do all kinds of stuff very quickly with it and 2) I can send input to the OS easily, which is not possible from the browser. But with the idea coming together I could do a javascript demo with a textarea to show the result of your input. I might come back to that when I am happier with the design.

            2. 1

              Once again the lobste.rs forum proves it worth! Thank you so much for all of your feedback.

              I am thinking of compiling all my findings into the book so if anybody wants to get an honorable mention that can be arranged! self-ublished via amazon so it’s doable within a few weeks for the first version.

              I am aiming for an iterative cooperative approach here. So crowdsourced if you want to.

              Along with this i want to create a software framework for comparing and creating new innovative models.

              This framework would provide:

              • text prediction
              • os level input support
              • input output file system
              • statistical analysis
              • configuration

              Prototypes will be coded in web to make access as easy as possible. Then a lot of user data could be sourced providing a good source of data for analysis and statistical comparison.

              Everything tied together via a simple JSON/websocket protocol to try to mimic the UNIX philosophy. I know some of you will start to shout UDP at me but bear with me.

              Doing this for a while could enable rapid prototyping of several input methods. The most successful alternatives could then be reimplemented in lower level language for that minimalistic low overhead code we all come to love and adore. This is text entry so an electron application using up half your RAM is a no go simply! But for prototyping using low level languages and handcrafted UI simply doesn’t cut it. Believe me I have tried:

              https://github.com/richard-jansson/veta https://m.youtube.com/watch?v=8kuppby-KJ0 this is quadboard in http://sigma.eruditenow.com Turned out as a looser in the sigma standofs: https://www.bitchute.com/video/DH97lqy3TH0Z/ however has application for math editing / latex https://youtu.be/qLRXs55MpOA usable example: roosevelt.eruditenow.com/bootstrap.html

              Thanks again! I am doing this full time with a very idealistic mindset so any help is greatly apricated! Once again the lobste.rs forum proves it worth! Thank you so much for all of your feedback.

              I am thinking of compiling all my findings into the book so if anybody wants to get an honorable mention that can be arranged! self-ublished via amazon so it’s doable within a few weeks for the first version.

              I am aiming for an iterative cooperative approach here. So crowdsourced if you want to.

              Along with this i want to create a software framework for comparing and creating new innovative models.

              This framework would provide:

              • text prediction
              • os level input support
              • input output file system
              • statistical analysis
              • configuration

              Prototypes will be coded in web to make access as easy as possible. Then a lot of user data could be sourced providing a good source of data for analysis and statistical comparison.

              Everything tied together via a simple JSON/websocket protocol to try to mimic the UNIX philosophy. I know some of you will start to shout UDP at me but bear with me.

              Doing this for a while could enable rapid prototyping of several input methods. The most successful alternatives could then be reimplemented in lower level language for that minimalistic low overhead code we all come to love and adore. This is text entry so an electron application using up half your RAM is a no go simply! But for prototyping using low level languages and handcrafted UI simply doesn’t cut it. Believe me I have tried:

              https://github.com/richard-jansson/veta https://m.youtube.com/watch?v=8kuppby-KJ0 this is quadboard in http://sigma.eruditenow.com Turned out as a looser in the sigma standofs: https://www.bitchute.com/video/DH97lqy3TH0Z/ however has application for math editing / latex https://youtu.be/qLRXs55MpOA usable example: roosevelt.eruditenow.com/bootstrap.html

              Thanks again! I am doing this full time with a very idealistic mindset so help in any shape or form is extremely welcome.

              I’ll update the github repo with some of your input.

              1. 1

                I’ll just mention a project I worked on with a friend a long time ago that was kind of a chorded keyboard.

                The idea was to make a single-hand entry device that didn’t require multiple keys to be pressed at once and didn’t require you to reposition your fingers. After a bit of manual testing we decided that three buttons would be a good number - one each for the three middle fingers.

                Since we didn’t want simultaneous keypresses, every character entered would be a sequence of key presses. To determine how to encode them, we wrote a simple input_server filter (for BeOS) and let it run for a few weeks. The filter stored every keypress in a database, and then we got a list of the most common characters entered.

                Determining the encoding for each character is a solved problem: Huffman coding - this time with a slight wrinkle since we were working with ternary instead of input. I wrote a program that converted the statistics to some sort of header file and an html file with a table of the characters and corresponding codes.

                Then we took the electronics from a joystick and wrote another addon for the input_server to convert the “joystick” keypresses to text input.

                I’ve tried to find the code and html cheat sheet, but it seems to be devoured by the bit rot gods…

                1. 1

                  We seem to have come to very similar conclusions. From the amount of fingers used to the use of compression. Starting out with the amount of fingers used, 3x2 seems optimal based on the following reasoning: I’d argue that market forces have optimized gamepad controllers to something reasonably close to peak optimization. So 4 fingers for analog inputs plus two thumbs for axis input. Leaving 4 fingers for support. I’d argue that even with keyboards there is a long tail on the frequency distribution that is so improbable that they are almost discardable if that makes sense.

                  As far as encoding I wholeheartedly agree. I started out by a grid based system, then I realized that this is just a tree and since I wanted it sorted by frequency basically what you end up with is a lesser used variation of a huffman tree. A non-binary one but the basic theory is still the same. See link below. Typing tests using this seems to imply me being correct.

                  I also always intended to do keylogging on myself to gather statistics but I’‘ve failed to do so!! So I’ve based my statistics on datasets such as selected wikipedia texts, gutenberg books and so on. For the sigma example below when you play one level or book the frequency profile is taken from that book. Somewhat cheating but I expect that a real implementation would have a sufficiently advanced prediction algorithm to make up for whatever difference. That or the fact that the user would have a set trained on himself and or whatever task he is operating on.

                  Where our approaches differ however is that in order not to force the user my approach involves presenting an onscreen referencee. This gives the advantage of making memorization errors easily recoverable. Just as you can easily look down upon your keyboard if you forget where a key is…

                  One project that also is based on a in user compression algorithm is the Dasher input method, which I will cover in another chapter. This one uses the more recent arithmetic encoding, that is of a more continous nature than the more digital huffman encoidng if that makes sense. He uses one digit input here but with continous input. Basically you fly through the space of all possible texts. Where the more probable letters have bigger hitboxes. Look up google talks with David MacKay, Dasher on youtube it is really worth the time, if you are into things like these! Otherwise wait for my comming post / chapter on the subject. He really attacks it with scientific rigor from an information science point of view. Providing estimets on bandwidth in bits per seconds and estimates on bps required for english language based on Shannon and much more.

                  Apart from that having an onscreen HUD allows for input in different languages as well as different compression tables if you will for different task. So me writing code would vary quite dramatically from me writing a friend on messenger and so forth…

                  With a dynamic approach there is a delay between having to go through the visual system instead of relying on what I believe to be responses located in the spine or whatever. I plan into this in more depth later on.

                  A short summary on my n-huffman tree coding. http://sigma.eruditenow.com/theory.html

                  I’d love to make an implementation of your idea and provide it as an example for side by side project. Of course I have a lot to implement so if you are willing to code help it would be very helpful!

                2. 1

                  not to forget for people with accessability issues

                  The article is mentioning VR a lot, so please don’t forget people (like me) who cannot use VR because of VR/motion sickness :)

                  1. 1

                    You bring up a good point! I go on a bit about the genius of John Carmack quite a bit, but his take on the VR situation being primarily a problem of latency is related to this.

                    (He has a writeup and there is a great talk on youtube with him which is brilliant, I’ll get some links when I can be bothered to look up sources, but I’’’m sure you can find it)

                    So compare using a bad architecture for VR with being drunk. While being intoxicated you are well as you can hear from the word poisoned, as such your sensory inputs are not handled properly. Introducing a latency gives exactly the same feeling.

                    As such eliminate lag and delays from a way to deep software stack and the problem is greatly mitigated.

                    One of my hopes is to make a prototype for cheap VR sets that can be used for practical work. I don’t think the problem is throwing money on the issue. Especially since we are dealing with text and simple objects for visualising huge graphs of data. Rather the key would be to create a clever piece of code that hides the latency.

                    1. 0

                      The older headsets used to make me extremely sick, and the new ones have no such effect. There will come a point where nobody gets sick from vr because it will be indistinguishable from your normal visual perception. We may already be there, provided you don’t move your avatar without moving your body :). The focus on VR in the article is likely because VR creates new needs in terms of UX. However I agree if this actually is planning on being a framework for ALL input methods it should represent all input methods. As I noted in a comment above, I was surprised to see that it missed the second most common text input device.

                      1. 3

                        The older headsets used to make me extremely sick, and the new ones have no such effect. There will come a point where nobody gets sick from vr because it will be indistinguishable from your normal visual perception.

                        No amount of graphical fidelity will fool peoples’ inner ears, which is the actual source of the issue for a lot of people. Graphical improvements might even intensify the nauseating impact of eye vs inner ear input conflicts.

                        1. 1

                          Really interesting point you are making. I thought most of it boiled down to a laggy experience, but this sounds really interesting.

                          I do have memorizes of when being really really young and introduced to 3d games they gave me nausea but now that has passed. Wonder why?

                          Surely there must be research on this subject?

                          1. 1

                            I think the main point here is about accessibility and inclusion. While some of the problems might be of technical nature, not all of them will be solved with better tech. Just think of those visually impaired. Using your favorite search engine will find you many results (including actual research :)) if you type in “vr 3d visually impaired”.

                            1. 1

                              The solution for VR interfaces for the blind is simple, better UX, more audio and tactile cues. In fact if I could have an effective non-visual VR solution I would be extremely excited. Any solution that lets me interface with a virtual space without eyes is one that lets me interface with a virtual space while existing in my normal space. Imagine if haptic feedback gave us information about the affordances we were working with, letting us know where in space different buttons and knobs are.

                            2. 1

                              From what I’ve been told motion sickness is your body thinking you’ve been poisoned, as poisons have a similar sensation of disagreement between inner ear and visual information. If you’ve ever been so drunk the room was spinning you might be able to relate to this. Similarly the reason why we can acclimate to 3d games and vr is the same reason we can acclimate to the sea. It takes as much as 36 hours though to acclimate to sea sickness or (vr sickness), and if you’re sensitive to motion sickness you may not desire to overcome that. In rare cases, some people can’t and they have to take an over the counter medicine if they want to enjoy those kinds of activities.

                            3. 1

                              You are completely correct. The issue comes from the disagreement with the visual information and the inner ear information. Removing all disagreement is entirely possible (provided you don’t move in vr without your body moving irl). For general purpose computing this is completely doable. For games, it may not be.

                              1. 1

                                VR does not imply movement decoupled from physical movement. In fact, many VR applications do not allow such movement, opting instead for a teleportation mechanic.

                                1. 1

                                  Yes, I’m aware that teleportation mechanics are an often-implemented mechanic to avoid triggering motion sickness/inner ear issues.

                                  Not sure why your phrasing indicates you’re disagreeing with anything I posted.

                                  1. 1

                                    Not disagreeing, just highlighting that VR doesn’t necessarily need to fool the inner ear to avoid motion sickness, particularly if all motion is also real-world motion.

                              2. 2

                                I don’t think we’ll get to the point where “nobody gets sick from VR”. Plenty of real-life activities give me motion sickness (a long bus ride for example). The motion sickness problem is a problem in our body and VR technology can get better at not exacerbating it as much, but it will not cure it, unfortunately.

                                I’ve tried some of the older and some of the newer VR sets and it’s true that the newer ones are better, but I still can use them for only a few minutes before feeling queasy.

                                1. 2

                                  To clarify, I meant nobody gets more sick from VR than they would from normal motions. Additionally our VR environments can be tailored to those needs so that you don’t need to move around so for general purpose computing. Motion sickness becomes a non issue because you aren’t moving. So for example I get queasy in VR when my body moves but I’m not moving, however I don’t actually have to do that for general purpose computing. I can just sit in a virtual chair managing a wide array of virtual interfaces in front of me. I still benefit from the new affordances however I am not moving and therefore not nauseous throughout the whole experience. I do have roomscale vr so I can get up and walk around without nausea, however typically I find it most comfortable to use while seated. The experience is still much more useful than no VR, even though I can’t necessarily use it to the extent that someone who had no nausea whatsoever could.

                                  1. 1

                                    Maybe for productive purposes it would be better. Basically what I want to produce here is visualisation for code and tree visualisations.

                                    Simplest approach present a grid of 6 workspaces at the same time.

                                    A lot of my own ideas and there are a lot of them. Revolves around visualising keyboards as trees. I do believe that this would have applications for visualising other data in a useful manner. Say File System, DOM object tree and just about anything within comp. sci.

                                    Again I do not expect to actually create the mother of all demos for VR but rather just to extend the bounds of what is thought to be possible ever so slightly and maybe maybe give some inspiration to some one else. Most of the work consists of compiling other peoples thought into one volume.

                                  2. 1

                                    Oh my God, there are so many things! I am loosely following an iterative approach with this project. My thought is to write something post it to great forums (read lobste.rs) adjust the aim. Then rinse and repeat. My aim is to create a publishable book from all of this in the mention.

                                    Half of it consisting of honourable mentions to people that have helped me online, at this rate! Every time I post I feel as if I have missed something huge. Can’t stress enough how much I value all of your input!!

                                    For a skeleton roadmap of the project, check out the README https://github.com/TBF-RnD/alpha/README.md

                                2. 1

                                  If I ever have the funds and time to pursue independent research, building something like a haptic input glove is near the top of the list of things I would like to pursue. Combined with a mobile phone strapped to one eye (with a lens), this would enable me to program anywhere, including while walking. I hate staring at and typing on tiny screens.

                                  It would be awesome if someone beat me to the punch and came up with a good solution.

                                  1. 1

                                    So there are quite a few attempts. Most of them are based around remembering chords. This would mean learning 36 chords for lower case a-z plus numbers. Bear in mind that z appears with a probability of a fraction of a percent so good luck memorizing that. One z appears your flow is completely broken.

                                    To continue to program efficiently you’ll need all the special characters as well so there is that. Not to get started on us who are abroad. My native language is Swedish so what is the chord for: å,ä,ö. I am in Turkey at the moment that adds: ş, ç, ğ, ü, to an otherwise latin alphabet. Then we have greek and Russian and shrug chinese or japanese.

                                    For coding you also need to control direction, copying line, a lot of shortcuts for editing text and compiling.

                                    My solution. Present the keyboard onscreen as a HUD. Change it according to the most probable letters and voila you have a shot of making it happen. To get 2 to the power of 9 or 10 input signals from your hands , you could use chorded input. Even more if you take rotation of the hand and analog input on the fingers.

                                    If properly adopted tapping into the full bandwidth of your hand would basically mean the bandwidth of a serial port / old modem especially with information theory and compression methods being applied.

                                    I’d love for you to help out with this! Help in any shape or form is much welcome. So if you feel like giving a hand at an early stage you might get to do this full time. Think of it this way. Even a one to million chance of creating the most used VR input system, would be a sensible bet as if there is a bullseye there would be a couple of billions of users so why not.