Using Python to Code by Voice

Your comment reminded me of an interesting video I saw a while back. The video is a talk [0] by a programmer explaining and demonstrating how he developed and used a voice recognition system to program in emacs, after a significant RSI injury. The commands and syntax he uses to interact with the system might be a decent approximation of the custom English syntax you suggest. [0]: ▶ this video

in “IBM is not doing "cognitive computing" with Watson (2016)” · View on HN

See this: ▶ this video > In a fast-paced live demo, I will create a small system using Python, plus a few other languages for good measure, and deploy it without touching the keyboard. The demo gods will make a scheduled appearance. I hope to convince you that voice recognition is no longer a crutch for the disabled or limited to plain prose. It's now a highly effective tool that could benefit all programmers. Turns out it's doable, you just need to invent your own language.

in “Something is wrong when the ‘telephone app’ on your phone becomes 3rd party” · View on HN

IMHO I don't think it would fly. The most difficult things of programming are too abstract for that. You may be interested in the guy that did all his coding with voice for a while because he couldn't use his hands, and still uses it to this day about half of the time. ▶ this video

in “How a Blind Developer uses Visual Studio [video]” · View on HN

[This talk by Tavis Rudd]( ▶ this video ) about a system he used when his hands were afflicted with RSI is pretty interesting. Not entirely a no-hands approach, but I can't seem to remember the developer who made his first app in the hospital by typing with one/two fingers. He wrote a blogpost about his process that was really inspiring.

in “Ask HN: Anyone using a “no hands” setup for programming?” · View on HN

There was an interesting talk at PyCon 2013 about programming using only your voice: ▶ this video Might be worth looking into for your case. It's worth mentioning that the guy did this for himself due to severe RSI in the hands, but there was another talk on using a stenographic keyboard to program and I remember this guy stood up and said he wished he'd known about it earlier as it's another approach for people who suffer from RSI since there's less typing.

in “Ask HN: I can barely read, is there a way I could still learn to program?” · View on HN

You might be interested in watching this talk: ▶ this video Although, if I recall correctly, he uses Dragon Naturally Speaking, which is a commercial product.

in “Annyang.js – Let visitors control your site with voice commands” · View on HN

One of my back-burner projects is putting some esp32s or similar around the house with microphones, streaming whatever audio they hear to Dragon NaturallySpeaking like in this video ▶ this video , but hooked up to home assistant scenes / automations instead of emacs commands.

in “Ask HN: Private Alternatives to Alexa?” · View on HN

Xah first wrote this over a decade ago. He’s had some RSI issues, and the real problem is simply typing in any form. I remember reading several of his articles thinking that surely within a decade, we’d have better voice assist for programming. Tavis Rudd’s demo looked promising in 2013: ▶ this video I’m not sure what’s the best solution. Here are a few that I known of: [link] [link] [link]

in “Dvorak vs Colemak (2010-2020)” · View on HN

I wonder if sub-vocal will happen anytime soon, and if it does, I am still not sure how it will solve voice being sub-optimal for some types of text. One example is plain old programming, and needs a fairly elaborate setup. The best example I've been able to find was Tavis Rudd's presentation [1], which seems to be a decent solution. I have tried to more or less replicate this setup, but found it less satisfactory. Aside from it demanding near-perfect audio quality to work remotely well in general (as well as a heavy time investment, initial and ongoing), you frequently run into edge cases where it just doesn't want to understand something you're saying. After the 5th or 6th mis-transcription of some specific phrase, you tend to just give up and reach for the keyboard, RSI be damned. (And this as a native English-speaker, with a relatively neutral accent.) I can definitely imagine better fleshed out solutions to either voice recognition for programming, or programming languages whose syntax lends themselves to easier and more productive voice recognition. I could see some definite productivity enhancements if it were achieved. [1] ▶ this video

in “Bill English has died” · View on HN

Tavis Rudd uses voice recognition to do his editing (in Emacs, no less): ▶ this video The key is that he basically made up his own language to handle some things like special characters.

in “Show HN: Emacs Anywhere” · View on HN

First, it doesn't feel unpleasant as I do it. There are a few reasons why it might look that way. I have very little practice at this point, especially considering I'm still changing my grammars regularly, so the editing video there is more a technical demo of the software's performance in spite of my own awkwardness while learning. That was my first and only take. I'm basically learning to type again, with cache misses as I try to remember commands and keep track of my position (like trying to remember where keys are, or the next note in a song as you pluck it out). It's also possible the most accurate form of voice input does not sound like natural language. Finally, it's a different mental exercise to copy text that is already written versus writing and editing my own code, so I'm triply out of my element when recording these early videos. Compare to Tavis Rudd's video from 2013, where he pauses between every command: ▶ this video I've recorded my own version of his demo today for comparison: [link]

in “How Apple fumbled a five-year lead in voice control” · View on HN

I have been suffering from RSI in both arms for a decade now. It started after coding 10+ hour days working at a game company. The pain ebbs and flows with usage, but the inflammation and tightness has never disappeared completely. Over the years, I've met a number of people who also also suffer from RSI in various forms. Fortunately, the majority of people I talk to suffer for some period of time, but do eventually recover, and are able to continue on with their lives having to be carful to avoid future flareups. A handful though, including myself, seem to fall into the "for life" category. I've tried everything you can think of to varying results: * 15+ doctors in various specialties * Physical therapy * Massage therapists * Chiropractors * Acupuncture * Most every ergonomic keyboard and pointing device on the market * Various desk/chair/keyboard tray ergonomic adjustments * Addressing TMS (See books by Dr. Sarno) * Taking a year off of work (multiple times) * NSAIDs This is resulting in me spending a lot of time looking into other career potabilities, but I keep ending up back writing code to some extent. Voice coding is literally the first thing that comes to everyone's mind when you tell them you have arm pain, but it's a dead end in my mind. I tried to follow in the footsteps of Tavis Rudd, but I just ended up with voice strain from talking too much. You might have better luck though: ▶ this video At one point in my life, I had an intern who knew how to code type for me. No

in “Ask HN: How many of you are suffering from RSI or similar?” · View on HN

I also have hand problems. I use speech recognition to type everything, including code, and a game controller to move around the mouse. The controller I'm currently using, a PS4 controller, has a trackpad in the middle, so I was able to mash that against my face and give her technique a shot. It works better than you'd think. The PS4 trackpad isn't exactly brilliant, but I can move around the mouse and click on what I want to with some accuracy. Of course, the trackpad on the controller is very small and not very accurate, so it's not really practical for artwork or anything. But, with a better, larger trackpad, I can imagine this technique actually working. I might give it a shot at some point. I am a bit worried about the inevitable neck and nose pain, though. I wish she had gone into a little more detail about how she avoids that. Maybe she just has a neck of steel? For the curious, these are some other resources I've found about people working around RSI. Most of these are about using Dragon NaturallySpeaking to code by voice, since that's what I'm most interested in, but I think it's still interesting. There really needs to be a list somewhere for open-source workarounds to disabilities. To the best of my knowledge, there really isn't one. Natlink + Dragon NaturallySpeaking: (NatLink, which lets you make custom speech commands for Dragon in Python, is currently being developed at [link] , but that site's pretty incomprehensible. The original author's site is at [link] .

in “Look, no hands” · View on HN

> I'd bet that, unless you had an optimized "dialect" for coding or a language designed to be efficiently spoken, a keyboard will beat a human speaking code every time. More or less what this guy did: ▶ this video It seems to me that languages with a more regular syntax are going to have a dramatic advantage here.

in “What VR could, should, and almost certainly will be within two years [pdf]” · View on HN