1. 14
  1.  

  2. 3

    One scenario is that the momentum and achievements behind current deep learning is strong enough that we won’t have the same sort of winter as the original AI winter. Rather, we’ll have a retrenchment/hunkering-down. An effort will be made to separate economical use-cases from everything else and shed the pure snake-oil parts.

    Until you have real intelligence or something, there are verifiably going to be a few cases where deep learning can beat every other solution. Old expert systems never went away but expert systems were much more like extensions of ordinary math. Deep learning systems are a new or a different way of doing things. Once their limits become visible, they seem like a kind of icky way to do thing but still. Building a ginormous black box and using sophisticated but ad-hoc training methods to get it predicting thing is only cool when is seems like it will open up new vistas. When/if it becomes clear it’s more like a cul-de-sac, that it works in a particular though impressive used case and then gradually hits diminishing return, it’s kind of ugly (imo). But it’s not going away.

    1. 1

      Well and lets be clear, these strategies actually do solve some problems meaningfully that we couldn’t solve before. The first AI golden age gave us chat bots.

      1. 4

        Remember like three or four years ago when chat bots were going to be the next big thing again? That sure didn’t last long.

    2. 3

      Excellent followup article, even while skirting most of the numerous (and serious) ethical issues faced by the field. Slow motion train wrecks are fascinating to watch, from a safe distance.

      Makes me miss @Shamar, back when he was on his earlier “ridiculous hobby-horse”.

      1. 1

        Ethical issues can be fixed by changing ethics or by fixing the problem, and is something that can be solved. Unlike how, fundamentally, most things humans do can’t be replicated by suitably advanced mathematical formulae

        1. 1

          most things humans do can’t be replicated by suitably advanced mathematical formulae

          Could you link to some actual proof, please? I mean, in principle, not just that it’s hard.

          1. 2

            I’d link you to the original article. Deep learning and all that it entails operates on the premise that if you just create a complicated enough formula, you’ll get what you want. In many surprising cases, that’s true. In the operation of motor vehicles, maybe it’s not.

            1. 2

              The original article is based on the idea that “the people pushing AI stuff don’t realize, or don’t care, that it isn’t as good as they say it is.” It never claims that human intelligence can’t be described using math. It just says it hasn’t been done yet.

              The premise of deep learning is that you can after a couple years, train a neural networks to be as good as the millions of years old human one (or, at least, good enough for purpose).

        2. 1

          Did he get banned for the ridiculous browser caching security talk? Thought he mostly spammed the projects’ bug trackers though, here I remember him as one of the users whose comments I usually liked

          1. 1

            Yeah, he had a… let’s say idiosyncratic view of browser security, was quite outspoken about it, hijacked threads and just wouldn’t back down, so @pushcx kicked him out. Before that he posted a lot of critical stuff about AI. Came across as a bit of a crackpot there too, but seemed concerned about ethics and usually had interesting things to say.

            1. 1

              @Shamar had interesting things to say, though maybe not always with evidence, also his way of saying it was usually not great. He’ll be missed by me, but mostly in the fondness that only memory can provide.

        3. 3

          I found this very interesting. I’ve noticed that in the media “A.I.” seems to often refer to just deep learning, so its failures will definitely affect the public perception of the whole field.

          1. 2

            That’s one of the sad (or funny) parts of the “AI winter” phenomenon: each cycle, there’s a different technology called “AI”. But let’s be honest, public perception owes more to Hollywood movies than scientific consensus. The interesting question is who’s handing out money, and what do they think will succeed or fail. Recently it’s been difficult to get funding for anything but deep neural nets.

            1. 1

              Recently it’s been difficult to get funding for anything but deep neural nets.

              For venture-capital funding, yes, but public funding bodies aren’t putting all their eggs in the deep-learning basket. The NSF, at least, continues to fund a pretty wide range of methods, including logic-based AI, symbolic planning, Bayesian methods, etc.

          2. 2

            I agree computer vision is slowing down, but natural language processing is progressing. See NLP’s ImageNet moment has arrived.

            1. 2

              I would say language processing already had it’s ImageNet moment. That was the moment that Watson won at Jeopardy. I’d actually almost forgotten.

              But I’d also say the way we forgot this stuff correlates with the way is gradually stops mattering. Winning at Jeopardy or classifying a ginormous but well-define image set or winning at the game of go suggests computers are catching up to humans and then a look at the wide range of human capacities suggests otherwise.

              In a sense, with deep learning, there hasn’t been any progress on vision as such. People have simply made progress on adding firepower to a very powerful but narrow pattern recognition system and turned this cannon on various particular problems.

              Of course, Watson’s victory didn’t involve any deep learning. It simply leveraged a few universities’ NLP libraries and choose a situation clever association was most of what mattered.

              1. 2

                My reading of that article is that its authors are eagerly anticipating progress in NLP similar to that seen in CV six years ago, and for similar reasons (we’re figuring out how to use pre-trained hierarchical representations in this domain too). So, it’s not a done deal – and even if the new approach does work out, it’s not a fundamentally novel technique, and there’s no reason to think that applications using it won’t encounter the same difficulties as deep-net CV approaches are now.

                1. 1

                  I agree about everything you wrote, but if NLP is to progress as much as CV did and encounters current CV difficulties, almost everybody will consider that a great success, even if it’s not fundamentally novel, etc. And such success seems likely.

                  By the way, since that article was written, Google BERT, using the same approach, broke all SOTA records by OpenAI GPT.

                  1. 2

                    I agree about everything you wrote, but if NLP is to progress as much as CV

                    Well, I couldn’t say it strongly since I’m not that much of an expert. But I could suggest, say that my hunch is, that NLP may have already “progressed as much as CV” has. In both NLP and CV, you have impressive seeming applications that get you associations between things (puzzles and potential-solutions, texts and sort-of-translations in the case of NLP). In neither CV nor NLP do you have “meaning-related” functionalities working very well.

                    The main thing may be that NLP naturally requires much more meaning-related tasked than vision.

                    1. 1

                      Cool! I look forward to seeing adversarial examples in this domain.

                2. 1

                  the only problem really worth solving in AI is the Moravec’s paradox

                  This isn’t a problem to be solved, it’s a step towards understanding what “intelligence” means. Engineers have always had a hard enough time doing science, even without fundamentally imprecise terminology and a heavy load of unexamined cultural assumptions. A bunch of clueless business people breathing down their necks while the money runs out won’t help any of that.

                  Somewhat ironically, the philosophers have been making more progress here than the computer people, with their turn towards theories of embodied cognition. When the snow melts in another couple decades or so, there will already be new seeds in the ground.

                  1. 1

                    “AI” and what it can do today is pretty exciting if you understand what was thought to be likely not that long ago. If you’re a suit with massively overblown expectations based on little evidence, then yeah maybe the winter is around the corner. I would argue that this winter will be caused entirely by the hypecycle. In the same way that the dot-com bubble happened, yet here we all are talking on the web I suspect that Neural Networks and other techniques will be used in almost invisible ways in the future.

                    1. 1

                      Yes. That’s why “winter” is a useful metaphor: it’s about hype and the funding that comes along with it, rather than the technology itself. Bubbles can deflate gently, or they can pop, but they can’t inflate forever. Government funding is a mixed bag, but I feel pretty confident that the VC funding in this field is almost entirely due to shallow technical understanding and unrealistic expectations on the part of those doling out the money. This can’t last.

                      There has been some great research done under the current funding regime, but that need not (and typically does not) translate directly into engineering success. Commercially successful applications of deep nets to date have been mostly based on fundamental research done like 50 years ago, but with massive amounts of cheap compute power and data.

                      1. 4

                        Commercially successful applications of deep nets to date have been mostly based on fundamental research done like 50 years ago, but with massive amounts of cheap compute power and data.

                        As someone who made such commercially successful application, I disagree. Yes, CNN is old, but various enhancements are not. Initialization (Glorot, He) is better, activation (ReLU) is better, optimization (Adam and adaptive learning rate methods) is better, regularization (Dropout, etc.) is better, normalization (batch normalization) is better, architecture (residual connection) is better, the list goes on and on. Value of these enhancements is obvious as soon as you try to make your application work without them.

                    Stories with similar links:

                    1. AI winter - update via orib 4 months ago | 35 points | 6 comments