1. 35
  1. 50

    This is a RIIR where I don’t see why you’d do that. The author seems to be serious about this, trying to add multiple sites. In my experience I always delegate the youtube & co crawling stuff to youtube-dl, because it has the biggest momentum and thus the best & fastest support for all those individually changing websites. Something you need crowdsourcing and a lot of maintainer time for.

    1. 17

      Yeah, the biggest value in youtube-dl is not in the git repo, it’s the community of people reverse-engineering websites and providing timely updates. Rewriting the engine in Rust is a simple matter of programming, but rebuilding that community might very well be impossible.

      1. 1

        can’t blame em for trying! or can you…

      2. 11

        Agree completely that it’s not necessarily “needed” by the community, but maybe the author simply finds the problem interesting. I’ve reimplemented stuff needlessly before. Not so seriously to open any of it up to outside contribution, but maybe the author just feels like doing that too.

        1. 7

          I would be tempted to use it just because it doesn’t rely on python as a dependency and ships as a single executable.

          1. 3

            fair enough, I also would like some changes to the “API” yt-dl has, but its definitely easier to hack around in for everybody when its written in python

            Edit: I just realized this project specifically doesn’t want to add any format/quality functions, so it’s kinda useless for most things

            1. 2

              I’d guess that the built with python is one of the reasons for the popularity of ytdl

              1. 1

                I can see that being the case - easy to write a new handler for a domain.

            2. 3

              To elaborate: I’ve got a yt-dl service (yes in rust) on my servers running since 2015 on kinda the same code since when it started. The only thing done daily in the background is updating youtube-dl, everything else just stays the same. But without that it won’t work.

              1. 3

                One possible reason it might be good to have is if the main youtube-dl project gets purged from the internet by copyright-holding corporations. It happened once, it could happen again. This less famous alternative, or simply different project run by different people, might be available in times and places where youtube-dl is not.

                Obviously it would be better if youtube-dl were to be generally available, and we should do what we can to make that happen, but some redundancy may be useful and wise.

                1. 7

                  youtube-dl was never really purged. afaik it was always accessible on repos of major distros, though the development infrastructure was all but gone.

                  if you are thinking of a more comprehensive attack by a group of tech giants or the government, it seems more likely that youtube will eventually not support methods of access that allow downloading files, rendering alternatives equally useless. I could be wrong though.

                2. 1

                  I started yaydl when youtube-dl had the massive disadvantage of having been taken offline. I chose Rust over Python for three reasons:

                  1. I can’t stand Python, neither its syntax nor its behavior when updating from 3.x to 3.y.
                  2. I needed an excuse to try Rust. (I try to know as many languages as possible - you’ll never know when they could be useful.)
                  3. I wanted to make it as easy as possible to add new sites, and Rust’s traits seem to be an awesome way to do that.

                  I tried to read youtube-dl’s source code and I could not figure out how to add new sites there easily, so that’s that… :-)

                  1. 1

                    I needed an excuse to try Rust.

                    To be fair that’s how my yayd project started, but since its just running since forever I don’t think it was a bad choice.

                    If I’d have a wish: Please add a format flag, 99% of my users want to download an mp3 of podcasts and playlists containing them (or songs). I’m currently using a hack of retrieving filename, then audio, then video to get live progress from youtube-dl & ffmpeg, which also means I’m doing the conversion manually. (And ffmpeg wants its own line endings Especially for audio you may have to use a different format than the “best” as youtube has some better audio codecs in others.

                    Definitely good luck with your project and if it works I’d be glad to adopt it, so I can drop all these std pipes. I think nobody here wants you to stop doing your hobby projects, its just a warning (at least from me) that yt-dl is a lot of work and creating a half-baked effort due to limited resources can leave a bad taste behind.

                    tried to read youtube-dl’s source code

                    To be fair it’s kinda bad, especially for the youtube section

                    1. 1

                      Hmm. I understand your wish, but I really don’t want that. There already are so many tools which do that. However, as yaydl calls FFMPEG anyway (if you tell it to), it could be easy to add that.

                      I doubt that this will be considered before 1.0 though.

                      Yes, my project needs manpower. I noticed that when fixing YouTube support recently. I was hoping that Rust would be popular enough for this to change. It does not, I guess.

                      1. 1

                        It does not, I guess.

                        Well there are many people working on different things, I’d say if you combine the overall amount of people involved in all the different projects (linux,OS,embedded,async,crypto,gamedev,hypervisor,databases,web,cs..) there are a ton of contributors. But youtube-dl is kind of a working solution where you don’t really gain anything from adding rust (security,performance..). And also probably also the next addition to my list above.

                        1. 1

                          The one thing that Rust adds for me is that the binary will survive a Python upgrade. :-) Also, a slightly better performance is a feature in my opinion, even if it only affects the general startup time.

                          Yes, youtube-dl is still a fine piece of software, but it has a very different intention from yaydl which aims to be as little complex as possible…

                          1. 1

                            will survive a Python upgrade

                            Is a python upgrade that bad ? Just using the debian python3 and haven’t had any problems for my tiny stuff.

                            1. 1

                              I don’t use Linux. On most systems I own which run Python, every upgrade breaks one or more pip modules, I’d have to reinstall them and this is something I do not want. (I know about virtualenv, but in my opinion, a language that requires a virtualenv to avoid its dependencies to fall flat on their - or your - face is broken by design.)

                              I cannot say for sure that I am not just too dumb for Python. :-) But Rust does not have these problems at all.

                              1. 2

                                tbf I also tried using virtualenv and gave up eventually

                3. 12

                  One thing that kinda like you in the eye under non-features:

                  No output quality choice. yaydl assumes that you have a large hard drive and your internet connection is good enough, or else you would stream, not download.

                  I would have thought that if I had a good connection, I wouldn’t care about downloading and could just stream at any time.

                  1. 3

                    When I’m downloading something then because I want a different quality or target format (mp3), but this is specifically unsupported by them. And more often that not you’ll realize there’s not “highest” version because there are at least two different formats that provide the same resolution and which you can’t compare just by their bitrate..

                    1. 1

                      Yeah, that’s all well, I agree it’s not as complete feature-wise. What I’ve meant is simply that they say “you have good connection or else you would stream”. And I thought, “well, if I had a good connection, then I might as well just stream, in whatever quality I choose”. Just seemed like a bogus argument for not implementing quality picker.

                    2. 2

                      I would have thought that if I had a good connection, I wouldn’t care about downloading

                      I have a good connection (Gbps fibre) and I frequently download rather than stream because YouTube mid-stream ads are now a plague - I’ve seen videos with 5 ad breaks in the first 15 minutes. Or I’m trying to listen to an ambient mix by Cryo Chamber and I get an un-skippable 30s ad for some new severely non-ambient dance pop act. Or just trying to follow a tutorial where every 2 minutes I have to stop what I’m doing and get the cursor over to “Skip Ad” because if I don’t, I’m going to get 3 minutes of [whatever guff it is this week] before the content comes back.

                      1. 5

                        I’ve been able to avoid ads with ublock origin on desktop and sponsorblock for ‘native’ ads. maybe yt is able to sneak past ublock but i haven’t seen it yet.

                        1. 2

                          I recommend also the “Enhancer for YouTube” on the firefox plugin store to disable autoplay, theme youtube, have scroll-volume-control, select certain video qualities and most importantly set a default volume as youtube ignores media.default_volume. (You can disable all the additional controls in the settings if you’re overhelmed)

                        2. 2

                          I’m subscribed so I don’t get to see ads usually, but yeah, I see how it can make a differnce.

                      2. 5

                        As the author of music player software that wants to have an automated import-from-youtube feature, I have some opinions about this.

                        The value-add of youtube-dl is the fact that it is regularly updated by humans doing labor. The fact that it is in Python or in Rust is actually problematic for this use case. Ideally the music player software would be able to fetch a youtube-dl update without having to rebuild the music player, and also ideally without the music player having a dependency on Python. Also, ideally, since new youtube-dl code would be fetched and run automatically, it should not need to be trusted and could be run in a sandbox. Perhaps something like WASI, if it supported networking, would be a good fit. In this case Rust could be used, but I don’t see why you would do that, since the WASI program could be run in a sandbox, which provides its own memory safety, making Rust’s borrow checker an unnecessary complication.

                        In conclusion, my selfish desires are:

                        • WASI hurry up and add networking please
                        • youtube-dl to switch from Python to something that can compile to WASI, and include wasi binaries with releases
                        • yarr harr fiddle-dee-dee, being a pirate is alright with me
                        1. 3

                          Write it in Haskell! You could probably design something clever using the ‘monad as DSL’ pattern so that your plugins don’t actually have any IO, and use Safe Haskell to make sure they’re not sneaking in an unsafePerformIO. Then the main binary just ‘executes’ your DSL, using a set of predefined primitives (fetch this URL and give it to the plugin, fetch this URL and save it to disk as the video, etc) that you know are safe.

                          Of course, Haskell isn’t a well-known language, I don’t think Safe Haskell has full sandbox guarantees, and I don’t know if loading plugins at runtime that were compiled on a separate machine is a thing you can do. But it’s a fun thing to think about!

                        2. 2

                          In the SiteDefinition there is find_video_direct_url() function. Is it a good abstraction, generic and flexible enough? What if some site requires composing the video from several pieces or does some other kind of streaming? Some sites may also require specific User-Agent headers or some cookies.

                          Maybe the SiteDefinition could return an input stream of bytes or get an output stream of bytes as a parameter and write to it.

                          BTW: I like your Code of Merit.

                          1. 1

                            I am not the author though

                            1. 1

                              You are right, downloading from several parts is not supported yet, resulting in a lack of support for (e.g.) YouTube playlists. I outlined a draft to fix this in issue #1 and I hope that someone helps me to make it. :-) Thank you for the suggestion.

                            2. 1

                              good luck!