Automating YouTube with yt-dlp
I spent last weekend thinking through and implementing some ideas that have been rattling around my head for months. I wanted a way to automatically download videos from every YouTube channel I followed and to download any ad hoc video I came across in my day-to-day life that I wanted to watch later. What I didn’t want was to manage any of this myself. I had been using my RSS reader to get new videos from the channels I subscribed to, but I had to then use my RSS reader, filter through the items, add each item to my YouTube list, then initiate the download command in the terminal myself. I explained how I did this in this post from November, but again, I wanted something more automated.
So I figured out how to make it more automated… by automating most of it!
Here’s how it works.
What you need
I won’t go through what yt-dlp is or why I set it up the way I did because I explained that in my previous post, so please read that if you need a quick explainer. What this post will go through is how to setup yt-dlp to “subscribe” to your own list of YouTube channels and to download your own list of ad hoc (or watch later) videos automatically.
I’m using a 2018 Intel-based Mac mini with
yt-dlp installed per their installation instructions. That means this setup is intended for people with a device running macOS. Optionally, to make this setup just that tad bit more awesome, you will also need:
- YouTube-Agent.bundle (a Plex plug-in to add YouTube metadata to each video)
- Apple Shortcuts
First things first
Before we can automate anything, we have to create a few files first. In my previous post, I created an
archive.md file to log all the videos I’ve downloaded. This is important. The archive file is the most important piece to this process, so create it now. Good? Okay. I then created a file called
subscriptions.md and placed that in my main
YouTube folder where all my files live. In this
subscriptions.md file, I added each channel I wanted to subscribe to, one channel per line. For example, a snippet of my file looks like:
https://www.youtube.com/@PeopleMakeGames https://www.youtube.com/@PeterMcKinnon https://www.youtube.com/@PickUpLimes https://www.youtube.com/@primitivetechnology9550
Essentially, that’s all you need. There’s nothing fancy to this. Keep in mind that this will download all the videos these channels produce, including shorts and other things. I don’t mind that, so this is good enough for me.
The next thing, and this is where the magic happens, is to simulate your
yt-dlp command and add each file to your
archive.md file. To do this, add both the
yt-dlp not to download anything and
--force-write-archive adds all the videos from all your subscriptions to your
archive.md file. Do you see why this is important? When you’re ready, you can now run your command. Mine looks like this:
yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" --simulate --force-write-archive -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/subscriptions.md'
Depending on how many subscriptions you’ve added, this process could take a long time, so go outside, hang out with friends, read a book, do something else while this process runs. By the end of this, your
archive.md file will most likely be huge. Mine has over 15,000 lines on it but barely cracks 300 KB. So really, you can leave this file alone forever and it’ll never cause you any issues (knock on wood).
This takes care of my subscriptions, but what about any ad hoc videos I may want to download and watch later? Well, I mostly answered that in my previous post (you should really read that post, it’s pretty good). Specifically, the important part is creating a
youtube.md file and including the
--batch-file command that points to it. Then, as I go along with my regular internet surfing life, I add links to any videos I want to watch later to this file using a simple Shortcut and my system automatically downloads it. How?
Well, I’m glad you asked!
Let’s automate the shit out of this
Time to create a few more files. I created four and called them:
youtube.plist. I placed these files in my main
YouTube folder, but the
plist files can be moved or copied to
/Library/LaunchAgents, which we’ll do later. I’ll focus on my subscriptions first.
subscriptions.sh file, I added:
#!/bin/bash /usr/local/bin/yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/subscriptions.md'
This is similar to the code above but without the
--force-write-archive commands. This is the real deal command, so if you haven’t run the
--simulate command, then you will be downloading everything in your
subscriptions.md file. Maybe that’s what you want, so you do you. I’m not your mom.
subscriptions.plist file is where the magic happens (lots of magic happening today). This file contains this bit of code:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>subscriptions</string> <key>ProgramArguments</key> <array> <string>/path/to/subscriptions.sh</string> </array> <key>StartInterval</key> <integer>21600</integer> <key>disabled</key> <false/> </dict> </plist>
I’ve simplified my
plist file compared to say, Jason, which I took a lot of inspiration from, but I customized his setup to fit my needs. He includes a few log files that are good practice to include, but I like living on the edge, so I didn’t include them. If you’d like to add them, include something like this in your
<key>StandardErrorPath</key> <string>/path/to/ytdl_error.log</string> <key>StandardOutPath</key> <string>/path/to/ytdl_st_out.log</string>
A few things to note:
Label key is essential if you plan on adding more than one file to the
LaunchAgents folder. I gave this one a label of
subscriptions to differentiate it from the others.
I set my
StartInterval to 21600, which is 21,600 seconds, or 6 hours. I set it to this because this is not something I want running all the time, and it’s not something I want to be checking my downloads folder for to see if there’s something new. That’s the behavior I wanted to eliminate, so putting it to 6 hours has helped me pull away from my devices while still ensuring I have something new to watch a few times throughout the day. Honestly, I could set this to 24 hours and still be happy, which might be something I do in the future. Stay tuned.
Once this is all done and setup to your liking, navigate to
/Library/LaunchAgents and copy your
plist file into it. macOS might notify you that
subscriptions.sh is an item that can run in the background. That’s exactly what you want. You may also have to add
bash to your
Files and Folders section in the
Privacy & Security pane in your
System Preferences. This ensures
bash has permission to run on your system. Boy did that one have me scratching my head for a while.
And that’s essentially it. Rinse and repeat with the
#!/bin/bash /usr/local/bin/yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/youtube.md'
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>youtube</string> <key>ProgramArguments</key> <array> <string>/path/to/youtube.sh</string> </array> <key>StartInterval</key> <integer>60</integer> <key>disabled</key> <false/> </dict> </plist>
And you have a fully automated YouTube setup. Note, I set the
StartInterval to 60 because I don’t mind this running all the time because I don’t add too many ad hoc videos anymore, so when I do, I’d like to watch it sooner rather than later.
Because you took the time to add all your channel videos to your
archive.md file, from this point forward, this setup will ensure you only download new videos as they’re posted to your subscriptions. Pretty nice, right?
But there’s one more piece to this that’s like icing on the cake. It’s not required, but I feel like this really completes the entire setup.
Tidying things up
The last thing I did was use a few apps to make this entire experience a tad nicer. First, I launched Hazel and added my
YouTube folder. I then created… many rules:
- Go into subfolders. I set the
Run rules on folder contents. This rule ensures every rule hereafter is run within all the folders contained in my main
YouTubefolder. This is important because:
- Delete empty folders. Each rule hereafter creates lots of folders, and this rule tidies things up by deleting any empty folder. This rule is also simple. First, I set the
Folder, and the
0. I then
Moveany matches to the
Trash. This is very important because:
- Sorting subscriptions by folder. I created a rule for each and every subscription I have. The rule here is simple. I set the
containsthe name of my subscription, e.g. MKBHD. I then used the
Sort into subfolderoption to sort my videos into its own folder. The pattern I used was
Subscriptions ▸ <name of subscription>. Why do this? Because I don’t have time to watch TV all the time, and sometimes I want to binge through one channel’s output, and this makes that easier. After adding a rule for each of my subscriptions, I created my final rule:
- Recently Added. This rule moves any file that hadn’t already been matched to a folder I called
Recently Added. Again, this rule is very simple. All it has is
Sort[s] into subfolderany matches to my folder called
Having this run 24/7 removes yet another cognitive weight from my life and just keeps things tidy. Work smarter not harder, right? And finally, on to the final step.
In Plex (you did download and install it, right?), I downloaded the YouTube-Agent.bundle plug-in and installed it per their installation instructions. This plug-in requires a bit of setup before you can take advantage of it. You have to create your own YouTube API key, which isn’t too tough, and each one of your movies needs to have the YouTube ID in its filename. Review my
yt-dlp command and you may notice I’ve added
[%(id)s] to my filename. This option is there to ensure
YouTube-Agent.bundle adds the correct metadata to my movies.
Finally, I created two libraries, one called
Subscriptions and the other called
Recently Added. I pointed each to its respective folder, and in the
Advanced section, I chose the
YouTubeMovie plug-in in the
Agent option. This is also where I added my API key.
If everything went well, then you truly have a (mostly) automated YouTube setup. I say mostly because I still have to add any ad hoc videos to my
youtube.md file, but it literally takes 2 seconds to do so and that’s the only actual work I have to do. For the most part, I haven’t had to troubleshoot this setup yet, but knowing the pace of technology, I will have to sometime in the future.
Until then, I’ve been really enjoying this setup. Dare I say, it feels a tad magical, and I love it. Maybe you will, too.