May 21, 2023

Automating YouTube with yt-dlp

Notes

I spent last weekend thinking through and implementing some ideas that have been rattling around my head for months. I wanted a way to automatically download videos from every YouTube channel I followed and to download any ad hoc video I came across in my day-to-day life that I wanted to watch later. What I didn’t want was to manage any of this myself. I had been using my RSS reader to get new videos from the channels I subscribed to, but I had to then use my RSS reader, filter through the items, add each item to my YouTube list, then initiate the download command in the terminal myself. I explained how I did this in this post from November, but again, I wanted something more automated.

So I figured out how to make it more automated… by automating most of it!

Here’s how it works.

What you need

I won’t go through what yt-dlp is or why I set it up the way I did because I explained that in my previous post, so please read that if you need a quick explainer. What this post will go through is how to setup yt-dlp to “subscribe” to your own list of YouTube channels and to download your own list of ad hoc (or watch later) videos automatically.

I’m using a 2018 Intel-based Mac mini with yt-dlp installed per their installation instructions. That means this setup is intended for people with a device running macOS. Optionally, to make this setup just that tad bit more awesome, you will also need:

Plex
YouTube-Agent.bundle (a Plex plug-in to add YouTube metadata to each video)
Hazel
Apple Shortcuts

First things first

Before we can automate anything, we have to create a few files first. In my previous post, I created an archive.md file to log all the videos I’ve downloaded. This is important. The archive file is the most important piece to this process, so create it now. Good? Okay. I then created a file called subscriptions.md and placed that in my main YouTube folder where all my files live. In this subscriptions.md file, I added each channel I wanted to subscribe to, one channel per line. For example, a snippet of my file looks like:

https://www.youtube.com/@PeopleMakeGames
https://www.youtube.com/@PeterMcKinnon
https://www.youtube.com/@PickUpLimes
https://www.youtube.com/@primitivetechnology9550

Essentially, that’s all you need. There’s nothing fancy to this. Keep in mind that this will download all the videos these channels produce, including shorts and other things. I don’t mind that, so this is good enough for me.

The next thing, and this is where the magic happens, is to simulate your yt-dlp command and add each file to your archive.md file. To do this, add both the --simulate and --force-write-archive commands. --simulate tells yt-dlp not to download anything and --force-write-archive adds all the videos from all your subscriptions to your archive.md file. Do you see why this is important? When you’re ready, you can now run your command. Mine looks like this:

yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" --simulate --force-write-archive -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/subscriptions.md'

Depending on how many subscriptions you’ve added, this process could take a long time, so go outside, hang out with friends, read a book, do something else while this process runs. By the end of this, your archive.md file will most likely be huge. Mine has over 15,000 lines on it but barely cracks 300 KB. So really, you can leave this file alone forever and it’ll never cause you any issues (knock on wood).

This takes care of my subscriptions, but what about any ad hoc videos I may want to download and watch later? Well, I mostly answered that in my previous post (you should really read that post, it’s pretty good). Specifically, the important part is creating a youtube.md file and including the --batch-file command that points to it. Then, as I go along with my regular internet surfing life, I add links to any videos I want to watch later to this file using a simple Shortcut and my system automatically downloads it. How?

Well, I’m glad you asked!

Let’s automate the shit out of this

Time to create a few more files. I created four and called them: subscriptions.sh, subscriptions.plist, youtube.sh, and youtube.plist. I placed these files in my main YouTube folder, but the plist files can be moved or copied to /Library/LaunchAgents, which we’ll do later. I’ll focus on my subscriptions first.

In my subscriptions.sh file, I added:

#!/bin/bash
/usr/local/bin/yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/subscriptions.md'

This is similar to the code above but without the --simulate and --force-write-archive commands. This is the real deal command, so if you haven’t run the --simulate command, then you will be downloading everything in your subscriptions.md file. Maybe that’s what you want, so you do you. I’m not your mom.

The subscriptions.plist file is where the magic happens (lots of magic happening today). This file contains this bit of code:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>subscriptions</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/subscriptions.sh</string>
    </array>
    <key>StartInterval</key>
    <integer>21600</integer>
    <key>disabled</key>
    <false/>
</dict>
</plist>

I’ve simplified my plist file compared to say, Jason, which I took a lot of inspiration from, but I customized his setup to fit my needs. He includes a few log files that are good practice to include, but I like living on the edge, so I didn’t include them. If you’d like to add them, include something like this in your plist file:

<key>StandardErrorPath</key>
<string>/path/to/ytdl_error.log</string>
<key>StandardOutPath</key>
<string>/path/to/ytdl_st_out.log</string>

A few things to note:

The Label key is essential if you plan on adding more than one file to the LaunchAgents folder. I gave this one a label of subscriptions to differentiate it from the others.

I set my StartInterval to 21600, which is 21,600 seconds, or 6 hours. I set it to this because this is not something I want running all the time, and it’s not something I want to be checking my downloads folder for to see if there’s something new. That’s the behavior I wanted to eliminate, so putting it to 6 hours has helped me pull away from my devices while still ensuring I have something new to watch a few times throughout the day. Honestly, I could set this to 24 hours and still be happy, which might be something I do in the future. Stay tuned.

Once this is all done and setup to your liking, navigate to /Library/LaunchAgents and copy your plist file into it. macOS might notify you that subscriptions.sh is an item that can run in the background. That’s exactly what you want. You may also have to add bash to your Files and Folders section in the Privacy & Security pane in your System Preferences. This ensures bash has permission to run on your system. Boy did that one have me scratching my head for a while.

And that’s essentially it. Rinse and repeat with the youtube.sh file:

#!/bin/bash
/usr/local/bin/yt-dlp -P "/path/to/YouTube/" -P "temp:tmp" -P "subtitle:subs" -o '%(uploader)s-%(upload_date)s-%(title)s [%(id)s].%(ext)s' --download-archive '/path/to/archive.md' -f 'bestvideo+bestaudio/best' --sub-langs all,-live_chat --embed-subs --yes-playlist --batch-file '/path/to/youtube.md'

And the youtube.plist file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>youtube</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/youtube.sh</string>
    </array>
    <key>StartInterval</key>
    <integer>60</integer>
    <key>disabled</key>
    <false/>
</dict>
</plist>

And you have a fully automated YouTube setup. Note, I set the StartInterval to 60 because I don’t mind this running all the time because I don’t add too many ad hoc videos anymore, so when I do, I’d like to watch it sooner rather than later.

Because you took the time to add all your channel videos to your archive.md file, from this point forward, this setup will ensure you only download new videos as they’re posted to your subscriptions. Pretty nice, right?

But there’s one more piece to this that’s like icing on the cake. It’s not required, but I feel like this really completes the entire setup.

Tidying things up

The last thing I did was use a few apps to make this entire experience a tad nicer. First, I launched Hazel and added my YouTube folder. I then created… many rules:

Go into subfolders. I set the Kind to Folder and selected Run rules on folder contents. This rule ensures every rule hereafter is run within all the folders contained in my main YouTube folder. This is important because:
‌Delete empty folders. Each rule hereafter creates lots of folders, and this rule tidies things up by deleting any empty folder. This rule is also simple. First, I set the Kind to Folder, and the Sub-file/folder Count to 0. I then Move any matches to the Trash. This is very important because:
Sorting subscriptions by folder. I created a rule for each and every subscription I have. The rule here is simple. I set the Name to contains the name of my subscription, e.g. MKBHD. I then used the Sort into subfolder option to sort my videos into its own folder. The pattern I used was Subscriptions ▸ <name of subscription>. Why do this? Because I don’t have time to watch TV all the time, and sometimes I want to binge through one channel’s output, and this makes that easier. After adding a rule for each of my subscriptions, I created my final rule:
Recently Added. This rule moves any file that hadn’t already been matched to a folder I called Recently Added. Again, this rule is very simple. All it has is Kind set to Movie and it Sort[s] into subfolder any matches to my folder called Recently Added.

Having this run 24/7 removes yet another cognitive weight from my life and just keeps things tidy. Work smarter not harder, right? And finally, on to the final step.

In Plex (you did download and install it, right?), I downloaded the YouTube-Agent.bundle plug-in and installed it per their installation instructions. This plug-in requires a bit of setup before you can take advantage of it. You have to create your own YouTube API key, which isn’t too tough, and each one of your movies needs to have the YouTube ID in its filename. Review my yt-dlp command and you may notice I’ve added [%(id)s] to my filename. This option is there to ensure YouTube-Agent.bundle adds the correct metadata to my movies.

Finally, I created two libraries, one called Subscriptions and the other called Recently Added. I pointed each to its respective folder, and in the Advanced section, I chose the YouTubeMovie plug-in in the Agent option. This is also where I added my API key.

If everything went well, then you truly have a (mostly) automated YouTube setup. I say mostly because I still have to add any ad hoc videos to my youtube.md file, but it literally takes 2 seconds to do so and that’s the only actual work I have to do. For the most part, I haven’t had to troubleshoot this setup yet, but knowing the pace of technology, I will have to sometime in the future.

Until then, I’ve been really enjoying this setup. Dare I say, it feels a tad magical, and I love it. Maybe you will, too.