Use sed to bulk edit Obsidian content

Introduction

The Obsidian knowledge management tool stores it’s content in Markdown, a lightweight markup language where the underlying content is plain text. And there is one thing that UNIX (and by inference Linux) got very good at, and that was managing plain text files.

So if Obsidian does not have an edit feature you need, you might be able to craft some UNIX commands to do it for you.

NOTES

• WARNING: These examples can rip through your vault making changes, so make sure you have a backup and test before using (more on testing later)

• Given the caveat above you may prefer to use the tag wrangler for a task like this instead

• This post assumes you understand how to use regular expressions to locate text in documents and are comfortable using the command line.

This 1980’s video introduction to UNIX explains some of the concepts we’ll be using and I suggest watching this clip. Even if you are familiar with the concepts, it’s nice piece of nostalgia.

Making changes to an Obsidian vault can be split up into three tasks:

1. Generate a list of all the Markdown files in our vault
2. Locate any information inside that we want to change, for example the tag used in a task description
3. Change the specific text, and only the specific text, to a new value

To help illustrate how to approach this, I am going to globally change the Obsidian-tasks tag from #todo to #td. However this approach can be used to edit any collection of text files. You can run these examples on Linux, macOS or WSL (Windows Subsystem for Linux).

Generate a list of all the Markdown files in our vault

An Obsidian vault can contain a lot of files of different type, and we only want to examine the Markdown files. In this case those files end with a .md extension. We can generate a list like this:

find /path/to/obsidian/vault -name \*.md -print

Note: For the rest of these examples I am going to assume that the current working directory is the Obsidian vault, so that /path/to/obsidian/vault will just be “.”.

The find command gives me a long list of Markdown files (that end in .md).

Handy Tips:

• If your collection of files have different extensions you extend the search like so.

  find . \( -name \*.md -o -name \*.markdown \) -print
  

• You can count the number of .md note files in Obsidian with:

  find . -name \*.md | wc -l
  

  If you watched the video then you might realise that we are simply asking the word count program (wc) to count the number of lines in the file list. I also skipped the -print option because it’s the default (unless you using a really old version of find).

The file names can contain spaces (and possibly other confusing characters) so we need to wrap the file list in some additional logic to take care of this.

find . -name \*.md | while read f ; do echo Filename: "$f" ;done

It looks a bit confusing, but I have found using a while read ... loop to be the most robust way to pass arbitrary text to commands as command parameters (in this example $f is the parameter to the echo command). Once the pattern is understood it works better than xargs in my opinion.

Locate any information inside that we want to change

When we discover a file of interest (i.e. a Markdown file), then we need to search in it for any occurrences of the text we want to change. I am going to the sed stream editor to this because once it finds the text we can also use sed to change it as well.

Going back to our example, we want to change all the #todo tags, that are part of a task, into #td. BUT, I also have #todo tags in my narrative text as well, to remind to come back and do something, and I don’t want to change those.

Note: I am going to be assuming that all our tags are lower case. In Obsidian that makes sense.

Let’s start with just finding all the tags no matter where

find . -name \*.md | while read f ; do sed -Ene '/\#todo/p' "$f" ;done

This gives us a list of lines that contain the text #todo. Because I only want to change my tasks tag, then I know that in fact the line will look like this

- [ ] #todo Process Note
- [ ] #todo Look at Pandrosion in more detail ⏳ 2022-12-26 📅 2023-01-03

The text on each line has the Obsidian checkbox marker - [ ] immediately before the tag. We can use this to only find the tags that appear on tasks.

sed -Ene '/^([[:space:]]*- \[.\] )#todo(.*)$/p'

This regex is looking for lines that consist of only:

• Optional whitespace

• Followed the by - [ ] marker ([ and ] are special regex characters and need to be escaped)

• and finally #todo.

There is a little wrinkle in there. The items inside the brackets () are captured for later reuse as \1 and \2. Hence the (.*)$ to capture the rest of the line.

So let’s test that

find . -name \*.md | while read f ; do sed -Ene '/^([[:space:]]*- \[.\] )#todo(.*)$/p' "$f" ;done

Which does gives us all the lines we need to change.

Change the specific text and only the specific to a new value

Now let’s see how we can use the sed substitute command (s) to make the needed changes.

Running find and processing every file for each test is laborious, so let’s pick one file and focus on that. I actually copied an existing file.

sed -Ene '/^([[:space:]]*- \[.\] )#todo(.*)$/p' testFile.md

gives my a single task line

- [ ] #todo Process Note

So far so good. Let’s use the s command

sed -Ene '/^([[:space:]]*- \[.\] )#todo(.*)$/s//\1#td\2/p' testFile.md

which printed

- [ ] #td Process Note

We have a winner.

There is a slightly simpler approach that does not use capture groups.

sed -Ene '/^[[:space:]]*- \[.\] #todo/s/#todo/#td/p' testFile.md

However, I prefer to remember the more complex version because I think it has wider application (but that is just me).

So far all these examples do not modify the underlying file, just displayed matching lines on the terminal. Let’s change that.

You can make sed print out all the lines it reads my removing the -n option (and also the p operand on the sed script otherwise some lines will appear twice). Like this

sed -Ee '/^([[:space:]]*- \[.\] )#todo(.*)$/s//\1#td\2/' testFile.md |less

Scroll through the new content to verify the tag has been changed. Depending on the file size you make see to scroll through a lot of content.

Now let’s actually update the file in place, instead of just displaying the new content to the terminal. The version of sed I am using (check your manual page), has a -i option that edits files in place.

For the moment we will only work on our test file

sed -i -Ee '/^([[:space:]]*- \[.\] )#todo(.*)$/s//\1#td\2/' testFile.md

If you examine the contents of your test file the correct change (and only those changes) should have been made.

- [ ] #td Process Note

This is the part were we can do serious damage to the Obsidian vault.

Seriously consider taking a copy of the whole vault before making changes. It’s easy to do (for example mkdir ../vaultBackup && cp -a . ../vaultBackup).

Make the change across the whole vault

find . -name \*.md | while read f ; sed -i -Ee '/^([[:space:]]*- \[.\] )#todo(.*)$/s//\1#td\2/' "$f" ; done

You can now change the settings in the Obsidian tasks plugin to use the new tag and verify the results. If something went wrong you can restore the backup copy of the vault and try again.