How to Browse Fastai Source Code Using Vim

A tutorial that breaks down the setup and usage of tags to navigate any source code in Vim

TL;DR

First, setup tagging and a few other things. Then get comfortable jumping all around any code using these commands:

Jump to tag by name→ :tag QuickSor + tab (for autocomplete)

Jump to current tag → Ctrl+]

Go back up the tag stack→ Ctrl-t

Searching → :ack or grep

Outlining (toggle, fold, unfold)→ Ctrl-za or Ctrl-o or Ctrl-c

There were lots of people in the Fastai v3 course from non-software backgrounds that got a bit confused when Jeremy started using Vim to jump around fastai source code. The intimidation is understandable but shouldn’t be a barrier. Especially because frequently in data science and machine learning disciplines we need to work on remote machines and therefore being comfortable with a terminal editor (like Vim) is really useful. It’s worth the effort to learn to do it. Alas, read on.

Setup Vim and Configs

Make sure you have Vim installed.

$ vim --version
Output of running vim — version in bash

Note: If you are not familiar with basic terminal usage, now would be a good time to google around for some tutorials to learn to navigate directory trees, use a package manager (eg. brew), and perhaps the basics of Vim like switching modes and basic navigation.

Setup Tags

Setting up tags will allow us to jump around to different function definitions.

First, install ctags

$ brew install ctags

If you are on osx and having trouble installing ctags you may need to set the alias by running these two commands:

$ alias ctags="`brew --prefix`/bin/ctags"
$ alias ctags >> ~/.bash_profile
$ source ~/.bash_profile

Note: If you are using zsh (comes with the latest osx Catalina update) you will be updating a file called .zshrc or .zprofile instead of .bash_profile.

Now, we are going to run the ctags tool which will go through any files we want and index them so that we can easily jump around to whatever has been indexed (function definitions, headers, etc)

Note: Before we do the next step, make sure that you have the fastai library installed. You can do this using pip install fastai. Then to find the package you can type $ find / -name "fastai" and use the root directory you see. In my case it was usr/local/lib/python3.7/site-packages/fastai.

With this command, we are telling ctags to recursively go through all the files in the /usr/local/lib/python3.7/site-packages/fastai directory and index things in the output file .tags .

$ ctags -R -o ~/.tags /usr/local/lib/python3.7/site-packages/fastai

You can do this for just the fastai library or for other modules that you use a lot and want to explore like matplotlib, pandas, etc. If you are using the same .tags file for all of your tags then you may want to use the -a flag like this so it appends to the same file when you run the command for other modules.

$ ctags -R -a -o ~/.tags /usr/local/lib/python3.7/site-packages/pandas

Note: The ctags tool is really flexible and when you get comfortable with its basic usage, you should check out how you can use it to generate tags for specific project folders and include to handy autogeneration specific to the type of work you do!

If you want to do a sanity check here you can pop open the .tags file and check out what was indexed. You should see a bunch of gobbly guck like this:

The last step to setting up tags is to let Vim know where to look for the tags you just created using ctags. Do this in your .vimrc using the following line:

" Setup tags to be able to jump around
set tags=~/.tags

Note: If you don’t have a .vimrc in your $HOME directory just create one using $ touch ~/.vimrc .

After you update your .vimrc you will need to reload it. You can do this from inside Vim by typing :source ~/.vimrc in normal mode. Or do it from bash terminal using source ~/.vimrc.

Browsing Source

Now for the fun part! So, there are a few things we need to be able to explore code efficiently.

  1. Jump to tag/symbol (with completions) — this will allow us to checkout certain functions or keywords even if we don’t remember the whole function name.
  2. Jump to current tag — once we are in a specific function chances are, it will call another function and we will want to jump to that function def and check it out. This will allow us to do that.
  3. Jump to library tags — the function may even call things in other libraries like matplotlib or something. We may want to be able to jump to those function definitions too. We just have to make sure they are indexed in the .tags file using the ctags utility.
  4. Go back — we will follow the function calls and at some point want to go back up the tag stack.
  5. Search — searching source code is critical…we will use ack. What is ack? ack finds things. You’ll see.
  6. Outlining/folding — it’s going to be impossible to see what is going on if you don’t have some kind of tool to help you fold and unfold code.

1–4. Can be solved via basic tagging commands

  • You can open Vim and type : tag QuickSort where Quicksort could be any tag (function name, etc) you are looking for. Hit tab to autocomplete if you don’t remember the whole function name.
  • To jump to a specific function that your cursor is on simply type Ctrl+] in normal mode and it will take you to the definition of that function.
  • If you’d like to jump to something in another library like matplotlib or some other library you are using you totally can! Guess how!? Just go back up and run the ctags command again with the files you’d like to be able to tag (eg. in this case you’d find the site-packages/matplotlib source)
  • Type Ctrl-t to go back up the tag stack (go back to previously jumped-to locations)
Using tagging to search for the DataBunch method in fastai source code. (Note: your code will not be folded like mine is in this pic until you install SimplyFold Plugin in #6 below)

5. Searching

Another really useful thing, is to be able to show all the places where a certain function (or any pattern) occurs. For finding things in source code we will use ack. Yes, you can use grep if you are more familiar and comfortable with it but ack is generally a faster way to search source code so…maybe step outside your comfort zone and learn it. If not, skip this section and keep using grep.

First, make sure you have ack on your machine (ack --version). Then install the ack.vim Plugin and we will be able to find all files containing a pattern using this syntax in Vim:Ack [options] {pattern} [{directories}]

For example, say we want to search for all files that contain the DataLoader function. We can just type:

:Ack DataLoader <directory>

Note: If you already have vim open and are in fastai source code you can just type :Ack DataLoader %:p:h to search in the directory of the file you already have open. You could also leave directory empty in which case it will default to searching in your current working directory (type :pwd to see what your current working directory is in Vim).

Results of running using :Ack DataBunch (search for everywhere DataBunch is used) in Vim

Use the up and down arrows to select a file you want to open, then hit enter (or t to open in new tab). Then of course close the newly opened buffer using :b#

Note: Feel free to add an .ackrc file to create custom configs that make your searching easier.

6. Outlining/Folding

Folding can be a little tricky in python since it uses indentations instead of brackets but there is a Plugin that helps vim fold Python code beautifully, SimplyFold.

Install the SimplyFold Plugin then you can use Ctrl-za, Ctrl-o, Ctrl-c in Vim normal mode to toggle, fold, unfold respectively. This way when you are jumping around you can hide things that you don’t need to see.

Additional Resources

  • All these commands hard to remember?! Checkout my my cheatsheets repo. Its a quick utility that I made so you can throw things into a quick custom cheatsheet then just type $ cheat vim (or more generally$ cheat <keyword>) in your terminal to check your cheatsheet!
  • Want all your configs to be the same across any machine you use? Checkout my tutorial on how to manage all of your config files (dotfiles) easily using git bare repositories.

~(°°) … ~(°°)