Pathway to a Better Electronic Notebook: A Markdown Experience

Scientists are relying more and more on electronic
lab notebooks.

One of the exciting parts of starting in a new lab is the chance to start establishing some better lab practices. As I wrote a few months ago, I was gaining a lot of traction using Microsoft OneNote as an electronic notebook. I liked it, but after a lot of use, I found it was not meeting my needs for customization, formatting, and most importantly, version control. As I started in the new lab, I decided to drop the ol' OneNote notebook and try out markdown on GitHub, as is used in my new lab. Ultimately I found markdown to be much more functional than OneNote. This week I want to share this experience so that you can try it out too.

Markdown is a simple documentation language that converts your plain text to HTML or a related language. In other words, it's an easy way to create a formatted document by writing up a simple text file. It has a lot of application including writing lab notebook posts.

To get started, you are going to need to download a program for editing markdown files. I tried a few and found the best to be the MacDown. This is an app for Mac that allows you to write your markdown with syntax highlighting and realtime rendering. So as the first step toward your markdown lab notebook, you are going to need download this program.

The next step is creating a GitHub repo to contain and organize your files. Instead of starting from total scratch, you can fork my lab notebook repo (OS-Note, which stands for Open Source Notebook) and build your notebook off of this foundation. This repo contains a 'posts' directory for the notebook posts, a 'figures' directory for your saved plots and graphs, a 'protocols'  directory for experimental protocols, and a 'bin' directory for a couple scripts I wrote to make the process a little easier. The directory also contains a README file that contains the table of contents, which facilitates easy browsing of the contents.

An example of markdown editing using MacDown.

So how do you put all of this together and actually use it? First you are going to need to open up your terminal (that's right, this is heavy on the command line, but that is what makes it great) and move to the notebook repo. From here you can easily create your first lab notebook post for the day using my the perl script. Note here that the notebook is organized chronologically with one post per day.

All you have to do is supply the name of the file, which is suggested to be today's date The script takes care of creating a properly formatted template that you can write on. Run the script like this:

$ perl ./bin/CreatePost.pl -o ./posts/2015-10-20.md

This automatically creates a post using the current date.

Fill in this file using the MacDown app. Don't be afraid to add content like syntax highlighted code snippets, links, and even figures (see the example post on GitHub). To add figures, you need to save the image into your 'figures' directory, and add the GitHub link to that file. Adding the link ensures the image will be shown when the markdown text is rendered and visualized in GitHub.

Once you get some notebook posts together, you are going to want to create a README file with a table of contents for your posts. This is also easy to generate using the other script I provided in the 'bin' directory. Just run the perl script while providing the README file as the output (-output) and a glob of the posts (-input). That's all it takes to create a nice table of contents. Here is an example of how to run it:

$ perl ./bin/CreateTableOfContents.pl -o ./README.md -i ./posts/*

Of course, once you get some of these files created, be sure to commit your changes to GitHub. This allows you to effectively backup your notebook, and to provide effective records of when the posts were created. These records can be important for intellectual  property (IP) issues that may arise down the road.

To wrap things up, this is just a simple foundation based on a notebook system that has worked well for me so far. It is definitely command line based, and requires a lot of your own maneuvering, but I have really liked this setup for my notebook, especially the effective version control. If you are looking for a nice way to record your research, and you don't mind opening up the Terminal, I definitely suggest building off of this repo.

Get onto GitHub and checkout the OS-Note repo!

Now this is just the start to a custom notebook system that I think could be further developed. If you think that this is a cool idea, or would like to practice some simple coding and organization, contact me about contributing to this project. I would love to collaborate with you on this on GitHub, and make an even better system for managing a lab notebook. I also think this might be cool to develop into an app, although I don't have the time to develop this myself right now. But again, let me know if you are interested in pushing this forward. After all, GitHub is about social coding, so it would be cool to actually do some coding as a social group. :)

Happy coding friends, and feel free to reach out with any questions, comments, or concerns.

<Take Me To OS-Note>