One thing I’ve needed to do was integrate a news feed that is easy for developers on the project to update, without having to mess with uploading new html, sharing the site password, etc. The solution I came up with was creating a wordpress blog (e.g., PyCogent, PyNAST); using feed.informer to generate a javascript feed digest; and integrating that javascript into the sidebar via custom layout.html page in Sphinx. See the ‘News and Announcements’ sections on the two sites I mentioned above.

The steps are as follows:

1. Create a blog (I chose wordpress, since that’s what I’ve used the most).
2. Create a free feed.informer account. There are a few different sites out there that will host your javascript-based ‘feed digest’, but I’ve found this one to be the least obnoxious in terms of including an advertisements (see the ‘Powered by Feed Informer’ note on my sites) and fairly customizable.
3. Run through the steps of adding your feed to a digest at feed.informer — for my PyNAST wordpress blog, the feed url is: http://pynast.wordpress.com/pynast.

   One option you should be sure to use is ‘Show Only Live Items’, which has the effect of refreshing the feed rather than reading it from feed.informer’s cache. This allows you to, for example, delete posts in your blog and have them no longer show up in your feed digest.

4. Paste the provided javascript into your layout.html file in your sphinx project if you’ve created one. If you haven’t created a custom layout.html file, you can work from the PyCogent layout.html or the PyNAST layout.html.

   layout.html is the only file you should need to modify to integrate javascript into a Sphinx sidebar. No changes were necessary to conf.py.

5. Re-build the html pages, and view the result. I usually end up going through a couple of iterations of steps 3-5 to get the feed digest looking exactly as I want it.
6. Post the html.

That should do it. You should now be able to update your blog, and have that automatically update the News and Annonucements section on your website. If other developers on the project need to make updates as well, you can have them create wordpress accounts, and then add them as contributors on the blog. You’ll then have multiple users able to update this without sharing any passwords.

PyCogent’s coevolution module supports several tree-aware and tree-ignorant methods for identifying pairs of coevolving pairs within and between biological sequences. Additionally it contains support for pre- and post-processing coevolution input and results, such as recoding multiple sequence alignments with reduced-state amino acid alphabets. The script included in PyCogent supports intramolecular coevolutionary analysis, while this new script supports intermolecular coevolutionary analysis.

• parsers for common file formats (MinimalFastaParser);
• support for common biological data types (alignments, sequences, and trees);
• interaction with external applications (MUSCLE and FastTree) via the application controller framework;
• and visualization (via the UnrootedDendrogram object).

The example illustrates, in about 20 or so lines of code, how to apply PyCogent to evaluate the idea that life on Earth clusters into three related domains, detectable by distances between their small-subunit ribosomal genes (i.e., their 16s rDNA sequences). Using sequences collections derived from the Silva database (filtered with cd-hit-est so the max pairwise identity between any two sequences is 90%), I randomly select sequences, align the sequences with MUSCLE, build a tree with FastTree, visualize the tree via a PDF.

To run this example, in addition to PyCogent, you’ll need MUSCLE, FastTree, and matplotlib installed. You can read more about the ideas behind this example in Woese 1987 and Woese 1990.

Let’s get started. I did this with the ipython interpreter, but the standard python interpreter will work fine too.


mentat@caporaso tol_example> ipython


First, we’ll load up the sequences. We’ll select only sequences that are at least 1000 bases long, and contain no N characters. As I mentioned above, the sequences I am importing here come from the Silva database of 16S sequences, and have been filtered with cd-hit-est at the 90% sequence identity. (Skipping the filtering step would probably result in more sequences being required to see the nice clusters.)


> from cogent.parse.fasta import MinimalFastaParser

> arch16s = []
> for seq_id, seq in MinimalFastaParser(open('archaeal_v11.fasta')):
|..> if len(seq) > 1000 and seq.count('N') < 1:
|..> arch16s.append((seq_id,seq))
> len(arch16s)
696

> euk16s = []
> for seq_id, seq in MinimalFastaParser(open('eukaryotic_v11.fasta')):
|..> if len(seq) > 1000 and seq.count('N') < 1:
|..> euk16s.append((seq_id,seq))
len(euk16s)
3902


Import shuffle from the random module so I can extract a random collection of sequences

> from random import shuffle
> shuffle(arch16s)
> shuffle(bac16s)
> shuffle(euk16s)


Take some random sequences from each domain:

> combined16s = arch16s[:3] + bac16s[:10] + euk16s[:6]
> len(combined16s)

Load the combined sequences into a SequenceCollection object. The SequenceCollection object has many useful attributes and methods associated with it. Call dir(seqs) (where seqs is a SequenceCollection object for a listing. getNumSeqs is one method of the SequenceCollection object.

> from cogent import LoadSeqs, DNA
> seqs = LoadSeqs(data=combined16s,moltype=DNA,aligned=False)
> len(seqs)
> seqs.getNumSeqs()

Get an aligner function — here we’ll align with MUSCLE via the MUSCLE application controller. The result of calling align_unaligned_seqs is an Alignment object.

> from cogent.app.muscle import align_unaligned_seqs
> aln = align_unaligned_seqs(seqs,DNA)


Get a tree-building function — here we’ll use FastTree. The result is a PhyloNode object.

> from cogent.app.fasttree import build_tree_from_alignment
> tree = build_tree_from_alignment(aln,DNA)

Next I import a drawing function to visualize the tree.

> from cogent.draw.dendrogram import UnrootedDendrogram
> dendrogram = UnrootedDendrogram(tree)

I can then generate a PDF of the tree, and save it to file:

> dendrogram.drawToPDF('./tol.pdf')


Here’s the final figure:

A quick-and-dirty tree of life, built using PyCogent and 16s rDNA sequences from the Silva database.

There you have it: about 20 lines of code to reproduce a classic bioinformatics experiment. As we can see the sequences (or tips in the tree), do appear to fall into three distinct clusters. By looking up these sequence identifiers in Silva, we can see that the three clusters match the three domains of life: archaea, bacteria, and eukarya. (We can also see that the number of tips in the clusters match the numbers of sequences we pulled from each of the sequence collections above.)

PyCogent is a very nice toolkit to work with because it lets you ignore the boring and/or frustrating aspects of the study, like parsing files, interacting with the different interfaces associated with different third-party tools, etc., and focus on the fun parts: the experiments. PyCogent is an open-source project with the core development team split between the University of Colorado at Boulder and Australia National University. I’ve been a developer on the project since about 2003.

NAST (originally published here) has primarily been used for aligning newly acquired 16s rDNA sequences against the Greengenes “core sets” via the Greengenes website. NAST has become a popular tool in microbial community analysis, but wider adoption has been limited by the difficulty of running the original implementation locally. Since users may need to align thousands or even hundreds of thousands of sequences, it is important for them to be able to run the software on their own laptops, servers, or clusters. PyNAST, which was developed in collaboration with some of the original NAST authors, provides a command line interface, an API, and a Mac OS X GUI (which will go live shortly) to provide convenient access in all of these environments. Additionally, because users can provide their own template alignments when running locally, PyNAST is not specific to 16s rDNA alignments.

We currently have an Applications Note which provides more details on PyNAST, in addition to some speed benchmarks, under review at Bioinformatics.

You can download count_seqs.py and test_count_seqs.py.

The way I use these is I have a scripts directory where I keep the script, and then have a symbolic link to it from my $HOME/bin directory. To do this, you could do the following:


> mkdir $HOME/scripts $HOME/bin


You’ll need to make sure that $HOME/bin is in your search path, or choose a different directory (e.g., /usr/local/bin) which will be in your search path.

Then save the script from the link above to $HOME/scripts. Make the file executable, and create a symbolic link in $HOME/bin so it’s in your search path.


> chmod 755 $HOME/scripts/count_seqs.py
> ln -s $HOME/scripts/count_seqs.py $HOME/bin/count_seqs


Once you’ve done that, you should be able to apply the script.

Get usage information:

> count_seqs -h
Usage: count_seqs [options] fasta_filepath1 [fasta_filepath2 [...]]

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-s, --suppress_errors
Suppress warnings about missing files [default: False]


Count the number of sequences in a set of fasta files in a hypothetical aln_size_tests directory:


> count_seqs aln_size_tests/*fasta not_a_real_file1.fasta /data/not_a_file.txt

If you’re not interested in a report about files which couldn’t be found, pass the -s option to suppress error messages.

10 : 10_seq.fasta
160 : 160_seq.fasta
1 : 1_seq.fasta
20 : 20_seq.fasta
2 : 2_seq.fasta
320 : 320_seq.fasta
3 : 3_seq.fasta
40 : 40_seq.fasta
4 : 4_seq.fasta
5 : 5_seq.fasta
80 : 80_seq.fasta
645 : Total


Note that the sequence counting functions exactly as using egrep -c to search for lines beginning with >. For example:


> egrep -c '^>' aln_size_tests/320_seq.fasta
320


Wrapping it in a python script allows for convenient application to a list of files, and convenient totaling of different sequence files. That’s useful, for example, if you have a 16S database split on domains of life, and you want to see how many fall into each category and how many there are total. Here’s an example using a local copy of the Silva 16S data set:


> count_seqs silva/*fasta

12635 : silva/archaeal.fasta
296754 : silva/bacterial.fasta
42239 : silva/eukaryotic.fasta
0 : silva/unclassified.fasta
351628 : Total


I hope this is useful!

I thought I’d also put a few of my favorite Costa Rica wildlife photos here.

Iguana on the beach.

A crocodile, from the bridge above it.

Toucan in the forest below Volcano Arenal.

White-faced monkeys in the forest below Volcano Arenal.

]]> http://blog.caporaso.us/2009/08/31/costa-rica-pictures-finally/feed/ 0 gregcaporaso Iguana Crocodile Toucan White-faced monkeys http://blog.caporaso.us/2009/08/21/when-will-i-ever-need-to-use-trigonometry-or-how-to-avoid-avalanche-terrain/ http://blog.caporaso.us/2009/08/21/when-will-i-ever-need-to-use-trigonometry-or-how-to-avoid-avalanche-terrain/#comments Sat, 22 Aug 2009 04:09:21 +0000 gregcaporaso http://blog.caporaso.us/?p=127 ]]> I sometimes tutor high school math, and as anyone who has done this knows, a common question is ‘But when will I ever need to use blank?’. This came up last year, where the blank was trigonometry.

When will I ever need to use trigonometry? It’s a legitimate question. As an instructor, you could explain how it’s a foundation of mathematics, or how you’ll need to understand it for any career in science or math, or so on… (They’re both true, and are probably close to what I said, but neither are an answer that your student is likely to care much about.) Think about it: what practical problem, in day-to-day life, have you solved by applying trigonometry? (Comment if you have a good one!)

So, when I was backcountry skiing last winter and my friend Erik and I actually applied trigonometry to decide if it was safe to ski a certain slope, we were proud.

It’s generally accepted that if a slope is 30 degrees or less, it’s probably not steep enough to avalanche. Slopes around 40-55 degrees are the most dangerous. We were about to cross a slope, but we weren’t really prepared to be in avalanche terrain: we didn’t have our beacons, probes, and shovels. In short, we wanted to avoid avalanche terrain by staying off slopes that were 30 degrees or greater.

We didn’t have a slope meter, but we did know trigonometry. If we could make a right triangle where one of the legs (i.e., sides not opposite the right angle) was double the length of the other leg, the slope of the opposite side (the hypotenuse) would be less than 30 degrees. (We did need a calculator to figure out exactly what that angle would be.) What could we use to make the two legs? Ski poles!

Figure 1: Identifying a 26 degree slope with two ski poles.

Here’s how it works. Plant your first pole vertically, and try to make a 90 degree angle with the second pole beginning halfway down the vertical pole. If the tip of the horizontal pole touches when the poles are at 90 degrees with one another, you’re on a roughly 26 degree slope (Figure 1). If you can’t make a 90 degree angle because the horizontal pole hits the slope too high, the slope is steeper than 26 degrees (Figure 2). If you can make an angle smaller than 90 degrees before the tip of the horizontal pole hits the snow, the slope is less than 26 degrees (Figure 3).

Figure 2: Identifying a slope greater than 26 degrees with two ski poles.

The rule-of-thumb is this: If the poles can form a right triangle with the horizontal pole exactly halfway down the vertical pole, and with the slope as the hypotenuse, the slope is less than 30 degrees and you are probably safe. This is the situation depicted in Figures 1 and 3. If you can’t form the 90 degree angle because the slope is too steep, as in Figure 2, you’re in possible avalanche terrain so make sure you know what you’re doing!

Figure 3: Identifying a slope less than 26 degrees with two ski poles.

Finally, you can also use this trick to identify a 45 degree slope (which is absolutely in avalanche terrain). To do that, this time place the horizontal pole at the top of the vertical pole rather than halfway down. If the tip of the horizontal pole just touches the slope when you have a 90 degree angle between the poles, the slope is just about 45 degrees (Figure 4). If it doesn’t touch, it’s less than 45 degrees. If you can’t make a 90 degree angle because the second pole hits the slope too high, you’re on some steep stuff!

Figure 4: Identifying a 45 degree slope with two ski poles.

There you have it: a field application of trigonometry. We’re not the first ones to figure this out — I’ve since heard that the Norwegians have been doing this forever. It’s a neat trick.

And of course, there is a lot more to avalanche safety than this. This is a basic rule, but by no means definitive. Be scared of avalanches, and don’t assume that if a slope is less than 30 degrees it can’t slide. If you’re planning to travel in avalanche country take a class, check the avalanche reports, and if you’re not scared of avalanches, watch some videos.

Skiing out from Ken’s Cabin

More photos from this trip.

The example script takes three optional parameters, -v, -s, -i, and one required parameter, output_filepath. (Note that I don’t need to define the -h or --help option — that functionality is automatically included.)

The -v parameter is a binary verbose flag defined as:

parser.add_option('-v','--verbose',action='store_true',\
        dest='verbose',help='Print information during execution -- '+\
        'useful for debugging [default: %default]')

When the script is called with -v or --verbose, True (because action='store_true') is stored in the verbose variable (because dest='verbose'). The help parameter defines the help string that will be presented when the user passes -h to the script. When the help string is printed, %default is replaced with the default value of the parameter.

Next I define an option which allows the user to pass a value to the script via -s:

parser.add_option('-s','--string_to_write',action='store',\
          type='string',dest='string_to_write',help='a string to write to '+\
          'file [default: %default]')

This allows for passing a -s or --string_to_write option to the script, the string value of which will be stored (because action='store') in a variable called string_to_write. Note that we explicitly define the type here (type='string') — something that is unusual in python.

The next option is similar, except that an int is stored to int_value:

    parser.add_option('-i','--int_value',action='store',\
          type='int',dest='int_value',help='an integer value to store '+\
          '[default: %default]')

After the options are defined I set defaults for them. The value for all options defaults to None, unless otherwise specified. I define the default for verbose to be False and the default for string_to_write as "Some example input.". I don’t define a default for int_value.

Finally, I parse the command line arguments into opts and args. I then have the values that the user provided, and am able to perform any necessary error checking.

The script requires that the user provides a single command line argument, output_filepath. So, I check that they have provided exactly one argument, and raise an error if not. Raising the error via parser.error is useful because it provides the user with the usage information in addition to the error message. Try calling the script with no options or arguments for an example.

If no errors are raised, opts and args are returned to the caller — in this case the main block of the code. The options and arguments can then be accessed as illustrated in the example.

You can see by experimenting with the script that the verbose flag prints some additional information to the terminal at runtime. Try running the script with and without -v, and you’ll see slightly different results. string_to_write is written to output_filepath. If it’s not provided by the user, the default value is used. Then int_value is written to output_file if provided.

That just about wraps up my optparse example. I’ve highlighted some of the optparse functionality in this tutorial, and provided a usage example that you can experiment with on the command line. A good way to learn more would be to start with my optparse example, and experiment with adding new functionality. You can find a lot more information about optparse here.

I previously mentioned that I start all new python scripts with a template file which includes optparse functionality. In my optparse template, which is a simplified version of my optparse example, I leave in some example options so it’s easy to add new ones in my script. Rather than having to remember how to instantiate the OptionParser object and define parameters, I can just modify what’s there and delete what isn’t necessary. Feel free to use and modify my optparse example to make it a template for your own python scripts.

Finally, you may have noticed that python has another module for parsing command line parameters: getopt. You should always use optparse over getopt as optparse provides a lot of additional functionality, and will continue to be supported in Python 3 (unlike getopt). (Scratch that about getopt not being supported in Python 3 — it looks like it is but the Python docs note that: “A more convenient, flexible, and powerful alternative is the optparse module.”.)

I think you’ll find that incorporating optparse-based parameter parsing to your python scripts will make them more useful to yourself and your users. I hope that my example will help you get up and running with optparse very quickly. Good luck!

This was a long but very fun hike, with four distinct sections: the Bierstadt ascent, the Sawtooth, the Evans ascent, and the descent. The hike from Guanella Pass to the Mt. Bierstadt summit is a 3.5 mile climb, from the trailhead at 11,669′ to the summit at 14,060′. Since it was a beautiful Saturday morning, this section was PACKED with people.

The Sawtooth and Bierstadt from the Bierstadt Trail.

From the summit, we scrambled across the Sawtooth. We ran into very few hikers in this 3.5 hour section of the hike (with a lunch and nap break), so it was a nice change from the Bierstadt Trail. This was sketchy in spots but doable.

Ingrid on the Sawtooth with the Bierstadt summit in the background.

Once across the Sawtooth, we hiked through a boulder field for about 2 hours up to the Mt. Evans summit at 14,264′. There is a paved road up to the Evans summit — the highest paved road in America — so, of course, the summit of Evans was packed.

Mt. Evans observatory from the Mt. Evans summit.

We quickly left the summit to make our way back to the Guanella Pass parking lot as we were starting to get tired. Rather than cross the Sawtooth again, we hiked down an alpine meadow, into a gully, and then through the infamous willows. The route finding was straight-forward in this section, and there was a decent trail through the willows (albeit a little soggy).

Waterfall just above the willows.

I now understand why everyone warns of how terrible it could be to get lost in the willows. Before there were decent trails through this section the willows could have been a slow, wet end to a long hike. The hike ended up taking us just under 12 hours, from trailhead to trailhead.

Twenty down, thirty-two to go… A map of Colorado 14ers with the ones I’ve hiked in red.

More photos from this hike and photos from my other 14er hikes.

$ python optparse_example.py -h
Usage: optparse_example.py [options] output_filepath

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         Print information during execution -- useful for
                        debugging [default: False]
  -s STRING_TO_WRITE, --string_to_write=STRING_TO_WRITE
                        a string to write to file [default: Some example
                        input.]
  -i INT_VALUE, --int_value=INT_VALUE
                        an integer value to store [default: none]

The first line we get is a usage statement telling us how to call the script. In this case it’s simple: there is one required argument, input_filepath, and several available options. This line can be really important when there are a few required arguments and you need to know the order of them. Then, we get a list of the available options to the program, the different ways we can call them, brief descriptions of each option, and the default value associated with each.

When you create your own version of parse_command_line_parameters() (or modify mine), you’ll write the usage statement yourself. To adhere to convention, everything optional should be enclosed in brackets and everything required should not be enclosed in brackets. Reviewing the example above we can see that [options] refers to optional parameters while input_filepath refers to the required parameter. (optparse_example.py is, of course, the name of the program and therefore also required so not enclosed in brackets.)

Let’s look at a more complex example. Imagine that a program required an integer followed by one or more input filepaths. The way I would write that usage statement would be:

optparse_example.py [options] input_int_value input_filepath1 [input_filepath2 [input_filepath3 [...]]]

This indicates again that optparse_example.py and input_filepath1 are required arguments, in addition to input_int_value (which must be provided before input_filepath1). Additionally however, we see that we can optionally pass an input_filepath2, and so on. The nesting of brackets is meant to indicate that if we want to include the next optional argument, the current one must be provided.

If there is a certain parameter where you always require a value, but where the majority of the time it will be the same, that’s where defining options with defaults is very useful. Focus on trying to make command line interaction with your script as simple as possible (don’t require 20 arguments in a specific order!), but harnessing the options with default values to give users full access to what your program can do.

Next we’ll jump into the code, which is pretty straight-forward, and look at how to define options and arguments, and the benefit of doing error checking in the parse_command_line_parameters() function.