Newspaper Seedling Pots
In our greenhouses, we grow a number of bedding plants in "peat pots"--small pots made from pressed peat moss. Unlike plastic pots, these are completely biodegradable. All you need to do is dig the hole and drop the entire pot into it. Easy, and environmentally friendly.
If you don't have peat pots handy, Eric from Japan has a neat origami trick for making planting pots from newspaper. You can see the article at http://www.geocities.com/newspaperpots/. The site is currently offline since it exceeded its bandwidth limit, so you may need to check back.
(hat tip: Lifehacker)