If a commodity is in "backwardization", that is, if futures prices are lower than current prices, does that imply that futures markets are discouraging storage (encouraging disgorgement)? Paul Krugman makes the case, here and here.

I'm going to challenge him with a low-down, dastardly kind of argument. The gentleman keeps asking for evidence, evidence, and in response I'm going to offer an unfalsifiable hypothesis. Krugman says that futures prices are too low to cause people to withhold physical oil and sell forward, as required to affect spot prices. But whatever forward price curve he shows me, I can posit an invisible "convenience yield" large enough to make hoarding oil worthwhile. I don't even have to be unreasonable about it. Extrapolating from historical data, we see that gently "backwardized" futures prices might be quite sufficient to encourage storage when convenience yields are taken into account.

Despite all of this, I agree with Krugman that futures markets can't explain the recent skyrocketing oil prices. Not unusually, he's been a voice of sanity and reason. But just as speculation in futures need not affect spot prices (if it is balanced long and short), speculative withdrawal of physical supply need not affect the shape of the future price curve. Futures market "signatures" can't be relied upon to distinguish speculative from fundamental demand, and inventory may be unmeasurable, especially if it is offshore or takes the form of withheld production. Further, both futures in contango and measured inventory build can signify known fundamental demand as well as speculation. (Suppose entrepreneurs are planning to fire up many new factories over the next few months. If they buy forward to hedge their exposure to energy prices, that would push futures into contango and promote storage with no one speculating on anything.)

But this isn't about that. This is a disquisition, and ode, a homage and a tribute to the marvelous, mysterious, misunderstood and maligned convenience yield:

The bedrock principle of futures markets is "no arbitrage", that is prices should be set such that there is no investment strategy that yields a risk-free profit higher than the risk-free rate of interest. For a physical commodity, this seems to imply that the forward price should equal the present price plus the risk-free rate of interest and any storage costs. Interest rates are never negative, and storing stuff always costs something, so you'd expect future prices of storable commodities to always be higher than current ("spot") prices. But check out the following graph of the four month "forward yield" of crude oil. (Data courtesy of the EIA, hat tip Krugman, calculations and errors are mine, click for a bigger version).

The "forward yield" is just the percentage by which the future price is higher than the spot price. I'm using four month futures, but I've converted the yield an annual rate. If the forward yield is 10% and the interest rate is 4%, an oil man would profit from buying oil now and selling it forward, as long as his storage costs are less than 6% per year. Tycoons would start buying spot and selling forward until yield dropped to interest + storage.

But didn't we say future prices should be higher than spot prices? Look at the average (the yellow line in the graph). It at -3.4%, well below zero! The yield is negative more often than it is positive. Whenever the yield is anything less than interest + storage, you'd expect oilmen to sell crude from storage and buy it back forward. He earns interest on his cash and saves the storage costs that way, which is better than the profit he'd make storing and selling forward. When the forward yield is negative, Texans should be selling physical and buying forward like mad. By doing so, they earn interest on cash, save storage costs, and lock-in an easy profit buying back their oil for less than than what they sold it for. Very quickly, all the selling should create a glut of physical oil, driving down spot prices, while the buying forward drives up future prices, until the yield is positive and sanity is restored. But negative forward yields for oil have persisted, sometimes for years at a time! We must be missing something. Hmmm...

Suppose someone offered to buy your vacuum cleaner today for $100, and sell it back to you next week for $80, with no risk of wear or breakage. Would you? It would depend how much you value the use of your snorter. If you refuse, we might infer that a week's access to the pleasures of vacuuming is worth 20 bucks to you. We call that value of temporary use a "convenience yield". It's as if having the vacuum cleaner around pays you $20, even if no cash changes hands. Maybe we observe negative forward yields on oil because the smell of oil in the morning is priceless to guys in cowboy hats. (Or not... see below for a more plausible account of oil's "convenience yield".)

Let's eliminate wandering interest rates, and take a look at the storage cost / convenience yield of oil historically. Below is a graph of a three-month forward yield (calculated from the first to the fourth month forward, because futures prices are more trustworthy than spot) with the concurrent 3-month T-bill rate backed out of it, leaving only storage costs. When storage costs are negative, that reflects, by definition, a convenience yield. (Again EIA monthly data, T-bill rates courtesy of FRED, my calculations)

From Jan 1986 through May 2008, oil futures have reflected a convenience yield of 8% per year on average. (This is in rough agreement with the overall mean of 0.021% per day calculated here by Milonas and Henker, see Table 3.)

Suppose that the current convenience yield is about 8% and three month interest rates are about 2%. Then a one-year futures contract should be about 6% cheaper than spot, and a four-month-out contract should be about 1.4% cheaper than a one-month-out contract (reflecting 3 months of storage). At the end of May, the 4-month-out contract was in "backwardation", but was only 0.5% cheaper than the 1-month, still too expensive given the convenience yield. Oil dudes could have earned (on an annualized basis) about 3.6% more than the risk free rate (about 5.6% overall) buying high and selling low, but enjoying the privilege of storage. Now that oil is in gentle contango (as of June 17, the 4 month contract costs about 1% more than the 1 month), buying forward and storing looks like a really fantastic deal.

What is this "convenience yield"? Is it real? It seems like it must be, the economics of an 8% return aren't subtle in the data. But when I first encountered this idea, it baffled me. So instead of talking oil, let's talk hotels.

Suppose you have a hotel, it's morning, and you've got a room that isn't yet booked for tonight. Empty rooms end up costing you about $10 a night, considering your rent, maintenance, utilities, etc. But, you estimate there's about a 50% chance that a weary last-minute traveler will come by and pay your walk-in rate of $150 for the room. So, the risk-neutral expected value of your empty room is $65 [(150 ÷ 2) - 10]. You're risk-averse, not risk neutral, though. You'd accept a certain $60 rather than a 50-50 chance of losing $10 or earning $140. That $60 is the "convenience yield" on your empty room, it's what having a room empty, in case opportunity strikes, is worth to you.

Oil is a "spiky" commodity. Every once in a while, someone really needs it, now, and will pay a premium for immediacy. The market for oil in Cushing, Oklahoma might be reasonably efficient, but what happens when someone in Peoria needs oil today? Opportunity! Instead of running a hotel, you build an oil tank in Peoria. Suppose that every month, there's a 10% chance a desperate client will offer a 5% premium for immediate delivery of all your oil, and that interest and storage cost you 0.2% per month. Then on average, you'd earn 0.5% (10% x 5%) each month from desperate clients, and pay 0.2% in expenses. You don't want to bear the risk of fluctuating oil prices, so you sell your oil forward. If you were risk neutral, you'd be willing to sell it for a discount of up to 0.3% less than you bought it for at the beginning of the month, at which price you'd just break even. But you're not risk-neutral: You attach a "certainty-equivalent" value of only 0.4% to the unpredictable income from needy customers, and would offer no more than a 0.2% discount on month-forward oil sales. In the end, you earn a risk-adjusted 0.4% per month "convenience yield" from desperate polluters, and pay 0.2% in interest and expenses, and 0.2% in hedging costs. If one-month oil futures pay more than 0.2%-less-than-spot, or (golden days!) if they are in contango, you'd buy as much oil as you could and sell it all forward, because every new barrel that you promise to buy high and sell not-so-low represents certain (well, "certainty equivalent") profit.

Putting aside Peoria and our artificial needy customer, sometimes oil spikes even on the wider market, so that anybody with physical oil can sell at a high price and while locking in low-priced near future purchases to replenish their stock quickly enough to meet any other contractual obligation to sell. If you estimate the profit you'd to earn from these occasional opportunities, and subtract a bit to come up with a "certainty-equivalent" value for this uncertain income stream, you'll have determined a convenience yield. It shouldn't be surprising that convenience yields are especially high for volatile commodities subject to frequent shortages and price spikes.

When futures markets are well-arbitraged (which might not always be the case!), the future price of a storable commodity is determined by the spot price plus the total cost of storage, defined as foregone interest, plus storage costs, minus any benefit of temporary ownership &mdash the convenience yield! When a storable commodity like oil is in backwardation, that doesn't mean that the markets are predicting that its price will fall. It means there is a convenience yield. And in order to decide whether futures markets are creating incentives to store or to sell physical stuff, you have to estimate the convenience yield.

Postscript: While I was writing...

Mark Thoma offers a nice extension of Krugman's model, showing how monetary policy, by affecting interest rates, would be expected to affect storage.

Yves Smith offers very pointed commentary on futures markets, speculation, and inventory (here and here).

If policymakers want to "do something" about commodity speculation, they should really start investigating passivity on the short side of the market rather than enthusiasm on the long. Given what's before them, I hope they ignore Michael Masters (ugh!) and Thomas Palley (whom I often like, but yuk), and go with Dean Baker's suggestion of a Tobin Tax (ht Mark Thoma).