Is talking on the phone less safe than talking to a passenger?

Probably. The University of Utah conducted a very recent study that compared the performance of drivers talking to a passenger and drivers talking on a hands-free phone.

It found that drivers on the phone with no passengers were more likely to drift, miss their exit and tailgate. However, performance of drivers talking to passengers directly rather than being on a cell-phone were more likely to slow the conversation when driving was difficult or talk about traffic, suggesting that they were actually helping drivers keep their minds on driving.