Whether or not we should, we do tend ot judge a book by its cover, which is why any author who is considering self-publishing should make time to watch the video below. The speaker is legendary designer Chip Kidd, who creates covers for books published by Random House.
I think you'll find this interesting even if you're not going to self-publish, because he makes some fascinating points about design in general. Along the way he talks about how he came up with the covers for Jurrasic Park, among others. It lasts about 17 minutes and if you're in a rush you can start at 1:20. If you can't see it fully below, you can also watch it by clicking on this link: http://www.ted.com/talks/view/lang///id/1410