Interesting article which attempts to answer the above question. Also some good, sound advice from a variety of expert sources:
Begin by collecting only what you like and intend to read. This way you'll get some value out of your collection even if you don't make money.
Condition means everything. If you're serious about collecting, buy the best condition you can afford.
Deal with a seller -- either local or online -- who has been recommended by a friend who's actually done business with that seller.
Check prices you're willing to pay for a specific title against national selling prices, which can easily be found online.
On a limited budget, say $500 to start? Buy four or five titles instead of springing for a single book at that price. You'll learn about the different points of collecting faster.
Nothing has more cache in this field than the words "first edition." But things are not always what they appear to be here -- you'll need to do a little research on your own into what the term "first edition" really means.
Handle with care. Truly rare books should be handled with gloves. Never pull a book off a shelf by the top of its spine. Store in a dry, temperature-controlled room.