Vinyl Availability

Note that a lot of this info is currently US centric.

The "market rate" for LPs depends primarily on the collector's value. Mass-produced and unpopular LPs will not cost more than a dollar. Original pressing Beatles LPs are usually over several hundred dollars. Jazz records tend to be very expensive. Classical records tend to be quite inexpensive. Rock, blues and electronica lie somewhere in the middle.

Depending on how clued the store is to the by-the-book market prices of vinyl, prices can range anywhere from 50 cents on up. Record stores tend to have the highest prices and charity stores the lowest, but there are always exceptions.

Many cities have local record stores that specialize in certain genres. Check your yellow pages and local music listings for ads.
Half Price Books (US) tends to have extremely good vinyl selections. Vinyl is not transferred between stores, so the quantity/quality varies widely even by the neighborhood.
Goodwill, Salvation Army, and other charity stores often have good deals on vinyl.
The Austin Record Convention http://www.austinrecords.com/ has a very, very large quantity of LPs. Biannual. Similar conventions exist in many other large cities.

Component Availability

Buying a turntable now is a lot harder than it used to be, but there are still plenty of options. A new turntable is usually much less hassle to set up than a used turntable for a new user. Maintenance and calibration is usually much less of an issue. However prices are much higher (30-100%) than equivalent used models.

As a general rule, do not purchase used cartridges, ever. They are the component that wears out the fastest on turntables during normal operation, and are the easiest to permanently damage. Buy them from reputable local or online sources. They do not have particularly difficult shipping requirements.

New

Hifi stores

The traditional place to obtain a turntable is at a local hi-fi shop. These are becoming an endangered species in the US - many have folded or switched to home theater and car audio - but most cities usually have at least one store selling audiophile kit.

Advantages:

Most hifi stores are locally owned and operated.
Shipping issues with turntables are already taken care of. Turntables are rather notoriously hard to ship correctly.
Hifi stores will usually tune/calibrate turntables free of charge on purchase.
You may be sending your table back to said store for maintenance, even if you purchase it elsewhere. Setting up a rapport is good.
Product support is often (possibly always) better from hifi stores than from any other source. A no-questions-asked tryout period between 7 and 30 days is standard for most stores.
They occasionally deal with both new and used gear and some very good deals are sometimes available.

Disadvantages:

Hifi stores have an extremely high markup, reflecting their small consumer base. All stores have noncompete clauses with neighboring stores in order to protect said customer bases. Therefore, there is often no competition whatsoever in most product lines.
Along the same lines, one of the promiment reasons why online stores cannot support certain products as well as hifi stores is that online stores are technically selling outside their 'designated area'. In other words, online support is prohibited because it enroaches on hifi stores' territories. Grado products, in particular, are notoriously bad for online support.
Expect to haggle.
Hifi stores usually cater to the extremely high end and will often not have much under $500 in the way of turntables.

Mass market stores

Best Buy, Circuit City, etc. They usually have a small selection of retro turntables, and perhaps more if they are ordered to the store beforehand.

Advantages:

Cheap (most turntables under $200)
Easily accessible

Disadvantages:

Very few turntables are sold that are considered any good
Very few available accessories and no cartridges
30-day return policy only for support. Hifi stores can actually repair things.

Online stores

Quite a number of online audiophile stores carry turntables of all makes and models.

Advantages:

Very inexpensive prices for most turntables
Shipping is usually of good quality; the hifi stores are going to get their turntables the same way you would.
Widest selection of turntables and accessories.

Disadvantages:

Turntable shipping costs are high. Hifi stores will roll in the shipping costs.
You may need to pay sales tax.
Very little to no support.

Used

There is a huge market for used turntables for any number of reasons. There is a high turnover rate in DJ turntables from people who have attempted to DJ as a hobby. Many people junk their turntables when upgrading their stereos. Audiophiles resell their turntables when upgrading.

Hifi stores sometimes resell turntables or demo units, and it is worth checking them for any deals. They are sometimes competitive.

In general, as a disadvantage, used turntable quality is completely unknown. At the very least you should expect to replace the cartridge and the drive train (belt). At the worst, there may be fundamental structural issues with the turntable, such as a damaged tonearm or bearing or motor, that may not be easily repairable.

Local stores

These include pawn shops and charity stores - Salvation Army, Goodwill, etc. Advantages:

Very, very good deals on some items.
Inexpensive.
Local; needs no shipping.

Disadvantages:

Completely unknown quality. Expect to replace the belt if necessary, cartridge, and possibly even the tonearm and wiring.
Selection is spotty, and the really good items can get snatched by the store workers.

Online stores and individuals

Notables here include eBay and Audiogon, but also check forums on places like Head-Fi, Audio Asylum, etc for sale posts. Advantages:

Large market in turntables of all price ranges.
Sellers are often fairly knowledgable.
Prices are quite competitive.

Disadvantages:

Stores take a rather large cut of the sale price. Audiogon is especially criticized for this. (This isn't a problem with forum interactions.)
The risk of shipping damage is quite high. 3rd-party sellers often have very little experience packaging and sending turntables.
Risky product quality.
No support.