Does it get cooler with altitude?

In most cases, yes. This is called the lapse rate and is typically slightly less than 1'C/100m of altitude. However, this does vary with humidity. Completely dry air may have a lapse rate of 0.65'C/100m. How water condenses out of the air as pressure changes with altitude can have a big impact on the lapse rate: at temperatures above freezing the rate is typically 0.5'C/100m.

Wind chill can have a far bigger effect than lapse rate and will vary with the terrain, though it is typically windier at altitude.

A temperature inversion reverses the lapse rate. This is when cold air sinks below warm air, typically in a valley, as in the beautiful picture above.

One of the problems of dressing for the hills is coping with the lapse rate, especially when wet weather makes changing layers difficult! Our clothing system employs massive vents to cope with the climb out of the valley towards cold mountain tops.

This link deals with the lapse rate in much greater detail.