Remember your short billed cycling cap? It's not just for staying warm. The bill can be invaluable to protect against both sun and rain. Flip the bill up out of the way when you don't need it, or just turn the cap backwards to protect your neck.
In very cold weather, use a heavy-duty winter cycling cap that has both a bill AND ear flaps. Worn under a helmet, the helmet strap holds the ear flaps down, keeping the head nice and warm.
In extreme conditions, use a balaclava (or full hood) which covers the head, face and neck and has a small opening for the eyes and nose. Just don't walk into a bank with this on!
More: Cyclocross: The Perfect Winter Sport
And don't forget the eyes. Traveling through cold air causes your eyes to tear, making it extremely difficult to see. Choose a good pair of cycling glasses that curve around the face and protect eyes from wind and other elements, without fogging up. Good eyewear, like all good cycling gear, is a good investment.
Respiration is another way to lose body heat, so if you're not wearing a balaclava, fold a bandana into a triangle and tie it to fit over your nose and mouth—just like the robbers in the old Westerns. This can make the difference between a comfortable ride and a miserable one. Just remember, don't go into a bank like this.
The Hands
I recommend cycling gloves for a couple of reasons. Most cycling gloves are cushioned on the palms, providing proper circulation in the various hand positions on your handlebars. Gloves also protect your hands from road rash if you fall. In the winter months, full fingered gloves are a good idea or cycling mittens for more extreme conditions.
More: 5 Bike Accessories for Winter
The Feet
Since the feet are pedaling circles and churning through the cold air more than the rest of your body, they need to be protected from the cold. Like the head, body heat is lost to a large degree through the feet. For cold-weather riding, use a heavier thermal cycling sock that wicks moisture and retains heat; choose socks made from synthetic fabrics.
Cycling booties slipped over your shoes are great in cold weather. The booties are designed to accommodate your pedal cleats, and insulate your foot and ankle as well. For days that aren't cold enough for booties, wear toe covers. Toe covers accomplish the same thing that a windbreaker does for your chest: they keep the cold air from penetrating your foot.
If you're on a ride without booties or covers and your feet get cold, get plastic bags from a convenience or grocery store and slip them over your feet (inside your shoes). While you're at it, you might as well ask for a hot cup of coffee—the hot coffee will help heat up your core from the inside.