We mean cool as in "you look like a badass." Yes, jerseys, worn tight and zipped up as far or low as the weather dictates, make you look like a serious cyclist (aka totally awesome). Conversely, if you're wearing a tee, sweating away with pumps and spare tubes bungeed to your bike, the only second glance you'll get is for looking ridiculous.
A general rule of thumb: For envious stares, wear a jersey. For objectionable sneers, wear a t-shirt.