When Jane was looking for a new blazer, one of the most important things was that it would be versatile. I paired it with denim, a white tee and a cotton scarf for a relaxed look that is perfect for lunch with friends or any day that just requires a little extra polish. When I am wearing a blazer with a casual, daytime look, I like to scrunch or cuff the sleeves to add a softer edge. As far as accessories go, a scarf is generally less formal than a necklace and still adds a finishing touch. When trying to dress down a blazer, go with cotton, not a silk (or other shiny) scarf. An untucked layering piece can also help tone down the formality of a blazer, but make sure it doesn't hang too far below- just about an inch is good and you get bonus points if there is an interesting hem detail!
A good rule of thumb is to soften your edges wherever possible for a casual look and crisp your edges wherever possible for a dressy look. A blazer will dress up most outfits, but don't rely on it completely to elevate the look- your clothing base should always reflect the look you are going for and the additional layer should be only for polish.