Your closet is probably full of what I refer to as "secondary neutrals." Your primary neutrals are those basic colors that go with everything: black, grey, navy, brown, taupe, white and ivory. Secondary neutrals are colors that coordinate most of the time. Lots of dark blues and teals fit into this category because of their proximity on the color spectrum to both navy and teal, but warmer colors such as burgundy and plum also lend themselves quite nicely as neutral foundation pieces. The beauty of secondary neutrals is that they allow you layer color without being overwhelming.
When looking through your closet to identify the secondary neutrals, it is important to remember that, just like your primary neutrals, that any pattern or texture should be minimal since these items are meant as base pieces to be built upon. Although there are colors that are more versatile than others, your secondary neutral colors will be unique to you because they are determined by how well they work with the majority of your wardrobe. If your closet has lots of warm, bright patterns in it, then coral and red might be secondary neutrals for you to try. However, if you gravitate more towards heathered jewel tones, olive and teal greens might be a better fit.