I never get rid of a piece of clothing ONLY because it doesn't fit me right now. I need a second strike against the item before I decide to donate it or throw it out. Anything that doesn't currently fit can be folded on a high shelf or hung in the dark recesses of my closet. I like having clothes in several sizes because my weight fluctuates and I want to have pieces I feel great in at any size, without shelling out a ton of money every time the hand on the scale moves.
Party or other specialty pieces should also stay in your closet, even if they don't fit right now. It is unlikely you will wear a sequin skirt on a regular basis, so even if it doesn't fit now, it may fit when you have another occasion to wear it. If you are short on closet space, give the party pieces that don't fit to a friend, with the agreement that you can borrow it back if you get back to that size.
Any sentimental or especially expensive pieces should also be kept. I have a beautiful beaded semi-formal dress that I don't currently fit into but will not get rid of because it reminds me of the occasions when I did wear it and I know that if I get back down to that size, I will want to wear it again. A well-tailored wool suit might also fit into this category. If it would be expensive to replace, but you would want to wear it again if you could, you should hang it in a garment bag at the back of your closet.
If a piece is in good condition, but doesn't fit, there are other reasons to get rid of it. The first is if it is a cheap basic, like a tee or a cotton sweater, basically any piece that you would replace on an annual basis. There is no point in saving a $7 tee in the hope that it might fit in another year when I will probably buy five more tees in that time. If it is easy to replace, then you don't need it taking up space in your closet.
If an item is more than two sizes too big or too small, you should also remove it from your closet (unless you have VERY positive feelings of a sentimental nature towards it). Having options when your weight fluctuates is different from having an entire size run of all the basics.
When cleaning out your closet, I suggest doing it in several stages. First, remove everything that is no longer in a wearable condition. That is the easy part because it is very objective and it will give you a better sense of what pieces you have and what space is available. Then remove any cheap basics, pieces you wouldn't wear, or items that are more than two sizes off from your current size. This will take care of everything that has two strikes against it (the first strike being that it doesn't currently fit). After this take your time going through the remaining pieces to see if they fit into any of the "to-keep" categories.