I would guess you have an older boat and now you have the dreaded corroded bailer . Rust remover/penatrating oil will not work as the parts are brass (the cap) and aluminum (the housing), that's what they appear to be. Go to Wind Line Sails "How-To" section, Removing a Metal Bailer. You have exactly what is described in that article. There, pictures and words will tell you how to remove your old bailer. You will need to get the new plastic bailer (P/N 79260) from you local Sunfish Parts dealer or order it on line. The bad news is the price . $35.20 (base on the Vanguard price list from last year). Follow the directions for installing the bailer and you will be back in business. Vanguard (www.teamvanguard.com) has both a parts locator and rigging manual (with bailer instructions) on line, downlaodable as PFD documents. I would not remove the old bailer until I got the new one in hand, just use a scoop bailer made from a plastic milk or clorox bottle to bail if you go sailing in the meantime. If you do not bail, (working bailer or scoop) the cockpit gets full and you will notice how sluggish your fish is. Good Luck
Make sure you install the new bailer with the open end facing forwards. This will ensure that on a hot day you will fill the cockpit with refreshing cool water and won't risk burning your bare feet on the cockpit floor.