When a swatch does not get removed by Select All Unused, it is because it is being used in some other palette content, such as a Brush, Graphic Style or Symbol.
Use in palette content is considered real use, even though it isn't on the artboard, because if the swatch were for a global color, deleting it would change the way the brush or symbol or graphic style that used it looked. It is true that deleting swatches for non-global colors will not change the way any other palette content looks, but then, it won't change the way anything looks on the artboard, either. So it is more consistent to use the same definition of "in use" for all swatches.
Without knowing which version of Illustrator you are using and which startup file, we can't say what particular palette elements are using those two colors.
It is a good idea to remove any palette content that you expect to seldom use from your startup files (most especially pattern brushes and symbols, since they tend to be bigger.) That way you will never have to delete them, and it will also make your New File command faster. The default palette content is always loadable as a library should you later decide you want to use one of the preset symbols or brushes.