Shame has much to do with the sense of failing to fit in with the social group, a sense of social undesirability, or social exclusion/ostracism. It has to do with how we anticipate we are seen through the eyes of others. Almost always this feeling is fed by an unconscious tendency within us all to measure ourselves against our images of how we should be, or should appear, through the eyes of the outside world, how we measure up against ideal outside norms.
Almost always, shame lessens when we come into an awareness/acceptance of ourselves as genuinely different from the norms we have previously self-identified with and fused with. Differentiating ourselves in this fashion not only relaxes feelings of shame but also tends to sharpen and clarify our inner sense of personal reality. Note that over-identification with social norms and ideals tends to both blur personal reality and leave us especially vulnerable to shame; akin to leading our psychological lives as if standing on stilts.