I have this problem, too. I never connected it to self-worth, but rather with emotional dysregulation.
Like something that happened to us made it impossible for us to gauge how much emotion to use in a given situation. So, we bottle it up and use no emotion or uncork it and it goes to 100%. No in-between.
Maybe emotional regulation is connected to self-worth? I don't know. I just know that I don't trust my emotions until someone else confirms or corrects me.
"There is a Catskill eagle in some souls that can alike dive down into the blackest gorges, and soar out of them again and become invisible in the sunny spaces... even in his lowest swoop the mountain eagle is still higher than other birds upon the plain, even though they soar." -- from Moby-Dick