I have this problem, too. I never connected it to self-worth, but rather with emotional dysregulation.
Like something that happened to us made it impossible for us to gauge how much emotion to use in a given situation. So, we bottle it up and use no emotion or uncork it and it goes to 100%. No in-between.
Maybe emotional regulation is connected to self-worth? I don't know. I just know that I don't trust my emotions until someone else confirms or corrects me.
Recovery is possible. Hang in there, brothers.