I made a sort of similar thread too recently talking about how frustrating it is that it's only "bits and pieces" that are remembered.

The way it's been explained to me: you're thinking like an adult and remember like an adult, but when you were younger your memories are different. A child that's 3-5 are more likely to center on sights, smells and such and less on language so it might seem different from more mature memories with language and stuff.