Your relatives are the ones being flaky here, not you. Calling someone the morning of an event is not enough notice for most people. Usually if it's a casual, spur-of-the-moment gathering, there shouldn't be pressure for the invited person to attend. If they really wanted to honor their daughter's birthday, they would have used better time management and thrown her a proper party. If they are giving you guilt trips for not dropping everything for their flighty plans, then it is a good thing to be assertive with them. Just simply say that you would have made time for the party if they gave you enough notice, but that it's disrespectful to you to send last minute invitations and make you feel obligated to drop everything for them.
“If a man wishes to be sure of the road he treads on, he must close his eyes and walk in the dark.”
- Saint John of the Cross