I do think it is VERY important to make sure that when you are visiting in Sunday, especially Sunday evenings, that things are planned and dinner is served at an appropriate time. People need to get home on Sundays and have time to relax and plan out stuff before starting work again on Monday, so if you are having someone over for Sunday dinner you need to be prepared to serve dinner early and not be offended if people want to leave to get home at a decent time.
There can't be guilt trips because people want to rest before starting a work week, especially if you yourself don't work. It's just the way things should be!
Saturday nights are the time for late dinners and hanging around after eating, but Sundays are the last moments of freedom before working again.