Dad gets distressed when i leave the care home, is it kinder to visit less often?

My dad moved into a care home a few months ago and his dementia has moved on quite a lot since then. He doesn't know who I am now. Sometimes he thinks I'm his brother, sometimes he just looks at me politely like I'm a stranger who has come to sit with him. I've made my peace with that, or I'm trying to.

The part I can't settle is what happens at the end. He's usually calm while I'm there, but the moment I stand up to go he gets very upset and tries to come with me. The staff are lovely about it and say he settles within twenty minutes or so, but I drive home in bits every time and spend the evening wondering whether I'm the reason he had a bad afternoon.

My sister thinks we should go less often so he isn't unsettled. I can't tell if that's kindness or if we'd just be making it easier on ourselves. Has anyone been through this?