I feel like I disappoint him when I end up not staying the night no matter how much he extends the invitation to me or asks me to.

And I hate disappointing people, so I sometimes find myself relenting.
But I should be able to say no, too, regardless of how lonely I feel most nights.