Myrth and Xiao are absolutely correct. Shielding and grounding are essential. For you and for him as well. If you have the kind of relationship that you can speak openly about things-then hopefully you guys can work out a code and you can lovingly let him know, without judgement, that it's happening in the moment....like giving a previously agreed upon cue.
If that is not the case you need to protect yourself. Working on shielding practice is essential. I visualize a crystal bubble more than a light. That's just my technique. I think of a membrane and let my energy go out...but it doesn't allow energy in.....maybe in some cases as an empath you need to let the membrane keep your energy in?
When in a case with a person who really is being draining I do the the following. I subtly cross my legs or feet. Easy to do sitting or standing and leaning. I join my hands together somehow and and while I listen I touch my tongue to the roof of my mouth. I slowly breathe in through my nose and out through my mouth either side of my tongue. This closes my energy circuit and helps me immeasurably. You can do this so it doesn't look like you are twisting yourself like a pretzel.
Here's the thing. You need to prepare yourself that if you do this correctly it may be that your partner senses the lack of energy coming off you. You may experience gnashing of teeth if you don't give them what they want, and they are having a moment in which they aren't particularly aware of themselves. I'm sure many of us have had this happen with a PV friend when you effectively learn to cut off the flow. You will see an increase in PV behaviour and frustration, but you won't be drained.
If you can, you may wish to visualize them tapping into the earth's energy and being filled that way....