Hello, Rosie!
This is going to be a long post and I deeply apologize. Hopefully it's worth it!
As someone who works heavily within the spirit realm, so to speak (rather, I can communicate with spirits, I've dealt a lot with them - good and bad), I don't think it's necessarily a bad thing that he's following you around. Based on what you're saying, I do agree that he's more than likely dead and that you accidentally called him in. Does that mean he's a negative entity? No, not at all. Not everything you accidentally call in is bad.
As Myrth said, shielding is important and casting out any negative energies is as well. If he stays around, then he's not bad. When working with someone far more experienced in this realm than I, we discussed shielding in depth and about how smudging/banishing/shielding will keep out the negative energies and they will not come through. It will not BANISH them, per say, but it will keep them away from YOU. Does that make sense?
As for how to shield, there are various ways to do so and not all ways work for everyone. I will list a few that might help you find what works best for you and you can try some out to see what you like. I'll put them at the end so that you don't have to scroll to find them, as I'm not done explaining yet.
I feel a very strong inkling that perhaps he had daughters and you remind him of them. Whether in the specific sense of his daughter/s or the fact that you are female (at least I think I saw you were, if you're not then please accept my apology) that he enjoys being around you because he never got to with his own children or something of that sort. There are always different reasons as to why a spirit likes to hang around.
You will know if you're dealing with something bad. Trust me. You'd feel it, even if you weren't sensitive but the fact that you're able to see him tells me you are. Those who are sensitive will know, and I have very, very strong confidence in saying so because negative beings, despite liking to hide themselves as pure and good, will give off negative vibes. Good spirits do not make us feel fear or like something's watching us in a more paranoid sense. You'd feel discomfort, yes, but nothing too extreme.
With negative energies, you'd feel very stiff, anxious, I don't know how else to explain it... All that I've spoken to who are sensitive get this same feeling. It's a very good sense to pay attention to, because it will tell you if you need to shield more or smudge again when you feel this discomfort.
If your spirit man is still hanging around after you exercise all that you can for protection, then chances are he means no harm. Whether or not he wishes to leave this realm is up to him and you could ask him if he wishes to go. If he does, then you can find ways to help him and if he doesn't, then you cannot force him to leave to whatever the final realm of rest is, depending on belief. He may be dead but I come from the belief that spirits still have thoughts and feelings just like a living, breathing human. You wouldn't want to be forced to leave somewhere you're comfortable with if you've done nothing wrong, right?
Anyways, here are some helpful things and the shielding exercises!
If you are unable to burn sage, I also have a very helpful bottle spray that you can try making. The recipe is found here:
Blossom Breeze Spray
Shielding Exercises:
1) Close your eyes and keep your feet flat on the floor. Imagine your feet sinking into the ground, becoming the roots of a tree (this is called Grounding). Feel them sink deeper and deeper below the ground. Eventually they will reach an element, and for each person the element is different. Sometimes it's water, sometimes it's fire, sometimes it's something else. Bring your element up through your body and feel it spreading through you as well as around you in a giant bubble of peace and protection. The bubble color is different for everyone as well and I've encountered with myself and others that our deities typically step in with their own elements at times to help with protection.
2) Close your eyes and imaging yourself bringing in white, gold and silver light through your crown chakra and through to your feet. Feel it cleansing you of all negativities and darkness. All the while imagine the same shining light surrounding your body and any area you wish to protect/shield. Think of happy things, and ask your guides as well as your deities you work with (if you work with any) to step in and offer their protection. Nothing that walks in the dark may enter and only those that walk in the light can. Imagine all darkness and dark beings being cast out to keep you safe.
3) Bless a ring or any jewelry piece by any manner you wish. I usually put mine in blessed water and then run over candle smoke from a candle that's been anointed with an oil of my choice. Then I put it back into the water and invest positive feelings into it by thinking happy thoughts while holding it. I ask my deities to come in and bless the ring or necklace so that while I wear it, I will be protected. Then what I also do is tap the jewelry piece (mine is a necklace and I tap the charm) and keep my eyes closed. I tap it only three times and imagine my element bubble surrounding me and getting larger with each tap.
I also have an entire thread dedicated to spirit communication and protection right here:
How to Communicate and Stay Protected.