I'm going to go ahead and pretend you said this:
Why are my votes disappearing when I downvote the same user repeatedly, but the upvotes remain? Is he somehow canceling out my votes? I don't want to jump to any absurdly paranoid conclusions, I'm just curious what's wrong
The engine has a number of ways to try and detect vote fraud. One of them involves a single user downvoting another user repeatedly in a short time span. The site assumes that user has pissed you off and you're just downvoting them to be mean, so it cancels out the downvotes automatically:
If you irritate another user, they might be having a bad day and decide to take it out on you by methodically going through and downvoting every post you’ve made. The impact of this is pretty limited on Stack Overflow, as you only get 30 votes per day, and upvotes are +10 while downvotes are only -2 (and -1 to the votee).
... it’s unacceptable behavior, and I’ve been getting several reports of this sort of revenge downvoting now, enough that we needed to take action to combat it.
Individual users don't have the ability to cancel downvotes against them, that would kind of defeat the point. Neither do moderators; they can't even see whether or not you downvoted a post. I'm assuming that's what happened in your case; it's hard to know for sure without you providing more details