It depends on the reason for banning, no? If the account was banned because it is a bot, it makes sense to remove all their activity including votes.
However, if the account was banned for misbehaviour, I think it makes more sense to remove only the offending posts and directly associated votes. E.g. all votes by the offending account in the thread in which the offence took place

I suppose that would address only a part of the issue and there are other, less intrusive ways to mitigate the effects of malicious early down voting. For instance, early down votes could be weighed less.

🇪🇺🇪🇺🇪🇺