I think that the fact that only the author can reply to a vote is rather dumb. Many times have I had to bite my tounge because I wasn't the author although I did have insightful and helpful replies to votes. However, there is the problem of people going around replying LOL U R STUPID (although that's not much of a problem here) to all the votes contradict theirs. To solve this, people should know that non author replying is a privilege, not a right. The first time they do it, they will be notified that any more of that kind of trolling action is not permitted, and further trolling will result in a loss of their replying privilege. All trol replies are open to deletion by any user. If they continue trolling, they wil be stripped of their replying privilege, and further replying actions are considered vandalism and are treated as such, regardless of the non-trollness of it. At that point, you would kind of have to be trying. If they feel that they have matured and want to be able to reply again, they will start an Arbitration case to request it back. 3 unbiased arbitors will decide if that person is to get their replying back. If 2/3 say yes, then they are allowed to reply to others' comments.
Note that none of this applies to authors replying to voters' votes, nor the person replied to replying to the author.