Supposedly there are kids on the forum who downvote for various reasons. However there is ample evidence that some mods are downvoting as they please, based on criteria only known to themselves. This is eroding the trust. Whenever I post a question I wonder how soon will I be downvoted for unknown reasons.
The problem with downvoting is not merely your reputation score. It makes users less likely to read the question and eventually answer it.