To me, the sets of "downvote-worthy" and "response-worthy" comments overlap very little. I downvote when a comment is not even wrong. When I comment in response to another comment, it's because I've seen some sign that the other person might be interesting, correct, amusing, curious, or any combination of those. A requirement to compose "justification" comments would just about end my downvoting.
Why? If it's so cut and dry, it should not be a problem to explain this.
I would of course assume that less people downvoting + more specific complaints would lead to more moderator action on the more obnoxious/distracting posters, if that wasn't the case I wouldn't recommend the change either.
