Let us assume for the sake of this thought experiment that Leon Brittan was, personally, utterly innocent of all the accusations that have been brought against him.
Leon Brittan was Home Secretary. That means he was in charge of the oversight of both the police and the security services. While he was in charge, somehow neither the police nor the security services ever told him that a serial sexual offender, Jimmy Saville, was regularly socializing with both the Prime Minister and the Royal family. Moreover, since Saville spent several Christmases at Chequers and so did most of the Cabinet, Brittan was, whether he knew it or not, in the somewhat invidious position of socializing with Saville on several occasions. Either he knew and was silent, or was utterly incompetent, to an extent that makes him complicit by omission,
Somehow, on his watch, Kincora was covered up. Jersey was covered up. Cyril Smith was able to bully journalists with people who posed as Special Branch while raping children with impunity. Parts of the BBC were full of rapists.
At least one dossier was handed to Brittan which he either lost or placed in the circular file.
People are upset with Tom Watson for calling Brittan evil. Even assuming his utter personal innocence as far as actions go, Brittan and his predecessors and successors in one of the high offices of state have to be held accountable to all the victims of abuse in those years for what he did not do to protect them.
I think evil covers it.