Your husband may be perfectly safe around kids. So may every other male your child comes into contact with. Don't depend on your husband's lack of desire to keep your child safe. Don't depend on your child not coming into contact with pedophiles in order that he not be groomed, seduced or raped.
Your husband's behavior is outside your control. So is every other adult your child comes into contact with. Let your husband deal with his own demons/fantasies/desires. Teach your son how to say, "NO." Teach your son how to get away from someone who is trying to hurt him. Learn about the techniques men use to isolate, groom, and manipulate boys and teach your son to be alert when someone (anyone) tries to force him into sex.
That is really the only way to guarantee his safety, give him the tools to fight back and get away.