If people leave the church, the pastor is to be blamed (or at least the pastor is ultimately responsible). - Paraphrasing my dad.
Many will try to defend pastors in this but pastors are still pastors. Though we must view God's anointed with all reverence; we should discern right from wrong from even the leaders.