The essential problem with AI safety: there will always be some people who are willing to roll the dice.
Smart people are often rather dumb about AI. Specifically, smart people tend to hang out with other smart people. So they say silly things like “AIs aren’t nearly as smart as humans”