The real safety payoff will be when manual driving is banned on public roads. We will have to fight the "freedom is doing whatever I want" people who are going to defend the right to manually drive on public roads.
The non-rich will adopt self driving cars faster than anyone. They will use these cars as a service and not need to own one or pay for insurance. It will be rich people with their sports cars and fancy motorcycles fighting us on this.