If you have a permit, I don’t think there is any way he could prevent you from proceeding.
If the city granted the permit, it means his permission or input is not necessary. In some places, it is. Building anything in La Jolla always seems to require neighbors’ approval. In Clairemont, not so much.
The permit process should surely weed out projects that could cause landslides. So, he can go to the city and say “hey, I wasn’t informed about this and it could cause a landslide” The city will say “it isn’t your property so you don’t have to be informed” and “Show us your engineering proof that it will cause a landslide and we’ll consider it. By the way, here are the engineered plans that show it won’t cause a landslide.” Unless he’s a brilliant engineer and knows how to work the city planning department, I’d give him zero chance of convincing them.
Anything is possible, though – so just make sure your insurance covers landslides caused by a new deck, built with permits. You should be OK there.
The nice thing to do would be to explain how the permit process protects him, assure him he’s safe, thank him for his concern, let him know you have his safety in mind, yada, yada. Then start building as soon as possible.