Israeli Prime Minister Benjamin Netanyahu has ordered the Mossad chief and his team to leave Qatar and return to Israel, after negotiations hit "dead end."