You are not supposed to listen to the majority. You are supposed to listen to your own common sense that knows what G-d has said and what He wants of us.
This is all a result of the sin of the 12 spies and it has dogged us throughout history.
Devarim 5778-Sparks of Rashi for the 9 days from Rabbi Sholom Gold on Vimeo.