Several studies show that, the more your site is slow, the more netizens want to leave and not return. This is directly related!
Since 2010 the position in search results of Google (and probably other search engines) takes into account loading time of pages. This means that if your site is slow, Google may place your page to lower positions and let the first ones to the fastest sites that may be your competitors
Cache decreases consumptions of CPU, RAM and database connections so you can choose a lower pack or you can decrease resources allocation in your hosting provider.