Making machine learning on HPC systems cost-effective and carbon-friendly