Geometric response in topological phases and exotic classical fluids