In this paper we challenge the common evaluation practices used for Virtual Machine (VM) consolidation, such as simulation and small testbeds, which fail to capture the fundamental trade-off between energy consumption and performance. We identify a number of over-simplifying assumptions which are typically made about the energy consumption and performance characteristics of modern networked systems.