(Bill Weinberg, EntrepriseLinux) Why don't we see more pervasive use of virtualization in HPC? Well, you may have heard this: There ain't no such thing as a free lunch. In some cases, virtualization technology may not (yet) meet legacy HPC requirements; in others, HPC systems providers and deployers are not comfortable departing from familiar (and expensive) technology acquisition paths and roadmaps. [...]

Virtualization overhead and increased latency
HPC architects have focused the greater part of their efforts on optimizing hardware to achieve maximum computing throughput. Investments included heavy parallelization of processing units and use of hierarchical memory design with high-speed interconnects to ensure maximum utilization of those CPUs. As such, defenders of traditional HPC attack virtualization for inserting a layer of abstraction between software loads and carefully tuned hardware. Virtualization, they claim, not only induces execution latency within and among parallel processors, and the delays can be highly variable or "jittery."

Advocates of modern virtualization technology respond that virtualization not only avoids many of these dreaded latency issues, it can actually enhance HPC performance. For example, virtualization facilitate use of specialized OSes, optimized for classes of HPC applications (e.g., Red Hat CHAOS or legacy mainframe OSes), A hypervisor can guarantee resource allocations to a VM with an HPC guest, dedicating a partition of physical memory or a percentage of CPU cycles, or guaranteeing a maximum latency to time-sensitive code (e.g., interrupt processing). Nodes of a virtual cluster can run concurrently, on multiple actual real nodes or using different processor cores of one or several physical nodes. Gang scheduling can allow a cluster-based HPC application, while running, to communicate between nodes in real time, as it would without virtualization on legacy HPC hardware.