Session

A co-resident VM refers to a virtual machine (VM) that shares physical hardware resources with other VMs on the same physical host. Co-locating containers pertain to the practice of deploying multiple containers within the same physical host or node in a containerized environment, such as Docker or Kata Containers.

First, VMM (virtunal machine monitor) offers significant advantages over native machines when VMs co-resident on the same physical host are non-competing in terms of network and computing resources. However, the performance of VMs is significantly degraded compared to that of native machine when co-resident VMs are competing for resources under high workload demands due to high overheads of switches and events in host/guest domain and VMM. Second, the communication overhead between co-resident VMs can be as high as the communication cost between VMs located on separate physical machines. This is because the abstraction of VMs supported by VMM technology does not differentiate whether the data request is coming from co-resident VMs or not. More importantly, when using TCP/IP as the communication protocol, the overhead of Linux networking stack itself is also significant for both VM and container cases.

Although vsock or traditional Unix domain sockets already provide an optimized alternative specifically for inter-VM or inter-container communications, we argue that lack of transparency to applications is the reason why vsock or Unix domain socket is not yet widely adopted. Instead of introducing more socket families, we propose a novel and more optimal solution using shared memory with eBPF to bypass TCP/IP stack completely and transparently to bring co-resident VM communications to optimal. Unlike SMC (Shared Memory Communications), it does not even require LD_PRELOAD.

More importantly, with the recent introduction of BPF arena, we could possibly build an eBPF map on top of Inter-VM Shared Memory (IVSHMEM). And with eBPF socket maps and sockops infrastructures, we are armed with all the foundations we need to establish shared memory communications for co-resident VM and containers.