Session

The future of SO_TIMESTAMPING

SO_TIMESTAMPING[1] is key to debugging network in kernel and end-to-end app latency. [2] states that by using SO_TIMESTAMPING, bugs that are otherwise incorrectly assumed to be network issues can be attributed to the kernel.

There are a few areas that need optimization for usability and performance[3]. These include: uAPI compatibility, extra system call overhead, and the need for application modification. Our initial solution to solve these issues constituted writing a kernel module that hooks various key functions. However, this approach is not suitable to land in the kernel officially, even though it has been deployed successfully in production. For this reason and based on feedback we took an approach of going with am eBPF extension approach.

In this talk, we will discuss our eBPF extension for SO_TIMESTAMPING. Our approach transparently equips every matched flow to enable such feature without modifying any application code. Such an approach will accelerate deployment in production because it is not intrusive at all. More than that, the feature has two other dominant merits we measured with the contrast of the traditional BPF related tracing tools: perfect anti-interference and much less performance impact of monitored application, which are clearly manifested by a group of experimental statistics.

One of our major goals in this work is cover all networking transmission paths. By utilizing eBPF, we’re able to capture all tx/rx path state of sockets and skbs as the last skb of each sendmsg traverses each key function in the hot path. This gives us an excellent insight into each flow than was ever possible before.

In the long run, the features we are enabling could potentially be used to trace every skb when administrators need to diagnose and debug by acquiring as many fields as needed analysis and as a result more issues that contribute to latency can be analyzed. It is our hope that this feature goes as far as to replace the core functionality of tcpdump in production and cope well with various complex issues appearing in kernel.

In conclusion, it can be a very promising feature in the world of tracing and debugging.

[1] https://netdevconf.info/0x13/session.html?talk-tcp-timestamping

[2] https://netdevconf.info/0x17/sessions/talk/so_timestamping-powering-fleetwide-rpc-monitoring.html

[3] https://netdev.bots.linux.dev/netconf/2024/jason_xing.pdf