Hyper-V Time Sync issues... FIXED!

I've been using Microsoft Hyper-V Server for a while now, and I've run into an issue in Linux operating systems where the time would skew VERY badly.  Installing NTPD didn't help, as it only updates the clock if it's within 128ms of the servers it is polling from... and running a cron job to update the time just wasn't accurate enough.
The time would skew more than a few minutes in the space of 1 hour.  This is VERY unacceptable.

But!  as the title suggests... there is a fix.  (for me it's a simple fix... for others... may not be so simple)  So, here we go!

The problem occurs because of an issue where the clock in an OS isn't based on the "hardware clock" that is kept alive by a battery... and is only sorta-accurate... but rather the OS's clock is based on a set number of cpu cycles.  In a physical machine, this provides a clock that is much finer-grained than the traditional hardware-time... which is only reports whole-second increments.  When you are doing highly time-sensitive things, you need a much more accurate clock.  For example,  VoIP (ulaw RTP audio streams) traditionally breaks audio data up into 20ms chunks of audio into 1 packet of data sends that.  On the other end, those packets are put into a special sort of buffer that takes those 20ms bits and reassembles them into a continuous stream of audio.  If you only had a clock accurate within 1 second... you'd have some SERIOUS delay in conversations.

Today, a VERY large number of things in computers require a highly accurate clock.  Rather than each application trying to have it's own clock, operating systems provide APIs that every applications can rely on for an accurate time source.  I am not 100% sure with all operating systems, but I do know that Linux has one such kernel-clock that is not based on the hardware-clock.  There are kernel options that can be set to define how accurate this clock is... (1/1000ms, 1/100ms, etc....) but that's not really very relevant to this topic.  In short, during the startup process, the kernel starts the os clock based on the hardware clock... and some sort of algorithm for defining a number of cpu cycles per "tick", and continues to count from there... and on shutdown sets the hardware clock to the OS's clock...  Typically, in the middle, services like NTPD can make the OS's clock much more precise with regards to the actual time as defined by the NIST.

So, what goes wrong in a Virtual environment? (not just hyper-v)  Well, cpu cycles are virtual.  There are several different things at play all of which can make the number of cycles per tick a variable rather than a constant.   Most virtual server frameworks (if not all) provide some sort of compensation to the guest OS'es to *appear* like they're getting a constant number of cycles, but this wreaks havoc if the guest OS doesn't quite understand what the host OS has done.  In the case of Hyper-V, extra cpu cycles are thrown at the guest OS to try & push the clock forward periodically when it thinks the guest OS might have missed some.  This *can* help, but in the case of most Linux OSes, they just steadily count the extra cpu cycles, and the OS clock skews forward.  The fix? Well, this is where it gets a bit more tricky.

A Kernel Module Saves the Day!!  Actually, this idea isn't as strange as it sounds.  Other virtualization frameworks have "integration" tools that do exactly this... and other functions which we really aren't worried about at this point.  We want Linux Guests in Hyper-V to keep time!  Microsoft was sooo thoughtful to provide us with the tools we need.  The "Linux Integration Services v2.1 for Windows Server 2008 Hyper-V R2" was written specifically for this purpose!  We're Saved!  ...or are we?  Well, if you read the fine print, it's only supported in a CRAZY-short list of Linux operating systems.

• SUSE Linux Enterprise Server 10 SP3 x86 and x64 (up to 4 vCPU)
• SUSE Linux Enterprise Server 11 x86 and x64 (up to 4 vCPU)
• Red Hat Enterprise Linux 5.2, 5.3, 5.4, and 5.5 x86 and x64 (up to 4 vCPU)

Well... that's a start, Microsoft appears to want to be friends with the Linux community... heck, they even went as far to get several of the pieces of the Linux Integration Services integrated into the kernel.  Wow!  Microsoft creating kernel drivers?   AMAZING!.... wait... why doesn't my OS work then?  Well the down side is that Microsoft managed to get the driver into the kernel, but failed to keep it there due to a GPL violation.  So, you can't expect to see the hyper-v bits in any mainstream linux repositories anytime soon...

But this is not the end!  This Linux Integration Services are still there and still can be useful.  There's a few *gotchyas* but we are mainly focused on 1 feature... time sync.

So, without any further-ado... here's what you need to do:

1) Download the Linux Integration Services package from Microsoft, and extract the files to someplace convenient.  We're only really interested in the LinuxIC v21.iso at this point.

2) Mount the .iso into your guest OS and mount the virtual cdrom to someplace convenient.

mkdir /mnt/cdrom; mount /dev/cdrom /mnt/cdrom

3) make a copy of the cdrom-stuff on the local guest OS.  (the cdrom isn't writable <shock>)

mkdir /opt/linux_ic_v21_rtm; cp /mnt/cdrom/* /opt/linux_ic_v21_rtm

4) get your guest OS ready to build a kernel module.  I'm using Debian 5, but your os should have something similar... (basically, just need to install the build-tools & kernel source)

apt-get install build-essential linux-source module-assistant
m-a update && m-a prepare 

5) Fix one line in the script/determine_os script.  Apparently, Microsoft didn't want to build the module for every kernel, just those 2.6.27 or greater.  Unfortunately, I'm running 2.6.26.

This may be a bit iffy, but for my kernel, all I needed to do was change line 40 from:

if [ $KERNEL_VER -ge 27 ]

to:

if [ $KERNEL_VER -ge 26 ]

This may work for other kernel versions, and the entire script might be better modified to support other kernels, but I am not 100% sure of what is & what is not supported.  I figured that 2.6.26 has very few (if any  ) differences in the system clock's functions.  (the entire package was designed to work with kernels 2.6.27 and kernel 2.6.9)

6)  Build *only* the hv_timesource module.  The other bits are very kernel-version specific.  On the other side of the coin... they do contain the other nifty paravirtual drivers.... but I am not a kernel developer (or any kind of programmer) and can't tell you how to fix the compile errors.

make hv_timesource

7)  If all goes well, the hv_timesource.ko module will be built!  Finally, we just need to load it.

insmod src/hv_timesource.ko

Final notes:  At this point, the module should be loaded, and the clock shouldn't drift anymore!  That being said, it may still be wrong, so it may be useful to set it.  You can use "ntpdate" or even pull the time from the hardware clock using "hwclock --hctosys".  This should at least get you started, but you'll still need to make this module auto-load on startup so it will be there after reboots... and if you should upgrade the kernel version, you may need to rebuild the module manually.

I'd be a very happy person if Microsoft would split up their "integration services" package into pieces.  They do have closed-source bits (which is why they were in trouble for violating the GPL) that they can keep as an add-on package... but I honestly can't see any reason why this bit should be kept from the mainstream kernel releases.  This is yet another example of Microsoft playing the "see we integrate with linux" game... without actually integrating with linux.