vr: release DHCP leases of expunged VMs on redundant routers

[calvix]

Description

With redundant routers, the lease of an expunged VM never gets released from dnsmasq. If a new VM later gets the same IP, the primary router refuses to give it out and the VM comes up without an address.

On a redundant router the guest NIC has the router's own IP as primary and the gateway (VIP) as secondary, and dnsmasq only listens on the VIP (listen-address=127.0.0.1,<VIP>), so the VIP is what it uses as its DHCP server identifier. On expunge CsDhcp.py calls dhcp_release, which puts the first address of the interface into option 54 - the router IP, not the VIP. dnsmasq ignores a DHCPRELEASE that isn't addressed to its own server identifier, without logging anything, so the lease just stays in memory.

The fallback from apache#13194 doesn't help either. It writes a new leases file over the old one and reloads dnsmasq, but dnsmasq only reads that file on startup, so the lease is still there, and from then on dnsmasq keeps writing to the deleted file.

Non-redundant routers aren't affected, there dnsmasq listens on the router IP, which is what dhcp_release sends.

The fixed IP in the steps below is only there to make it reproducible. In normal use the IP is picked at random, and a freed IP is a normal candidate, so it happens whenever a new VM happens to land on the IP of an expunged one. Leases are infinite, so these stuck ones stay until dnsmasq is restarted.

How to reproduce

Create an isolated network offering with redundant routers and a network from it:

cmk create networkoffering name=redundant displaytext=redundant guestiptype=Isolated traffictype=GUEST \
  supportedservices=Dhcp,Dns,SourceNat \
  "serviceproviderlist[0].service=Dhcp" "serviceproviderlist[0].provider=VirtualRouter" \
  "serviceproviderlist[1].service=Dns" "serviceproviderlist[1].provider=VirtualRouter" \
  "serviceproviderlist[2].service=SourceNat" "serviceproviderlist[2].provider=VirtualRouter" \
  "servicecapabilitylist[0].service=SourceNat" "servicecapabilitylist[0].capabilitytype=RedundantRouter" "servicecapabilitylist[0].capabilityvalue=true" \
  "servicecapabilitylist[1].service=SourceNat" "servicecapabilitylist[1].capabilitytype=SupportedSourceNatTypes" "servicecapabilitylist[1].capabilityvalue=peraccount"
cmk update networkoffering id=<offering> state=Enabled
cmk create network name=rnet displaytext=rnet zoneid=<zone> networkofferingid=<offering>

Deploy a VM with a fixed IP and expunge it:

cmk deploy virtualmachine zoneid=<zone> serviceofferingid=<so> networkids=<network> templateid=<template 1> ipaddress=10.1.1.71
cmk destroy virtualmachine id=<A> expunge=true

Then deploy another one on the same IP:

cmk deploy virtualmachine zoneid=<zone> serviceofferingid=<so> networkids=<network> templateid=<template 2> ipaddress=10.1.1.71

One catch: dnsmasq looks leases up by client identifier first, so if the second VM sends the same client id as the first one, it just takes over the old lease and you won't see the problem. Clones of a template with a baked-in /etc/machine-id do exactly that. If your templates regenerate the machine-id the same template is fine, otherwise use a different one for the second VM.

The second VM gets no IP, and /var/log/dnsmasq.log on the primary router has:

dnsmasq-dhcp[3606]: not using configured address 10.1.1.71 because it is leased to 02:01:00:ce:00:04
dnsmasq-dhcp[3606]: DHCPDISCOVER(eth0) 02:01:00:ce:00:05 no address available

There's no DHCPRELEASE line for the first VM anywhere in the log. ls -l /proc/$(pidof dnsmasq)/fd shows /var/lib/misc/dnsmasq.leases (deleted), and reading that fd still shows the old lease.

You can also see the release being dropped directly on the primary router with dhcp_release eth0 <ip> <mac> for any VM - the lease stays and nothing shows up in the log.

What the fix does

• The DHCPRELEASE is now sent with the address dnsmasq actually listens on in the VM's network, taken from listen-address in cloud.conf (the VIP on a redundant router, the router IP otherwise). CsHelper.send_dhcp_release() builds the same packet as dhcp_release and sends it the same way, just with the right server identifier. Guests still see the VIP as the DHCP server, so Windows passwords cannot be set or reset when the system is on an isolated network with dual routers apache/cloudstack#11877 / Password server on redundant VRs binds only the VRRP VIP, so guests never receive their password apache/cloudstack#13720 stay fixed.
• Before falling back, it waits up to 2 seconds for dnsmasq to drop the lease from the file. Before, the check ran right after sending.
• The Ensure VM IP is removed from VR DHCP records in dnsmasq.leases after expunge VM apache/cloudstack#13194 fallback now removes the line in place instead of replacing the file, and does systemctl try-restart dnsmasq instead of a reload. On the backup router dnsmasq isn't running, so try-restart doesn't do anything there.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial