OpenStack nova VM migration (live and cold) call flow

OpenStack nova compute supports two flavors of Virtual Machine (VM) migration:

Cold migration -- migration of a VM which requires the VM to be powered off during the migrate operation during which time the VM is inaccessible.

Hot or live migration -- zero down-time migration whereupon the VM is not powered off during the migration and thus remains accessible.

Understanding these VM migration operations from an OpenStack internals perspective can be a daunting task. I had the pleasure of digging into these flows in the latter part of 2013 and as part of that effort created a rough outline of the internal flows. Other's I've worked with found these flow outlines useful and thus they're provided below.

Note -- The outlines below were created based on the OpenStack source in late 2013 and thus reflect the state of OpenStack at that point in time.

Live Migration Flow:

nova.api.openstack.compute.contrib.admin_actions._migrate_live()

nova.compute.api.live_migrate()

update instance state to MIGRATING state

call into scheduler to live migrate (scheduler hint will be set to the host select (which may be none)).

nova.scheduler.manager.live_migration()

nova.scheduler.manager._schedule_live_migration()

nova.conductor.tasks.live_migrate.LiveMigrationTask.execute()

check that the instance is running

check that the instance's host is up

if destination host provided, check that it..
is different than the instance's host

is up

has enough memory

is compatible with the instance's host (i.e. hypervisor type and version)

passes live migration checks (call using amqp rpc into nova manager check_can_live_migrate_destination)

else destination host not provided, find a candidate destination host and check that it...
is compatible with the instance's host (i.e. hypervisor type and version)

passes live migration checks (call using amqp rpc into nova manager check_can_live_migrate_destination)

call using amqp rpc into nova manager live_migration
Note: Migration data is initially set by check_can_live_migrate_destination and can be used for implementation specific parameters from this point.

nova.compute.manager.check_can_live_migrate_destination()

driver.check_can_live_migrate_destination()

call using amqp rpc into nova manager check_can_live_migrate_source

driver.check_can_live_migrate_destination_cleanup()

nova.compute.manager.check_can_live_migrate_source()

determine if the instance is volume backed and add result to the migration data

driver.check_can_live_migrate_source()

nova.compute.manager.live_migration()

if block migration request then driver.get_instance_disk_info()

call using amqp rpc into nova manager pre_live_migration

Error handler: _rollback_live_migration

driver.live_migration()

nova.compute.manager.pre_live_migration()

get the block device information for the instance

get the network information for the instance

driver.pre_live_migration()

setup networks on destination host by calling the network API setup_networks_on_host

driver.ensure_filtering_rules_for_instance()

nova.compute.manager._rollback_live_migration()

update instance state to ACTIVE state

re-setup networks on source host by calling the network API setup_networks_on_host

for each instance volume connection call using amqp rpc into nova manager remove_volume_connection

if block migration or volume backed migration without shared storage

call using amqp rpc into nova manager rollback_live_migration_at_destination

nova.compute.manager._post_live_migration()

driver.get_volume_connector()

for each instance volume connection call the volume API terminate_connection

driver.unfilter_instance()

call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start

call using amqp rpc into nova manager post_live_migration_at_destination

if block migration or not shared storage driver.destroy()

else driver.unplug_vifs()

tear down networks on source host by calling the network API setup_networks_on_host

nova.compute.manager.post_live_migration_at_destination()

setup networks on destination host by calling the network API setup_networks_on_host

call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish

driver.post_live_migration_at_destination()

update instance to ACTIVE state

setup networks on destination host by calling the network API setup_networks_on_host

nova.compute.manager.rollback_live_migration_at_destination()

tear down networks on destination host by calling the network API setup_networks_on_host

driver.destroy()

nova.compute.manager.remove_volume_connection()

call _detach_volume

driver.get_volume_connector()

remove the volume connection by calling the volume API terminate_connection

nova.compute.manager._detach_volume()

driver.detach_volume()

Since the live migration failed the VM should not be on the destination host. So this should be a no-op.

If there is an exception detaching the volume then rollback the detach by calling the volume API roll_detaching

Cold Migration Flow:

nova.api.openstack.compute.servers._resize()

nova.api.openstack.compute.contrib.admin_actions._migrate()

nova.compute.api.resize()

if flavor_id is not passed, migrate host only and keep the original flavor

else flavor_id is given, migrate host and resize to new flavor

lookup the image for the instance by calling the image API show

check quota headroom and reserve

update instance to RESIZE_PREP state

determine if the instance's current host should be ignored as a migration target and update filter properties for the scheduler accordingly

call into scheduler to prep_resize

nova.scheduler.manager.prep_resize()

call scheduler driver to schedule_prep_resize

if no valid host was found then update instance to ACTIVE state and rollback quota reservation

if error occurred then update instance to ERROR state and rollback quota reservation

nova.scheduler.filter_scheduler.schedule_prep_resize()

run through scheduler filters to select host

call using amqp rpc into nova manager prep_resize

nova.compute.manager.prep_resize()

if no node specified call driver.get_available_nodes()

call _prep_resize

if an exception occurs then call into scheduler to prep_resize again if possible

nova.compute.manager._prep_resize()

if same host is used then ensure that the same host is allowed (as per configuration)

call using amqp rpc into nova manager resize_instance

nova.compute.manager.resize_instance()

get network and instance information

update instance to RESIZE_MIGRATING state

get block device information

call driver.migrate_disk_and_power_off()

call _terminate_volume_connections

call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start

update instance to RESIZE_MIGRATED state

call using amqp rpc into nova manager finish_resize

nova.compute.manager._terminate_volume_connections()

if there is a volume connection to terminate

driver.get_volume_connector()

for each volume connection remove the connection by calling the volume API terminate_connection

nova.compute.manager.finish_resize()

call _finish_resize

if successful commit the quota reservation

else rollback the quota reservation and update instance to ERROR state

nova.compute.manager._finish_resize()

if the flavor is changing then update the instance with the new flavor

setup networks on destination host by calling the network API setup_networks_on_host

call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish

update instance to RESIZE_FINISHED state

refresh and get block device information

driver.finish_migration()

update instance to RESIZED state

Cold migration confirm flow:

nova.api.openstack.compute.servers._action_confirm_resize()

nova.compute.api.confirm_resize()

reserve quota for decrease in resource usage

call amqp rpc into nova manager confirm_resize

nova.compute.manager.confirm_resize()

tear down networks on source host by calling the network API setup_networks_on_host

driver.confirm_migration()

update instance to ACTIVE (or possibly STOPPED) state

commit the quota reservation

Cold migration revert flow:

nova.api.openstack.compute.servers._action_revert_resize()

nova.compute.api.revert_resize()

reserve quota for increase in resource usage

update instance task state to RESIZE_REVERTING

call amqp rpc into nova manager revert_resize

nova.compute.manager.revert_resize()

tear down networks on destination host by calling the network API setup_networks_on_host

call into conductor to network_migrate_instance_start which will eventually call the network API migrate_instance_start

get block device information

driver.destroy()

call _terminate_volume_connections

drop resize resources claimed on destination

call amqp rpc into nova manager finish_revert_resize

nova.compute.manager.finish_revert_resize()

update instance back to pre-resize values

re-setup networks on source host by calling the network API setup_networks_on_host

refresh and get block device information

driver.finish_revert_migration()

update instance to RESIZE_REVERTING state

call into conductor to network_migrate_instance_finish which will eventually call the network API migrate_instance_finish

update instance to ACTIVE (or possibly STOPPED) state

commit the quota usage

Source: http://bodenr.blogspot.com/2014/03/openstack-nova-vm-migration-live-and.html