This guide describes how to power up a previously-installed CORD POD that has been powered down (cleanly or otherwise). The end goal of the power up procedure is a fully functioning CORD POD. The guide applies to the CORD 3.0 release.
Boot the head node
Physical POD: power on the head node
CiaB: bring up the prod VM:
$ cd ~/cord/build; vagrant up prod
Check the head node services
Verify that mgmtbr and fabric interfaces are up and have IP addresses
Verify that MAAS UI is running and accessible:
Troubleshooting: MAAS UI not available on CiaB
If you are running a CiaB and there is no webserver on port 8080, it might be necessary to refresh port forwarding to the prod VM. Run
ps ax|grep 8080and look for an SSH command (will look something like this):
31353 pts/5 S 0:00 ssh -o User=vagrant -o Port=22 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ForwardX11=no -o IdentityFile="/users/acb/cord/build/targets/cord-in-a-box/.vagrant/machines/prod/libvirt/private_key" -L *:8080:192.168.121.14:80 -N 192.168.121.14
A workaround is to kill this process, and then copy and paste the command above into another window on the CiaB server to set up a new SSH port forwarding connection.
Verify that the following Docker containers are running: mavenrepo, switchq, automation, provisioner, generator, harvester, storage, allocator, registry
sudo lxc listto ensure that juju lxc containers are running. If any are stopped, use
sudo lxc start <name>to restart them.
- Verify that juju services are running with
juju status --format tabular. They should all have a status of
Unit is ready. If they do not have this status, with the service name run
juju resolved --retry <servicename>/0, and then rerun the
juju status ...to see if the issue is resolved.
Verify that the following OpenStack commands work:
nova list --all-tenants
Troubleshooting: OpenStack commands give SSL error
Sometimes Keystone starts up in a strange state and OpenStack commands will fail with various SSL errors. To fix this, it is often sufficient to run:
$ ssh ubuntu@keystone sudo service apache2 restart
Power on leaf and spine switches
On a physical POD: power on the switches.
On CiaB: bring up the switch VMs:
$ cd ~/cord/build; vagrant up leaf-1 leaf-2 spine-1
Check the switches
On the head node (i.e., prod VM for CiaB):
- Get switch IPs by running:
cord prov list
- Verify that
pingworks for all switch IPs
Boot the compute nodes
On a physical POD and CiaB: Log into the MAAS UI and power on the compute node.
Check the compute nodes
Once the compute nodes are up:
- Login to the head node
nova service-listshows the compute node as “up”.It may take a few minutes until the node's status is updated in Nova.
- Verify that you can log into the compute nodes from the head node as the ubuntu user
Verify that XOS UI is running and accessible:
$ cd /opt/cord_profile; docker-compose -p rcord restart
- Make sure the compute nodes have COMPLETE status.
- Prior to rebooting existing OpenStack VMs:
- Make sure some ports show up
- If not, try this:
onos> cordvtn-sync-neutron-states <keystone-url> admin admin <password>
onos> cordvtn-sync-xos-states <xos-url> firstname.lastname@example.org <password>
Boot OpenStack VMs
To bring up OpenStack VMs that were running before the POD was shut down:
- Get list of VM IDs:
nova list --all-tenants
- For each VM:
nova start <vm-id>
nova console-log <vm-id>
- Inspect the console log to make sure that the network interfaces get IP addresses.
To restart a vSG inside the vSG VM:
SSH to the vSG VM
sudo rm /root/network_is_setup
Save the vSG Tenant in the XOS UI
Once the synchronizer has re-run, make sure can ping 220.127.116.11 from inside the vSG container
sudo docker exec -ti vcpe-222-111 ping 18.104.22.168