This tutorial uses ExampleService to illustrate how to write and on-board a service in CORD. ExampleService is a multi-tenant service that instantiates a VM instance on behalf of each tenant, and runs an Apache web server in that VM. This web server is then configured to serve a tenant-specified message (a string), where the tenant is able to set this message using CORD's administrative interface. From a service modeling perspective, ExampleService extends the base Service model with two fields:
service_message, a string that contains a message to display for the service as a whole (i.e., to all tenants of the service).
tenant_message, a string that is displayed for a specific Tenant.
ExampleService: The snippets of code used throughout this page are available in GitHub.
The result of preparing ExampleService for on-boarding is the following set of files, all located in the xos directory of the ExampleService repository. (There are other helper files, as described throughout this page.)
|Component||Source Code (in https://github.com/opencord/exampleservice/)|
For this tutorial we recommend using CORD-in-a-Box (CiaB) as your development environment. By default CiaB brings up OpenStack, ONOS, and XOS running the R-CORD collection of services. This tutorial demonstrates how to add a new customer-facing service to R-CORD.
CiaB includes a build machine, a head node, switches, and a compute node all running as VMs on a single host. Before proceeding you should familiarize yourself with the CiaB environment.
CORD-in-a-Box: Instructions on how to install a development environment.
Once you’ve prepared your CiaB, the development loop for changing/building/testing service code involves these stages:
- Make changes to your service code and propagate them to your CiaB host. There are a number of ways to propagate changes to the host depending on developer preference, including using gerrit draft reviews, git branches, rsync, scp, etc.
Build XOS container images on the build machine (corddev VM) and publish them to the head node (prod VM). For this step, run the following commands in the
Launch the new XOS containers on the head node (prod VM). For this step, run the following commands in the
prodVM (after the aliases have been defined for the first time, it's only necessary to run line 4):
- Test and verify your changes
- Go back to step #1
CiaB Development Workflow: A description of the XOS development workflow using CiaB.
Define a Model
Create a file named exampleservice.xproto in your service's xos directory. This file encodes the models in the service in a format called xproto which is a combination of Google Protocol Buffers and some XOS-specific annotations to facilitate the generation of service components, such as the GRPC and REST APIs, security policies, and database models among other things.
It consists of three parts:
- The Service model, which manages the service as a whole.
- The Tenant model, which manages tenant-specific (per-service-instance) state.
- Custom model extensions, which let you add arbitrary methods and properties to the generated Django models. Note that the preferred strategy for implementing any extension to your model is to do so via the API, GRPC or REST. However, when placing such code with a model is unavoidable, a custom model extension can be used.
Defining the Service model
A Service model extends (inherits from) XOS' base Service model. At its head is a set of option declarations: the name of the service as a configuration string, and as a human readable one. Then follows a set of field definitions.
Defining the Tenant model
Your tenant model will extend the core TenantWithContainer class, which is a Tenant that creates a VM instance:
The following field specifies the message that will be displayed on a per-Tenant basis:
Think of this as a tenant-specific (per service instance) parameter.
Custom Code Insertions
The declarative xproto part of models is used to generate model definitions in the ORM (Django). It may sometimes be desirable to extend these ORM model definitions with arbitrary code, such as computed fields that process the stored fields in an object and provide a computed result. There are three parts to doing this. The first part is to add a "legacy" option to your xproto file:
The second part is to generate an extension stub for yourself. To do so, run the following commands:
If you run into any dependency errors while generating the models.py stub, please install the following and retry:
$ sudo pip install git+https://firstname.lastname@example.org
$ sudo pip install jinja2
The above command should generate a stub called models.py, which you will fill in to add your custom model extensions. Within this file, you will find definitions for each of your models as Python classes. You must add any additional functionality as methods to these models. Refer to the Django documentation to get a list of special methods, such as save() and delete().
The save() method is called when a model is saved. In the code below, it is overridden to call a "model policy" which in turn creates a container.
Similarly, the delete() method is called to clean up this container.
Similarly, code that goes into the ExampleTenant class:
In CORD 3.0, a view in the GUI is auto-generated for your model. Once example service has been on-boarded you'll find a new view in the side navigation that references it.
In CORD 3.0, the REST API is auto-generated when your service is on-boarded.
Define a TOSCA API
The TOSCA API will also be auto-generated, but not until the next release. For Dangerous-Addition, it must still be coded by hand.
Creating a TOSCA resource for your service allows your service to be configured using Tosca. The first step in creating a TOSCA API is to create any TOSCA custom types used by your service. ExampleService's custom types are in a file called xos/exampleservice.m4: (TODO: move this to tosca/custom_types)
The above is written in the m4 macro language, and uses a couple of macros (xos_base_props, xos_base_service_props). You'll have to remember to run the m4 tool on it whenever you edit it,
m4 exampleservice.m4 > exampleservice.yaml.
You will typically create a python file for each important model of your service, and that python file tells how to translate between the TOSCA and REST representations. Here is ExampleServices' Tosca resource for the ExampleService object, located in tosca/resources/exampleservice.py:
Define a Synchronizer
Synchronizers are processes that run continuously, checking for changes to models. When a synchronizer detects a change, it will apply that change to the underlying system. For exampleservice, the Tenant model is the model we will want to synchronize, and the underlying system is an Instance. In this case, we’re using TenantWithContainer, which creates a Virtual Machine Instance for us.
XOS Synchronizers are typically located in the xos/synchronizer directory of your service.
Create a file named
model-deps with the contents:
NOTE: This is used to track model dependencies using
tools/dmdot, but that tool currently isn’t working.
Create a file named
To configure this module, create a file named
exampleservice_from_api_config, which specifies various configuration and logging options:
NOTE: Historically, synchronizers were named “observers”, so s
/observer/synchronizer/when you come upon this term in the XOS code/docs.
Create a directory within your synchronizer directory named
steps, create a file named
Bring in some basic prerequities. Also include the models created earlier, and SyncInstanceUsingAnsible which will run the Ansible playbook in the Instance VM.
Two optional sections, "watches" and "handle_service_monitoringagentinfo_watch_notifications" can be added to tie the service in with the A-CORD monitoring service. If you don't need monitoring, then you may omit those sections. If you wish to use the monitoring service, then please consult the monitoring service documentation for information on how to create those sections of the sync step.
Next, create a run-from-api.sh file for your synchronizer.
Finally, create a Dockerfile for your synchronizer, name it "Dockerfile.synchronizer" and place it in the synchronizer directory with the other synchronizer files:
Create Ansible Playbooks
In the same
steps directory, create an Ansible playbook named
exampletenant_playbook.yml which is the “master playbook” for this set of plays:
This sets some basic configuration, specifies the host this Instance will run on, and the two variables that we’re passing to the playbook.
In this case, there are two roles, one which installs Apache, and one which creates the
index.html file from a Jinja2 template.
Create a directory named roles inside
steps, then create two directories named for your roles,
install_apache, create a directory named
tasks, then within that directory, a file named
main.yml. This will contain the set of plays for the
install_apache role. To that file add the following:
This will use the Ansible apt module to install Apache.
create_index, create two directories,
templates. In templates, create a file named
index.html.j2, with the contents:
These Jinja2 Expressions will be replaced with the values of the variables set in the master playbook.
tasks directory, create a file named
main.yml, with the contents:
This uses the Ansible template module to load and process the Jinja2 template then put it in the
dest location. Note that there is no path given for the
src parameter - Ansible knows to look in the
templates directory for templates used within a role.
As a final step, you can check your playbooks for best practices with ansible-lint if you have it available.
Define an On-boarding Spec
Each service should have an on-boarding recipe. By convention, we use <servicename>-onboard.yaml, and place it in the xos directory of the service.
The on-boarding recipe is a TOSCA specification that lists all of the resources for your synchronizer. It's basically a collection of everything that has been created above. For example, here is the on-boarding recipe for ExampleService:
You will also need to modify the profile-manifest in platform-install to on-board your service. To do this, modify the xos_services and xos_service_sshkeys sections as shown below:
The above modifications to the profile manifest will cause platform-install to automatically install an ssh key for your service, and to onboard the service at build time.
Service On-boarding (Dangerous-Addition): Describes the internals of how services are on-boarded.
Test Your Service
This section still needs to be updated from 2.0.
Execute the makefile target you created in the previous step, for example,
For example, to test exampleservice do the following:
In the Admin web UI, navigate to the Slice ->
<slicename> -> Instances, and find an IP address starting with
10.11.X.X in the Addresses column (this address is the “nat” network for the slice, the other address is for the “private” network).
curl <10.11.X.X address>, and you should see the display message you entered when creating the ExampleTenant.
user@ctl:~/xos/xos/configurations/devel$ curl 10.11.10.7 ExampleService Service Message: "Example Service Message" Tenant Message: "Example Tenant Message"
After verifying that the text is shown, change the message in the “Example Tenant” and “Example Service” sections of the Admin UI, wait a bit for the Synchronizer to run, and then the message that
curl returns should be changed.
This section still needs to be updated from 2.0.
XOS isn’t coming up after making changes
Verify that the docker containers for XOS are running with:
sudo docker ps
If you need to see log messages for a container:
sudo docker logs <docker_container>
There’s also a shortcut in the makefile to view logs for all the containers:
If you want to delete the containers, including the database, and start over, run:
Which will delete the containers.
“500 Internal Server Error” when navigating the admin webpages
This is most likely Django reporting a problem in
Django’s debug log is located in in
/var/log/django_debug.log on the xos container, so run
make enter-xos and then look at the end of that logfile.
“Ansible playbook failed” messages
The logs messages for when the Synchronizer runs Ansible are located in
/opt/xos/synchronizers/<servicename>/sys in its synchronizer container. There are multiple files for each Tenant instance, including the processed playbook and stdout/err files . You can run a shell in the docker container with this command to access those files:
sudo docker exec -it devel_xos_synchronizer_<servicename>_1 bash