Introduction

The XOS Monitoring service provides a generic platform to support real time network observability for SDN fixed and mobile networks in a Telco Central Office, such that any 3rd party Analytic solution can be brought into this framework.

The platform is being developed as part of CORD (Central Office Re-architected as Datacenter), and focuses on fixed/residential network environment in the initial phases. The main goals of this platform are:

As shown in Figure 1, the Monitoring service fetches probe information from different CORD network elements, including compute servers, white-box switches, I/O devices, and software-based services running on the compute servers, and make it available to the one or more Analytics applications running on top of CORD.  Initially, this platform is being integrated with CORD residential services such as vSG (Virtualized Subscriber Gateway), vOLT (Virtualized OLT), vRouter (Virtualized Router), although architecturally it is extendable to other services such as Mobility and Enterprise services.

Figure 1. High-level organization of the XOS Monitoring Service.

Architecture

The Monitoring service leverages the open source OpenStack Ceilometer framework, and enhances it with few key features, as depicted in Figure 2 ( http://docs.openstack.org/developer/ceilometer/).

Figure 2. Openstack Ceilometer with A-CORD enhancements.

As shown in Figure 3, the service provides both Notification-based and Polling-based measurement collection mechanisms on its southbound side (interfacing to the underlying services and devices), and on the northbound side, provides Query-based and Publish/Subscribe-based interfaces towards analytic applications. The analytic application correlate these events coming from different network sources into more meaningful events for performing any real-time closed feedback loop operations, but may also feed those events back into the Monitoring framework, such that other applications can make use of them in their analysis.

In addition, the Monitoring service provides a platform to dynamically control the probe information available in the underlying services and devices. As part of this functionality, it also provides flexibility to analytics applications to instantiate “application-specific” probe functions in the network, and potentially mirrors selective traffic to those functions. This would be done to perform deeper levels of instrumentation that would otherwise not available from the underlying devices. The probe functions can directly feed the probe data directly to applications or feed them back into the Monitoring framework, such that other application can make use of those events.

Figure 3. Internal implementation based on OpenStack’s Ceilometer.

In aligning with XOS core principle of Everything-as-a-Service (XaaS), the Monitoring framework is imported into XOS as an elastically scalable, multi-tenant service. It does this by taking advantage of XOS service construction toolkit (e.g., Data Model, Synchronizer). Each of the components can then be scaled up and down based on workload. Multi-tenancy, as shown in Figure 4, is achieved by creating a lightweight proxy container for every tenant of the Monitoring service, such that each tenant is able to access only to the instrumentation data of the network resources belonging to that tenant. Typically the tenants would be the analytics applications running on top of this service.

Figure 4. Monitoring service integrated into XOS.

The following sections details out usage of the XOS Monitoring service  and briefly touches on internal details of this service.

Setting Up the Service

Refer to Build A-CORD components for setting up Monitoring service.

Creation of Service Tenants

The tenants for the Monitoring service can be created either using GUI or using REST API (The REST URL: http://<xos endpoint>/xoslib/monitoringchannel/)

Create Tenant: curl -X POST -v -G -H "Accept: application/json; indent=4" -u padmin@vicci.org:pass http://<xos endpoint>/api/tenant/monitoring/monitoringchannel/

Query Tenants: curl -v -G -H "Accept: application/json; indent=4" -u padmin@vicci.org:pass http://<xos endpoint>/api/tenant/monitoring/monitoringchannel/


When a Monitoring service tenant is created, the backend synchronizer picks an available openstack resource and launches a docker container for this tenant and pushes the list of access controls to be applied for this tenant.

The tenant container hosts a simple proxy web server that translates ceilometer queries between tenant application and backend openstack ceilometer module after applying configured access controls.

The URL to proxy web server along with list of authorized openstack project IDs is returned back to the tenant as part of the Monitoring service tenant creation (Tenant application should poll for this information using the GET method). Example ceilometer tenant info is shown below

{
    "humanReadableName": "ceilometer-tenant-8",
    "id": 8,
    "provider_service": 4,
    "service_specific_attribute": "{\"instance_id\": 4, \"creator_id\": 1}",
    "ceilometer_url": "http://10.11.10.20:8888/",
    "tenant_list_str": "f28d2e522e1f466a95194c10869acd0c, d41cdd2ade394e599b40b9b50d9cd623",
    "creator": "padmin@vicci.org",
    "instance": "mysite_ceilometer-4",
    "computeNodeName": "cp-1.xos-ceilometer.xos-pg0.clemson.cloudlab.us"
} 

 

The tenant application should use the "ceilometer_url" specified in the return value to use ceilometer service. Below are few example ceilometer service queries:

GET all tenant meters: curl http://10.11.10.20:8888/v2/meters 

GET all tenant resources: curl http://10.11.10.20:8888/v2/resources

GET statistics for a given meter type: curl http://10.11.10.20:8888/v2/meters/vcpe/statistics 

GET samples for a given meter type: curl 'http://10.11.10.20:8888/v2/samples?q.field=meter&q.op=eq&q.value=vcpe'

GET statistics for a given meter type based on a query: curl 'http://10.11.10.20:8888/v2/meters/vcpe/statistics?q.field=timestamp&q.field=timestamp&q.op=ge&q.op=le&q.type=&q.type=&q.value=2015-10-19+01%3A08%3A11.478998%2B00%3A00&q.value=2015-10-20+01%3A08%3A11.478998%2B00%3A00&period=86400' 


Adding New Meters

Ceilometer supports two mechanisms to collect statistics data from different components as shown in below Figure 4:

Figure 4. Openstack Ceilometer data collection mechanism.

The following section describes the mechanisms used to integrate meters from CORD “vcep” service into ceilometer. (The “vcpe” service is now called vSG.) The same procedure can be reused for other services as well.

The Push mechanism is used to integrate meters from CORD “vcpe” services into ceilometer. A high level workflow and set of components involved in collection of “vcpe” measurements is shown below:

Figure 5. Components involved in vcpe data collection.

As shown above, adding new meters to the ceilometer framework using Push mechanism needs the following components:

Typical workflow to define new meters using Push mechanism:

  


Software Components

The XOS Monitoring service has the following components:

Figure 6. Software components of XOS Monitoring service.

Models

There are two Django models defined for Ceilometer service: CeilometerService and MonitoringChannel, also known as CeilometerTenant. The relationship of these two models with XOS core models is shown below.

Figure 7. Ceilometer service data model.

The MonitoringChannel model is inherited from the core XOS data model TenantWithContainer  and holds list of authorized tenants along with ceilometer proxy URL for a given tenant.  

Synchronizer

The role of the Ceilometer synchronizer is to monitor the associated data model for any changes and ensure the state of backend system is up-to-date with latest data model changes. Ceilometer synchronizer uses Ansible playbook to reflect any data model changes to the backend.

When a new Ceilometer tenant data model is created or updated, the synchronizer performs the following operations:

Front-end APIs

Like any other XOS service, Ceilometer provides three front-end mechanisms to interact with this services:

Summary

The Monitoring service runs in CORD infrastructure, providing real-time instrumentation data from different elements. This enables network operators to run a one or more Analytics applications to monitor and react to various aspects of the network, including anomaly detection and mitigation, congestion avoidance in the fabric, customer care troubleshooting/diagnosis, and so on.