Monday, September 26, 2016

Keystone notifications: integration with Jenkins and Slack

We are going to set up such notification system:

Notifications generated by Keystone are generated in JSON format. Currently, all notifications are immediate, meaning they are generated when a specific event happens. Notifications have a specific top level format:
    "event_type": "identity.<resource_type>.<operation>",
    "message_id": "<message_id>",
    "payload": {},
    "priority": "INFO",
    "publisher_id": "identity.<hostname>",
    "timestamp": "<timestamp>"
Where <resource_type> is a Keystone resource, such as user or project, and <operation> is a Keystone operation, such as created, deleted.
The key differences between the two notification formats (Basic and CADF), lie within the payload portion of the notification.
The payload portion of a Basic Notification is a single key-value pair:
    "resource_info": <resource_id>
Where <resource_id> is the unique identifier assigned to the resource_type that is undergoing the <operation>.
CADF notifications include additional context data around the resource, the action and the initiator.  The payload portion of a CADF Notification is a CADF event, which is represented as a JSON dictionary. For example:
    "typeURI": "",
    "initiator": {
        "typeURI": "service/security/account/user",
        "host": {
            "agent": "ceilometer-polling keystoneauth1/2.12.1,
            "address": ""
        "user_id": "e5ac866ebfce4595a707efd97c342b36",
        "id": "e5ac866ebfce4595a707efd97c342b36"
    "target": {
        "typeURI": "service/security/account/user",
        "id": "f026aee7-20f7-5a7f-965d-300ec50c4686"
    "observer": {
        "typeURI": "service/security",
        "id": "9275459bf1e84ecb8aaaa135b4239bf6"
    "eventType": "activity",
    "eventTime": "2016-09-23T11:46:27.616983+0000",
    "action": "deleted.user",
    "outcome": "success",
    "id": "bdfdb6c5-f8b8-50f5-b161-c9af3e85a852"
Where the following are defined:
  • initiator:  user that performed the operation
  • target: resource that was created/updated/deleted
  • action: The action being performed, typically: <operation>. <resource_type>
The following table displays the compatibility between resource types and operations.

Resource type
Supported operations
create, update, delete
create, update, delete
create, update, delete
create, update, delete
create, update, delete
create, delete
create, update, delete
create, update, delete
create, update, delete
create, update, delete
role assignment
add, remove

Sending event notifications

To turn on notifications in keystone we should update keystone config /etc/keystone/keystone.conf .
First of all set notification driver(s) to handle sending notifications. or drivers.
Possible values are:
  • messaging - send notifications using the 1.0 message format,
  • messagingv2 - send notifications using the 2.0 message format (with a message envelope),
  • routing - configurable routing notifier (by priority or event_type),
  • log - publish notifications via Python logging infrastructure,
  • test - store notifications in memory for test verification,
  • noop - disable sending notifications entirely.
We are going to select messagingv2 and log:
driver = messagingv2
driver = log
Set the format of messages:
notification_format = cadf
Restart Keystone.
If you are using devstack you should have RabbitMQ installed and configured. Otherwise you should install and configure RabbitMQ.
Example configuration in /etc/keystone/keystone.conf:
transport_url = rabbit://stackrabbit:123456@localhost:5672/

Reading event notifications

Trigger an event. For example, delete user from Horizon dashboard:


Check Keystone logs. You should have something like this:
2016-09-26 07:08:01.212 6631 INFO oslo.messaging.notification.identity.user.deleted [req-0d291906-6fea-4966-af9a-99ccca202745 21ef42a576bd4c80a0d7f49096dca89f eabeb9e0a689433f90d0004517a8a14e - default default] 
{"event_type": "identity.user.deleted", "timestamp": "2016-09-26 07:08:01.211592", 
"payload": {"typeURI": "", 
"initiator": {"typeURI": "service/security/account/user", "host": {"agent": "python-keystoneclient", "address": ""}, "project_id": "eabeb9e0a689433f90d0004517a8a14e", "user_id": "21ef42a576bd4c80a0d7f49096dca89f", "id": "21ef42a576bd4c80a0d7f49096dca89f"}, 
"target": {"typeURI": "data/security/account/user", "id": "4a58f45bd3344a8fbf082643df1edc92"}, 
"observer": {"typeURI": "service/security", "id": "9275459bf1e84ecb8aaaa135b4239bf6"}, 
"eventType": "activity", "eventTime": "2016-09-26T07:08:01.210879+0000", 
"action": "deleted.user", "outcome": "success", "id": "d5712427-3fa8-5037-a1d9-7e3ffe94c957", "resource_info": "4a58f45bd3344a8fbf082643df1edc92"}, 
"priority": "INFO", "publisher_id": "identity.celo", "message_id": "8f3ea446-e6e5-49ff-a4d0-6874f89ec2b6"}


Check RabbitMQ queue “” via web interface:

Read message by clicking at “Get Message(s)” button:

Messages in log file and in the RabbitMQ contains same data.

Consuming event notifications

We are going to use OpenStack Aodh to consume Keystone notifications. Aodh's goal is to enable the ability to trigger actions based on defined rules against sample or event data collected by Ceilometer.
Aodh allows users to define alarms which can be evaluated based on events passed from other OpenStack services. The events can be emitted when the resources from other OpenStack services have been updated, created or deleted.

Install and configure Ceilometer and Aodh

To enable aodh and ceilometer in the devstack add following lines to local.conf before running
enable_plugin ceilometer
enable_plugin aodh master
Config the Ceilometer to be able to publish events to the queue the aodh-listener service listen on. The event_alarm_topic config option of Aodh identify which messaging topic the aodh-listener on, the default value is “alarm.all”.
In Ceilometer side, a publisher of notifier type need to be configured in the event pipeline config file(event_pipeline.yaml as default), the notifier should be with a messaging topic same as the event_alarm_topic option defined.
    - name: event_source
          - "*"
          - event_sink
    - name: event_sink
          - notifier://
          - notifier://?topic=alarm.all
In the event_definitions.yaml you can configure json fields of the event data:
- event_type: ['identity.user.*', 'identity.project.*', '*', 'identity.role.*', 'identity.OS-TRUST:trust.*', 'identity.region.*', 'identity.service.*', 'identity.endpoint.*', 'identity.policy.*']
  traits: &identity_crud
      fields: payload.resource_info
      fields: payload.initiator.project_id
      fields: payload.initiator.domain_id
We are keeping default devstack Aodh configuration.

Create event alarm in Aodh

The alarming component of Aodh, first delivered in Ceilometer service during Havana development cycle then split out to this independent project in Liberty development cycle, allows you to set alarms based on event.
Alarm has an action. There can be multiple form of actions, but only several actions have been implemented so far:
  • HTTP callback: you provide a URL to be called whenever the alarm has been set off. The payload of the request contains all the details of why the alarm was triggered.
  • log: mostly useful for debugging, stores alarms in a log file.
  • zaqar: Send notification to messaging service via Zaqar API.
Let’s create alarm for a Keystone ‘identity.user.deleted’ event. Aodh calls such alarms “event alarms” and has a special type for them. We are going to use HTTP callback action for the our alarm.
Aodh uses HTTP callback action to send POST HTTP request to some address. At this address you should have some HTTP server (we will show simple example below in this document). It is very important to understand that you can not specify HTTP request headers or data. Everything is not configurable.
To create event alarm run command:
root@celo:~# aodh alarm create --name user_deleted --description "Keystone user deleted event" --event-type identity.user.deleted --type event --severity critical --repeat-actions True --alarm-action 'http://localhost:5123' --ok-action 'http://localhost:5123' --insufficient-data-action 'http://localhost:5123'
| Field                     | Value                                |
| alarm_actions             | [u'http://localhost:5123']           |
| alarm_id                  | ceba3ec0-d519-4d10-8915-4d027a3353e1 |
| description               | Keystone user deleted event          |
| enabled                   | True                                 |
| event_type                | identity.user.deleted                |
| insufficient_data_actions | [u'http://localhost:5123']           |
| name                      | user_deleted                         |
| ok_actions                | [u'http://localhost:5123']           |
| project_id                | eabeb9e0a689433f90d0004517a8a14e     |
| query                     |                                      |
| repeat_actions            | True                                 |
| severity                  | critical                             |
| state                     | insufficient data                    |
| state_timestamp           | 2016-09-21T09:57:53.664273           |
| time_constraints          | []                                   |
| timestamp                 | 2016-09-21T09:57:53.664273           |
| type                      | event                                |
| user_id                   | 21ef42a576bd4c80a0d7f49096dca89f     |
Now we can test Aodh. Just trigger event and look at the Aodh logs.Aodh-listener:
2016-09-26 06:56:27.927 11713 DEBUG aodh.event [-] Received 1 messages in batch. sample /opt/stack/aodh/aodh/
2016-09-26 06:56:27.927 11713 DEBUG aodh.evaluator.event [-] Starting event alarm evaluation: #events = 1 evaluate_events /opt/stack/aodh/aodh/evaluator/
2016-09-26 06:56:27.928 11713 DEBUG aodh.evaluator.event [-] Evaluating event: event = {u'event_type': u'identity.authenticate', u'traits': [[u'typeURI', 1, u''], [u'eventTime', 1, u'2016-09-26T06:56:27.871206+0000'], [u'outcome', 1, u'success'], [u'initiator_typeURI', 1, u'service/security/account/user'], [u'service', 1, u'identity.celo'], [u'target_id', 1, u'c92f2859-2fb0-5d74-b84d-9de183830135'], [u'observer_id', 1, u'9275459bf1e84ecb8aaaa135b4239bf6'], [u'initiator_id', 1, u'e5ac866ebfce4595a707efd97c342b36'], [u'target_typeURI', 1, u'service/security/account/user'], [u'eventType', 1, u'activity'], [u'observer_typeURI', 1, u'service/security'], [u'action', 1, u'authenticate'], [u'initiator_host_addr', 1, u''], [u'initiator_host_agent', 1, u'ceilometer-polling keystoneauth1/2.12.1 python-requests/2.11.1 CPython/2.7.12'], [u'id', 1, u'7b125fcc-d580-572d-83c7-0d237460e911']], u'message_signature': u'8336d3329dfca599e612bd03d73e52b0405504172ea5825a2386f7d792155862', u'raw': {}, u'generated': u'2016-09-26T06:56:27.872018', u'message_id': u'8f4f79f0-089b-4974-84f6-7159a51b28da'} evaluate_events /opt/stack/aodh/aodh/evaluator/
2016-09-26 06:56:27.943 11713 DEBUG aodh.evaluator.event [-] Finished event alarm evaluation. evaluate_events /opt/stack/aodh/aodh/evaluator/
2016-09-26 07:13:26.350 13304 DEBUG aodh.notifier [-] Received 1 messages in batch. sample /opt/stack/aodh/aodh/notifier/
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier [-] Unable to notify alarm ceba3ec0-d519-4d10-8915-4d027a3353e1
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier Traceback (most recent call last):
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier ConnectionError: HTTPConnectionPool(host='localhost', port=5123): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe88c4ee150>: Failed to establish a new connection: [Errno 111] Connection refused',))
We have an error “Failed to establish a new connection”. It is because we don’t have any web server at http://localhost:5123.

Proxy web server

We created an event alarm in Aodh and it’s sending POST request to http://localhost:5123.
Now we have to write our web server. It is possible to use some application instead of writing your own but remember that Aodh don’t allow your to change headers or data of the request. So it means that we need to have some simple proxy web server.
For example:
from flask import Flask, request
import jenkins
import json

server = jenkins.Jenkins('',
app = Flask(__name__)

@app.route("/", methods=['POST'])
def jenkins():
    traits = json.loads(['reason_data']['event']['traits']
    data = {"INITIATOR_ID":
                filter(lambda x: x[0] == 'initiator_id', traits)[0][-1],
                filter(lambda x: x[0] == 'resource_id', traits)[0][-1],
                filter(lambda x: x[0] == 'project_id', traits)[0][-1]
    server.build_job('transfer_ownership', data)
    return "Jenkins job was started"

if __name__ == "__main__":
We use Flask( and Openstack python-jenkins ( libraries.
This example web server listen on port 5123 for POST requests and starts Jenkins job “transfer_ownership” with parameters INITIATOR_ID, USER_ID, PROJECT_ID.


Create an parameterized project “transfer_ownersip”  with string parameters INITIATOR_ID, USER_ID, PROJECT_ID:

Add build actions, for example a python script that uses novaclient:

This script just lists instances for deleted user. The output of build will be:

Here should be a trigger for Nova task “transfer ownership”.

Transfer ownership

The most reasonable action for deleted user is to transfer ownership of nova entities to other user. We always have an user id and an id of user who perform the action (initiator_id). So it looks like a good idea to transfer ownership from deleted user to the initiator.
There is a spec in Nova: about transferring ownership.
Unfortunately it is an “Abandoned” status right now but it makes sense to update it and to continue work in this direction.

Slack messages

We can also add some post-build action. For example, add Slack notifications:

And we’ll have a messages in Slack channel:


It is possible to configure in the OpenStack notification system that will process notifications about Keystone event like user/project created/updated/deleted. It is also possible to use such notification system to run some post-actions in OpenStack.
The most reasonable actions are “transfer ownership” if user/project was deleted or to notify users about Keystone events via messengers like Slack.
Right now Nova doesn’t have ability to perform transfer ownership but it makes sense to implement such feature in Nova project.
OpenStack Aodh is easy to integrate with different enterprise solutions like Jenkins. With such systems like Jenkins we can cover a large variety of use cases.

No comments:

Post a Comment