Notifications generated by Keystone are generated in JSON format. Currently, all notifications are immediate, meaning they are generated when a specific event happens. Notifications have a specific top level format:
{
"event_type": "identity.<resource_type>.<operation>",
"message_id": "<message_id>",
"payload": {},
"priority": "INFO",
"publisher_id": "identity.<hostname>",
"timestamp": "<timestamp>"
}The key differences between the two notification formats (Basic and CADF), lie within the payload portion of the notification.
The payload portion of a Basic Notification is a single key-value pair:
{
"resource_info": <resource_id>
}CADF notifications include additional context data around the resource, the action and the initiator. The payload portion of a CADF Notification is a CADF event, which is represented as a JSON dictionary. For example:
{
"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event",
"initiator": {
"typeURI": "service/security/account/user",
"host": {
"agent": "ceilometer-polling keystoneauth1/2.12.1,
"address": "172.18.186.212"
},
"user_id": "e5ac866ebfce4595a707efd97c342b36",
"id": "e5ac866ebfce4595a707efd97c342b36"
},
"target": {
"typeURI": "service/security/account/user",
"id": "f026aee7-20f7-5a7f-965d-300ec50c4686"
},
"observer": {
"typeURI": "service/security",
"id": "9275459bf1e84ecb8aaaa135b4239bf6"
},
"eventType": "activity",
"eventTime": "2016-09-23T11:46:27.616983+0000",
"action": "deleted.user",
"outcome": "success",
"id": "bdfdb6c5-f8b8-50f5-b161-c9af3e85a852"
}- initiator: user that performed the operation
- target: resource that was created/updated/deleted
- action: The action being performed, typically: <operation>. <resource_type>
Resource type
|
Supported operations
|
group
|
create, update, delete
|
project
|
create, update, delete
|
role
|
create, update, delete
|
domain
|
create, update, delete
|
user
|
create, update, delete
|
trust
|
create, delete
|
region
|
create, update, delete
|
endpoint
|
create, update, delete
|
service
|
create, update, delete
|
policy
|
create, update, delete
|
role assignment
|
add, remove
|
None
|
authenticate
|
Sending event notifications
To turn on notifications in keystone we should update keystone config /etc/keystone/keystone.conf .First of all set notification driver(s) to handle sending notifications. or drivers.
Possible values are:
- messaging - send notifications using the 1.0 message format,
- messagingv2 - send notifications using the 2.0 message format (with a message envelope),
- routing - configurable routing notifier (by priority or event_type),
- log - publish notifications via Python logging infrastructure,
- test - store notifications in memory for test verification,
- noop - disable sending notifications entirely.
[oslo_messaging_notifications]
driver = messagingv2
driver = lognotification_format = cadfIf you are using devstack you should have RabbitMQ installed and configured. Otherwise you should install and configure RabbitMQ.
Example configuration in /etc/keystone/keystone.conf:
transport_url = rabbit://stackrabbit:123456@localhost:5672/Reading event notifications
Trigger an event. For example, delete user from Horizon dashboard:
Log
Check Keystone logs. You should have something like this:2016-09-26 07:08:01.212 6631 INFO oslo.messaging.notification.identity.user.deleted [req-0d291906-6fea-4966-af9a-99ccca202745 21ef42a576bd4c80a0d7f49096dca89f eabeb9e0a689433f90d0004517a8a14e - default default] {"event_type": "identity.user.deleted", "timestamp": "2016-09-26 07:08:01.211592", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "initiator": {"typeURI": "service/security/account/user", "host": {"agent": "python-keystoneclient", "address": "192.168.1.21"}, "project_id": "eabeb9e0a689433f90d0004517a8a14e", "user_id": "21ef42a576bd4c80a0d7f49096dca89f", "id": "21ef42a576bd4c80a0d7f49096dca89f"}, "target": {"typeURI": "data/security/account/user", "id": "4a58f45bd3344a8fbf082643df1edc92"}, "observer": {"typeURI": "service/security", "id": "9275459bf1e84ecb8aaaa135b4239bf6"}, "eventType": "activity", "eventTime": "2016-09-26T07:08:01.210879+0000", "action": "deleted.user", "outcome": "success", "id": "d5712427-3fa8-5037-a1d9-7e3ffe94c957", "resource_info": "4a58f45bd3344a8fbf082643df1edc92"}, "priority": "INFO", "publisher_id": "identity.celo", "message_id": "8f3ea446-e6e5-49ff-a4d0-6874f89ec2b6"}RabbitMQ
Check RabbitMQ queue “notifications.info” via web interface:
Read message by clicking at “Get Message(s)” button:
Messages in log file and in the RabbitMQ contains same data.
Consuming event notifications
We are going to use OpenStack Aodh to consume Keystone notifications. Aodh's goal is to enable the ability to trigger actions based on defined rules against sample or event data collected by Ceilometer.Aodh allows users to define alarms which can be evaluated based on events passed from other OpenStack services. The events can be emitted when the resources from other OpenStack services have been updated, created or deleted.
Install and configure Ceilometer and Aodh
To enable aodh and ceilometer in the devstack add following lines to local.conf before running stack.sh:enable_plugin ceilometer https://git.openstack.org/openstack/ceilometer.git
enable_plugin aodh https://git.openstack.org/openstack/aodh masterIn Ceilometer side, a publisher of notifier type need to be configured in the event pipeline config file(event_pipeline.yaml as default), the notifier should be with a messaging topic same as the event_alarm_topic option defined.
---
sources:
- name: event_source
events:
- "*"
sinks:
- event_sink
sinks:
- name: event_sink
transformers:
publishers:
- notifier://
- notifier://?topic=alarm.all- event_type: ['identity.user.*', 'identity.project.*', 'identity.group.*', 'identity.role.*', 'identity.OS-TRUST:trust.*', 'identity.region.*', 'identity.service.*', 'identity.endpoint.*', 'identity.policy.*']
traits: &identity_crud
resource_id:
fields: payload.resource_info
initiator_id:
fields: payload.initiator.id
project_id:
fields: payload.initiator.project_id
domain_id:
fields: payload.initiator.domain_idCreate event alarm in Aodh
The alarming component of Aodh, first delivered in Ceilometer service during Havana development cycle then split out to this independent project in Liberty development cycle, allows you to set alarms based on event.Alarm has an action. There can be multiple form of actions, but only several actions have been implemented so far:
- HTTP callback: you provide a URL to be called whenever the alarm has been set off. The payload of the request contains all the details of why the alarm was triggered.
- log: mostly useful for debugging, stores alarms in a log file.
- zaqar: Send notification to messaging service via Zaqar API.
Aodh uses HTTP callback action to send POST HTTP request to some address. At this address you should have some HTTP server (we will show simple example below in this document). It is very important to understand that you can not specify HTTP request headers or data. Everything is not configurable.
To create event alarm run command:
root@celo:~# aodh alarm create --name user_deleted --description "Keystone user deleted event" --event-type identity.user.deleted --type event --severity critical --repeat-actions True --alarm-action 'http://localhost:5123' --ok-action 'http://localhost:5123' --insufficient-data-action 'http://localhost:5123'
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| alarm_actions | [u'http://localhost:5123'] |
| alarm_id | ceba3ec0-d519-4d10-8915-4d027a3353e1 |
| description | Keystone user deleted event |
| enabled | True |
| event_type | identity.user.deleted |
| insufficient_data_actions | [u'http://localhost:5123'] |
| name | user_deleted |
| ok_actions | [u'http://localhost:5123'] |
| project_id | eabeb9e0a689433f90d0004517a8a14e |
| query | |
| repeat_actions | True |
| severity | critical |
| state | insufficient data |
| state_timestamp | 2016-09-21T09:57:53.664273 |
| time_constraints | [] |
| timestamp | 2016-09-21T09:57:53.664273 |
| type | event |
| user_id | 21ef42a576bd4c80a0d7f49096dca89f |
+---------------------------+--------------------------------------+2016-09-26 06:56:27.927 11713 DEBUG aodh.event [-] Received 1 messages in batch. sample /opt/stack/aodh/aodh/event.py:48
2016-09-26 06:56:27.927 11713 DEBUG aodh.evaluator.event [-] Starting event alarm evaluation: #events = 1 evaluate_events /opt/stack/aodh/aodh/evaluator/event.py:165
2016-09-26 06:56:27.928 11713 DEBUG aodh.evaluator.event [-] Evaluating event: event = {u'event_type': u'identity.authenticate', u'traits': [[u'typeURI', 1, u'http://schemas.dmtf.org/cloud/audit/1.0/event'], [u'eventTime', 1, u'2016-09-26T06:56:27.871206+0000'], [u'outcome', 1, u'success'], [u'initiator_typeURI', 1, u'service/security/account/user'], [u'service', 1, u'identity.celo'], [u'target_id', 1, u'c92f2859-2fb0-5d74-b84d-9de183830135'], [u'observer_id', 1, u'9275459bf1e84ecb8aaaa135b4239bf6'], [u'initiator_id', 1, u'e5ac866ebfce4595a707efd97c342b36'], [u'target_typeURI', 1, u'service/security/account/user'], [u'eventType', 1, u'activity'], [u'observer_typeURI', 1, u'service/security'], [u'action', 1, u'authenticate'], [u'initiator_host_addr', 1, u'172.18.186.212'], [u'initiator_host_agent', 1, u'ceilometer-polling keystoneauth1/2.12.1 python-requests/2.11.1 CPython/2.7.12'], [u'id', 1, u'7b125fcc-d580-572d-83c7-0d237460e911']], u'message_signature': u'8336d3329dfca599e612bd03d73e52b0405504172ea5825a2386f7d792155862', u'raw': {}, u'generated': u'2016-09-26T06:56:27.872018', u'message_id': u'8f4f79f0-089b-4974-84f6-7159a51b28da'} evaluate_events /opt/stack/aodh/aodh/evaluator/event.py:167
2016-09-26 06:56:27.943 11713 DEBUG aodh.evaluator.event [-] Finished event alarm evaluation. evaluate_events /opt/stack/aodh/aodh/evaluator/event.py:1842016-09-26 07:13:26.350 13304 DEBUG aodh.notifier [-] Received 1 messages in batch. sample /opt/stack/aodh/aodh/notifier/__init__.py:98
...
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier [-] Unable to notify alarm ceba3ec0-d519-4d10-8915-4d027a3353e1
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier Traceback (most recent call last):
...
2016-09-26 07:13:26.367 13304 ERROR aodh.notifier ConnectionError: HTTPConnectionPool(host='localhost', port=5123): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe88c4ee150>: Failed to establish a new connection: [Errno 111] Connection refused',))Proxy web server
We created an event alarm in Aodh and it’s sending POST request to http://localhost:5123.Now we have to write our web server. It is possible to use some application instead of writing your own but remember that Aodh don’t allow your to change headers or data of the request. So it means that we need to have some simple proxy web server.
For example:
from flask import Flask, request
import jenkins
import json
server = jenkins.Jenkins('http://172.18.66.118:8080',
username='admin',
password='123456')
app = Flask(__name__)
@app.route("/", methods=['POST'])
def jenkins():
traits = json.loads(request.data)['reason_data']['event']['traits']
data = {"INITIATOR_ID":
filter(lambda x: x[0] == 'initiator_id', traits)[0][-1],
"USER_ID":
filter(lambda x: x[0] == 'resource_id', traits)[0][-1],
"PROJECT_ID":
filter(lambda x: x[0] == 'project_id', traits)[0][-1]
}
server.build_job('transfer_ownership', data)
return "Jenkins job was started"
if __name__ == "__main__":
app.run(port=5123)This example web server listen on port 5123 for POST requests and starts Jenkins job “transfer_ownership” with parameters INITIATOR_ID, USER_ID, PROJECT_ID.
Jenkins
Create an parameterized project “transfer_ownersip” with string parameters INITIATOR_ID, USER_ID, PROJECT_ID:
Add build actions, for example a python script that uses novaclient:
This script just lists instances for deleted user. The output of build will be:
Here should be a trigger for Nova task “transfer ownership”.
Transfer ownership
The most reasonable action for deleted user is to transfer ownership of nova entities to other user. We always have an user id and an id of user who perform the action (initiator_id). So it looks like a good idea to transfer ownership from deleted user to the initiator.There is a spec in Nova: https://review.openstack.org/#/c/105367/ about transferring ownership.
Unfortunately it is an “Abandoned” status right now but it makes sense to update it and to continue work in this direction.
Slack messages
We can also add some post-build action. For example, add Slack notifications:
And we’ll have a messages in Slack channel:
Conclusions
It is possible to configure in the OpenStack notification system that will process notifications about Keystone event like user/project created/updated/deleted. It is also possible to use such notification system to run some post-actions in OpenStack.The most reasonable actions are “transfer ownership” if user/project was deleted or to notify users about Keystone events via messengers like Slack.
Right now Nova doesn’t have ability to perform transfer ownership but it makes sense to implement such feature in Nova project.
OpenStack Aodh is easy to integrate with different enterprise solutions like Jenkins. With such systems like Jenkins we can cover a large variety of use cases.
No comments:
Post a Comment