diff --git a/source/_static/images/resource-alert-create.png b/source/_static/images/resource-alert-create.png new file mode 100644 index 0000000000..6f90402716 Binary files /dev/null and b/source/_static/images/resource-alert-create.png differ diff --git a/source/_static/images/resource-alert-details.png b/source/_static/images/resource-alert-details.png new file mode 100644 index 0000000000..c251d923dd Binary files /dev/null and b/source/_static/images/resource-alert-details.png differ diff --git a/source/_static/images/resource-alert-history.png b/source/_static/images/resource-alert-history.png new file mode 100644 index 0000000000..e28cbffa8c Binary files /dev/null and b/source/_static/images/resource-alert-history.png differ diff --git a/source/_static/images/resource-alert-rules.png b/source/_static/images/resource-alert-rules.png new file mode 100644 index 0000000000..da9d1fee17 Binary files /dev/null and b/source/_static/images/resource-alert-rules.png differ diff --git a/source/_static/images/resource-alerts-tab.png b/source/_static/images/resource-alerts-tab.png new file mode 100644 index 0000000000..d67852ca26 Binary files /dev/null and b/source/_static/images/resource-alerts-tab.png differ diff --git a/source/adminguide/events.rst b/source/adminguide/events.rst index b3444730e0..8aaaf97225 100644 --- a/source/adminguide/events.rst +++ b/source/adminguide/events.rst @@ -417,3 +417,9 @@ Webhooks -------- .. include:: events/webhooks.rst + + +Resource Alerts +--------------- + +.. include:: events/resource_alerts.rst diff --git a/source/adminguide/events/resource_alerts.rst b/source/adminguide/events/resource_alerts.rst new file mode 100644 index 0000000000..b6677ca40b --- /dev/null +++ b/source/adminguide/events/resource_alerts.rst @@ -0,0 +1,273 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information# + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +Resource alerts let you get told when a resource goes over (or under) a +value you pick. For example, "CPU of this Instance is above 80%" or +"this storage pool is more than 90% full". + +You create a rule. CloudStack checks the rule every minute. When the +condition is true, an alert is fired. The alert is saved in the alert +history and can be sent out by webhook, email and the event bus. + +Rules can be set on: + +- Instances +- Volumes +- Hosts +- Storage pools + +A rule can watch one resource, or all resources of that type that the +rule owner can see. + +Instance and Volume rules only cover user Instances and their volumes. +System VMs, virtual routers and their volumes are not included. + +Resource alerts can be managed using both API and UI. CloudStack provides +the following APIs: + + .. cssclass:: table-striped table-bordered table-hover + + ======================== ================================= + API Description + ======================== ================================= + createResourceAlertRule Creates a resource alert rule + listResourceAlertRules Lists resource alert rules + updateResourceAlertRule Updates a resource alert rule + deleteResourceAlertRule Deletes a resource alert rule + listResourceAlerts Lists fired resource alerts + ======================== ================================= + +In the UI, rules are under *Monitoring > Resource Alerts*. + +|resource-alert-rules.png| + + +Who can create rules +~~~~~~~~~~~~~~~~~~~~ + +Every account role can create rules, but each one only sees and manages +its own. + +- **User**: rules on their own Instances and Volumes. + +- **Domain Admin**: rules on Instances and Volumes in their domain. A + domain admin can also create a rule for another account in the + domain. + +- **Root Admin**: rules on any resource, including Hosts and Storage + pools. Only the root admin can turn on email for a rule. + +A rule on "all resources" follows the same limits. For a user it covers +only their own resources. For a domain admin it covers the domain. For +the root admin it covers the whole cloud. + +Users and domain admins cannot see or change rules that belong to +someone else. + + +Creating a rule +~~~~~~~~~~~~~~~ + +#. Go to *Monitoring > Resource Alerts* and click *New Resource Alert*. + +#. Fill in the form: + + - **Name**: a name for the rule. + + - **Domain** and **Account**: shown to admins. Pick them to create the + rule for another account. Leave empty to create it for yourself. + + - **Resource type**: Virtual Machine, Volume, Host or Storage Pool. + + - **Resource**: one resource, or *All resources*. + + - **Metric**: what to watch. The list depends on the resource type. + + - **Condition** and **Threshold**: for example *Is above* and *80*. + + - **Severity**: Critical, High, Medium or Low. + + - **Message**: optional text that is added to the alert. + + - **Email**: root admin only. Also send the alert by email. + + - **Cooldown (seconds)**: how long to wait before the same rule + alerts again for the same resource. Leave empty to use + ``resourcealert.repeat.interval.default``. + + - **Webhooks**: optional. Webhooks the alert is sent to. Only webhooks + the rule owner can use are listed. + +#. Click *OK*. + +|resource-alert-create.png| + +The rule details page shows the rule and the webhooks it sends to. It can +be edited, disabled or deleted from there. + +A disabled rule is not checked and fires no alerts, but it keeps its alert +history. Enable it again to start checking. This is useful during planned +maintenance. A disabled rule on one resource does not stop your *All +resources* rule for that resource. + +|resource-alert-details.png| + +Metrics +~~~~~~~ + + .. cssclass:: table-striped table-bordered table-hover + + =============== ========================================================== + Resource type Metrics + =============== ========================================================== + Virtual Machine CPU Utilization %, Memory Utilization %, Disk Read IOPS, + Disk Write IOPS, Disk Read KB/s, Disk Write KB/s, + Network In KB/s, Network Out KB/s + Volume Volume Used (GB), Volume Used % + Host CPU Utilization %, Memory Utilization %, Load Average, + Network In KB/s, Network Out KB/s + Storage Pool Storage Utilization %, Storage Used IOPS + =============== ========================================================== + +Values come from the stats CloudStack already collects. If a resource does +not report a metric, the rule is skipped for it and no alert is fired. For +example, NFS storage pools do not report IOPS, and Instance memory needs +the guest to report it. + +Volume Used is the space the volume takes on the storage, not its disk +size. Volume Used % is that space out of the disk size. + +Disk metrics are only on Instances. CloudStack does not collect disk +reads and writes per volume by default, so a rule on a Volume cannot use +them. + +For metrics that are a percentage, the threshold cannot be more than 100. + + +How alerts are fired +~~~~~~~~~~~~~~~~~~~~ + +- Rules are checked every ``resourcealert.evaluation.interval`` seconds. + +- When a rule's condition is true, an alert is fired. After that, the + same rule does not alert again for the same resource until the + cooldown is over. The check still runs every interval, the cooldown + only controls how often an alert goes out. + +- With more than one management server, only one of them checks the + rules. Alerts are not doubled. + +- If you have a rule on one resource for a metric, your *All resources* + rule for the same metric skips that resource. Rules of other accounts + are not affected. + +- To keep a resource out of *All resources* rules, add the tag + ``resource.alert.opt.out`` with the value ``true`` to it. Rules on that + one resource still work. + +- When a resource or an account is removed, its rules are removed too. + + +Alert history +~~~~~~~~~~~~~ + +Fired alerts can be seen in two places: + +- The *Alert History* tab of a rule. + + |resource-alert-history.png| + +- The *Alerts* tab on the Instance, Volume, Host or Storage pool page. It + lists the alerts for that resource from the rules you can see. + + |resource-alerts-tab.png| + +Old alerts are removed after ``resourcealert.history.retention.days`` +days. Deleting a rule also deletes its alert history. + + +Where alerts are sent +~~~~~~~~~~~~~~~~~~~~~ + +Every alert is saved in the alert history. It can also be sent to: + +- **Webhooks**: to the webhooks picked on the rule. The payload is JSON + with the rule, the resource, the metric, the value, the threshold and + the severity. The event type is ``RESOURCE.ALERT``, so webhook filters + can include or exclude it. The deliveries show in the webhook's + *Recent deliveries* tab and can be sent again from there. See the + Webhooks section above for setting up webhooks. + + Example payload: + + .. code:: json + + { + "event": "RESOURCE.ALERT", + "id": "2695b7b3-7500-4a0f-93ab-dae8b405cae2", + "ruleid": "e6a44e57-a087-4d53-bf29-ac092ee69fa0", + "rulename": "web-01 high CPU", + "resourcetype": "VirtualMachine", + "resourceid": "3e5f346d-7dd8-4a05-af64-9c356d4fce34", + "resourcename": "web-01", + "metric": "CPU_UTILIZATION", + "condition": "GT", + "threshold": 80.0, + "value": 86.9, + "severity": "HIGH", + "message": null, + "timestamp": "2026-09-29T16:11:03.940Z" + } + +- **Email**: when *Email* is on for the rule. The email goes to the + addresses in ``alert.email.addresses`` and uses the same mail server + settings as other CloudStack alerts (``alert.smtp.host``, + ``alert.smtp.port`` and so on). + +- **Event bus**: every alert is published as an alert event with type + ``RESOURCE.ALERT``, when an event bus such as RabbitMQ or Kafka is + set up. See the Notification section above for setting up the event bus. + +Creating, updating and deleting rules also creates the events +``RESOURCE.ALERT.RULE.CREATE``, ``RESOURCE.ALERT.RULE.UPDATE`` and +``RESOURCE.ALERT.RULE.DELETE``. + + +Settings +~~~~~~~~ + + .. cssclass:: table-striped table-bordered table-hover + + ======================================== ======= ====================================================== + Setting Default Description + ======================================== ======= ====================================================== + resourcealert.evaluation.interval 60 Seconds between rule checks. Needs a management + server restart. + resourcealert.repeat.interval.default 600 Cooldown in seconds for rules that don't set one. + resourcealert.history.retention.days 30 Days to keep fired alerts. 0 keeps them forever. + resourcealert.per.user.limit 20 Most rules an account can own, admin accounts + included. 0 is unlimited. Can be set per account. + ======================================== ======= ====================================================== + + +.. Images + + +.. |resource-alert-rules.png| image:: /_static/images/resource-alert-rules.png +.. |resource-alert-create.png| image:: /_static/images/resource-alert-create.png +.. |resource-alert-details.png| image:: /_static/images/resource-alert-details.png +.. |resource-alert-history.png| image:: /_static/images/resource-alert-history.png +.. |resource-alerts-tab.png| image:: /_static/images/resource-alerts-tab.png