Versions Compared

Old Version 2

changes.mady.by.user Antti Kallio

Saved on

compared with

New Version 3

changes.mady.by.user Antti Kallio

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The ID service may have already added Continuous Queries and Retention Policies that meet your needs. Check the existing queries with the influx prompt:

Code Block
breakoutModefull-width
> use oneportal
Using database oneportal

> show continuous queries
name: _internal
name query
---- -----

name: oneportal
name                                         query
----                                         -----
openid_api_requests_hourly                   CREATE CONTINUOUS QUERY openid_api_requests_hourly ON oneportal BEGIN SELECT count(auth_header_exists) INTO oneportal.one_year.openid_api_requests_hourly FROM oneportal.one_day.openid_api_request GROUP BY time(1h), path, method END
openid_api_requests_daily                    CREATE CONTINUOUS QUERY openid_api_requests_daily ON oneportal BEGIN SELECT sum(count) INTO oneportal.five_years.openid_api_requests_daily FROM oneportal.one_year.openid_api_requests_hourly GROUP BY time(1d), path, method END
mgmt_api_requests_hourly                     CREATE CONTINUOUS QUERY mgmt_api_requests_hourly ON oneportal BEGIN SELECT count(auth_header_exists) INTO oneportal.one_year.mgmt_api_requests_hourly FROM oneportal.one_day.mgmt_api_request GROUP BY time(1h), path, method, api_client_id, oauth2_client_id END
mgmt_api_requests_daily                      CREATE CONTINUOUS QUERY mgmt_api_requests_daily ON oneportal BEGIN SELECT sum(count) INTO oneportal.five_years.mgmt_api_requests_daily FROM oneportal.one_year.mgmt_api_requests_hourly GROUP BY time(1d), path, method, api_client_id, oauth2_client_id END
log_entry_hourly                             CREATE CONTINUOUS QUERY log_entry_hourly ON oneportal BEGIN SELECT count(event_id) INTO oneportal.one_year.log_entry_hourly FROM oneportal.one_day.log_entry GROUP BY time(1h), target_namespace_id, target_type, source, log_level, log_life END
log_entry_daily                              CREATE CONTINUOUS QUERY log_entry_daily ON oneportal BEGIN SELECT sum(count) INTO oneportal.five_years.log_entry_daily FROM oneportal.one_year.log_entry_hourly GROUP BY time(1d), target_namespace_id, target_type, source, log_level, log_life END
filterable_service_find_max_results_hourly   CREATE CONTINUOUS QUERY filterable_service_find_max_results_hourly ON oneportal BEGIN SELECT max(results) INTO oneportal.one_year.filterable_service_find_max_results_hourly FROM oneportal.one_day.filterable_service_find GROUP BY time(1h), service_class END
filterable_service_find_count_queries_hourly CREATE CONTINUOUS QUERY filterable_service_find_count_queries_hourly ON oneportal BEGIN SELECT count(results) INTO oneportal.one_year.filterable_service_find_count_queries_hourly FROM oneportal.one_day.filterable_service_find GROUP BY time(1h), service_class END

name: ruuvitag
name query
---- -----

...

This existing continuous query data can be used in Grafana with the following configuration:

...

Creating a

...

new RP and CQ

Let’s suppose there isn’t a CQ and RP that meet our requirements.

...

We need to have a Grafana display which shows Management API request counts by path, and we need to show data for the last 30 days. We need to be able to display data for quite a short interval, so let’s pick an interval of 5 minutes.

In summary:

  • We need to store metrics data about Management API requests

  • For 30 days

  • With 5 minute interval accuracy

  • This means we will have 30 * 24 * (60/5) = 8640 measurements per any stored tag combination. This is a reasonable amount of data to process.

What we already have

We can show a graph for the last 24 hours with the following Grafana Panel configuration:

...

Create a Continuous Query

Let’s create a CQ that collects everything usable from default mgmt_api_request data. We can find the tag and field names with commands like SELECT * FROM mgmt_api_request LIMIT 1.

Code Block
breakoutModefull-width
> CREATE CONTINUOUS QUERY "mgmt_api_requests_5min_for_1month" ON "oneportal" BEGIN
  SELECT count(auth_header_exists) AS "count_requests", sum("request_duration") AS "sum_request_duration", mean("request_duration") AS "mean_request_duration"
  INTO "one_month"."mgmt_api_requests"
  FROM "mgmt_api_request"
  GROUP BY time(5m), "path", "method", "api_client_id", "oauth2_client_id"
  END

We could have created a CQ that collects only the number of requests and groups them by path, but in anticipation of other needs, we collect more data and more tags.

Note:

  • The data will be stored with RP one_month and name mgmt_api_requests

  • It will have the following values:

    • count_requests: The number of requests made during 5 minutes. This is needed in our example.

    • sum_request_duration: A sum of the duration of request processing during 5 minutes. This can be used to display which requests are slowest to process and might be the causes of performance problems.

    • mean_request_duration: The mean duration of request processing during 5 minutes. This can be used to display how long requests take on average.

    • Tags path, method, api_client_id and oauth2_client_id. Of these path is needed for our example, others can be used in other graphs.

  • The fields response_code, request_uri apparently cannot be used for grouping in a CQ, so they are not included.