Configuration

Cloudmon accepts yaml file with the configuration. This config is then being split and proper configuration is being used to deploy certain component in certain environment.

In order to improve openess of the configuration (public vs sensitive information) it is possible to split it into 2 parts: git repository with publicly accessible elements and a local config with sensitive part. Content of the git repository, specified with –config-repo parameter, is then taken as a base and local config is being applied on top to keep possibility to override fake information with sensitive elements. Merging of lists is being done based on the name attribute of the list item.

Note

When using –config-repo parameter config file name can only be same as in config-dir location.

Sample configuration

%YAML 1.2
---
monitoring_zones: # Defining from where we are monitoring
  - name: internal
    graphite_group_name: graphite
    statsd_group_name: statsd
    #  external:
    #    graphite_group_name: graphite
    #    statsd_group_name: statsd

environments: # What we monitor and which credentials to use from every monitoring location (zone)
  # NOTE: not allowed to use same cloud names in same zone for different envs
  - name: production
    env:
      OS_CLOUD: production
    monitoring_zones:
      - name: internal
        clouds:
          - name: production
            ref: p1
          - name: swift
            ref: swift1
      - name: external
        clouds:
          - name: production
            ref: p2
          - name: swift
            ref: swift1

# Known CloudMon plugins with their basic settings
plugins:
  - name: apimon
    type: apimon
    scheduler_image: quay.io/opentelekomcloud/apimon:change_35_latest
    executor_image: quay.io/opentelekomcloud/apimon:change_35_latest
    epmon_image: quay.io/opentelekomcloud/apimon:change_35_latest
    tests_projects:
      - name: apimon
        repo_url: https://github.com/stackmon/apimon-tests
        repo_ref: main
        exec_cmd: ansible-playbook -i inventory/production %s -vvv
        scenarios_location: playbooks
        grafana_dashboards_location: dashboards
  - name: epmon
    type: epmon
    image: quay.io/opentelekomcloud/apimon:change_35_latest
    config: sample_config_epmon.yaml
  - name: globalmon
    type: globalmon
    image: quay.io/stackmon/globalmon:change_12_latest
  - name: lb
    type: general
    image: quay.io/opentelekomcloud/cloudmon-plugin-lb
    init_image: quay.io/opentelekomcloud/cloudmon-plugin-lb-init

graphite:
  host: localhost

database:
  # using ha_mode (patroni) requires having multiple hosts in the postgres
  # group. For now it is disabled since there is certain instability in the
  # setup.
  # ha_mode: true
  postgres_postgres_password: ChangeMe123$
  databases:
    - name: grafana
      users:
        - name: grafana
          password: &grafana_database_password ChangeMe!123$
    - name: apimon
      users:
        - name: apimon
          password: &apimon_database_password ChangeMe!123$

grafana:
  datasources:
    - name: cloudmon
      type: graphite
    - name: apimon_db
      type: postgres
      database: apimon
      user: apimon
      jsonData:
        postgresVersion: 14
        sslmode: disable
      secureJsonData:
        password: *apimon_database_password
  config:
    grafana_image: quay.io/opentelekomcloud/grafana:9.1.5
    grafana_renderer_image: quay.io/opentelekomcloud/grafana-image-renderer:3.6.1
    grafana_security_admin_password: fake_password_change_me
    grafana_enable_renderer: "true"
    grafana_grafana_host: grafana
    grafana_renderer_host: renderer
    grafana_database_type: postgres
    grafana_database_name: grafana
    grafana_database_user: grafana
    grafana_database_password: *grafana_database_password
  dashboards:
    - name: main
      repo_url: https://github.com/stackmon/apimon-tests.git
      repo_ref: main
      path: dashboards/grafana

matrix:
  # Mapping of environments to test projects
  # Regular apimon project in env ext
  - env: production
    monitoring_zone: internal
    # TODO: placing db_url here feels questionable
    # db_url or db_entry as ref to database.databases
    db_entry: apimon.apimon
    plugins:
      - name: apimon
        schedulers_inventory_group_name: schedulers
        executors_inventory_group_name: executors
        #epmons_inventory_group_name: epmons
        tests_project: apimon
        tasks:
          - scenario1_token.yaml
      - name: epmon
        epmons_inventory_group_name: epmons
        cloud_name: production # env in zone has few creds. We need to pick one
        config_elements:
          - compute
          - dns
          - identity
          - image
          - network
          - object_store
          - orchestration
          - volume
      - name: globalmon
        globalmons_inventory_group_name: globalmons
        cloud_name: production # env in zone has few creds. We need to pick one
        config: globalmon-config.yaml

clouds_credentials:
  - name: p1
    auth:
      auth_url: https://fake.com
      username: fake_user
      password: fake_pass
      project_name: fake_project
      user_domain_name: fake_domain
  - name: swift1
    profile: otc
    auth:
      auth_url: https://fake.com
      username: fake_user
      password: fake_pass
      project_name: fake_project
      user_domain_name: fake_domain

class cloudmon.types.CloudCredentialModel(*, name: str, profile: str = None, auth: dict, region_name: str = None, **extra_data: Any)

Cloud Credentials

auth: dict

Auth block (as in clouds.yaml)

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Credential name (for reference)

profile: str

Optional OpenStack profile to use with credentials

region_name: str

Optional OpenStack profile region name

class cloudmon.types.CloudCredentialsModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.ConfigModel(*, clouds_credentials: CloudCredentialsModel, database: DatabaseModel, environments: EnvironmentsModel, grafana: GrafanaModel = None, matrix: List[MatrixModel], metrics_processor: List[MetricsProcessorModel] = [], monitoring_zones: MonitoringZonesModel, plugins: List[PluginModel], status_dashboard: List[StatusDashboardModel] = [])

CloudMon Config

clouds_credentials: CloudCredentialsModel

Cloud Credentials section

database: DatabaseModel

Databases configuration

environments: EnvironmentsModel

Environments to be tested

grafana: GrafanaModel

Grafana configuration

matrix: List[MatrixModel]

Testing matrix (where to, from where and what)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

monitoring_zones: MonitoringZonesModel

Monitoring zones from which to test

plugins: List[PluginModel]

Registered plugins to enable for testing

status_dashboard: List[StatusDashboardModel]

Status dashboard configuration

class cloudmon.types.DatabaseInstanceModel(*, name: str, users: List[DatabaseUserModel])

Database instance

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

DB name

users: List[DatabaseUserModel]

DB users list

class cloudmon.types.DatabaseModel(*, postgres_postgres_password: str, ha_mode: bool = False, databases: List[DatabaseInstanceModel])

Database configuration

databases: List[DatabaseInstanceModel]

Databases list

ha_mode: bool

HighAvailability mode (based on Patroni, requires SSL certificates to be managed externally)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

postgres_postgres_password: str

Password of the postgres user (used to create Databases)

class cloudmon.types.DatabaseUserModel(*, name: str, password: str)

Database user

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

DB user name

password: str

DB user password

class cloudmon.types.EnvMonitoringZoneModel(*, name: str, clouds: List[EnvZoneCloudModel])

Configuration of the monitoring zone for certain environment

clouds: List[EnvZoneCloudModel]

List of cloud credentials to be deployed

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Zone name

class cloudmon.types.EnvMonitoringZonesModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.EnvZoneCloudModel(*, name: str, ref: str)

Environment Zone Cloud credentials entry

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Cloud name (clouds.yaml) to use while deploying

ref: str

Reference to the cloud_credentials name to use

class cloudmon.types.EnvZoneCloudsModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.EnvironmentModel(*, name: str, env: dict, monitoring_zones: EnvMonitoringZonesModel)

Target environment to be tested

env: dict

Environment variables to be set for this env

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

monitoring_zones: EnvMonitoringZonesModel

Monitoring zones from which environment will be tested

name: str

Environment name

class cloudmon.types.EnvironmentsModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.GitRepoModel(*, repo_url: str, repo_ref: str = 'main')

Git repository

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

repo_ref: str

Git repo reference (branch)

repo_url: str

Git repo url

class cloudmon.types.GrafanaDashboardRepoModel(*, repo_url: str, repo_ref: str = 'main', name: str, path: str = 'dashboards/grafana')

Grafana dashboard repository configuration

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Repository name

path: str

Path to the dashboards definitions inside the repository

class cloudmon.types.GrafanaModel(*, api_url: str = None, api_token: str = None, datasources: List[dict], k8_config: dict = None, config: dict, dashboards: List[GrafanaDashboardRepoModel])

Grafana configuration

api_token: str

API token

api_url: str

API url

config: dict

Configuration options to use for deploying on VMs

dashboards: List[GrafanaDashboardRepoModel]

List of dashboards to be managed in the instance

datasources: List[dict]

List of datasources to install into the Grafana instance

k8_config: dict

Kubernetes configuration to use for deploying Grafana to K8

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.Kustomization(root: RootModelRootType = PydanticUndefined)

Basic Kustomization properties to use for overlay building

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.MatrixModel(*, env: str, monitoring_zone: str, db_entry: str, plugins: List[PluginApimonRefModel | PluginEpmonRefModel | PluginGlobalmonRefModel | PluginGeneralModel])

Testing Matrix entry

db_entry: str

Which DB to use for storing logs

env: str

Environment to test

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

monitoring_zone: str

From which monitoring zone to test

plugins: List[PluginApimonRefModel | PluginEpmonRefModel | PluginGlobalmonRefModel | PluginGeneralModel]

Which plugins to use for testing

class cloudmon.types.MetricProcessorEnvironmentModel(*, name: str, attributes: dict)

Metric Processor Environment configuration

attributes: dict

Status Dashboard attributes linked to the environment

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Environment name

class cloudmon.types.MetricsProcessorModel(*, name: str, kube_context: str, kube_namespace: str, datasource_url: str, datasource_type: str = 'graphite', domain_name: str, environments: List[MetricProcessorEnvironmentModel], kustomization: Kustomization, status_dashboard_instance_name: str = None)

Metrics Processor configuration

datasource_type: str

Datasource type

datasource_url: str

URL to the datasource (the one where graphite is deployed)

domain_name: str

FQDN

environments: List[MetricProcessorEnvironmentModel]

Environments configuration

kube_context: str

Kubernetes context to use for deployment

kube_namespace: str

Kubernetes namespace name for deploy

kustomization: Kustomization

Kustomize overlay options

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Instance name

status_dashboard_instance_name: str

Reference name of the associated Status Dashboard instance name

class cloudmon.types.MonitoringZoneModel(*, name: str, graphite_group_name: str = 'graphite', statsd_group_name: str = 'statsd')

Monitoring Zone

graphite_group_name: str

ansible group name of the graphite hosts to use

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Zone name

statsd_group_name: str

ansible group name of the statsd hosts to use

class cloudmon.types.MonitoringZonesModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.PluginApimonModel(*, name: str, type: Literal['apimon'], scheduler_image: str, executor_image: str, tests_projects: List[dict])

ApiMon plugin configration

executor_image: str

Executor image to use

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Plugin name

scheduler_image: str

ApiMon scheduler image to use

tests_projects: List[dict]

List of git repositories with tests

class cloudmon.types.PluginApimonRefModel(*, name: str, schedulers_inventory_group_name: str = 'schedulers', executors_inventory_group_name: str = 'executor', tests_project: str, tasks: List[str] = [])

ApiMon plugin invocation

executors_inventory_group_name: str

ansible group name to deploy executors

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name (as used in the plugins section)

schedulers_inventory_group_name: str

ansible group name to deploy scheduler component

tasks: List[str]

Optional list of tasks (playbooks) to schedule

tests_project: str

Name of the project with tests (one from defined in the plugin configuration)

class cloudmon.types.PluginEpmonModel(*, name: str, type: Literal['epmon'], image: str, config: str)

Endpoint Monitoring plugin

config: str

Path to the epmon configuration elements (which service which endpoints)

image: str

Epmon image to use

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name

class cloudmon.types.PluginEpmonRefModel(*, name: str, cloud_name: str = None, config_elements: List[str] = [], epmon_inventory_group_name: str = 'epmons')

Epmon plugin invocation

cloud_name: str

Cloud name to test in the environment

config_elements: List[str]

List of configuration entries of the epmon

epmon_inventory_group_name: str

ansible group name to deploy epmon process

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name

class cloudmon.types.PluginGeneralModel(*, name: str, type: Literal['general'], init_image: str)

General plugin

init_image: str

Init image (this image is invoked to optionally initialize infrastructure for further testing)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name

class cloudmon.types.PluginGlobalmonModel(*, name: str, type: Literal['globalmon'], image: str)

Endpoint Monitoring plugin

image: str

Globalmon image to use

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name

class cloudmon.types.PluginGlobalmonRefModel(*, name: str, cloud_name: str = None, globalmons_inventory_group_name: str = 'globalmons', config: str)

Globalmon plugin invocation

cloud_name: str

Cloud name to test in the environment

config: str

Path to the globalmon configuration elements (which service which endpoints)

globalmons_inventory_group_name: str

ansible group name to deploy globalmon process

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

plugin name

class cloudmon.types.PluginModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.PluginRefModel(root: RootModelRootType = PydanticUndefined)

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class cloudmon.types.StatusDashboardModel(*, name: str, kube_context: str, kube_namespace: str, api_secret: str = None, domain_name: str, kustomization: Kustomization)

Status Dashboard configuration

api_secret: str

API secret

domain_name: str

FQDN

kube_context: str

Kubernetes context to use for deployment

kube_namespace: str

Kubernetes namespace name for deploy

kustomization: Kustomization

Kustomize overlay options

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

Instance name