Configuring Monitoring with Grafana LGTM

You can optionally set up monitoring for AtScale using the grafana-lgtm stack, which includes Loki, Grafana, Tempo, and Mimir.

Note

This procedure assumes you have already installed AtScale.

Prerequisites

• A Kubernetes cluster with AtScale already deployed.
• Helm installed on your local machine.
• kubectl configured to interact with the cluster.

Configure monitoring

To configure monitoring with grafana-lgtm:

1. On your Kubernetes cluster, create a monitoring namespace:

   kubectl create namespace <monitoring_namespace>

   Where <monitoring_namespace> is the namespace you want to use.

2. Configure the Grafana dashboards:

   1. Create a ConfigMap in the monitoring namespace containing the AtScale dashboard JSON files. To download the dashboard files, click here.

   2. Add the following to the values file for the grafana-lgtm chart. This enables Grafana to load the AtScale dashboards automatically. Note that the default datasources are used as provided by Grafana.

      grafana:
        dashboardProviders:
          dashboardproviders.yaml:
            apiVersion: 1
            providers:
            - name: 'atscale'
              orgId: 1
              folder: ''
              type: file
              disableDeletion: false
              editable: true
              options:
              path: /var/lib/grafana/dashboards/atscale
      
        dashboardsConfigMaps:
          atscale: atscale-custom-dashboards

3. Configure OpenTelemetry to export data:

   1. Export the default telemetry-config from the AtScale namespace:

      kubectl get configmap -n <atscale-namespace> telemetry-config -o yaml > otel-config.yaml
      Example

      To download an example ConfigMap, click here.

   2. In otel-config.yaml, add exporters for Loki and Mimir:

      exporters:
        # Loki exporter using OTLP HTTP (required from OpenTelemetry v0.131.x onward)
        otlphttp/loki:
          # Previous exporters/loki is deprecated in v0.131.x+
          # Use otlphttp/loki instead with appropriate Loki endpoint
          ## NOTE:
          ## From version 0.131.x of OpenTelemetry Collector Contrib, the 'exporters/loki' is no longer available.
          ## To continue exporting logs to Loki, you should use the 'otlphttp/loki' exporter with the appropriate configuration.
          ##
          ## Migration instructions and discussion:
          ## - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/release/v0.130.x/exporter/lokiexporter/README.md#migration-instructions
          ## - https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33916
          ##
          endpoint: http://loki.example.com:3100/otlp/v1/logs
      
        # Prometheus remote write for Mimir
        prometheusremotewrite:
          endpoint: http://<mimir-nginx-service>.<monitoring-namespace>.svc.cluster.local/api/v1/push

   3. Make the following changes to the service.pipelines section of otel-config.yaml:

      service:
        pipelines:
          logs:
            receivers: [otlp]
            processors: []
            exporters: [file/logs, otlphttp/loki]
          metrics:
            receivers: [prometheus,otlp]
            processors: []
            exporters: [file/metrics, prometheusremotewrite]

   4. Apply the configMap in the AtScale namespace:

      kubectl apply -f otel-config.yaml

4. Point AtScale to the new ConfigMap. In your values override file for AtScale, add the following:

   telemetry:
     configMap:
       create: false
       existingName: "otel-config"

5. Apply your changes:

   1. Upgrade AtScale with the updated configuration:

      helm upgrade atscale -n <atscale-namespace> -f <override-file>

   2. Deploy the LGTM stack:

      helm repo add grafana https://grafana.github.io/helm-charts
      helm install lgtm-distributed grafana/lgtm-distributed --version 2.1.0 -f override-file.yaml

Note

Grafana allows for further configuration (LoadBalancer service, Ingress resources, environment variables, etc.). For more information, refer to the Grafana documentation.