{"revision":"c10276e953456835cdbb1f3a60d58ae1980be353","document":{"id":30485,"path":"it-operations-management/event-management/metric-collection-otel.md","title":"Metric collection from OpenTelemetry \\(Otel\\) metrics","bundle":null,"doc_type":null,"release":"australia","body":"\n# Metric collection from OpenTelemetry \\(Otel\\) metrics\n\nThe MID WebService metric Collector enables you to collect JSON and protobuf formatted metrics sent from OpenTelemetry \\(Otel\\).\n\n## Before you begin\n\nEnsure that you have installed the Event Management Connectors \\(sn\\_em\\_connector\\) plugin on the ServiceNow AI Platform instance and the ServiceNow version is Australia or above.\n\nEnsure that you:\n\n-   Deploy and start the MID Server. See [MID Server configuration](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/servicenow-platform/c_MIDServerConfiguration.md).\n-   Configure and start the MID Web Server. See [Configure the MID Web Server extension](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/it-operations-management/event-management/configure-mid-web-server-extension.md).\n-   To activate metric collection, ensure that the MID Server that retrieves metrics is configured with the Metric Intelligence extension and that the extension is in Started mode. See [Manually configure the Metric Intelligence extension](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/it-operations-management/metric-intelligence/configure-itoa-metric-extension.md).\n-   Enable the rest listener.\n\nRole required: evt\\_mgmt\\_admin\n\n## About this task\n\nProtobuf and JSON formatted metrics are sent from Opentelemetry. The MID Server transforms the collected metrics by parsing them using the**TransformMetrics\\_MidOpenTelemetry** script include, located at **Event Management** &gt; **Integrations** &gt; **PushConnectors**. In the Push Connectors page, select **OpenTelemetry**.\n\nThe default format of the URL to push event messages from the GCP to the MID Server is: `http://{MID_Server_IP}:{MID_Web_Server_Port}/api/mid/sa/inbound_metrics?transform=TransformMetrics_MidOpenTelemetry`\n\n|Variable|Description|\n|--------|-----------|\n|MID\\_Server\\_IP|IP address of the MID Web Server Extension.|\n|MID\\_Web\\_Server\\_Port|Listening port of the MID Web Server Extension.|\n|MID\\_Web\\_Server\\_User|Username for the MID Web Server Extension.|\n|MID\\_Web\\_Server\\_Password|Password of the user of the MID Web Server.|\n\n## Procedure\n\n1.  CONFIGURE RECEIVERS: HOSTMETRICS RECEIVER - SYSTEM METRICS COLLECTION\n\n    Add the following hostmetrics receiver under the \"receivers:\" section to collect system metrics:\n\n    ```\n    receivers:\n      hostmetrics:\n        collection_interval: 10s\n        scrapers:\n          cpu:\n          memory:\n          disk:\n          filesystem:\n          network:\n          processes:\n          paging:\n    \n    ```\n\n    **Note:**\n\n    -   Collection\\_interval set to 10 seconds for frequent metric updates.\n    -   Adjust the interval based on your monitoring requirements \\(higher values reduce overhead\\).\n    -   Enable only the scrapers you need to minimize resource usage.\n2.  CONFIGURE PROCESSORS\n\n    TRANSFORM PROCESSOR - REMOVE UNNECESSARY FIELDS\n\n    1.  Add the following transform processor under the \"processors:\" section to remove unnecessary fields from the payload:\n\n        ```\n        transform:\n          metric_statements:\n            - context: scope\n              statements:\n                - set(name, \"\")\n                - set(version, \"\")\n        \n        ```\n\n    2.  MEMORY LIMITER PROCESSOR - PREVENT MEMORY ISSUES\n\n        Add the following memory\\_limiter processor under the \"processors:\" section to prevent out-of-memory errors:\n\n        ```\n        memory_limiter:\n          limit_mib: 5\n          spike_limit_mib: 1\n        \n        ```\n\n        **Note:**\n\n        -   limit\\_mib set to 5 MB of memory usage\n        -   spike\\_limit\\_mib adjusted to 1 MB to handle temporary spikes\n        -   Adjust these values based on your system's available memory\n    3.  FILTER PROCESSOR - SELECT SPECIFIC METRICS\n\n        Add the following filter processor under the \"processors:\" section to include only selected metrics:\n\n        ```\n        filter/minimal:\n          metrics:\n            include:\n              match_type: strict\n              metric_names:\n                # CPU Metrics\n                - system.cpu.time\n                - system.cpu.load_average.1m\n                - system.cpu.load_average.5m\n                - system.cpu.load_average.15m\n                # Memory Metrics\n                - system.memory.usage\n                - system.paging.usage\n                # Disk Metrics\n                - system.filesystem.usage\n                - system.disk.io\n                - system.disk.io_time\n                # Process Metrics\n                - system.processes.count\n                - system.processes.created\n                # Paging Metrics\n                - system.paging.operations\n                # Network Metrics\n                - system.network.io\n                - system.network.packets\n                - system.network.errors\n                - system.network.dropped\n                - system.network.connections\n                # MySQL Metrics\n                - mysql.operations\n                - mysql.locks\n                - mysql.uptime\n                - mysql.threads\n                # HTTP Check Metrics\n                - httpcheck.duration\n                - httpcheck.status\n                - httpcheck.error\n                # Prometheus node_exporter Metrics\n                - node_cpu_seconds_total\n                - node_memory_MemTotal_bytes\n                - node_memory_MemAvailable_bytes\n                - node_disk_read_bytes_total\n                - node_disk_written_bytes_total\n                - node_disk_io_time_seconds_total\n                - node_network_receive_bytes_total\n                - node_network_transmit_bytes_total\n        \n        ```\n\n        **Note:** Histogram metrics are not supported by ServiceNow MID Server.\n\n        Even if histogram metrics are sent to the MID Server, the ServiceNow transform script will automatically ignore them. However, as a best practice, it is recommended to filter out histogram metrics at the collector level to optimize network bandwidth and reduce unnecessary data transmission.\n\n        To exclude histogram metrics, add the following filter:\n\n        ```\n        filter/drop_histograms:\n            metrics:\n              metric:\n                - 'type == METRIC_DATA_TYPE_HISTOGRAM'\n        ```\n\n3.  CONFIGURE EXPORTERS\n\n    Choose one of the following authentication methods based on your requirements:\n\n    1.  OPTION A: API KEY AUTHENTICATION\n\n        Add the following exporter under the \"exporters:\" section:\n\n        ```\n        otlphttp/opentelemetry:\n          metrics_endpoint: \"http://{mid_server_ip}:{port}/api/mid/sa/inbound_metrics?transform=TransformMetrics_MidOpenTelemetry\"\n          headers:\n            Authorization: \"Key {api_key}\"\n          timeout: 30s\n          compression: none\n          tls:\n            insecure: true\n          sending_queue:\n            enabled: true\n            num_consumers: 4\n            queue_size: 100\n          retry_on_failure:\n            enabled: true\n            initial_interval: 5s\n            max_interval: 30s\n        \n        ```\n\n        **Note:** Compression:gzip is not supported. Replace \\{mid\\_server\\_ip\\}, \\{port\\}, and \\{api\\_key\\} with your actual values.\n\n    2.  OPTION B: BASIC AUTHENTICATION\n\n        For Basic Authentication, you need to configure both an extension and an exporter.\n\n        1.  Configure Basic Auth Extension\n\n            Add the following under the \"extensions:\" section:\n\n            ```\n            extensions:\n              basicauth/mid:\n                client_auth:\n                  username: \"otel\"\n                  password: \"Otel@123\"\n            \n            ```\n\n            **Note:** Replace username and password with your actual credentials.\n\n        2.  Configure Basic Auth Exporter\n\n            Add the following under the \"exporters:\" section:\n\n            ```\n            otlphttp/opentelemetryBasic:\n              metrics_endpoint: \"http://{mid_server_ip}:{port}/api/mid/sa/inbound_metrics?transform=TransformMetrics_MidOpenTelemetry\"\n              auth:\n                authenticator: basicauth/mid\n              timeout: 30s\n              compression: none\n              tls:\n                insecure: true\n              sending_queue:\n                enabled: true\n                num_consumers: 4\n                queue_size: 100\n              retry_on_failure:\n                enabled: true\n                initial_interval: 5s\n                max_interval: 30s\n            \n            ```\n\n            **Note:** Compression:gzip is not supported. Replace \\{mid\\_server\\_ip\\} and \\{port\\} with your actual values.\n\n4.  CONFIGURE SERVICE PIPELINE\n\n    Add the configured processors and exporters to the metrics pipeline under the \"service:\" section:\n\n    ```\n    service:\n      pipelines:\n        metrics:\n          receivers: [<your_configured_receivers>]\n          processors: [memory_limiter, transform, filter/minimal]\n          exporters: [otlphttp/opentelemetry, otlphttp/opentelemetryBasic]\n    \n    ```\n\n    **Note:**\n\n    -   Replace &lt;your\\_configured\\_receivers&gt; with your actual receiver names \\(e.g., hostmetrics\\).\n    -   Include only the exporters you have configured \\(API Key or Basic Auth, or both\\).\n    -   Processor order matters: memory\\_limiter should be first, followed by transform and filter.\n    -   Ensure the processor names match those defined in Step 2.\n    -   If using histogram filtering, add filter/drop\\_histograms after filter/minimal.\n5.  ADDITIONAL CONFIGURATION OPTIONS\n\n    By default, OpenTelemetry sends data in Protocol Buffers \\(protobuf\\) format.\n\n    To send data in JSON format instead, add the following header to your exporter configuration:\n\n    ```\n    headers:\n      Content-Type: \"application/json\"\n    \n    ```\n\n    COMPLETE CONFIGURATION EXAMPLE.\n\n    Below is a complete example configuration file combining all components:\n\n    ```\n    receivers:\n      hostmetrics:\n        collection_interval: 10s\n        scrapers:\n          cpu:\n          memory:\n          disk:\n          filesystem:\n          network:\n          processes:\n          paging:\n     \n    processors:\n      memory_limiter:\n        limit_mib: 5\n        spike_limit_mib: 1\n      \n      transform:\n        metric_statements:\n          - context: scope\n            statements:\n              - set(name, \"\")\n              - set(version, \"\")\n      \n      filter/minimal:\n        metrics:\n          include:\n            match_type: strict\n            metric_names:\n              - system.cpu.time\n              - system.cpu.load_average.1m\n              - system.memory.usage\n              - system.network.io\n      \n      filter/drop_histograms:\n        metrics:\n          include:\n            metric:\n              - 'type != METRIC_DATA_TYPE_HISTOGRAM'\n     \n    extensions:\n      basicauth/mid:\n        client_auth:\n          username: \"otel\"\n          password: \"Otel@123\"\n     \n    exporters:\n      otlphttp/opentelemetry:\n        metrics_endpoint: \"https://10.0.0.100:8443/api/mid/sa/inbound_metrics?transform=TransformMetrics_MidOpenTelemetry\"\n        headers:\n          Authorization: \"Key your_api_key_here\"\n        timeout: 30s\n        compression: none\n        tls:\n          insecure: false\n        sending_queue:\n          enabled: true\n          num_consumers: 4\n          queue_size: 100\n        retry_on_failure:\n          enabled: true\n          initial_interval: 5s\n          max_interval: 30s\n     \n    service:\n      extensions: [basicauth/mid]\n      pipelines:\n        metrics:\n          receivers: [hostmetrics]\n          processors: [memory_limiter, transform, filter/minimal, filter/drop_histograms]\n          exporters: [otlphttp/opentelemetry]\n    \n      \n    ```\n\n\n**Parent Topic:**[Integrate with push connectors](https://raw.githubusercontent.com/ServiceNow/ServiceNowDocs/australia/markdown/australia/it-operations-management/event-management/configure-listener-transform-script.md)\n\n"}}