Menu

Stream AWS CloudWatch metrics with CloudFormation

Use an AWS CloudFormation template to set up CloudWatch metrics streaming to AppSignal. The template creates all required resources in a single deployment, instead of configuring each one manually through the AWS Console.

For an overview of CloudWatch metrics in AppSignal, including magic dashboards and metric naming, see AWS CloudWatch metrics. If you prefer to set up each resource manually, see the console setup guide.

Before you start

Have the following information ready:

  • Your app's App-level Push API key. To find it, open the API keys settings page, or navigate to your organization settings and select Dev zone > API keys.
  • The AWS region where you want to deploy the stack

What the template creates

The CloudFormation template creates the following resources:

  1. An S3 bucket to store records that fail to deliver, with a 30-day lifecycle expiration and a retain-on-delete policy.
  2. An IAM role that allows Amazon Data Firehose to write failed records to the S3 bucket.
  3. A Firehose delivery stream that sends metrics to the AppSignal endpoint over HTTPS.
  4. An IAM role that allows CloudWatch Metric Streams to write to the Firehose delivery stream.
  5. A CloudWatch metric stream that exports metrics in OpenTelemetry 1.0 format.

CloudFormation template

Copy the following template and save it as appsignal-cloudwatch-metrics.yaml:

YAML
AWSTemplateFormatVersion: "2010-09-09"
Description: >
  Stream CloudWatch metrics to AppSignal using Amazon Data Firehose.
 
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: AppSignal Configuration
        Parameters:
          - AppLevelPushApiKey
    ParameterLabels:
      AppLevelPushApiKey:
        default: "App-level Push API Key (find at https://appsignal.com/redirect-to/app?to=api_keys&key_tab=app)"
 
Parameters:
  AppLevelPushApiKey:
    Type: String
    NoEcho: true
    Description: >
      Your App-level Push API key from AppSignal.
 
Resources:
  # 1. S3 bucket for failed deliveries
  FailedDeliveryBucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      BucketName: !Sub "appsignal-firehose-${AWS::StackName}-${AWS::AccountId}-${AWS::Region}"
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      LifecycleConfiguration:
        Rules:
          - Id: ExpireFailedDeliveries
            Status: Enabled
            ExpirationInDays: 30
 
  # 2. IAM role for Firehose to write to S3
  FirehoseS3Role:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: FirehoseS3Access
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "s3:AbortMultipartUpload"
                  - "s3:GetBucketLocation"
                  - "s3:GetObject"
                  - "s3:ListBucket"
                  - "s3:ListBucketMultipartUploads"
                  - "s3:PutObject"
                Resource:
                  - !GetAtt FailedDeliveryBucket.Arn
                  - !Sub "${FailedDeliveryBucket.Arn}/*"
 
  # 3. Firehose delivery stream
  MetricsDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: "https://appsignal-endpoint.net/metrics/aws-cloudwatch"
          AccessKey: !Ref AppLevelPushApiKey
          Name: AppSignal
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 1
        S3BackupMode: FailedDataOnly
        S3Configuration:
          BucketARN: !GetAtt FailedDeliveryBucket.Arn
          RoleARN: !GetAtt FirehoseS3Role.Arn
        RoleARN: !GetAtt FirehoseS3Role.Arn
 
  # 4. IAM role for CloudWatch Metric Streams to write to Firehose
  MetricStreamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: streams.metrics.cloudwatch.amazonaws.com
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: MetricStreamFirehoseAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "firehose:PutRecord"
                  - "firehose:PutRecordBatch"
                Resource: !GetAtt MetricsDeliveryStream.Arn
 
  # 5. CloudWatch metric stream
  CloudWatchMetricStream:
    Type: AWS::CloudWatch::MetricStream
    Properties:
      FirehoseArn: !GetAtt MetricsDeliveryStream.Arn
      RoleArn: !GetAtt MetricStreamRole.Arn
      OutputFormat: "opentelemetry1.0"
 
Outputs:
  DeliveryStreamName:
    Description: Name of the Firehose delivery stream.
    Value: !Ref MetricsDeliveryStream
 
  MetricStreamName:
    Description: Name of the CloudWatch metric stream.
    Value: !Ref CloudWatchMetricStream
 
  FailedDeliveryBucketName:
    Description: S3 bucket for failed delivery records.
    Value: !Ref FailedDeliveryBucket

Deploy the stack

  1. Open the AWS CloudFormation console.
  2. Select Create stack and choose With new resources (standard).
  3. Under Specify template, select Upload a template file and upload appsignal-cloudwatch-metrics.yaml.
  4. Select Next.
  5. Enter a stack name, for example appsignal-cloudwatch-metrics.
  6. For AppLevelPushApiKey, enter your app's App-level Push API key.
  7. Select Next twice, then check I acknowledge that AWS CloudFormation might create IAM resources and select Submit.

The stack takes a few minutes to create. Once the status shows CREATE_COMPLETE, CloudWatch begins streaming metrics to AppSignal.

Filter specific AWS services

By default, the template streams metrics from all AWS services in the account. To stream only specific services, add IncludeFilters to the CloudWatchMetricStream resource:

YAML
CloudWatchMetricStream:
  Type: AWS::CloudWatch::MetricStream
  Properties:
    FirehoseArn: !GetAtt MetricsDeliveryStream.Arn
    RoleArn: !GetAtt MetricStreamRole.Arn
    OutputFormat: "opentelemetry1.0"
    IncludeFilters:
      - Namespace: AWS/EC2
      - Namespace: AWS/RDS
      - Namespace: AWS/ELB

Replace the namespaces with the AWS services you want to monitor. See the AWS services that publish CloudWatch metrics for a full list of available namespaces.

Verify the deployment

  1. In the AWS Console, search for "CloudFormation" and select Stacks. Open your stack and select the Resources tab to confirm all resources show CREATE_COMPLETE.
  2. In the AWS Console, search for "CloudWatch". In the left sidebar, go to Metrics > Streams and confirm the metric stream status is Running.
  3. In the AWS Console, search for "Firehose". Open the delivery stream and use the Test with demo data function to verify connectivity.
  4. In AppSignal, open your app's dashboard and check that CloudWatch metrics appear on your custom dashboard.

If metrics do not appear after a few minutes, check the S3 bucket for failed delivery records. If you need help, contact us.

Multi-region deployments

The template deploys all resources in a single AWS region. If you run workloads in multiple regions, deploy a separate copy of this stack in each region. This keeps data transfer within each region and avoids cross-region transfer fees.

Clean up

To remove all resources created by this template, delete the stack in the CloudFormation console. The S3 bucket for failed deliveries has a DeletionPolicy of Retain, so it is kept when the stack is deleted to avoid losing any failed delivery records. Failed delivery records in the bucket are automatically deleted after 30 days by the lifecycle rule. You can delete the bucket manually once you no longer need it.