Menu

Stream AWS CloudWatch metrics with CloudFormation

You can automate the setup of CloudWatch metrics streaming to AppSignal using an AWS CloudFormation template. This creates all the required resources in a single deployment instead of configuring each one manually through the AWS console.

If you prefer to set up each resource manually, see the CloudWatch metrics setup guide.

Before you start

Have the following information ready:

  • Your app's App-level Push API key. To find it, open the API keys settings page, or navigate to your organization settings and select Dev zone > API keys.
  • The AWS region where you want to deploy the stack

What the template creates

The CloudFormation template creates the following resources:

  1. An S3 bucket to store records that fail to deliver.
  2. An IAM role that allows Amazon Data Firehose to write failed records to the S3 bucket.
  3. A Firehose delivery stream that sends metrics to the AppSignal endpoint over HTTPS.
  4. An IAM role that allows CloudWatch Metric Streams to write to the Firehose delivery stream.
  5. A CloudWatch metric stream that exports metrics in OpenTelemetry 1.0 format.

CloudFormation template

Copy the following template and save it as appsignal-cloudwatch-metrics.yaml:

YAML
AWSTemplateFormatVersion: "2010-09-09"
Description: >
  Stream CloudWatch metrics to AppSignal using Amazon Data Firehose.
 
Parameters:
  AppLevelPushApiKey:
    Type: String
    NoEcho: true
    Description: >
      Your App-level Push API key from AppSignal.
 
Resources:
  # 1. S3 bucket for failed deliveries
  FailedDeliveryBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub "appsignal-firehose-${AWS::StackName}-${AWS::AccountId}"
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
 
  # 2. IAM role for Firehose to write to S3
  FirehoseS3Role:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: firehose.amazonaws.com
            Action: "sts:AssumeRole"
            Condition:
              StringEquals:
                "sts:ExternalId": !Ref "AWS::AccountId"
      Policies:
        - PolicyName: FirehoseS3Access
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "s3:AbortMultipartUpload"
                  - "s3:GetBucketLocation"
                  - "s3:GetObject"
                  - "s3:ListBucket"
                  - "s3:ListBucketMultipartUploads"
                  - "s3:PutObject"
                Resource:
                  - !GetAtt FailedDeliveryBucket.Arn
                  - !Sub "${FailedDeliveryBucket.Arn}/*"
 
  # 3. Firehose delivery stream
  MetricsDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    DependsOn:
      - FirehoseS3Role
      - FailedDeliveryBucket
    Properties:
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        EndpointConfiguration:
          Url: "https://appsignal-endpoint.net/metrics/aws-cloudwatch"
          AccessKey: !Ref AppLevelPushApiKey
          Name: AppSignal
        RequestConfiguration:
          ContentEncoding: GZIP
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 1
        S3BackupMode: FailedDataOnly
        S3Configuration:
          BucketARN: !GetAtt FailedDeliveryBucket.Arn
          RoleARN: !GetAtt FirehoseS3Role.Arn
        RoleARN: !GetAtt FirehoseS3Role.Arn
 
  # 4. IAM role for CloudWatch Metric Streams to write to Firehose
  MetricStreamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: streams.metrics.cloudwatch.amazonaws.com
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: MetricStreamFirehoseAccess
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - "firehose:PutRecord"
                  - "firehose:PutRecordBatch"
                Resource: !GetAtt MetricsDeliveryStream.Arn
 
  # 5. CloudWatch metric stream
  CloudWatchMetricStream:
    Type: AWS::CloudWatch::MetricStream
    DependsOn:
      - MetricsDeliveryStream
      - MetricStreamRole
    Properties:
      FirehoseArn: !GetAtt MetricsDeliveryStream.Arn
      RoleArn: !GetAtt MetricStreamRole.Arn
      OutputFormat: "opentelemetry1.0"
 
Outputs:
  DeliveryStreamName:
    Description: Name of the Firehose delivery stream.
    Value: !Ref MetricsDeliveryStream
 
  MetricStreamName:
    Description: Name of the CloudWatch metric stream.
    Value: !Ref CloudWatchMetricStream
 
  FailedDeliveryBucketName:
    Description: S3 bucket for failed delivery records.
    Value: !Ref FailedDeliveryBucket

Deploy the stack

  1. Open the AWS CloudFormation console.
  2. Select Create stack and choose With new resources (standard).
  3. Under Specify template, select Upload a template file and upload appsignal-cloudwatch-metrics.yaml.
  4. Select Next.
  5. Enter a stack name, for example appsignal-cloudwatch-metrics.
  6. For AppLevelPushApiKey, enter your app's App-level Push API key.
  7. Select Next twice, then check I acknowledge that AWS CloudFormation might create IAM resources and select Submit.

The stack takes a few minutes to create. Once the status shows CREATE_COMPLETE, CloudWatch begins streaming metrics to AppSignal.

Filter specific AWS services

By default, the template streams metrics from all AWS services in the account. To stream only specific services, add IncludeFilters to the CloudWatchMetricStream resource:

YAML
CloudWatchMetricStream:
  Type: AWS::CloudWatch::MetricStream
  Properties:
    FirehoseArn: !GetAtt MetricsDeliveryStream.Arn
    RoleArn: !GetAtt MetricStreamRole.Arn
    OutputFormat: "opentelemetry1.0"
    IncludeFilters:
      - Namespace: AWS/EC2
      - Namespace: AWS/RDS
      - Namespace: AWS/ELB

Replace the namespaces with the AWS services you want to monitor. See the AWS services that publish CloudWatch metrics for a full list of available namespaces.

Verify the deployment

  1. In the AWS Console, search for "CloudFormation" and select Stacks. Open your stack and select the Resources tab to confirm all resources show CREATE_COMPLETE.
  2. In the AWS Console, search for "CloudWatch". In the left sidebar, go to Metrics > Streams and confirm the metric stream status is Running.
  3. In the AWS Console, search for "Firehose". Open the delivery stream and use the Test with demo data function to verify connectivity.
  4. In AppSignal, open your app's dashboard and check that CloudWatch metrics appear on your custom dashboard.

If metrics do not appear after a few minutes, check the S3 bucket for failed delivery records. If you need help, contact us.

Clean up

To remove all resources created by this template, delete the stack in the CloudFormation console. Note that the S3 bucket must be empty before the stack can be deleted. Empty the bucket first, then delete the stack.