Menu

Stream CloudWatch logs with CloudFormation

Do not send Personal Identifiable Information (PII) to AppSignal. Filter PII (e.g., names, emails) from logs and use an ID, hash, or pseudonymized identifier instead.

For HIPAA-covered entities, more information about signing a Business Associate Agreement (BAA) can be found in our Business Add-Ons documentation.

You can automate the setup of CloudWatch logs streaming to AppSignal using an AWS CloudFormation template. This creates all the required resources in a single deployment instead of configuring each one manually through the AWS Console.

If you prefer to set up each resource manually, see the CloudWatch logs setup guide.

Before you start

Have the following information ready:

  • Your log source's API key. If you do not have a log source yet, create a new log source first.
  • The name of the existing CloudWatch log group you want to stream to AppSignal.
  • The AWS region where the log group lives. Deploy the stack in the same region and account as the log group.

What the template creates

The CloudFormation template creates the following resources:

  1. An S3 bucket to store records that fail to deliver, with server-side encryption and public access blocked.
  2. An IAM role that allows Amazon Data Firehose to write failed records to the S3 bucket and to write its own delivery error logs to CloudWatch.
  3. A Firehose delivery stream that sends logs to the AppSignal endpoint over HTTPS.
  4. An IAM role that allows CloudWatch Logs to write to the Firehose delivery stream.
  5. A CloudWatch log subscription filter on your existing log group that forwards log events to the Firehose delivery stream.

CloudFormation template

Copy the following template and save it as appsignal-cloudwatch-logs.yaml:

YAML
AWSTemplateFormatVersion: 2010-09-09
Description: >-
  Configures a CloudWatch log subscription and Amazon Data Firehose
  delivery stream to send logs to AppSignal from an existing CloudWatch log group.
 
  It must be deployed in the same region and same account as the CloudWatch log group.
 
  Implements all the steps from AppSignal documentation all at once:
  https://docs.appsignal.com/logging/platforms/cloudwatch.html
 
Parameters:
  LogSourceApiKey:
    Type: String
    Description: Your AppSignal log source API key (39-character hexadecimal string).
    NoEcho: true
    MinLength: 39
    MaxLength: 39
  CloudWatchLogGroupName:
    Type: String
    Description: >-
      Name (not the ARN) of the CloudWatch log group to send to AppSignal.
      Group logs list: https://console.aws.amazon.com/cloudwatch/home#logsV2:log-groups
    MinLength: 1
 
Resources:
  S3FirehoseEventsBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Join
        - "-"
        - - "appsignal-firehose"
          - !Ref AWS::StackName
          - !Ref AWS::AccountId
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
 
  FirehoseRole:
    Type: AWS::IAM::Role
    Properties:
      Description: >-
        Role to allow firehose stream to put events into S3 backup bucket
      RoleName: !Join
        - "-"
        - - "appsignal-firehose"
          - !Ref AWS::StackName
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - firehose.amazonaws.com
            Action:
              - "sts:AssumeRole"
            Condition:
              StringEquals:
                "sts:ExternalId": !Ref AWS::AccountId
      Policies:
        - PolicyName: !Join
            - "-"
            - - "appsignal-firehose"
              - !Ref AWS::StackName
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - "s3:AbortMultipartUpload"
                  - "s3:GetBucketLocation"
                  - "s3:GetObject"
                  - "s3:ListBucket"
                  - "s3:ListBucketMultipartUploads"
                  - "s3:PutObject"
                Resource:
                  - !GetAtt S3FirehoseEventsBucket.Arn
                  - !Join ["", [!GetAtt S3FirehoseEventsBucket.Arn, "/*"]]
              - Effect: Allow
                Action:
                  - "logs:PutLogEvents"
                Resource:
                  - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/kinesisfirehose/appsignal-firehose-${AWS::StackName}:*"
  FirehoseDeliveryStream:
    Type: AWS::KinesisFirehose::DeliveryStream
    Properties:
      DeliveryStreamName: !Join
        - "-"
        - - "appsignal-firehose"
          - !Ref AWS::StackName
      DeliveryStreamType: DirectPut
      HttpEndpointDestinationConfiguration:
        RequestConfiguration:
          ContentEncoding: GZIP
        EndpointConfiguration:
          Name: AppSignal
          Url: "https://appsignal-endpoint.net/logs/aws-kinesis"
          AccessKey: !Ref LogSourceApiKey
        BufferingHints:
          IntervalInSeconds: 60
          SizeInMBs: 1
        RetryOptions:
          DurationInSeconds: 60
        S3Configuration:
          CompressionFormat: GZIP
          BucketARN: !GetAtt S3FirehoseEventsBucket.Arn
          RoleARN: !GetAtt FirehoseRole.Arn
        RoleARN: !GetAtt FirehoseRole.Arn
 
  LogsStreamRole:
    Type: AWS::IAM::Role
    Properties:
      Description: Role to allow stream put into a firehose
      RoleName: !Join
        - "-"
        - - "appsignal-cloudwatch"
          - !Ref AWS::StackName
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - !Sub "logs.${AWS::Region}.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Policies:
        - PolicyName: !Join
            - "-"
            - - "appsignal-firehose"
              - !Ref AWS::StackName
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - "firehose:*"
                Resource:
                  - !GetAtt FirehoseDeliveryStream.Arn
 
  SubscriptionFilter:
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      LogGroupName: !Ref CloudWatchLogGroupName
      FilterName: "AppSignal"
      FilterPattern: ""
      DestinationArn: !GetAtt FirehoseDeliveryStream.Arn
      RoleArn: !GetAtt LogsStreamRole.Arn

The logs:PutLogEvents statement allows Firehose to write its own delivery error logs to CloudWatch. It targets the Firehose error log group (/aws/kinesisfirehose/<STREAM_NAME>), which is separate from the log group you want to stream to AppSignal. See Controlling access with Amazon Data Firehose for more details.

Deploy the stack

  1. Open the AWS CloudFormation console.
  2. Select Create stack and choose With new resources (standard).
  3. Under Specify template, select Upload a template file and upload appsignal-cloudwatch-logs.yaml.
  4. Select Next.
  5. Enter a stack name, for example appsignal-cloudwatch-logs.
  6. For LogSourceApiKey, enter your log source's API key.
  7. For CloudWatchLogGroupName, enter the exact name of the log group you want to stream.
  8. Select Next twice, then check I acknowledge that AWS CloudFormation might create IAM resources and select Submit.

The stack takes a few minutes to create. Once the status shows CREATE_COMPLETE, CloudWatch begins streaming logs to AppSignal.

Verify the deployment

  1. In the AWS Console, search for "CloudFormation" and select Stacks. Open your stack and select the Resources tab to confirm all resources show CREATE_COMPLETE.
  2. In the AWS Console, search for "CloudWatch". In the left sidebar, open your log group and select the Subscription filters tab to confirm the AppSignal filter is listed.
  3. In the AWS Console, search for "Firehose". Open the delivery stream and use the Test with demo data function to verify connectivity.
  4. In AppSignal, open the log management screen and confirm log entries from your log group appear there.

If logs do not appear after a few minutes, check the S3 bucket for failed delivery records. If you need help, contact us.

Clean up

To remove all resources created by this template, delete the stack in the CloudFormation console. Note that the S3 bucket must be empty before the stack can be deleted. Empty the bucket first, then delete the stack.