Send CloudWatch logs to AppSignal through Amazon Data Firehose

Complete the following steps to send CloudWatch logs to AppSignal through Amazon Data Firehose (formerly Kinesis Data Firehose):

1. Create a log source
2. Set up an S3 bucket for failed deliveries
3. Set up an IAM role for Data Firehose
4. Create a Firehose delivery stream
5. Set up an IAM role for CloudWatch
6. Create a CloudWatch log subscription

Before you start, have the following information ready:

• Your log source's API key. If you do not have a log source yet, create a new log source first.
• AWS account ID
• AWS region
• S3 bucket name for failed delivery storage
• Data Firehose stream name
• IAM role name for the S3 bucket
• IAM role name for the CloudWatch subscription

Create a log source

Create a log source before proceeding. See Logging configuration for instructions.

Set up an S3 bucket for failed deliveries

Amazon Data Firehose requires an S3 bucket to store records that fail to deliver. Create an S3 bucket through the AWS Console or CLI before proceeding, you can't create a Firehose delivery stream without one.

Set up an IAM role for Data Firehose

Create an IAM role that allows Data Firehose to write to the S3 bucket and, optionally, to CloudWatch Logs for delivery error logging. Use the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "firehose.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Assign the following permissions policy to the role as an inline policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>",
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:<AWS_REGION>:<AWS_ACCOUNT_ID>:log-group:<FIREHOSE_LOG_GROUP>:*"
      ]
    }
  ]
}

Replace the placeholders with your own values:

• <AWS_ACCOUNT_ID> — your AWS account ID.
• <AWS_REGION> — the region your Firehose stream will run in.
• <FIREHOSE_S3_BUCKET> — the S3 bucket you created in the previous step.
• <FIREHOSE_LOG_GROUP> — the log group Firehose uses for its own delivery error logs (separate from the log group you want to stream to AppSignal). AWS uses the convention /aws/kinesisfirehose/<FIREHOSE_STREAM_NAME>, so decide your Firehose stream name now — you'll reuse it in the next step. If you don't plan to enable error logging on the Firehose stream, you can omit the logs:PutLogEvents statement entirely.

Create a Firehose delivery stream

1. Open the Amazon Data Firehose console and select Create Firehose stream.
2. Select Direct PUT as the source.
3. Select HTTP Endpoint as the destination.
4. Under "Transform records", leave data transformation turned off.
5. Enter the following URL as the HTTP endpoint URL:

https://appsignal-endpoint.net/logs/aws-kinesis

6. Enter your AppSignal log source API key from creating a log source as the Access key. This is an AppSignal key, not an AWS access key.
7. Under "Content encoding", enable GZIP.
8. Under "Backup settings", select the S3 bucket you created in the previous step.
9. Under "Advanced settings > Service access", select Choose existing IAM role and choose the IAM role you created in the previous step.
10. (Optional) Under "Advanced settings > Amazon CloudWatch error logging", enable error logging to use the logs:PutLogEvents permission you granted in the previous step.
11. Use the same Firehose stream name you referenced when setting up the IAM role, then select Create Firehose stream.

To verify the connection, you can use the Test with demo data button in the Firehose console. The stock-ticker JSON payloads it sends appear in AppSignal under Logging as individual log lines — a useful end-to-end check before you hook up real log groups. You can also watch DeliveryToHttpEndpoint.Success climb on the Firehose stream's Monitoring tab.

Set up an IAM role for CloudWatch

Create an IAM role that allows CloudWatch to send logs to the Firehose delivery stream. Use the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "logs.<AWS_REGION>.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

Assign the following permissions policy to the role as an inline policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:*"],
      "Resource": [
        "arn:aws:firehose:<AWS_REGION>:<AWS_ACCOUNT_ID>:deliverystream/<FIREHOSE_DELIVERY_STREAM>"
      ]
    }
  ]
}

Replace <AWS_REGION>, <AWS_ACCOUNT_ID>, and <FIREHOSE_DELIVERY_STREAM> with your own values.

Create a CloudWatch log subscription

1. In the CloudWatch console, navigate to the log group whose logs you want to stream.
2. Open the Subscription filters tab.
3. Select Create Amazon Data Firehose subscription filter.
4. Choose the Firehose delivery stream from step 4 and the IAM role from step 5.
5. Enter a Subscription filter name and select Start streaming.

After saving the subscription, logs appear in AppSignal under the log source you've configured above. If you see an error saving the subscription, verify that the IAM role in step 5 uses the correct region, account ID, and delivery stream name.

If logs do not appear in AppSignal:

• Open your Firehose stream's Monitoring tab and check DeliveryToHttpEndpoint.Success. The value should stay near 1; a drop or zero indicates AppSignal is rejecting records.
• Check the S3 failure bucket from step 2. Failed records land there, and each object contains the rejection reason.
• If records are failing with an authorization error, verify that the Access key on the Firehose stream matches your AppSignal log source API key.

If you're still stuck, contact us for support.