AWS Data Firehose / CloudWatch Configuration

You must complete the following steps to allow AppSignal to receive CloudWatch logs through AWS Data Firehose (formerly Kinesis):

1. Create a log source
2. Setup an S3 bucket for failed log deliveries
3. Setup an IAM role to allow Data Firehose to use the S3 bucket
4. Setup a Kinesis Firehose Delivery Stream
5. Setup an IAM role to allow CloudWatch to send logs to Kinesis Firehose
6. Setup a CloudWatch log Subscription

Before you start, we recommend you have the following information to hand:

• Your log source's API key
• AWS account ID
• AWS Region
• S3 bucket name for the failed delivery log storage
• Kinesis stream name
• IAM role name for the s3 bucket
• IAM role name for CloudWatch subscription to the delivery stream

Create a log source

Before proceeding, you first need to create a log source. Read our Logging Configuration documentation for more information on how to do this.

Setup an S3 bucket for failed log deliveries

Either through the AWS Console UI or with the CLI, create an S3 bucket that Kinesis can use to store failed log deliveries. Without this bucket, it's not possible to create a delivery stream.

Setup an IAM role to allow Kinesis to use the S3 bucket

To allow Kinesis to write to the S3 bucket, create a new IAM role with the following statement:

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "firehose.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "<AWS_ACCOUNT_ID>"
      }
    }
  }
}

Assign this new IAM role needs the following policy:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>",
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:<AWS_REGION>:<AWS_ACCOUNT_ID>:log-group:<FIREHOSE_LOG_GROUP>:*"
      ]
    }
  ]
}

Setup a Kinesis Firehose Delivery Stream

You can now proceed to set up a Kinesis Firehose Delivery stream. Use "Direct PUT" as the source and "HTTP Endpoint" as the destination. Do not transform the lines, and use the following address as the endpoint:

https://appsignal-endpoint.net/logs/aws-kinesis

Use the API Key provided when creating a log source as "Access key". Enable GZIP and use the S3 bucket and IAM role created in the steps above.

Once this configuration is in place, you can use the demo data function to send some demo logs our way. If everything is set up correctly, these will show up in AppSignal.

Setup an IAM role to allow CloudWatch to send logs to Kinesis

Now that we have a delivery stream that can store failed payloads on S3, we have to set up another IAM role; CloudWatch can use that to send logs to the Delivery stream.

{
  "Statement": {
    "Effect": "Allow",
    "Principal": { "Service": "logs.<AWS_REGION>.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }
}

The role needs the following policy:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:*"],
      "Resource": [
        "arn:aws:firehose:<AWS_REGION>:<AWS_ACCOUNT_ID>:deliverystream/<FIREHOSE_DELIVERY_STREAM>"
      ]
    }
  ]
}

Setup a CloudWatch log Subscription

In the final step, it's time to create a CloudWatch log subscription:

• Navigate to the desired log group and go to the "Subscription filters" tab.
• Create a new Kinesis Firehose Subscription, and use the delivery stream created in Step 3 and the role in Step 4.

After saving the subscription, logs should appear in AppSignal. If there's an error saving the delivery subscription, verify that the role in Step 4 has been set up with the correct region, account, and delivery stream variables. If you cannot see any logs, please contact us for support.