AWS Kinesis / CloudWatch Configuration

Your log data must not contain any personal data, such as names, email addresses, etc. It is your responsibility to ensure this data is filtered out before being sent to AppSignal, and when identifying a person is necessary that your application uses alternative forms of identification such as a user ID, hash, or pseudonym.

You must complete the following steps to allow AppSignal to receive CloudWatch logs through AWS Kinesis:

Create a log source
Setup an S3 bucket for failed log deliveries
Setup an IAM role to allow Kinesis lot to use the S3 bucket
Setup a Kinesis Delivery Stream
Setup an IAM role to allow CloudWatch to send logs to Kinesis
Setup a CloudWatch log Subscription

Before you start, we recommend you have the following information to hand:

Your log source's API key
AWS account ID
AWS Region
S3 bucket name for the failed delivery log storage
Kinesis stream name
IAM role name for the s3 bucket
IAM role name for CloudWatch subscription to the delivery stream

Create a log source

Before proceeding, you first need to create a log source. Read our Logging Configuraiton documentation for more information on how to do this.

Setup an S3 bucket for failed log deliveries

Either through the AWS Console UI or with the CLI, create an S3 bucket that Kinesis can use to store failed log deliveries. Without this bucket, it's not possible to create a delivery stream.

Setup an IAM role to allow Kinesis to use the S3 bucket

To allow Kinesis to write to the S3 bucket, create a new IAM role with the following statement:

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "firehose.amazonaws.com"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": {
        "sts:ExternalId": "<AWS_ACCOUNT_ID>"
      }
    }
  }
}

Assign this new IAM role needs the following policy:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:ListBucketMultipartUploads",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>",
        "arn:aws:s3:::<FIREHOSE_S3_BUCKET>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["logs:PutLogEvents"],
      "Resource": [
        "arn:aws:logs:<AWS_REGION>:<AWS_ACCOUNT_ID>:log-group:<FIREHOSE_LOG_GROUP>:*"
      ]
    }
  ]
}

Setup a Kinesis Delivery Stream

You can now proceed to set up a Kinesis Delivery stream. Use "Direct PUT" as the source and "HTTP Endpoint" as the destination. Do not transform the lines, and use the following address as the endpoint:

https://appsignal-endpoint.net/logs/aws-kinesis?api_key=YOUR_LOG_SOURCE_API_KEY

ReplaceYOUR_LOG_SOURCE_API_KEY with the API Key provided when creating a log source.

Do not enable GZIP, and use the S3 bucket and IAM role created in the steps above.

Setup an IAM role to allow CloudWatch to send logs to Kinesis

Now that we have a delivery stream that can store failed payloads on S3, we have to setup another IAM role, CloudWatch can use that to send logs to the Delivery stream.

{
  "Statement": {
    "Effect": "Allow",
    "Principal": { "Service": "logs.<AWS_REGION>.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }
}

The role needs the following policy:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["firehose:*"],
      "Resource": [
        "arn:aws:firehose:<AWS_REGION>:<AWS_ACCOUNT_ID>:deliverystream/<FIREHOSE_DELIVERY_STREAM>"
      ]
    }
  ]
}

Setup a CloudWatch log Subscription

In the final step, it's time to create a CloudWatch log subscription:

Navigate to the desired log group and go to the "Subscription filters" tab.
Create a new Kinesis Firehose Subscription, and use the delivery stream created in Step 3 and the role in Step 4.

After saving the subscription, logs should appear in AppSignal. If there's an error saving the delivery subscription, verify that the role in Step 4 has been set up with the correct region, account and delivery stream variables. If you are still unable to see any logs, please contact us for support.