Enabling CloudWatch Logs for your Pipes will allow us to trace when and which targets were triggered while the pipe is working. For example, if you have a pipe that triggers a Step Function when DynamoDB table rows are updated, the following are the different stages captured in CloudWatch Logs:

  1. ExecutionStarted
  2. PollingStageSucceeded
  3. TargetStageEntered
  4. TargetTransformationStarted
  5. TargetTransformationSucceeded
  6. TargetInvocationStarted
  7. TargetInvocationSucceeded
  8. TargetStageSucceeded
  9. ExecutionSucceeded

The above logs will help us trace the events that occurred during the pipe execution.

Following is an example of how to setup Cloudwatch logging with pipe defintion


## Log group definition
DynamoDBSfnPipeLogGroup:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: /aws/vendedlogs/${self:service}-${self:provider.stage}-dynamodb-sfn
    RetentionInDays: ${self:provider.logRetentionInDays}

## Pipe definition
RefreshTokenDynamoDBSfnPipe:
  Type: AWS::Pipes::Pipe
  Properties:
    Name: ${self:service}-${self:provider.stage}-dynamodb-sfn
    Description: 'Pipe to connect DDB stream to Step function'
    RoleArn: !GetAtt DynamoDBSfnPipeRole.Arn
    Source: !GetAtt Table.StreamArn
    LogConfiguration:
      Level: TRACE
      CloudwatchLogsLogDestination:
        LogGroupArn: !GetAtt DynamoDBSfnPipeLogGroup.Arn
    SourceParameters:
      FilterCriteria:
        Filters:
          - Pattern: '{ "dynamodb": { "NewImage": { "PK": { "S": [{ "prefix": "EMAIL#" }] }, "SK": { "S": [{ "prefix": "DEPARTMENT#" }] } } } }'
      DynamoDBStreamParameters:
        StartingPosition: LATEST
        BatchSize: 1
    Target: !GetAtt RefreshTokenStateMachine.Arn