AWS recently introduced logging from the pods directly to CloudWatch; see user guide. The FluentBit setup could achieve that, but it’s a bit tricky. For those who would expect a side container in the pod the way it’s described at “How to capture application logs when using Amazon EKS on AWS Fargate”, you might be surprised.

The user guide helped a lot, and after a setup of the namespace, I was almost done. The first thing which seemed to be a bit odd was the difference between the user guide and the blog post. Both described the OUTPUT configuration of FluentBit differently. The guide has the following snippet:

 

```

[OUTPUT]

    Name cloudwatch_logs

    Match *

    region <us-east-1>

    log_group_name fluent-bit-cloudwatch

    log_stream_prefix from-fluent-bit-

    auto_create_group true

``` 

 

The problem is that the configuration didn’t work until I changed the region to `us-east-1` instead of `<us-east-1>`. I am not sure if that was the root cause, but it worked for me. I ended up with:

 

```

[OUTPUT]

    Name cloudwatch_logs

    Match *

    region us-east-1

    log_group_name my-beloved-project

    log_stream_prefix app-

    log_key log

    auto_create_group On

```

 

If you were able to put the original configuration into work, please, let me know in the comments. I wasn’t that lucky. The other thing I had to add was the parser configuration:

 

```

[PARSER]

    Name docker

    Format json

    Time_Key time

```

 

Without the parser, I was not able to make JSON logs work. It was always displayed in the CloudWatch as `stdout`. At the end, I ended up with this config map:

 

```

apiVersion: v1

data:

  output.conf: |

    [OUTPUT]

        Name cloudwatch_logs

        Match *

        region us-east-1

        log_group_name my-beloved-project

        log_stream_prefix app-

        log_key log

        auto_create_group On

  parsers.conf: |-

    [PARSER]

        Name docker

        Format json

        Time_Key time

kind: ConfigMap

metadata:

  name: aws-logging

  namespace: aws-observability

```

If you wouldn’t like to do the setup on your own, use the published terraform module at https://registry.terraform.io/modules/AckeeCZ/fargate-logs/aws/latest

Are you interested in working together? We wanna know more. Let’s discuss it in person!

Get in touch >