Recently I've been working on a project that is hosted on the Amazon Web Services - Elastic Container Service, this is new area to me and one thing that took me a while to get used to was looking for error logs within CloudWatch.
The first thing to understand about logs in CloudWatch is that there isn't a simple feed of the logs from your application. Each singular logged message belongs to a "Log stream" which in turn belongs to a "Log group".
In this post, I'm just going to skip straight past Log streams and instead we are going to focus on being able to list all logs in a Group and then how to search for a specific log.
Where is CloudWatch?
If you are as new to AWS as I am, you'll likely be quite overwhelmed by all the information that gets thrown at you when you are logged in.
Once you start using AWS it will show services that you have recently visited, but before then you have to find them for yourself.
You could open up that "All services" section, but that will show you a massive list of everything AWS offers and is a bit of an eyesore. Instead, I recommend just using the search to find what you are looking for.
Luckily the search is a little more forgiving, you should be able to find CloudWatch in there either by searching directly or even by searching for just "Logs"
How do I view all the logged messages for my app?
Now that we are at the CloudWatch "home page" there is a lot to take in. But to keep things simple, let's just navigate to Logs > Insights in the sidebar on the left.
You should be presented with a page like:
If you hit "Run query" now, you will be prompted to select a log group. Ue the search field to find the log group that contains the logs for your application.
Note: You can select multiple groups here, so once you are happy with your selection, just click out of the dropdown.
Hitting "Run query" will now return 20 of the most recent logged messages from the selected log group(s). Depending on the application and the level of logging this might be all you need. However, my application is a web app and we log EVERYTHING.
How do I filter the logs?
You can read more about querying the logs here. But I will show some simple examples that I find extremely helpful.
One thing that constantly catches me out is that there is a default date/time range applied to the query of 1hr. This means that all queries you run will only return results if they were logged in the past hour. This can be altered up the top right of the page using either their predefined options or by clicking "Custom" and selecting your own.
Finding a string in the message
If you know what some of the log message will contain then the following is likely all you will need:
fields @timestamp, @message
| filter @message like "SOMETHING_TO_SEARCH"
| sort @timestamp desc
| limit 20
The query above will return any message that contains the string inside the quotes.
For example, if I run:
fields @timestamp, @message
| filter @message like "404"
| sort @timestamp desc
| limit 20
I expect to se any error logs that include the text "404"...
For most users, this is likely enough to get started with finding the logs they need. However, the application I am working on has loads of logs and I want to be able to do some more specific filtering.
In the application, we define how the logs are formatted. Our logs are formatted like so:
[asctime][process][levelname][name] message
Since I know this I can use `parse` to split the @message field into ephemeral fields that we can perform further filters on.
fields @timestamp, @message
| parse @message "[*][*][*][*] * {*}" as asctime, process, levelname, name, message
| filter message like "SOMETHING_TO_SEARCH"
| filter name like "python.app.name"
| sort @timestamp desc
| limit 20
The above example splits the @message field into asctime, process, levelname, name and message, which cal all be used in filters for restricting your results even further.
Summary
I really recommend having a play around with parsing the message and filtering on your parsed data. After wrapping my head around this I now find that the CloudWatch logs are not as scary as I initially thought and I can now find most things really quickly which helps with efficiently debugging issues.