As a new engineer, understanding the concept of logging and monitoring is important for several reasons.
First, logging and monitoring are key components of cloud-native application development. They involve collecting, analyzing, and visualizing data from an application and its underlying infrastructure. By understanding how logging and monitoring work, you can build, deploy, and manage cloud-native applications more effectively.
Second, logging and monitoring provide better visibility and troubleshooting. By collecting, analyzing, and visualizing data, you can understand the behavior and performance of the application, identify problems, and troubleshoot issues quickly.
Third, logging and monitoring promote better collaboration and DevOps culture. By providing insights into the performance and behavior of the application, it becomes easier for different teams and developers to work together on the same application.
Fourth, logging and monitoring allow for better security. By collecting and analyzing data from different sources, you can detect and respond to security threats quickly.
In summary, as a new engineer, understanding the concept of logging and monitoring is important because it is a key component of cloud-native application development, provides better visibility and troubleshooting, promotes better collaboration and DevOps culture, and allows for better security. It is a powerful tool for building and deploying applications in a cloud environment and is essential for any engineer working in the field today.
Learning Materials
Here’s a list to get you started learning about logging and monitoring. Note that some of these links may not be free and may require a subscription or payment. I receive no affiliate payments for these links.
Beginner:
- “Introduction to Cloud Monitoring” by Google: https://cloud.google.com/monitoring/docs/monitoring-overview
- “AWS Monitoring and Logging” by AWS: https://docs.aws.amazon.com/whitepapers/latest/introduction-aws-security/monitoring-and-logging.html
- “AWS Monitoring Strategies” by O’Reilly: https://learning.oreilly.com/videos/aws-monitoring-strategies/9780136611899/
Intermediate:
- “Logging and Monitoring with Elastic Stack” by Microsoft: https://learn.microsoft.com/en-us/dotnet/architecture/cloud-native/logging-with-elastic-stack
- “Using Prometheus and Grafana to Monitor K8s” by Gardener Cloud: https://gardener.cloud/docs/guides/applications/prometheus/
- “Log Monitoring” by Datadog: https://www.datadoghq.com/log-monitoring/
Advanced:
- “Google Cloud’s Operations Suite” by Google: https://cloud.google.com/products/operations
- “Kubernetes Logging and Monitoring Explained” by kublr: https://kublr.com/blog/kubernetes-logging-and-monitoring-explained/
- “Cloud-Native Logging and Monitoring Pattern” by weaveworks: https://www.weave.works/blog/cloud-native-logging-and-monitoring-pattern
Videos to Watch
Monitoring, Logging, And Alerting In Kubernetes
This video provides an overview of the different components of a self-managed monitoring and logging stack for Kubernetes clusters, such as Prometheus, Node Exporters, Push Gateway, and Alert Manager. Robusta is also mentioned as a platform for Kubernetes notifications, troubleshooting, and automation.
A Possible Learning Path
Hands-on experience: Start by setting up a simple Kubernetes cluster and experimenting with different logging and monitoring tools such as Prometheus, Grafana, and Elasticsearch. This can be done by following tutorials and guides, and deploying these tools on a cloud platform like AWS, Azure, or GCP.
Theoretical learning: Once you have a basic understanding of logging and monitoring, you can begin to explore the underlying concepts and technologies such as Kubernetes API objects, metrics, and alerting. This can be done through online resources such as tutorials, courses, and documentation provided by Kubernetes, as well as books and blogs on the topic.
Understanding the principles and best practices: Logging and monitoring are important aspects of a microservices architecture, so it’s important to understand the key principles and best practices of logging and monitoring such as observability, log retention, and monitoring dashboards.
Joining a community: Joining a community of Kubernetes enthusiasts will help you connect with other people who are learning and working with logging and monitoring for Kubernetes. This can be done through online forums, meetups, and social media groups.
Practice, practice, practice: As with any new technology, the best way to learn is by doing. The more you practice deploying and using logging and monitoring tools in a Kubernetes cluster, the more comfortable and proficient you will become with the technology.
A Note from the Architect
Believe it or not, in my early days of development a rarely created logs in production. I’m not sure if it was “bro science,” but many of us were of the opinion that writing logs would slow down our applications. It was almost as though many of us believed you could drive faster if you kept your eyes closed.
Logging and monitoring are like a detective and a spy for your application. They help you to understand what’s happening inside your application and detect any issues that might arise.
In a cloud-native environment, logging and monitoring are essential for understanding the health and performance of your application. Logging is the process of capturing information about what’s happening inside your application, such as error messages or performance metrics. Some of the common tools used for logging in cloud-native environments are Elasticsearch, Logstash, and Kibana (ELK stack), Fluentd, and Graylog. A typical log file might look like this:
[2022-11-01T09:15:00Z] [INFO] Server started on port 80
[2022-11-01T09:15:02Z] [ERROR] Failed to connect to database
[2022-11-01T09:15:03Z] [INFO] Successfully connected to database
Monitoring is the process of collecting and analyzing data about the performance and health of your application. Some common monitoring tools used in cloud-native environments are Prometheus, Grafana, and InfluxDB. These tools allow you to collect metrics about your application, such as CPU usage, memory usage, and network traffic. They also provide visualizations of the metrics, which makes it easy to understand what’s happening inside your application.
In Kubernetes, logging and monitoring are built into the platform and can be easily configured with Kubernetes resources. This allows you to collect and analyze data about your applications and systems.
