Log Aggregator
Techsurf 2023
Part of my submission for the hackathon Techsurf 2023 organised by Contentstack, the hackathon provided 3 problem statements one of them was to build a log aggregator service prototype (something like logstash) that will aggregate logs and provide a way to visualize them, they had bonus points for using opentelemetery to collect logs from various sources like AWS CloudWatch, Kafka, various frontend and backend applications etc. I implemented a solution using React.js, Express.js, BullMQ, NGINX and Opentelemetry's OTLP.
DETAILS
The first major system design project that I took up made me work with various kinds of technologies and think about various aspects that one has to look at while building a system.
As one of the 42 finalists for Techsurf 2023 out of the 34000+ applicants, this was my final submission for the problem statement that I chose, I iterated through multiple designs, identified issues with each and every one of them, and finally arrived at this solution.
The solution I proposed is a distributed system for log aggregation and visualization, the job of log collection or instrumentation is handled by OpenTelemetry, and the data is then sent over to my aggreagation service via OTLP/HTTP.
Aggregation service:
- Routes the request to one of the servers via an NGINX loadbalancer.
- A server then pushes the request onto a BullMQ job queue, which in turn stores it on Redis to make sure the server is never fully occupied and ends up dropping a request.
- When the server load is reduced, BullMQ starts a worker thread to then process the data and store it on a cloud-hosted ElasticSearch Database instance.
Visualization service:
- Serves a React app via an NGINX loadbalancer that allows you to visualize all the log entries so far.
- Allows querying logs based on multiple parameters, enabled with full text search from ElasticSearch DB.
- Provides a basic graphical visualziation.
The project helped me understand optimizing systems to function effectively, efficiently and in a fault tolerant manner.
OTHER PROJECTS
Bus Ticketing System
PR301 SEMESTER PROJECT
Taken up as part of the PR301 semester project at IIITDM Jabalpur, the project was built with the intention to make an attempt to digitize the current system used for transit to and from the campus, the result was a system that provides real time updates about seat vacancy, a convenient wallet system, and security features like single device sign in. To prevent hogging a sophosticated QR based system was implemented to unqiuely identify user and validate tickets.
View details
