Data Engineering

Exploring Data Engineering on AWS

Migrating your infrastructure to another location can be a stressful and fiddly experience, not helped by the fact you can’t just simply move your files, scripts and database.

I started exploring options on moving our company data ops infrastructure onto a more flexible and scalable environment. As far as cloud solutions go, AWS and Google Cloud are some of the cost effective options I could think of. I created a playlist of good material I found as part of the research.

Data Engineering Data Visualization

Dashboarding with the ELK(Elasticsearch, Logstash, Kibana) Stack

My recent internship at Audience Partners presented me an opportunity to create a dashboard which would help in “Display Reach Analysis” on Campaign operations data. The primary source of the data being the DSPs that help in serving ADs with the publishers. As a proof of concept to create a data pipeline, I utilized the ELK stack(Elasticsearch, Logstash, Kibana) to come up a neat dashboard with a scalable infra at the back-end. We will walk-thru the creation steps of the design of the framework and familiarize ourselves with the components of the ELK stack.

Kibana Dashboard

To be added …