Inside the life of a Curology intern

Building a top-notch product requires teamwork and collaboration. For Curology’s distributed engineering team, we’ve carefully selected tools through trial and error to encourage a team-centric, collaborative culture. In this post, I’d like to share our current set of tools we use on our team. For each tool, I’ve also provided more context on other tools we’ve used and why we migrated away from them!

At Curology we use the Serverless Framework to build and deploy a myriad of AWS Lambda services that support everything from real-time data ingress to NLP topic labeling. Some of these services are written in Rust and recently we open sourced a piece of this infrastructure as a very small library called Vicuna.

For nearly two years Airflow has been an integral piece of Curology’s data stack. Along the way we have adopted patterns for Airflow development that have greatly accelerated our workflow. In this article, we would like to share some of the learnings from our journey developing with and operating Airflow at Curology.