Summary
Building a data pipeline that is reliable and flexible is a difficult task, especially when you have a small team. Astronomer is a platform that lets you skip straight to processing your valuable business data. Ry Walker, the CEO of Astronomer, explains how the company got started, how the platform works, and their commitment to open source.
Preamble
- Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at www.dataengineeringpodcast.com/linode?utm_source=rss&utm_medium=rss and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
- You can help support the show by checking out the Patreon page which is linked from the site.
- To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
- This is your host Tobias Macey and today I’m interviewing Ry Walker, CEO of Astronomer, the platform for data engineering.
Interview
- Introduction
- How did you first get involved in the area of data management?
- What is Astronomer and how did it get started?
- Regulatory challenges of processing other people’s data
- What does your data pipelining architecture look like?
- What are the most challenging aspects of building a general purpose data management environment?
- What are some of the most significant sources of technical debt in your platform?
- Can you share some of the failures that you have encountered while architecting or building your platform and company and how you overcame them?
- There are certain areas of the overall data engineering workflow that are well defined and have numerous tools to choose from. What are some of the unsolved problems in data management?
- What are some of the most interesting or unexpected uses of your platform that you are aware of?
Contact Information
Links
- Astronomer
- Kiss Metrics
- Segment
- Marketing tools chart
- Clickstream
- HIPAA
- FERPA
- PCI
- Mesos
- Mesos DC/OS
- Airflow
- SSIS
- Marathon
- Prometheus
- Grafana
- Terraform
- Kafka
- Spark
- ELK Stack
- React
- GraphQL
- PostGreSQL
- MongoDB
- Ceph
- Druid
- Aries
- Vault
- Adapter Pattern
- Docker
- Kinesis
- API Gateway
- Kong
- AWS Lambda
- Flink
- Redshift
- NOAA
- Informatica
- SnapLogic
- Meteor
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA