Data Serialization Formats with Doug Cutting and Julien Le Dem - Episode 8
Summary
With the wealth of formats for sending and storing data it can be difficult to determine which one to use. In this episode Doug Cutting, creator of Avro, and Julien Le Dem, creator of Parquet, dig into the different classes of serialization formats, what their strengths are, and how to choose one for your workload. They also discuss the role of Arrow as a mechanism for in-memory data sharing and how hardware evolution will influence the state of the art for data formats.
Preamble
- Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
- When you’re ready to launch your next project you’ll need somewhere to deploy it. Check out Linode at dataengineeringpodcast.com/linode and get a $20 credit to try out their fast and reliable Linux virtual servers for running your data pipelines or trying out the tools you hear about on the show.
- Continuous delivery lets you get new features in front of your users as fast as possible without introducing bugs or breaking production and GoCD is the open source platform made by the people at Thoughtworks who wrote the book about it. Go to dataengineeringpodcast.com/gocd to download and launch it today. Enterprise add-ons and professional support are available for added peace of mind.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
- You can help support the show by checking out the Patreon page which is linked from the site.
- To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
- This is your host Tobias Macey and today I’m interviewing Julien Le Dem and Doug Cutting about data serialization formats and how to pick the right one for your systems.
Interview
- Introduction
- How did you first get involved in the area of data management?
- What are the main serialization formats used for data storage and analysis?
- What are the tradeoffs that are offered by the different formats?
- How have the different storage and analysis tools influenced the types of storage formats that are available?
- You’ve each developed a new on-disk data format, Avro and Parquet respectively. What were your motivations for investing that time and effort?
- Why is it important for data engineers to carefully consider the format in which they transfer their data between systems?
- What are the switching costs involved in moving from one format to another after you have started using it in a production system?
- What are some of the new or upcoming formats that you are each excited about?
- How do you anticipate the evolving hardware, patterns, and tools for processing data to influence the types of storage formats that maintain or grow their popularity?
Contact Information
- Doug:
- Julien
- @J_ on Twitter
- Blog
- julienledem on GitHub
Links
- Apache Avro
- Apache Parquet
- Apache Arrow
- Hadoop
- Apache Pig
- Xerox Parc
- Excite
- Nutch
- Vertica
- Dremel White Paper
- CSV
- XML
- Hive
- Impala
- Presto
- Spark SQL
- Brotli
- ZStandard
- Apache Drill
- Trevni
- Apache Calcite
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA