Speed Up Your Analytics With The Alluxio Distributed Storage System
Summary
Distributed storage systems are the foundational layer of any big data stack. There are a variety of implementations which support different specialized use cases and come with associated tradeoffs. Alluxio is a distributed virtual filesystem which integrates with multiple persistent storage systems to provide a scalable, in-memory storage layer for scaling computational workloads independent of the size of your data. In this episode Bin Fan explains how he got involved with the project, how it is implemented, and the use cases that it is particularly well suited for. If your storage and compute layers are too tightly coupled and you want to scale them independently then Alluxio is the tool for the job.
Introduction
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute.
- Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
- To help other people find the show please leave a review on iTunes, or Google Play Music, tell your friends and co-workers, and share it on social media.
- Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
- Your host is Tobias Macey and today I’m interviewing Bin Fan about Alluxio, a distributed virtual filesystem for unified access to disparate data sources
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by explaining what Alluxio is and the history of the project?
- What are some of the use cases that Alluxio enables?
- How is Alluxio implemented and how has its architecture evolved over time?
- What are some of the techniques that you use to mitigate the impact of latency, particularly when interfacing with storage systems across cloud providers and private data centers?
- When dealing with large volumes of data over time it is often necessary to age out older records to cheaper storage. What capabilities does Alluxio provide for that lifecycle management?
- What are some of the most complex or challenging aspects of providing a unified abstraction across disparate storage platforms?
- What are the tradeoffs that are made to provide a single API across systems with varying capabilities?
- Testing and verification of distributed systems is a complex undertaking. Can you describe the approach that you use to ensure proper functionality of Alluxio as part of the development and release process?
- In order to allow for this large scale testing with any regularity it must be straightforward to deploy and configure Alluxio. What are some of the mechanisms that you have built into the platform to simplify the operational aspects?
- Can you describe a typical system topology that incorporates Alluxio?
- For someone planning a deployment of Alluxio, what should they be considering in terms of system requirements and deployment topologies?
- What are some edge cases or operational complexities that they should be aware of?
- What are some cases where Alluxio is the wrong choice?
- What are some projects or products that provide a similar capability to Alluxio?
- What do you have planned for the future of the Alluxio project and company?
Contact Info
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
- Alluxio
- Carnegie Mellon University
- Memcached
- Key/Value Storage
- UC Berkeley AMPLab
- Apache Spark
- Presto
- Tensorflow
- HDFS
- LRU Cache
- Hive Metastore
- Iceberg Table Format
- Java
- Dependency Hell
- Java Class Loader
- Apache Zookeeper
- Raft Consensus Algorithm
- Consistent Hashing
- Alluxio Testing At Scale Blog Post
- S3Guard
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA