Summary
Building an API for real-time data is a challenging project. Making it robust, scalable, and fast is a full time job. The team at Tinybird wants to make it easy to turn a continuous stream of data into a production ready API or data product. In this episode CEO Jorge Sancha explains how they have architected their system to handle high data throughput and fast response times, and why they have invested heavily in Clickhouse as the core of their platform. This is a great conversation about the challenges of building a maintainable business from a technical and product perspective.
Announcements
- Hello and welcome to the Data Engineering Podcast, the show about modern data management
- When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
- RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
- Ascend.io — recognized as a 2021 Gartner Cool Vendor in Enterprise AI Operationalization and Engineering—empowers data teams to to build, scale, and operate declarative data pipelines with 95% less code and zero maintenance. Connect to any data source using Ascend’s new flex code data connectors, rapidly iterate on transformations and send data to any destination in a fraction of the time it traditionally takes—just ask companies like Harry’s, HNI, and Mayvenn. Sound exciting? Come join the team! We’re hiring data engineers, so head on over to dataengineeringpodcast.com/ascend and check out our careers page to learn more.
- Your host is Tobias Macey and today I’m interviewing Jorge Sancha about Tinybird, a platform to easily build analytical APIs for real-time data
Interview
- Introduction
- How did you get involved in the area of data management?
- Can you start by describing what you are building at Tinybird and the story behind it?
- What are some of the types of use cases that your customers are focused on?
- What are the areas of complexity that come up when building analytical APIs that are often overlooked when first designing a system to operate on and expose real-time data?
- What are the supporting systems that are necessary and useful for operating this kind of system which contribute to the overall time and engineering cost beyond the baseline functionality?
- How is the Tinybird platform architected?
- How have the goals and implementation of Tinybird changed or evolved since you first began building it?
- What was your criteria for selecting the core building block of your platform, and how did that lead to your choice to build on top of Clickhouse?
- What are some of the sharp edges that you have run into while operating Clickhouse?
- What are some of the custom tools or systems that you have built to help deal with them?
- What are some of the performance challenges that an API built with Tinybird might run into?
- What are the considerations that users should be aware of to avoid introducing performance issues?
- How do you handle multi-tenancy in your platform? (e.g. separate clusters, in-database quotas, etc.)
- For users of Tinybird, can you talk through the workflow of getting it integrated into their platform and designing an API from their data?
- What are some of the most interesting, innovative, or unexpected ways that you have seen Tinybird used?
- What are the most interesting, unexpected, or challenging lessons that you have learned while building and growing Tinybird?
- When is Tinybird the wrong choice?
- What do you have planned for the future of the product and business?
Contact Info
- @jorgesancha on Twitter
- jorgesancha on GitHub
Parting Question
- From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA