1/14/2024 0 Comments Redshift ra3 pricing![]() With DS2 nodes, there was pressure to offload or archive historical data to other storage because of fixed storage limits. Amazon Redshift RA3 with managed storage is the latest generation node type and would allow Tophatter to scale compute and storage independently. To improve stability and reduce engineering maintenance, Tophatter decided to migrate from DS2 to RA3 nodes. While the issue was still being resolved, the latencies resulted in downstream issues for the analytics team, such as certain tables being out of sync with others, resulting in incorrect query results. When the issue couldn’t be resolved internally, they notified Etleap support, who typically recommended either canceling or reducing the frequency of certain long-running model queries. These events caused distress for the on-call engineers. Latencies and errors began to pop up more frequently-at least once or twice a week. Finding the optimal cluster size often required multiple iterations.ĭue to the time-sensitive nature of data needed for live online auctions, Tophatter used automated monitoring to notify on-call engineers when data pipeline latency had exceeded the desired threshold. When Amazon Redshift usage increased, Tophatter had to resize the cluster manually or reduce the frequency of certain analytics queries or models. But Tophatter’s Amazon Redshift cluster-which they managed internally-wasn’t as easy to scale. In response, Etleap seamlessly scaled to support their increased volume of ingestion pipelines and materialized data models. The following diagram illustrates how Tophatter uses Etleap’s AWS-native extract, transform, and load (ETL) tool to ingest data from their operational databases, applications, and Amazon S3 into Amazon Redshift.īefore the migration, Tophatter’s team operated 4 DS2 Reserved Instance (RI) nodes (ds2.xlarge) in Amazon Redshift, which use HDD drives as opposed to relatively faster SSDs.Īs their user base expanded and online auction activity increased exponentially, Tophatter’s ETL needs grew. Ultimately, these critical data pipelines fuel Tophatter dashboards that both internal analysts and users interact with. Tophatter configures the update schedule of the model to happen at defined times or when certain source tables have been updated with new data. Tophatter relies on Mode Analytics and Looker for data analysis, and uses Etleap’s model feature based on Amazon Redshift materialized views to persist the results of frequently used business intelligence (BI) queries. Every hour (according to the schedule configured by Tophatter), Etleap extracts all the new data that has been added or changed in the source, transforms the new data according to the pipeline rules defined by the user in the UI, and loads the resulting data into Amazon Redshift. Additionally, some of their vendors drop data directly into dedicated S3 buckets. Tophatter stores the majority of their product data in MySQL databases, while sending some webhook and web events to Amazon Simple Storage Service (Amazon S3). Tophatter’s data storage and ETL architecture We highlight the steps they took, how they improved stability and lowered costs, and the lessons other companies can follow. In this post, we walk through how Tophatter-a virtual auction house where buyers and sellers interact, chat, and transact in diverse categories-recently migrated from DS2 to RA3 nodes in Amazon Redshift. However, a migration path that is painless, flexible, and supported is not always available. Migrating to a new architecture is often a key component of this. Etleap, an AWS Advanced Tier Data & Analytics partner, is an extract, transform, load, and transform (ETLT) service built for AWS.Īs a company grows, it continually seeks out solutions that help its teams achieve better performance and scale of their data analytics, especially when business growth has eclipsed current capabilities. ![]() Tophatter is a mobile discovery marketplace that hosts live auctions for products spanning every major category. This is a guest post co-written by Julien DeFrance of Tophatter and Jordan Myers of Etleap.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |