Gearbox is an American video game development company based in Frisco, Texas. With tens of thousands of concurrent online gamers sending data to Gearbox in realtime, their data infrastructure was beginning to become stressed. They called Full 360 to evaluate, redesign, build and tune a new solution.
Gearbox data warehouse running on Amazon Redshift was part of a complex stack:
The existing Redshift ingestion process and database design at Gearbox could not scale to the 20TB per day of raw source data. Load performance was inadequate. The disk footprint in Redshift was too large. Handling of new attributes challenged the original design. Historical queries were unacceptably slow. This raised general questions about cost, performance and staffing.
Full 360 worked with Gearbox to streamline the data model, change the source data format, create a new loader process using ECS, and tune the tables to perform properly. First we identified pros and cons of the initial state. Then we evaluated two new design proposals, Event Based CSV and KVP from CSV.
Our recommendation was KVP from CSV which employed a hybrid storage model giving the following characteristics.
The creation of a new ingestion process enabled Gearbox to handle the volume of transactions as needed. Full 360 was able to analyze the problem and understand the basic constrains of their design and implementation and are able to expand their capabilities on a going forward basis if needed. Significant price reduction in Redshift spending, increased performance, and satisfied customer.
Fanatical Dedication to fulfilling the Cloud's Promise, Mastering its complexity, and Delivering the Impossible.