At Quantcast, data is king, and the Scale & Transport team knows how to process it. The unique features of our custom map-reduce implementation include a revolutionary petabyte sort mechanism, an efficient and flexible petabyte join and lookup, based on innovative data indexing and partitioning mechanisms, dynamic task size adjustment in running jobs, and a self-tuning system that learns and improves job performance without developer intervention. In the last 5 years it has allowed Quantcast to scale from processing 400 to 40,000 terabytes of data per day. For a workload consisting primarily of ETL, analytics, and data mining, our big data processing platform is on average 4 to 5 times more efficient than the most popular open source map-reduce system.
Now Quantcast is looking to marry the ergonomics of Spark with the adaptability and performance of our home-grown stack. Are you the Staff Engineer who can bring us into the next generation of distributed computing? You would integrate and improve new open source technologies and invent your own.
It's easy, and free! Add jobs from any website! Get recommendations from your friends! Start by adding this job...