Fast, Reliable File Uploading: A Technical Saga

Since the very beginning, report cards have been a core feature in Gingr. Report cards allow Gingr customers (your business) to keep in touch with their customers (pet parents) while they're away. In fact, it's Lee's (Gingr's founder) favorite feature! He loves getting pictures of Ginger while he's visiting a PB&D trade show.

However, file uploads have been a constant scalability concern. Not only is file uploading one of the most resource intensive things our servers do, we handle a lot of file uploads! Yesterday (March 21), we handled well over 100,000 file uploads with a peak of 4 per second. Once we receive an upload, we also have to process the image to make it suitable for use in a web browser.

Over the years, we've tried many approaches to handling file uploads. They've all worked for a while...until Gingr grew to a point where the current system couldn't handle the increased load. We think we've finally achieved the ultimate solution- infinite scalability. Let's review the history of file uploads in Gingr:

Upload to Gingr Directly, Process & Save Locally

Initially, Gingr took the "traditional" approach to file uploads. A user would upload a file directly to Gingr, which would then process the file (resize and rotate as needed) and save it on a hard drive shared amongst the servers.

Upload to Gingr Directly, Process Asynchronously & Save in Cloud

Once our second hard drive was filled to the brim, we knew we needed to find a more scalable solution. Users would upload files directly to Gingr, which would temporarily store the file locally. We'd then notify another server (using Redis pub/sub) about the upload. This other server would process the file and save it in the cloud. It would then update the database with the new file location. 

Upload to Gingr Directly, Process & Save in Cloud

After about a year, activity increased to the point where the temporary hard drive was unhappy with all the reads/writes we were throwing at it every second. We started getting failing disk warnings. That's scary, and it was time to find another solution. Now, users would upload files to Gingr which would process the file and save to the cloud. 

Upload, Process & Save in Cloud Directly

After a few months, our servers started to get unhappy with all that work. Resizing images takes up a lot of RAM and CPU, so our servers had to work harder and harder every day. Although nothing was "wrong", it was clear this solution wouldn't scale.

We turned to Google Cloud Platform to help us scale file uploads infinitely and to solve this problem once and for all. Using a mixture of Serverless cloud functions and Cloud Storage, users now upload files directly to Google. Serverless functions then process the file asynchronously.

There are some amazing benefits to this solution:

  • File uploads don't touch Gingr's production servers at all.
  • Uploading directly to Google is much faster than using Gingr as a middle-man.
  • Serverless architecture means Gingr's operation team doesn't have to maintain new infrastructure. Google handles it all for us.

We turned this feature on earlier this week and noticed some huge improvements. Most importantly, our users are now experiencing much faster uploading. Up to a 50% improvement! Internally, our average API response time decreased 22% and our server load was down 30%.

This solution will allow Gingr to scale file uploads for years to come and provide an improved experience for our users. We're very proud to roll this out to all our customers. You probably didn't notice a thing :).

Have any questions? Like this technical blog post? Leave a comment below!

 

-Lee

 

PS - Since you made it to the end, we want to share some inside info with you! We're currently working on extending our new file upload architecture to achieve better video streaming. In the next few weeks, you should notice huge improvements in report card video streaming. It'll be just as good as YouTube!