You may have heard that 2021 is Shopify’s biggest Black Friday Cyber Monday (BFCM) ever. These four days have been huge for both Shopify merchants and our engineering teams.
Last year number take a moment in time but also help us predict what is to come next year. In our cloud in 2021, our maximum BFCM traffic exceeded 32 million app server requests per minute (RPM). In the same time period, our load balancers reached over 34 million RPM. To put that in perspective, this means that the equivalent of the total Texas population hits our load balancers in a given minute. A flash sale — a short-term sale that exceeds our checkout per minute threshold — even generated enough load to use more than 20% of our total computing capacity at its maximum.
During BFCM 2021, we will also:
-
sent nearly 145 million emails
-
averaged 30 TB per minute of network outlet traffic
-
handled 42 billion API calls and delivered 13 billion webhooks
-
wrote 3.18 PB and read 15 PB of data from our storefront caching infrastructure
-
performed over 11 Million queries per second and delivered 11 terabytes per second read I/O using our MySQL database fleet
The next year brings even greater challenges for our engineers, data scientists, user experience designers, and product managers. More BFCM sales are happening on mobile devices. More people are shopping on social media. Commerce takes place across a growing range of platforms and consumers expect a fast and consistent experience. If the metaverse becomes a reality, there will be commercial opportunities within that world that need to be explored. What does a flash sale look like in the metaverse and how does that perform?
If the above data and trends tell us anything, it is that there is no avoiding the fact that flash sales, huge floods of web traffic, and many different buying environments are a large part of the future of commerce. The questions for me are: What are the lingering challenges for engineering teams working to enable incredible growth over the next five to ten years? How do we build scalable products and infrastructure so that millions of merchants can go from zero to IPO — and beyond? Engineering at Shopify is about solving challenges and developing resilient systems so that merchants can focus on their business rather than on technology.
Here are some things we plan to do in 2022 to work quickly in a world that is rapidly growing, becoming more global, and at the same time getting closer to where merchants do business and where consumers shop.
We are making more modular code. Shopify is famously one of the largest Rails monolith codebases in the world. We have been actively transforming monolith architecture into a stately, modular monolith for several years. And recently, we’ve been changing our architectural patterns as we deconstruct monolith components for better developer productivity.
As an example, we split our storefront rendering process from modular monolith repo to ensure merchants (and their customers) get the fastest online shopping experience possible. When we’re done with the split and some code refactoring work, the results are four times faster cache fill rate and five times faster page rendering. Also, removing the storefront renderer means it can now be deployed in geographies across the planet without having to deploy our entire Rails monolith. The closer we can deliver the storefront to the buyer, the fewer round-trips between the store and the browser need to be made, further improving the overall performance of the storefront. In 2022, we will continue to explore majestic monoliths. We see that engineers working with repos directly improve merchant performance, such as storefront rendering, iterate and deploy quickly. This model also allows us to prioritize our developer experience and provide simpler setup with tighter integration with our debugging and resiliency tools.
We use new cloud development platforms to work better on a global scale. This year, we’ll spend a lot of time making sure developers can make an impact quickly — in minutes not hours. We move most of our developers to our cloud development environment, called Spin. Devs (pun intended) can rotate an entire development environment in seconds versus minutes. You can also have plenty of environment for experimentation to share what is currently being done with teammates. (We plan to share more about Spin in the future.)
Another big part of this year was about building on this cloud development platform foundation to make our developer workflow faster and smoother. We’ve also shifted all of our engineering to working on Apple M1 Macbook Pro laptops and these powerful devices, along with Spin, are making developers more productive. Spin creates opportunities for us to develop improved IDEs and browser extensions for enhanced productivity and enjoyment, and an exciting opportunity for us to explore new ways to solve problems. of the developer to a scale that was not possible in our previous paradigm in the local development environment.
We make load testing a more natural part of the development process. To prepare for BFCM 2021, we started the load test in July and we ran the highest load test in Shopify’s history: a peak load balancer of 50.7 million RPM. But, flash sales that rise within minutes are unpredictable in their load requirements as a seasonal growth pattern like BFCM. To help prepare our infrastructure and products to handle larger and more spikier sizes, we continue to improve our load testing. These load tests, built in-house, help our teams understand how the products handle larger surge scenarios across platforms. Our load test helps test product sales whether they are exclusively online, personally using our retail POS products, or a combination of both. Automating and integrating load tests as part of our product development processes is really critical to avoid performance issues as we measure with our merchants.
Here are some ways we make it as easy as possible for developers to do the best work in their lives. We want to have the right tools so we can be creative about commerce — not “How do I set up my environment?” or “How is my code generated?” Engineers like to work to scale, send significant changes to a regular rhythm, and work with a great team.
Speaking of talented teams, a team of engineers from Shopify and Github developed YJIT, a new Just-in-time (JIT) compiler that will merge with Ruby 3.1. This is 31% faster than the interpreted CRuby and 26% faster than the MJIT, reaching almost maximum performance after a iteration of any benchmark. This has a huge impact on the Ruby community inside and outside of Shopify and speeds up a lot of production code execution time.
What doesn’t change in 2022: We remain opinionated about our tech stack. We’re all on Rail and doubling down on React Native for mobile. We will continue to make large bets on our infrastructure, on building satisfying developer environments, and ensuring that we build for the success of all our merchants. BFCM 2022? Take it on.
Allan Leinwand is Chief Technology Officer at Shopify leading the engineering and data teams. Allan is formerly SVP of Engineering at Slack and CTO at ServiceNow. He has founded and held senior leadership positions at numerous companies, authored books, and ventured into the dark side as a venture capital investor for seven years. He is passionate about helping Shopify become the best commerce platform for everyone!
Wherever you are, this is where your next journey begins! If building systems from scratch to solve real -world problems interests you, our Engineering blog has stories about other challenges we’ve encountered. intrigued? Visit our Engineering careers page to learn about our open positions and learn about Digital by Design.
.