We built Document360 in a bootstrapped model by making use of the right toolset for each purpose. And we chose Algolia as our search partner. Here I’m sharing how we bumped into Algolia QPS Violation and how did we fix it.
Algolia as a Search Partner
Algolia is one of the most reliable platforms for building search capabilities. The hosted search API caters to billions of queries for different websites every month.
Algolia offers different tiers of subscription and with a higher tier, they offer a higher Query per second (QPS). QPS is the highest number of hits received for any given second.
Let’s say, you have 10 customers searching for something on your website at the same time and 10 requests arrive in Algolia for information – it will be counted as 10 QPS. Read more on how Algolia calculates the max QPS.
With our current subscription, our allowed limit is 75 QPS. This means that 75 requests are allowed to hit Algolia server per second. This value is more than enough unless you have a high volume e-commerce application.
Problem faced
We received a notification from Algolia about the QPS violation. We were quite surprised to see this and were clueless. Our immediate attention went to the customer-facing documentation website.
The first question we had in mind was “Do we have enough customers searching for something at the same time simultaneously?” The answer was No. We then started to dig deeper into the problem.
What caused the Algolia QPS violation?
We use a background job for operational backup of all the projects (one at a time). This job also includes Algolia indexes for the backed up project. We use Algolia’s C# .NET SDK to manage the indexes from our codes.
The implementation performed a.BrowseAll() operation for given index. What we did not realize was that the index.BrowseAll() operation was very similar to performing a “Search” operation on the website. In fact, the method is used to retrieve all indexed data, without a limit on the number f records it returns (Search has a 1k record limit).
This resulted in the Algolia QPS violation.
We did not know of this violation until there were enough documents in any project. With a growing number of customers and the documents in their projects, the number of Algolia index objects were also gradually increasing. We looked into the dashboard and found out that the count was 192 QPS for a certain date and between 80 to 100 on some other dates.
After an investigation, we found that ~94% of the indexing operations were triggered by the C# .NET SDK. The Algolia Dashboard was of real help in finding out the root cause.
This gave us the confirmation that the index.BrowseAll() operation was the culprit. We had to change the way we implemented the Algolia Index backup process to fix the issue. We removed the usage of index.BrowseAll() completely and this sorted out the Algolia QPS violation issue.
Summary
If you are using any of Algolia’s SDK in the backend, you must cut down the usage of index.BrowseAll() operation to zero. This might not hurt for occasional usage. But for a SaaS application, as the number of users grows, it will become trouble sooner or later.
Have you faced something like this before? Share your experience with us by leaving a comment in the comments section below.
–