Welcome to the latest roundup of our march updates from the team at Androidacy!
We’re excited to share some news regarding our current Patreon offerings, and the further optimization of our API, servers, and general infrastructure.
Let’s dive on in, there’s a lot to cover!
Infrastructure updates
This month saw the biggest leap in growth that we have seen yet – between our blossoming third party integrations, our website, and more, we over tripled our traffic this month. In total, we saw an estimated 270% increase in website visitors, and our forecasts for new users were off by a negative margin of about 170%.
While that may sound like a lot of numbers (it really is), it means one thing: we grew – a lot. We’d like to take this moment to say a big welcome to all of our new users, and we hope everyone stays along for the ride! While user growth was extremely good, it also showed a couple holes in our infrastructure.
Our proprietary scaling systems
Earlier this year and late last year, we were busy at work developing out our infrastructure to be able to scale and do it fast. Until this month, however, the new systems saw little real-world testing, and over the past week, as load increased, we were not able to scale fast enough. Unfortunately for our users, that meant a lot of 503 errors from our origins. Other times responses were slow.
We have always strived for sub 500 ms responses from our API – that’s half a second! – so that you can get what you need done fast, and aren’t waiting on our apps. Until we maxed out on load, we were hitting that. Then suddenly we weren’t. Between two (2!) massive DDoS attacks in the same month, and the unexpected sharp rise in user growth, the defects in the old system used for scaling our services on-demand were front and center.
Our systems use a mixture of load data from the qemu Guest Agent (qemu-ga) running on each of our servers and our own proprietary algorithm. Due to a programming defect in our code, this wasn’t returning the expected data and caused the automated system to not scale just when it was most critical. We’ve patched that up, though, so you should experience less slowdowns as our system learns our traffic patterns.
March Updates for security
As we mentioned previously, March has seen two of our largest DDoS attacks to date – both originating from Russian IP addresses. In the aftermath of the second attack, as we were left reeling, we made the difficult decision to temporarily block Russian IP addresses entirely. This was not politically motivated, but done out of a need to keep our servers online.
We’re happy to say the ban has been lifted and Russian users may now access our services normally, albeit with an occasional extra CAPTCHA or two.
We’ve also implemented better rate limiting controls on our servers. This might mean you see more 429 or 503 errors initially, but as we find the balance in the controls, we foresee less and less issues. We’ve already reduced such false positives ~90% from when we first implemented those shields a few days before this post.
API updates
We’ve optimized both API clients and the API server code to streamline performance and reduce traffic usage.
In addition, we about halved the database lookups required on each request, and about three quarters of the lookups still necessary should hit our in-memory cache system – that means your limited to the speed of the server memory, not the bottleneck of the connection to the database.
We’ve also made improvements to the JavaScript SDK in regards to connectivity checking and error display. The bash client has been updated to better handle the new rate limits in place on the API. Also, we fixed a bug on the backend that could group users together when applying rate limiting.
Module March Updates Madness
Okay, so maybe we exaggerated on the madness part, but hey – artistic license or something like that.
Both Font Manager and WebView Manager saw a minor patch update to incorporate new API changes. Font Manager is now updated with Android 12 support.
Patreon march updates
Starting this month, fast downloads require a Patreon tier of $5 or more. Ad-free is still $3 a month.
We felt this change would help us build out our infrastructure, and would further deal with bandwidth hogs. Users will need to upgrade their tier if this affects them.
Support for Patreon login for off website integrations is currently in limited testing. We hope to have a final solution rolled out to users by the end of April.
And bunnies
Yes, I know, I know, you saw bunny in the title and now you’re near the end. Where are the promised bunnies???
Well, unfortunately, bunnies and servers don’t mix, but since it is close to Easter, here’s a picture of some:
