How can you design a cloud application to handle auto-scaling?

It is important that we do not let autoscaling be the answer to everything, however. Outsourcing strategic aspects of the complete solution to SaaS providers, ensuring efficiency, and spreading load through caches, CDNs and other tools in front of your autoscaled components are also important. Finally, as a rule of thumb, start with serverless and only take a step down where a need arises that current serverless offerings are not able to fill. It is easier to find reasons why you need to drop down a level than to find arguments for moving up one.

From my experience, I found that understanding application workload patterns and desired non-functional properties (e.g., performnce, reliability, secuirty) are among the most crucial steps to handle auto-scaling. While auto-scaling decisions can be automated, but this may lead to inefficient use of cloud resources, and application's qualities. Understanding application's workload and qualities can help to choose the right architecture and auto-scaling services. Often, there is no one-size-fits-all solution and thus fine-tuning the architecture and cloud services would be essential to optimize resources and application's performance. Continouos monitoring will be also key to ensure optimal performance and cost of cloud resources.

Use Domain Driven Design with Reactive Architecture to create microservices which are mostly self contained, independently deeployable and executable

In the context of auto-scaling, the serverless model offers unique advantages. With Serverless Computing, scaling is inherently dynamic, as the platform automatically adjusts resources based on demand. This eliminates the need for manual intervention in scaling decisions, enhancing efficiency and responsiveness to varying workloads. Additionally, serverless architectures often follow a pay-as-you-go pricing model, aligning costs closely with actual resource consumption. Leveraging serverless for auto-scaling not only simplifies operational complexities but also optimizes resource utilization, making it a compelling choice for applications with fluctuating workloads.

It's important to identify the features of the cloud application that will receive high requests. The following points should be considered for such an application: 1. I recommend a microservice architecture (though in some cases starting a project with a microservice architecture may not be feasible) 2. Containerization helps to encapsulate an application along with its dependencies such that it can be deployed seamlessly on any machine/environment 3. Container orchestration can help automatically adjust the number of running containers based on the current workload. Orchestration tools also help with service discovery, load balancing, health monitoring and so on which are crucial in applications that are designed to auto-scale.

Any session-related data should be stored in a shared database or cache that all instances can access. This allows any instance to handle any request, providing the flexibility needed for autoscaling. Each application component should be able to scale independently based on its own demand. Use distributed data and storage systems that can scale as your application does. These systems should not maintain any state information.

Projetar para apatridia é um princípio fundamental para garantir escalabilidade e resiliência. No entanto, é importante não negligenciar a experiência do usuário final, que pode ser afetada pela ausência de estado. Encontrar um equilíbrio entre manter um estado mínimo necessário para a funcionalidade do usuário e a escalabilidade é uma arte que pode diferenciar um bom design de nuvem de um excelente.

In the realm of complex systems, especially those handling critical business processes like payments, going fully stateless poses security challenges. Offloading states and in-process persistence to client-side may open the door to vulnerabilities, considering client apps run in uncontrolled environments susceptible to manipulations. The solution? Striking the right balance between cryptography, consensus solutions, and state-persisting services. These services, designed to be session-dependent, orchestrate stateless instruments while providing the crucial ability for rollbacks and adjustments at critical steps. It's the intricate dance of ensuring security without compromising functionality.

Use combination of DB- Cosmos, Azure Blob, Postgress- to manage source of truth & Redis cache for fast read & write operations. Do no share tables between microservices

To facilitate auto-scaling, applications should follow stateless designs. Stateless components can launch new instances as needed without relying on persisting local state information. This means leveraging external data stores rather than in-memory data that cannot be redistributed easily. Session data should be offloaded to caches providing universal accessibility. Overall system architectures should aim for decoupled services and context-free communication patterns that remove binding to specific application nodes.

This is not always true as the cloud services on offer can often handle the load balancing for you, allowing you to only focus on the business and application logic.

Whether you use cloud native services or you configure a container orchestrator, making sure the the right load balancing algorithm is in place is key to a high performance application. Health checks is a feature that most of orchestrators or cloud native services provide out of the box. It should be leveraged correctly to make sure that the containers app issues are raised and catched correctly during the health checks.

Cloud platforms, such as Microsoft Azure, allow you to implement public and private load balancers to help with this. Public load balancers manage traffic from the internet to your servers while private ones, handle internal traffic only. You can configure the health probe features for these load balancers in the Azure portal. We typically scale out and in, when working with load balancers. Scale up is an indicator if increasing compute resources such as adding more memory or increasing the number of virtual CPUs. Ensure your cloud platform supports automatically scaling up. Azure can, and changes the pricing plan model.

Implement load balancing to evenly distribute traffic across instances. Utilize auto-scaling groups to dynamically adjust capacity based on demand, ensuring resources are efficiently utilized. Implement health checks to monitor instance status, automatically replacing or reallocating resources when issues arise. Integrate robust logging and monitoring solutions for real-time insights. This combination ensures a scalable, resilient, and responsive cloud application.

Enquanto o balanceamento de carga e as verificações de integridade são essenciais, eles também introduzem complexidade no sistema. Uma abordagem interessante pode ser a utilização de soluções de IA para otimizar esses processos, permitindo que o sistema se adapte dinamicamente às mudanças de carga e mantenha o desempenho sem intervenção manual constante.

When scaling, data is the most valuable asset. First, scale invariant metrics are to be considered: preferring total seconds all CPU spent on computations rather than CPU utilization. Second, refining the data so it can most accurately hint of the need to scale, based on your use case will improve accuracy. For example, if there are multiple nodes, one admin and other worker nodes, it's essential to evalue their CPU differently.

Cloud computing provides virtually unlimited amount of resources for applications to use them on demand. Auto-scaling policies specify rules that trigger adding or removing new instances. Defining and testing these policies to trigger the scaling up and down is critical. We use Kubernetes in our deployments and define auto-scaling policies in helm charts. Collecting metrics and adjusting policies accordingly helps us increase the efficiency of resource usage.

To control costs and guarantee high availability, scaling policies are a very important component to consider. In the case of K8s, Pod-level auto-scaling can be achieved through Horizontal Pod Autoscaler (HPA). HPA allows to specify the memory usage, CPU utilization or number of requests rules/parameters to scale up or down efficiently. A validation phase is necessary using performance testing through simulating different loads and confirming that the Pods are scale up/down as expected.

While scaling horizontally, one can try to "be responsive" while scaling-out and "don't be too responsive" while scaling-in. Cloud provisioning is not instantaneous and hence, if user-experience is a consideration, try to front load the capacity by adding some buffer e.g. if calculations indicate to go for N nodes if a particular metric is breached, one can follow the 'N+1' rule. This provides a buffer in case of a sudden spike in activity, while providing extra insurance in the event of an unexpected node failure. While scaling-down, try to not to go down all the way to a single node, for the same reasons as stated above.

Looking at auto-scaling from a broader perspective, it's about establishing a dynamic framework that responds to your application's needs in real time. However, it's not a one-and-done setup. Regular testing and fine-tuning of these policies and metrics are essential. This ensures that the scaling actions triggered are not excessive (over-scaling) or insufficient (under-scaling), which could negatively impact performance, cost-effectiveness, or even the stability of your application. In essence, successful auto-scaling is a continuous journey of observation, adjustment, and refinement tailored to meet the evolving needs of your application while balancing performance, efficiency, and cost.

In my experience, microservices and containers are two fundamental building blocks of scaleable applications. Microservices simplify software development by serving a well-defined functionality, and facilitate resource efficient scaling. For example, if the system receives higher number of queries from clients, only the microservices handling these queries scale up. System resources are saved via keeping the remaining microservices untouched. Similarly, containerization increases modularity via decoupling runtime environment of microservices, and saves resource consumption via enabling microservices share operating system resources. My teams achieve granular scalability, modularity, and platform independence by using Kubernetes and Dockers.

Microservices and containers are fundamental ways to scale the entire system horizontally by adding more cost-effective nodes as demand increases. However, there are important caveats and nuances to consider: 1. Observability and Transaction Lineage: Incorporating observability and transaction lineage within microservices is crucial. This addition inevitably increases the complexity of the microservices architecture. 2. Database Segmentation and Scaling: While many teams focus on transitioning from a monolithic architecture to a microservices approach for scaling, a common oversight is relying on a single, centralized database. Each microservice should ideally have its own segmented data store or database for more scale and resilience.

While microservices and containers offer scalability benefits, their adoption introduces complexities that may not be essential for every application. The decision should align with your project's specifics; smaller, less complex projects may find the overhead of managing microservices and containers outweighing the benefits. In such cases, a monolithic architecture could be more straightforward and easier to maintain, especially if the workload is relatively stable. Consider factors like team expertise, project size, and long-term scalability vision before committing to microservices and containers, recognizing that one size does not fit all in application design.

Microservices are not a golden hammer, without proper autoscaling mechanisms (HPA, VPA, etc) in place, one might not fully benefit from a distributed architecture.

Absolutely, use containers and microservices. Ensure the microservices are not fine grained, but self contained and use bounded context paradigm of Domain Driven Design and Reactive Architecture

With microservices and more services, the attack surface can increase. Therefore, implementing robust security practices and ensuring compliance with data protection regulations is crucial for each service. Threat modeling in the development process can have the rigor and design for the secure system by identifying surface areas for risk, threats, and mitigation plans for each threat. not all threats are equally important, and that depends on the threat and the mitigation.

As we add more services, surface attack increases. During dev phase, it is crucial to adopt security practices as soon as possible in the process. Shifting left security is an important component to "secure" applications. Adopting methods like DevSecOps will definitely help catch vulnerabilities early int he process and help correct them before going to production. Post-production, security needs to be continuously monitored by SOC & Prod teams. Any unusual traffic needs to be analyzed and understood. Action plan should be ready and available for Prod and SOC teams to deploy in case of any type of attack.

If you are letting the cloud platform automatically scake your application, ensure you have policies appropriate to the data it is handling in place. Having an application handling GDPR data scale I to the wrong country can land you in all sorts of strife.

Posters got so deep into this without addressing a base concept. Security in depth. If security is addressed through the software development lifecycle, not as a sub point, it will be part of the applications DNA and fundamental to the customer experience.

This is also incomplete. The primary security concerns related to autoscaling and load balancing are availability and robustness, allowing the system to be spread across multiple physically separated datacentres, heal automatically and handle different kinds of attacks on the service. That is not to say that other security aspects are not important, but the way the above text was worded I must disagree with the picture being presented.

In a microservices architecture, individual services are decoupled and communicate over the network. This communication is typically done using HTTP/1.1 over TCP. Every new HTTP/1.1 request traditionally opens a new TCP connection. This process involves a three-way handshake (SYN, SYN-ACK, ACK) introducing latency. By enabling the KeepAlive flag in TCP, connections can be reused for multiple requests. This means that once a connection is established, subsequent requests don’t need to go through the handshake process again, reducing latency.

Consider Costs, Bottlenecks, and protect your systems. While this should happen in your design phase, it is not uncommon for engineers to underestimate how customers will interact with the products and services. While your primary services may scale and be cost effective, things like API gateways or SASS auth providers can contribute to significant costs. Also consider your IDP could be a bottleneck especially for external facing APIs. I worked for an org where we found clients were re-authenticating for every request as opposed to caching tokens. And get a good WAF that does bot protection.

There will be dragons! What ever solution you choose you will have problems. Try to identify the drawbacks of a solution and select the one that with the problems you are willing/able to handle. If this can be done early you won't be going through a bunch of re-writes later.

Monitoring is the vigilant guardian of a cloud application's health in the dynamic realm of auto-scaling. Like a watchtower, it provides real-time insights into performance, resource utilisation, and potential issues. With robust monitoring, teams can swiftly identify, analyse, and address any anomalies, ensuring the seamless orchestration of auto-scaling processes. It's the ever-watchful eye that empowers proactive decision-making, enhancing the reliability and efficiency of the entire cloud ecosystem, e.g Spring Boot Actuator acted as the watchtower, providing real-time insights. Tools like Prometheus and Grafana served as vigilant observers, offering detailed analytics.

I think that serverless is the easiest solution, especially for building MVPs for startups. GCP, AWS and Azure, for example, offer powerful infrastructure to build a high quality serverless system on the cloud. #azure #gcp #aws