What strategies do you use to design complex systems?

One fundamental element that is often missed within the problem definition phase that is CRITICAL to the success is how it's going to be tested or measured. An engineer should never write a line of code without first writing the unit test for how "success" is going to be measured of the component and how that component interacts with the rest of the overall solution. If we could all write unit tests at every level of the system, prior to the development phase, then I strongly believe we would write better software. Implementation of quality gates and architecture review cycles are two additional best practices.

In my experience maintaining bilateral traceability across all stages of system design by having clear links between MRD, PRD, system arch, Sys design, FW/SW/components design implementation and corresponding tests phases has great advantages. This ensures a seamless flow and visibility of crucial information and verification/validation throughout the cycle. e.g. consider a product intended to operate for X years. If this requirement is only well thought during system arch and system integration test and but not considered or not thoroughly considered during component design or component selection or associated tests for verification/validation, it can result in complex bugs uncovering late in cycle. Leading to longer product release cycle

Start by clearly defining the problem you are trying to solve and the goals you want to achieve. Understand the requirements and constraints of the system to ensure that your design meets the needs of the stakeholders.

Undoubtedly, good design should start with Product Discovery. Know the problem that you are trying to solve, know the environment, know the limitations, the key aspects, etc. There is a lot of books about that topic, one I'd recommend is "Inspired: How to Create Tech Products Customers Love". It is important to work out the concept properly, before you even start thinking about its implementation.

Toda inovação necessita contribuir com algum benefício para a sociedade. Identificar oportunidades torna-se mais fácil quando desenvolvemos nossa observação no dia a dia, principalmente pela identificação de problemas coletivos. Tendo o problema foco identificado, podemos definir os objetivos com maior precisão e agilidade.

Create models of the system to visualize its structure, components, and interactions. Use tools such as UML (Unified Modeling Language) diagrams to represent the system architecture, data flow, and behavior

Modelling behaviours is important but I tend to leave it for later. When talking about really complex systems I strongly prefer to start with foundational assumptions, with specifying actors, their responsibilities, all basic rules of the system. Think about it as a set of rules of the world you are building for a game. And then make sure you will stick to those rules! Only after having the rules defined, I follow with defining processes (behaviours).

A aplicação dos princípios de projeto contribuem a garantir que o sistema seja eficiente, confiável, escalável e que atenda às necessidades dos usuários. Enxerga-lo com um todo dificulta seu controle, por isso uma das melhores estratégias é decompô-lo em diversas etapas ou "mini projetos", a fim de desenvolvê-los segmentados e posteriormente integrados em um resultado final. A cada avanço, revisitar a origem do sistema e confirmar que os objetivos nativos continuem sendo atendidos.

Use established systems design principles such as modularity, abstraction, encapsulation, and separation of concerns to break down the system into manageable components. This helps to reduce complexity and improve maintainability.

It helps tremendously to go back to point 1 and think again about the environment in which the system runs. Define it as much as you can to get estimations (!) of the requirements you will need to meet. I think what is most challenging at this point is not to dive too much into technologies and implementations just yet. Think about the teams who will work on certain parts of the design. Perhaps they will want to make their own decisions? Perhaps, given enough data, they will make better decisions?

For improving your system, you will need to know how its performing and what performance criteria matter most to you and how would you measure them. I believe the old saying of "Devil lies in the detail" is changed to "Devil lies in the data" for the new world. Your success criteria would also depend on what industry or customers you are serving to. A healthcare application may care much more about the application downtime or how quickly they can recover from a failure than a new start-up where rolling out new features quickly might be more important. So, finding your success criteria and measuring your system's performance against those, would be critical in iterating and improving your system in the long term.

Designing complex systems is an iterative process. Continuously review and refine your design based on feedback and new insights. Be willing to make changes to improve the system's performance, scalability, and reliability.

Key question is: how to actually improve the overall design by iterating? We tend to iterate by changing certain small parts, adding features and not tying them back to overall design or not checking against the foundational assumptions. This is where the design often deteriorates, sometimes beyond salvaging. I think the key concept to address this problem is to always go back to point 1 and improve each phase. Or at least check if it needs additional input.

O sistema/projeto desejado não deve ser planejado para atingir o resultado final apenas, mas sim dividi-lo em diversas etapas, em que a cada avanço, seja testado e avaliado por seus usuários/clientes, a fim de que os avanços não percam de vista o objetivo básico de atender as necessidades de quem utilizará a ferramenta. Não realizar tais testes e medições a fim de controlar o desenvolvimento do projeto ou sistema, maximiza e muito para a necessidade de revisões e grandes alterações como retrabalho, desperdiçando tempo e recursos.

Often with multi functional teams, or large business silos - there is an element of shared understanding of the problem. Different perspectives to the same element are often framed in different ways. Taxonomy or labelling becomes an additional hurdle to a shared viewpoint. Time taken to firstly understand and map the various actors perspective, identifying variations that can be often framed in a common way - if discussed and agreed. Work done here will ease communication and ultimately ensure that the complex system is framed correctly for all.

Complex systems are often quite big and expensive. There is a lot of factors to include in the design and it's often created by a bunch of teams and individuals. Every team and every person will have their own view on the topic, their own arguments and their own way of doing things and their own expectations. At the same time, maintaining the design of a system created by more than one team is crucial to its success. Which means that it's necessary to establish processes around design changes that all those teams and individuals comply with. Make evolution of the overarching design a key part of every implementation task.

Strategies for Designing Complex Systems Modular Design: Break down the system into smaller, manageable modules that can be developed, tested, and deployed independently Use of Design Patterns: Implement proven design patterns that solve common problems Prototyping and Simulation: Early-stage prototypes and simulations can help validate assumptions Incremental Development: Adopt an agile approach, allowing for incremental development of features Emphasize on Scalability and Flexibility: Design the system with scalability in mind Implement Rigorous Testing: Use a combination of unit tests, integration tests, and end-to-end tests Prioritize Security and Privacy Continuous Integration and Deployment (CI/CD)

Consider non-functional requirements: In addition to functional requirements, consider non-functional requirements such as performance, scalability, reliability, security, and maintainability. Design your system to meet these requirements while balancing trade-offs between them. Use design patterns and best practices: Utilize design patterns and best practices to address common design challenges. Design patterns provide proven solutions to recurring design problems and can help you design more robust and maintainable systems. Collaborate and communicate: Designing complex systems often requires collaboration with stakeholders, developers, and other team members.