#9: Move fast and break nothing! - Rapid Application Development

Through this series of articles, I look to share my experience of building a team that crafted software solutions to extend and enhance the organization's core consulting offerings.

Read the introductory part here #6: Move fast and break nothing!, and the parts on Talent and Teaming.

The seamless coming together of People - Processes - Tooling is key to a team's success in the conception, development, and delivery of quality software, repeatedly and consistently.

A process, very simply put, is a defined and structured set of tasks that transform inputs to desired outputs. However, a process can be complicated to define crisply and can be harder to apply successfully. Several approaches have evolved by various individuals and organizations through the decades of software development. These can range from a high-level set of principles and values to detailed and prescriptive definitions of day-to-day work. 

Having started to write software a couple of decades ago, I have worked with several software development approaches during this time. These ranged from the very structured, arduous stage gated processes to those based on a very lightweight set of principles. Many of these approaches, in their time, were considered to be that silver bullet, with the position often taken that one ought to know how to work with that particular approach to be successful. The history and the evolution of software development methodologies are beyond the scope of this article. However, briefly, in my opinion, the evolution is tied to becoming more people-centric, customer-focused, static to dynamic discovery and change, complexity, and technology (hardware, languages, frameworks, among others). In my considered opinion, there have been two major inflection points in the evolution of software development approaches. The first inflection point was during the mid-1980s, with Barry Boehm introducing the spiral model and Takeuchi and Nonaka's article on The New New Product Development Game that lead to the Scrum framework; and the second inflection point in the early 2000s, with the publishing of the Manifesto for Agile Software Development.

Code at the speed of thought

I came across this inspiring tag line quite a long time ago, when Delphi was my favorite all-in-one IDE, language (actually a dialect of Object Pascal), and framework. From my early days with Borland IDEs in the 1980s, and that inspires me even today, the right developer/engineer experience with tooling is crucial to efficiency and efficacy.  

As I mentioned in the introductory part of this series, we had to overcome some challenges to fully realize the benefits of bringing management consulting and software engineering together. The fundamental challenge was to define our approach to software engineering. Our methodology had to effectively complement and collaborate with the hypotheses-driven approach our management consulting colleagues took. Additionally, we had to consider that project teams could split across geo-locations and time-zones.

We took a conscious decision to spend the first 8-9 months operating on a loosely defined set of principles that one could attribute broadly as Agile/scrum. During this time, as the team grew exponentially and our individual and collective experiences from various project engagements, so did our ability and knowledge of what works and what does not.

We then embarked on defining our methodology and, about two months later, released v0.9. We christened it by the challenge it was meant to solve; Rapid Prototyping and Development, or RPD in short. Of course, like everything that we humans do, learn, and adapt, it changed and matured as our projects and experiences grew. After all, we were not looking to implement or follow Agile; we needed to be agile! 

Processes are never written in stone.

RPD is heavily influenced by and combines Agile principles, Scrum practices, RAD approaches, and DevOps melded in our unique way, built on our individual and collective experiences. Speed, Quality, Risk, and UX are essential considerations. Inspired by Steve Job's quote, 

Design is not just what it looks like and how it feels. Design is how it works.

RPD is designed with and for the experiences of all its practitioners, our team, consulting colleagues, and clients. At the core, RPD addresses two essential definitions,

• Workflow - A lightweight definition of When, What, Why, Who, and How across a project's life cycle, from discovery to handover. 
• Responsibilities - Clarity among roles, expectations, and handshakes.

and strongly powered by,

• Tooling - Stitching the workflow and the responsibilities together, expediting development and delivery, ensuring quality, and eliminating risks.

The workflow heavily incorporated acceptable scrum practices to support our need for rapid, iterative, and incremental development and delivery. We distilled various crucial aspects of the workflow into crisp and easy to apply sets of guidelines and checklists, supported by concise definitions. The conciseness helped get new members off the ground quickly and collaborate seamlessly; this also served as easy, single-page cheat-sheets that we could pin to our desks for in-context and ready reference. Incorporated into the workflow were also essential considerations such as security, data confidentiality, and data privacy. One crucial aspect that we considered upfront is to be in-synch with the iterations of a consulting case engagement, and hence we worked in 1-week sprints and at times in daily sprints. Making progress and delivering incremental value are vital metrics for each iteration. Quantum of work or hours are not!

Similarly, the clarity in the roles deployed on a project, the responsibilities, the expectations from other roles, and the handshakes between roles and across every stage of a project and a sprint is distilled into a simple, easy to follow set of Dos and Donts.

The approach to tooling and the tools themselves brings all of this together in a powerful way. It dramatically increases process efficiency, ensures quality, adherence to practices, and compliance with organizational policies. Our tooling approach is along with the following.

• High-level architecture - Patterns and techniques used to design, build, test, and deploy applications.
• Technology stacks - Languages and frameworks across the applicable layers and form factors of applications.
• Reusable code - Code snippets, scripts, and libraries
• Accelerators - Automation scripts, CLI and IDE extensions, testing tools, and others.
• Essential ops - Staffing, Costing, Project/sprint planning and management, and others

Taking such a holistic approach to our methodology has been the winning formula; perhaps brought out best via the following three examples:

Time to first demo

Time to first demo is a crucial metric for us; time from confirmation of a project win to the first client demo. It is also the first step to showing how quick we can be. To break this down, assuming the intended solution has an Angular frontend, node.js backend, MongoDB database, and TypeScript as the language of choice, it would mean

• Ops: Setup of a git repo, CI/CD pipes, a Jira project, a Confluence space, a Slack channel, an update to our projects database, and a few other things.
• Codebase: Containing the frontend code in Angular, using Milligram and backend code in node.js with express and Mongoose from pre-prepared seeding code.
• Deployment environments: Dev, Test, and Production with requisite components, like NGINX, MongoDB, and others.

The first part of this metric is the setup of a scaffold, which covers all the above and more in other situations. The setup time is a few minutes from checking the box in the projects database to a Win. The second part is in the scaffold's inflated code, which are the bare minimums based on the project creation-time choices; the language and frameworks, and minimum functional features, including authentication, the login page, user management, among others. We can quickly put other variants of the above, based on the architecture, language, frameworks, and libraries of choice, and sets of functional features in place.

Total time to first demo, roughly 5-15 minutes. The net result is a massive gain in not just time but also quality, consistency, adherence to essential minimum standards, and upfront credibility with the client.

Server and Client synch

Our applications are web, mobile, or desktop and, at the most abstract level, comprise a rich, smart client that communicates with its server via REST end-points. Our engineers staffed in pairs, ensure the end to end delivery of any functionality. They work together at one workstation to define the client/server interface signatures in the server code before separately developing in parallel the client and server sides of the required functionality, and then pair together again to test the functionality end to end. We created custom IDE add-ins to support working in parallel and eliminate "integration" issues. Our add-in uses language parsing and transformation templates to generate or update several parts of the source code, as the engineers modified the server API signature or data models.

• Client-side methods and data models based on the server-side API signature and data models
• Postman tests and test suite for server-side testing
• Mock API server tests for client-side testing (we adapted WireMock with some customization)
• Mock data for the client and the server tests

The value realized through the savings in efforts and time, ensuring quality, and adherence to standards is tremendous. It also eliminates several mundane tasks, which typically are areas where developers tend to be error-prone. Regression testing is a breeze, as the test scripts are part of the project code and hooked up via the already generated CI/CD pipes. 

In sprint test automation

Effective testing is a must in ensuring quality deliverables. Efficient testing is an equal must for us. In our world, with the need to be flexible and agile, we need to ensure that our testing approach is equally efficient and effective. One of our techniques draws inspiration from Behavior-driven Development, Robot automation framework, and Test-Driven Development. The aspiration is for an English-like language and structure to write user stories and acceptance criteria that would lend itself as closely as possible to a test script that can be automated. 

We developed a custom test tool that used Selenium (and Appium) under the hood. It used an English-like language and syntax, allowing anyone, including non-engineers, to write tests. Tests were essentially a logical sequence of steps strung together into scenarios and scenarios into suites. The tool sported its custom IDE with syntax highlighting, language and syntax intelligence, ability to step through, trace, debug and run parts or whole of any test scenario or suite. Further customization was quickly possible, with the ability to write scripts to create new language keywords or change the meaning of existing keywords.

With this, the author of the user story and acceptance criteria can quickly and easily write the expected behavior to test and ensured that tests were ready before the code. Such tests were easy to read and understand by anyone. These test scripts are part of the project code and hooked up via the CI/CD pipes, and meant that regression testing, even for end-to-end scenarios, is easy.

There are several other such tooling ideas that we continually experiment with and employ. Our holistic approach to combining talent, teaming, and methodology, fostered by our cultural values, ensured that we continued to reap and improve our successes.

It is necessary to keep in mind that a specific approach, methodology, or process is not necessarily suitable for any team or any project. A team choosing to implement Agile, Scrum, XP, or any other approach is unlikely to tap into its full potential. It is imperative that leaders and teams put the effort to define how they want to work, what norms they wish to adhere to, what and how they create and deliver their work, and that supports the culture, the people, and teaming. It forms the very definition of what it means to be them.

Organizations often underestimate the importance and criticality to invest in the initial and ongoing efforts to ensure foundational agility in their methodology. Wholesome considerations and continuous investment is a must.

In the next and concluding part, I will discuss our governance approach that helped support the fantastic team we built to achieve its many successes.

Click here to continue reading, on Governance