What is the best way to choose mobile devices for testing?

"Know Your audience!" This statement says everything about selecting mobile devices for testing. The primary matrix of the device selection should be driven by the platform used by most of the audience and then it can be bifurcated by the brands and the models. The focus has to be of course on the widely used devices and the platform but the least commonly used platform version should also be considered (with light effort weightage) to make the app as robust as possible.

I agree with this statement. e.g. if your target audience is USA and you are planning to develop android then you should go with Samsung devices. My recommendation is go with different android version like version 11, version 13. There were a lot architecture related changes in these two versions. For iPhone devices I think you can purchase two devices one below iPhone X version one iPhone X or above (to handle notch concept which impacts UI too).

We need to know the most relevant devices (brand/model) used by our customers who use our application. Through metrics or directly with the customer, if it is a B2B customer. In the latter case, there can generally be a device approval process.

Some best practices for selecting mobile testing devices: 1: Develop a device matrix comparing the specifications of each device, including screen resolution, CPU, memory, RAM, and other hardware parameters that may affect the app's performance 2: Incorporate a combination of high-end and low-end devices, as well as popular device brands, to ensure thorough testing coverage

Understanding your target audience is paramount. Identify the devices they commonly use and prioritize those in your testing strategy. Consider factors like demographics, geography, and device preferences to tailor your testing efforts to your user base. This ensures that your app is optimized for the devices your audience actually uses.

For testing on iOS, you want to ensure you’re testing for both iPhones and iPads, and thinking about various screens and device layouts. This includes testing on devices with Touch ID, Face ID, or even Dynamic Island. You also want to consider devices sizes from the smaller iPhone SE to the larger iPhone Pro Max. By adopting a robust testing approach across diverse iOS devices and layouts, you can deliver an exceptional app experience for your users.

On Android we have general rule: the most no-name phone you have, the most effective your tests will be. You indeed should check on the most popular brands, but having a worst phone ever made always will be an advantage.

If it's a B2B customer, there may generally be a device approval process. Therefore, our application will be tested and approved on a specific device, whether approved or not. But in cases where the application is for all users (individuals) it will be a little more challenging It is recommended to check global metrics for the most used devices (brand/model) in the world and operating system versions. Generally, a minimum version of the device's Operating System is established to be compatible with our application And be very clear with the permissions required for the correct use of the application, clearly informing the user if their device has any limitations that will prevent the application from fully functioning.

Take into account the diversity of mobile devices in the market. This includes variations in screen sizes, resolutions, operating systems, and hardware capabilities. Cover a spectrum of devices to ensure a broad testing coverage. This approach helps uncover potential issues that might arise on different devices, leading to a more inclusive and user-friendly app.

Device farms can also be used to automatically test a mobile app across different form factors and operating systems. It's impossible to manually test and app on all device types and form factors especially when you're considering the Android OS that has a heavily fragmented OEM base.

Apply the Pareto principle, also known as the 80/20 rule, to testing. Focus on the 20% of devices that contribute to 80% of your user base. Analyze usage data, user feedback, and market research to identify the most popular devices. Prioritize testing on these devices to maximize the impact of your testing efforts. This pragmatic approach allows you to efficiently allocate resources and address the majority of your users' needs. In essence, tailoring your device testing strategy involves a mix of understanding your audience, accommodating device diversity, and applying principles like the Pareto principle. This ensures a more effective and targeted testing process that aligns with user expectations and device usage patterns.

Applying the Pareto Principle, or the 80/20 rule, is a smart way to optimize device selection for testing. By focusing on the 20% of devices responsible for 80% of your user base, you can achieve significant coverage with less effort. Identifying these key devices involves considering factors like: market share, user feedback, and compatibility and edge cases. By prioritizing testing on these devices, you'll achieve efficient coverage and identify important bugs without testing every possible combination.

Focusing on just 20% of available devices may be in fact wrong. This needs to be in 'sync' with your Analytics. Just because you focused on 20% of devices may not be enough and maybe you haven't consider enough devices based on your Analytics data. This principle is right, but it's not be solely considered. There are multiple factors involved.

Keeping your device list updated regularly is crucial as there’s new devices and OS updates each year. Both iOS and Android release major updates annually, introducing new features. Additionally, Apple releases new iPhone models annually, and yearly or every 18 months for the iPad. Staying informed about these updates allows you to anticipate potential compatibility issues and proactively adapt your testing strategy. This could involve prioritizing newer devices, incorporating updated testing frameworks, or addressing user feedback related to specific updates. By taking a proactive approach, you can ensure your app remains compatible, performant, and delivers a robust user experience across different devices and software versions.

It is extremely important to monitor the evolution of the operating systems of the most relevant brands/models Keeping a periodic track of each change log documentation can help you stay preventive of problems due to OS updates

I don't think it's necessary to just update your device list without considering that maybe you still have users using 'old' devices. Maybe those users can bring more revenue that those with newer devices. Of course there are cases in which new app features, API, etc are not supported by older devices. This is the case in which you need to convince your users that using your app is worth also for them to update their devices as well, in order to continue using your app.

Emulated devices should be only used initially to test your logic that mostly not connected to the platform-specific APIs. There's no actual need to check new network API calls or some kind of business logic on the real device, it's uneffective. Real devices are useful when you're checking perfomance and overall stability of your app.

Take into consideration that emulated devices cannot simulate everything that already works on a physical device. A simple example could be taking a photo with your camera. So yes, a mix of real and emulated devices will help you better develop and test your app. Just leave critical features for real devices and for simple features/implementations you can use emulators. This will speed up your entire process.

The testing process is always quite challenging due to the wide variety of device brands/models. Important to mix between physical devices and automated tests The automated tests tested practically all of the application's functionalities, but some will certainly have to be on physical devices.

Real Device Testing: • Explanation: Conduct tests on actual physical devices to ensure authentic user experience. • AI-Powered Insight: AI can analyze real-time device data, identifying performance bottlenecks and hardware-specific issues.

The most common problems will not be due to the device (brand/model) itself but will be due to operating system version updates For example, OS Android 11 brought major changes in the use of background location compared to versions 10 and below.

Reviewing your testing results and evaluating device selection is important for continuous app improvement. This involves analyzing your app's quality, usability, and compatibility across different devices. You can compare these results with your initial expectations, and ultimately refining your app and optimize your device selection strategy for future app updates.

Reviewing test results can be very helpful especially in an automated CI/CD pipeline that runs UI tests. UI tests can take snapshots of your app's screens and UI elements and you can access those to see how your app's UI behaves on different devices. Gather these UI snapshots and check for any errors, glitches, etc.

Use ready-to-go platforms for testing such as Firebase test lab to avoid manual checking or running your own test devices (sometimes it's a good option, but not always).

Analyzing market trends and statistics can be an effective way to choose mobile devices for testing. This involves researching the most commonly used devices, operating systems, and screen sizes among your target audience. By understanding which devices and OS versions are most popular in your target market, you can prioritize those for testing, ensuring that your app performs well on the devices most likely to be used by your potential users. This data-driven approach helps in making informed decisions about device selection for comprehensive testing.

Explore market share data to prioritize testing on popular devices, Consider regional preferences, Analyze user reviews to address issues specific to certain specific devices, Considering industry standards for device compatibility can contribute to a comprehensive testing strategy.

It is advisable to possess a device capable of receiving Beta updates promptly for new OS releases, ensuring the future-proofing of your application and preventing any disruptions in functionality with subsequent OS updates. As an illustration within the Android ecosystem, Google Pixel enjoys early access to the latest Android OS updates and beta releases, facilitating the timely implementation of necessary changes and migrations to fortify your application for the future. Furthermore, Google guarantees seven years of Android updates for the Pixel 8 series, making it a prudent investment for prolonged OS support.