Categories
Uncategorized

How to report on testing

testing-meme
QA testing meme

The article is based on How is the testing going by Michael Bolton.

Imagine that a Project Manager has approached you with a question “how is the testing going?”. We will let you know how and what to answer in this article.

An inexperienced tester will dive into numbers straight away, and his answer will sound like this: “Why, everything’s cool. I’ve completed 234 test cases out of 500. 16 automated tests out of 100 have gone down”. This here is a bad answer. Dry numbers with no context whatsoever do not reflect the state of the product, thus, useless. They do not help the manager decide what to do next and how the team should proceed. 

An experienced tester has to provide his team with useful information that will help assess risks correctly and set up the priorities.

How to present information?

At Fora Soft, we present the useful information in this order:

  1. Explain the state of the product. What serious problems we’ve encountered, why they are serious, and how they can affect our customers. This information helps the team understand what to deal with first
  2. Explain how the testing is going. What still needs testing, what has already been tested, what we will not test and why is that. It’s important to mention how the testing was being done, what environment was used and why. This data is mandatory to assess the risk of receiving problems in the untested product areas and correct the testing plan, should the need arise
  3. Explain why we test the way we do it. Why the tests we’ve chosen are more effective than those we haven’t. When time and resources are limited, it’s crucial that we choose the right testing areas and sort out priorities
  4. Let the team know about the problems we’ve encountered during testing. Namely, what makes testing harder, what may cause us to miss bugs, what could help us make testing faster and simpler. If your team knows about your problems, they can help you

The main task the tester has is finding problems that put the product’s value in danger and reporting on those problems to the project manager and the team. Providing this information in a timely manner allows creating a high-quality product without missing deadlines.

To catch any problem that endangers the product’s value quickly, we use test plans and test strategies. Stay tuned to find out how the plan and strategy are created!

Do you want to learn more about our processes and how we do things? Do not hesitate to contact us via our contact form! It is right here 🙂

Categories
Uncategorized

Testing automation in iOS

According to Businessofapps, mobile apps become more and more popular. They offer more functions that are in need of testing. This is a crucial process in development. No one wants to use a lousy app that keeps crashing or working bad, right? Also, what if there’s a payment function in an app? Working with a product that can let you down when you send money somewhere isn’t a good idea.

To improve the quality of an app, we need to test it. It’s believed that the developer’s eye eventually starts swimming and they can miss some problems. So, we have testers for that. To increase the speed of testing and implement a system in it, testers automate the process.

Let’s take a look at testing automation using iOS apps as an example. Nowadays, mobile apps have way more functionality than before, so testing takes more time, too. And don’t forget that there are many kinds of iOS devices out there, which increases the time spent on testing even further! To guarantee that an app works correctly on all devices, many tests need to be done, which makes the process longer and more expensive. Testing automation is out there to deal with it because testers won’t need to check some functions manually. It’s enough to write a script and edit it from time to time. Sounds good, doesn’t it? Nothing is perfect, though, so let’s turn to the pros and cons of this method:

Advantages

  • Concurrent testing on multiple devices
  • Faster testing process
  • No human factor. Sometimes, bugs might appear that are difficult to catch. Even if a tester is able to catch those, they can not always understand what the cause was. Automated testing may help with the precision of understanding
  • Testing transparency. When the testers swap, old scripts stay, and the testing process will continue as intended. Regression testing stays the same, too. If you need to change something, or if a new tester wants to check the application logic, the script will work as documentation. This is one of the main advantages

Disadvantages:

  • When iOS updates, you have to wait for the automated testing tools to update, too
  • The tester needs to have special knowledge about automated testing scripts. The company needs to teach employees to do that or hire more expensive ones

The auto testing tools have pros and cons about them. It’s difficult to find a perfect tool, and oftentimes you need to sacrifice convenience in working with a tool or with its possibilities.

When you need automated testing

Do you even need it? The answer is “yes”, if:

  • Your app has too many functions and features, and you are going to support it while adding new things along the way. Why do you need auto testing then? New functionality can conflict with the old one. For example, you’ve introduced a chat and calls broke. To find out the problem, the tester has to test the whole app manually. It takes a lot of time, and the tester is at risk of missing something else! Auto testing helps avoid that problem, as the tester won’t have to test everything after introducing each feature.  Whatever stayed the same will be checked automatically, as long as you launch them and then collect the data. It helps reduce the testing time and the development costs
  • You are going to adapt the app for each new iOS version and take advantage of new system possibilities. Every iOS update can break something within the app. Even if you never planned to update the app in the near future, it might be that you have to. In that case, automated testing will help with that. After that, you’ll understand what it was that broke and be able to solve the problem. Obviously, you wouldn’t add automated testing with this sole purpose, but they will help a great deal if they are already implemented
  • There are testers on your team that possess some knowledge of automated testing. They at least have to know some popular programming language if you go for, say, Appium. Or, they have to know Swift if your choice is XCTest / Earl Gray / KIF. Testers also need to know all the possible testing methods and needed tools. If your employees only know how to manually test apps and have no knowledge of programming languages whatsoever, you will either have to teach them or hire new ones

However, writing automated tests is programming, although you’re not writing new functions for your app but rather a program that goes through your product and checks it. It is expensive. It won’t be worth it to add automated tests if:

  • The app is small. It doesn’t have lots of functions and it is easy to test it manually. With that being said, you are not planning on adding new functions on a constant basis
  • The app is supposed to be developed and distributed within a short period of time, such as those for the World Cup-2018 or the Olympics-2014
  • The app changes frequently. The functionality is unstable. For instance, a startup that looks for its client and keeps changing the main features

Tools for automated testing in iOS

After finding out the main advantages and disadvantages of this approach, let’s take a look at the tools.

XCUITest/XCTest

Apple developed a fully native tool that is out there only for testing iOS apps. Since it is native, external dependencies won’t be there. You develop tests in Swift or Objective-C, which helps developers and testers interact more effectively. However, developing in those languages isn’t that simple. It might be that testers will turn to developers to ask for help far too often, which will make work a bit chaotic. 

There is a test recorder, too. It records real actions with an app and creates a test out of them, but using it is actually quite hard. It isn’t very accurate and it’s best to use it as an assisting tool while developing main tests in Swift or Objective-C. XCUITest/XCTest also works in a separate stream, doesn’t read the state of an app. Therefore, delays in updating the data may lead to an impossibility of seeing requested elements.

EarlGray

The Framework by Google. It requires tests in Objective-C or Swift. The framework synchronizes requests, UI, and streams – that’s the advantage of it. However, EarlGray isn’t very popular because you can only test iOS apps with it. It isn’t very different from XCUITest, yet it is not native, so testers would rather use XCUITest.

KIF

KIF is a framework that has to be added to the project to use it. Objective-C or Swift are the testing languages. Its realism is its main competitive edge. KIF can simulate interaction with a user, therefore it’s very good for UI testing.

You see the iOS-only tools above but when mobile development is in question, oftentimes the developers go for both iOS and Android apps. So no wonder there are cross-platform tools for automated testing.

Detox

JavaScript is a language for tests with Detox. It can access the memory and control ongoing processes. The framework works with emulators and real devices and uses native methods right on the device. It also uses EarlGray and is considered to be really good at testing apps written in React Native, because React Native uses JavaScript, just like Detox. It allows for writing the same tests for Android and iOS.

Appium

Appium is the most popular tool nowadays. It allows testing apps regardless of the platform, type, and system version. Writing tests for each platform is possible using unified API, without transforming an app into a special, network-compatible kind. Appium doesn’t require adding to the app source code. It’s working as a separate tool. Let’s take a look at its advantages:

  • A big number of languages for tests: Java, C#, Python, Ruby. It means that Appium doesn’t only work with Objective-C or Swift, so all testers will be able to create tests
  • An app doesn’t need re-compiling or changing it for automation’s sake. It’s important because the test source code and the app source code aren’t in the same project, and they are developed separately. These two don’t depend on each other, so one can avoid many problems. For example, if somebody wrote the tests incorrectly and they don’t compile, it won’t affect the app in general
  • It is cross-platform. The testers can develop tests for iOS and Android in the same environment, in the same language. They can even re-use the code. It saves time and money
  • Wide functionality. You can launch and stop the app, check the visibility of elements on the screen, and use gestures. Simulator and real devices work with Appium

Appium has some disadvantages, too. It is essentially a superstructure over native iOS and Android drivers. The tests can break more often due to the mistakes in the superstructure code. It’s important to notice here that Appium is very popular, develops quickly, so arising problems will be likely solved in the future.

Conclusion

Automated tests gain popularity in mobile development. They have advantages and disadvantages. Introducing automated tests is worth it when the benefit outweighs the costs. It is not magic, it’s still development, but you’re not developing new functionality but the ways to test the old. If an app has many functions and you keep updating it, but the majority of functions stay the same with each version, take a look at automated testing. After spending money once, you will save more later on.

Testers won’t have to test everything manually with each update. You only changed the teacher’s profile, but all courses, login, payment, booking, admin dashboard have to work the way they did?  The automated tests will have to work for it then. If they didn’t, it means that your small update broke something, now you have to figure out what. If the tests went through – be happy, because the tester will only manually check the teacher’s profile instead of the whole functionality.

We took a look at the tools that enable automated testing. It’s worth saying that we didn’t talk about all the tools out there. There are still paid, unpopular solutions or solutions that are not supported by the developer anymore. Whenever you choose a tool, consider the application’s peculiarities, check if there will be an Android version of the app. Also, consider your tester team! 🙂

We at Fora Soft also use automated testing for some of our projects and find success with it. Do you want to learn more about this topic? Message us using the Contact us form!

Categories
Uncategorized

7 ways to Simulate Low Network Speed to Test Your Mobile Application

a graphic hand holding a phone with the article title on the left

When testing mobile apps, newbies QA frequently forget to check the app with an unstable Internet connection. But in many cases this is critical: connection speed directly influences user experience and workability of the main functions. It is especially true for applications where geolocation and mobile Internet are heavily in use. For example, video chats, messengers, and other multimedia products we specialize in.

In this article, we’ll show how to simulate slow Internet connection on a test device with no hassle. 

Network Link Conditioner

Let’s start with a standard utility Network Link Conditioner for iOS apps testing. It lets the QA adjust the Internet connection as they need.

To switch on this function on iPhone, you need a MacOS device:

  1. Download and install Xcode for Mac
  2. Open Xcode on Mac
  3. Connect iPhone to Mac
  4. Allow Mac access iPhone
  5. Open Settings on iPhone
  6. Scroll down
  7. Tap Developer
iPhone settings
  • Tap Network Link Conditioner
Developer section setting options
  • Pick network preset or create your own
  • Switch on the Enable toggle
Developer section interface and enable toggle
Enable toggle in Network Link Conditioner

iOS lets us choose one of pre-installed presets of connection quality – or create our own preset.

For our own preset these settings are available:

Settings for preset connection quality

Here we see that Apple took care of testing apps with different levels of connection quality and gave us almost all the needed settings.

Having got acquainted with Network Link Conditioner for iOS, we’ve been sure such a feature would be on Android too. God, how much we’ve been mistaken.

Network Link Conditioner on Android

It appeared to be impossible to emulate a slow or unstable connection on a real Android with the help of standard tools. Therefore, I faced 2 paths: download some apps from Google Play that emulate slow connection, or use a specifically precise adjustment of the Internet connection access point.

Apps didn’t work out for me ☹ All the apps that give this function require Root access, and this breaks the concept of testing in real-world conditions.

So, having left the Root access as the last resort, I decided to look closer at path #2 — adjustment of the access point.

In the past, when I was a student, mobile internet traffic was ending up quickly (and we needed to read, watch something while on the lesson), and we used an iPhone as an access point. The idea came to mind: to mix the student experience and recently gathered knowledge.

Using Network Link Conditioner and an access point made of macOS or iOS doesn’t require any extra knowledge and are easy to adjust. Exactly what’s needed if we want to save time. 

So, to emulate bad connection on Android we need and Android device and… an iPhone with mobile internet Developer Tools switched on.

  1. Make iPhone the access point (Settings > Mobile Data > Personal Hotspot)
  2. Adjust connection with Network Link Conditioner
  3. Connect to the access point with the Android device
  4. Ready. You’re awesome 🙂

Cloud device farms

You don’t have to own a device farm to test an app on a wide range of mobile devices. It’s also quite inconvenient if you or your employees mostly work remotely. We started using cloud device farms, and let us tell you — it’s a big gamechanger. You should try it for better mobile testing. 

Some farms (e.g. Browserstack or LambdaTest) allow connection throttling — artificial connection slowdown — for testing on mobile devices. 

Briwserstack interface
Throttling options on Browserstack

It might require a paid subscription to throttle the connection. For Browserstack the individual plan price starts from $29.

Android emulator testing

Another way to simulate slow connection is to use an emulator. It’s software for a PC that copies another OS. Android emulator imitates the performance of a real smartphone, so you can test your mobile app on it if you don’t have an actual device.  

Testing on emulators could never compare to testing on real devices but it’ll do for the first development stages. 

With Android studio emulators you can set the limit based on the cellular type (LTE, EDGE, etc.). Each type has its own speed limitations (in kbps): 

gsm — GSM/CSD (up: 14.4, down: 14.4)
hscsd — HSCSD (up: 14.4, down: 57.6)
grps — GRPS (up: 28.8, down: 57.6)
edge — EDGE/EGPRS (up: 473.6, down: 473.6)
umts — UMTS/3G (up: 384.0, down: 384.0)
hsdpa — HSDPA(up: 5760.0, down: 13,980.0)
lte — LTE (up: 58,000, down: 173,000)
evdo — EVDO (up: 75,000, down: 280,000)
full — No limit, the default (up: 0.0, down: 0.0)

How to test an app on Android emulator with throttled connection?

1. Download Android Studio

2. Create an emulator using Android Virtual Device Manager 

3. When creating / editing the emulator pick the cellular type: 

  • Tap Show advanced settings
Android Studio interface
Advanced Settings in Android Studio
  • Select speed in the Network section
Android Studio interface
Picking the Speed in the Network section

4. Run the emulator

5. Launch your app in the emulator

6. Turn off the WiFi on the emulator for it to use cellular connection settings

Changing cellular type in the device settings 

You can pick a certain cellular type on an actual Android or iOS device. To do that you’ll also need a SIM-card with Internet access.

How to switch to another network on an Android device? 

  1. Go to Settings
  2. Then Network & Internet 
  3. Open Internet
  4. Go to SIM settings
  5. Tap Preferred network type
Instruction on how to switch cellular types on Android
Instruction on how to switch cellular types on Android

Works for Android 12. Section names and placings may differ depending on the device and OS version. 

How to switch to another network on an iOS device? 

  1. Go to Settings
  2. Then to Mobile Data
  3. Open Mobile Data Options
  4. Tap Voice & Data
Instruction on how to switch cellular types on iOS
Instruction on how to switch cellular types on iOS

Works for iOS 15.

Bandwidth throttling in web debugging tool Charles

If cloud farms are expensive, launching an emulator is complicated, and you don’t have any SIM, you can use the network throttling function in Charles. 

Basically, you can use Charles to keep track of the mobile devices traffic when you carry out functional testing or debugging. But you can also use it to simulate slow internet connection for testing.  

How to imitate slow connection with Charles?

  1. Setup your host machine with Charles installed as a proxy for the device and the app you’re testing. Learn more about it from the Charles documentation
  2. When in Charles go to Proxy > Throttle settings
  3. Tap Enable Throttling
  4. Select the speed limitation preset or set it up manually
  5. Save the settings
  6. Click Proxy > Start throttling
Charles interface
How to start throttling on Charles

Throttling the speed in the router settings

You can also go for a radical way and throttle the speed in your router settings. It will definitely work if you test your app at home, on a real device, and you have access to the router admin panel. Keep in mind that having such an option depends on the router.

How to throttle the connection on a router?

  1. Go to the router settings. To do that, enter the special URL or the router IP in the address bar of your browser (192.168.1.1 or 192.168.0.1)
  2. Enter login and password. You can find them on the router itself (usually both are “admin” by default) 
  3. Open the Bandwidth control section
Router admin panel
What a standard Admin panel looks like

Don’t forget to cancel the limitation once you’re done 🙂

Conclusion

There’re many ways to test an app and fake bad internet connection. The most convenient one, at least for us, is Network Link Conditioner. But you’re free to choose the best for yourself. 

Tested everything, it all worked out? Don’t forget to report on it 🙂