Nowadays, mobile development is not only about creating apps by small companies or startups. The majority of enterprises are biting into the cake, too. Therefore, many different devices are in use—especially when it comes to Android.

But how you do you know if your application looks and behaves appropriately on every device in existence?

The short answer: Xamarin Test Cloud.

With Xamarin Test Cloud, you can test your mobile apps on a variety of different devices and take advantage of detailed reports.

Let’s dig into the options.

What Do I Need to Start With Xamarin Test Cloud?

This is going to be a rather short paragraph. Without further ado:

• Xamarin Test Cloud account (you can use a free trial version),
• Xamarin Test Recorder (either for Windows or Mac OS).

Go!

Before you proceed to testing, let’s cover the basics.

Xamarin Test Cloud is a cloud-based service that provides an automatable way for UI Acceptance Testing of mobile apps. You’ll find thousands of various devices within the cloud. As it happens usually, different functionalities of an application will be checked by the software.

It is not a mystery that manual tests are expensive. Sometimes, there are just too many features for manual testing—you might have no choice but to automate. That’s where automated UI tests step in.

Xamarin UITest

The Xamarin UITest is a framework based on the popular NUnit testing library, which enables developers to write tests in C#.

It is especially recommended for developers who have previous experience with automating their tests. To read more about NUnit library, please refer to this link.

Calabash

Calabash is a framework that enables developers to write their tests in Ruby, using the Cucumber tool. These tests are very close to behavior-driven development methodology. The big advantage of using this framework is the ability to write the tests in a business language. Literally—anyone following grammar rules imposed by the Cucumber tool can write the tests.

The decision about which framework you should use is up to you—both of them will test the proper behavior of the application.

Also, remember that they’re based on Calabash.

Continuous Integration

Developers can submit app packages with tests manually. With larger apps, they can also use continuous integration:

Test Cloud Frameworks. How Do They Work?

The frameworks mentioned above (UITest or Calabash) test the mobile app functionality, but they do not interact with the app directly. For that, you need something called Xamarin Test Cloud Agent.

And so, Xamarin Test Cloud Agent is responsible for running tests on the devices. It’s a separate app installed by Xamarin Test Cloud alongside your app.

Calabash and UITest frameworks use a client-server architecture:

• Test Cloud Server Application—a lightweight server app runs on the device which listens for clients to initiate a selected test. It communicates over HTTP via JSON.
• Test Cloud Agent—responsible for providing information for the server about which the test should be executed.

Xamarin Test Cloud Agent on Android

Test Cloud Server Application runs as a separate application with permission to cooperate with the actual application for tests.

Such a scenario is possible because Test Cloud signs both apps with the same key.

You do not have to add any additional references in your code to make it run.

On iOS, Test Cloud looks different. Test Cloud Server Application is built into the tested application—but in order to achieve that, you have add a NuGet package (Xamarin Test Cloud Agent).

The agent should be included in debug mode only. When you proceed to releasing your app, it should be removed by Xamarin.iOS Linker.

Run Tests in Xamarin Test Cloud

By now, we’ve talked about the testing framework structure and how it works. When you upload your application to the Test Cloud, it hosts the tests and runs them on different devices (user does not have to run tests manually from the command prompt):

Calabash or UITest—Which Framework Should I Choose?

There is no one good answer for that—you will achieve what you have planned with both of them.

The diagram below, provided by Xamarin, should help you choose the right way:

If you have your application already written in C# (Xamarin), you probably should choose UITest framework because you are familiar with the language.

What’s more, you can run all tests from the IDE (Visual Studio or Xamarin Studio). UITest is based on the popular NUnit framework, so there is also big support from the community.

For native applications (Java, Swift, Obj-C, or hybrid apps), Calabash framework will be a good choice. You can create cross-platform tests in any text editor (Ruby IDE, too) without any special tools.

What Kind of Tests Do We Have in Mobile Development?

Let’s start from the bottom.

• Unit Tests—this is the type of test that is the most costly. The purpose of it is to test a small unit of the application (often this is a method), isolating it and verifying if it’s working properly.
• Components Tests —this is the type of test that verifies if components (like classes or objects) are working properly. For example, testing the whole functionality of “Car” class is called a component test.
• Integration Tests—the purpose of this type of test is to check proper integration between classes—for instance if class “Car” behaves properly with “Tire” class.
• Acceptance Tests—these tests are carried out to validate the functionality of the application. They just check if the app functionality meets the expectations related to the functionality agreed at the beginning of development.
• UI Tests—these tests are very often done by a quality assurance (QA) team manually. They check if the user interface is implemented correctly (by clicking buttons, testing different flows). This is something that can be done with Xamarin Test Cloud.

The last type of test is called a Regression Test, and it is not included in the picture. This type of test is being used to check if our current code works with the previous versions (for instance, if newly a written class works properly with others).

Now you know what the possible test frameworks are and how tests are performed on Android and iOS.

Long story short, I described the kinds of tests that are available with reference to mobile development.

Now, let’s dig deeper into how to actually write the tests and run them with Xamarin Test Cloud to create automated UI Tests.

How to Create Automated UI Tests with Xamarin Test Cloud

Let’s Start!

I’ll show how to use it with NUnit framework and write tests that can be deployed on Xamarin Test Cloud.

This is a sample fragment of UITest:

As you can see, the instructions are quite understandable.

Let me remind you of the Xamarin UITest architecture:

Tests that you wrote are being launched by Xamarin Test Cloud Client, and through the JSON, they are passed to the Test Cloud Agent HTTP Server. Automation APIs run on your app (so for instance, button click is performed).

What Are the Possible Options for Adding UITests for Applications?

All the operations were performed in Xamarin Studio, but all of them are also possible to do in Visual Studio.

There are two possible options to add UITests:

• UITest App for cross-platform (Xamarin Forms)

• UITest App per each platform (Android and iOS)

It just depends on your solution structure and on the way you decide to create your apps (Xamarin Forms or Xamarin Platform).

Configure the Tests

In this step, I will show how to prepare a solution, configure tests, and run them. It is valuable to go through step by step how to do it.

First of all, download a sample with Android, iOS, and UITests projects to start working on them:

• Xamarin UITest GitHub sample

Once you open the solution, it should look like below:

There are three options to connect our tests to the existing apps (identifying them):

• Bundle (.app), ipa or .apk – we need to just paste the path to the package on the disc like below:
  • IApp app = ConfigureApp.iOS .AppBundle(“../../path/mybundle.app”);
  • IApp  app = ConfigureApp.iOS .AppBundle(“../../path/mybundle.apk”);

• Installed app name – we just need to define the package name of the installed app (on the emulator/simulator or on the device):
  • IApp app = ConfigureApp.iOS // or Android .InstalledApp(“com.xamarin.taskypro”);

• Defining project in the solution – in the Test panel you have to choose which application project you would like to test:

IApp is an interface to communicate with the currently tested application.

In this example, we will use the second option—we will pass the package name.

Configure Android Tests

First of all, you we need to set the name of the application’s package—in this case this will be:

• com.xamarin.example.creditcardvalidator

On Android, there is no special configuration to enable testing. Once you have the package name set correctly, you can launch the application on the emulator (yes, do it before tests). In my case, I am using Xamarin Android Player emulator (I advise you do the same; you can download it here).

The second important thing is that INTERNET permission has to be set—this is crucial for communication with the test server.

Once you complete all of the above, we can launch the application on the emulator—you have to always first run your app normally on the emulator to install it and then you can test it; if you do not install the app before tests, there an exception will show:

Now we can start writing UITests. The flow will be like below:

• Wait for the Activity to load
• Enter 15 digits for the credit card number
• Tap the Validate Credit Card button
• Confirm the Error Message

Open “AppInitializer” class. Paste the code below to configure app package:

As you can see, we have to set “Platform” parameter to indicate on which platform we want to test the app.

• Open “Tests” class and paste below code (with comments):

• Show “Unit Tests” and “Test Result” pads to see the result of our tests:

• On the right side of the Xamarin Studio, you should see “Tests” class for Android and iOS (because we precised two platforms earlier):

• Right click on the “Tests(Android)” and select “Run tests”:

REMEMBER that the app needs to be installed on the emulator before you start testing.

Once you run the test, the app should be opened on the emulator, and the REPL console should be visible like below:

It is worth to stop here for a while and say few words about REPL tool – Read Evaluate Print Loop.

With this tool user can type expressions and commands to test user interface. REPL will evaluate these expressions and return with the result. It allows us to explore the user interface and create the queries and statements so that the test may interact with the application. You can reed more uner REPL docs.

Now type “tree” in the command and see the result:

As you can see, the REPL returned all views for the current activity. It is really helpful especially when exploring the view hierarchy.

We can test our first command to flash the action bar:

We can take this code and paste it in the “Tests” class—this is very useful because you can use it either in the code or in the REPL.

• Add below method to the “Tests” class just below “AppLaunches()” method:

Result should look like below:

In the “Test Results” pad, you should see if the test passed or not:

Configure iOS tests

• To enable tests on iOS, you should add a NuGet package called Xamarin TestCloud Agent.

This package is required to configure the server responsible for running tests on iOS.

• Remember to add the below code to “AppInitializer” class (if you did not do it before). Your app package name should be exactly like below:

• Change the “CreditCardNumber_TooShort_DisplayErrorMessage” to look like below:

As you can see, we are going to test the iOS user interface now. The flow is the same as for Android:

• Wait for the UINavigationBar to load
• Enter 15 digits for the credit card number
• Tap the Validate Credit Card button
• Confirm the Error Message

You can test your app either on a physical device or on the simulator. To enable tests on the real device, go to “Settings” -> “Developer” -> enable “Enable UI Automation”:

This time we will use a simulator for tests. Before we launch tests, there is one more thing to do. Sometimes you can encounter problems with running tests on simulators. To solve a problem, please take the below steps:

• Open the “Tools” tab in Xamarin Studio and select “Instruments”
• Then choose “Automation”
• In the Instruments app, choose the launched simulator and the app you would like to test:

• Click the red “record” button to launch the app on the simulator
• Once it’s ready, you can close it and get back to the Xamarin Studio

Launch the tests from the “Tests Pad”:

Now you can see that the app is launched on the simulator and the REPL console is opened:

As you can see, UITests are performed and the result is displayed in the “Test Results” pad:

Of course, there are many ways to prepare an abstraction layer for tests to not duplicate test methods, but for this article, I decided that the most important place to start is to understand how it works in general.

If you are interested in broadening your knowledge about cross-platform tests (but not only), I recommend checking the “XTC101” and “XTC102” courses available at the Xamarin University.

So far, we’ve covered how to use UITests with Android and iOS applications. Now you know how to use the REPL tool and how to launch tests on the Android emulator and iOS simulator. In the third and last part, I will show how to deploy tests on the Xamarin Test Cloud.

Test Your App on Thousands of Devices

Before we start, it is worth it to mention that there are two ways to submit your application for tests:

• Using IDE—you can directly submit your tests via Visual Studio or Xamarin Studio
• Using the command line

What Are the Requirements?

Xamarin Test Cloud requires IDE Xamarin Studio 5.9 or Xamarin for Visual Studio 3.11.

You need also to add your email to the team and (I will explain further) and have the Team Api Key.

Firewall traffic should be possible for the below Test Cloud addresses (with ports either 80 or 443):

• 195.249.159.238
• 195.249.159.239

Last thing for Android apps: INTERNET permission has to be enabled in the app manifest.

Briefly About Test Cloud Structure

In Xamarin Test Cloud, users and applications are grouped into Organizations. One Organization can have many teams:

• Organization—coordinate teams, subscriptions, users, and applications. Each organization can have many users as administrators who are responsible for creating teams.
• Team—always, a team consists of users and at least one application used by these users. Each team always has the API Key used to associate an application and the users that are testing it (with their accounts in Test Cloud).

Three types of users:

• User—anyone registered with Xamarin Test Cloud. A user can be assigned to one or more teams. A user can also belong to many organizations.
• Team Managers—users with “Manage Permissions”—it is possible for them to add or remove users from the team. Each team has always at least one manager.
• Administrator—this is the highest level in the hierarchy. Admins are responsible for organizations—deleting, adding, and modifying teams is their role. An organization has always at least one Administrator. Admin has rights to:
  • Add or invite new users
  • Add more administrators
  • Create more teams

Configure Organization

Each new Xamarin Test Cloud account has automatically one Team with one User created who is both—Team Manager and Administrator.

Explore Xamarin Test Cloud Portal

• Sign in to Test Cloud
• Click your account in the right top corner and select “Account settings”

As you can see, there is one team created, and you are included in it.

Please note two options:

• New test run
• Show API Key

These are crucial to enable testing through the Test Cloud.

If you click on “member” dropdown you will be able to add users to your team:

It is possible to rename a tea

m, of course:

Submit Your Tests to Xamarin Test Cloud

Once you have your application ready for tests, you can submit it to the Test Cloud via Visual Studio or Xamarin Studio.

IDE rebuilds the application and the tests and uploads the binaries, placing them in a queue for execution on Test Cloud.

We will use the sample from the previous article—“CreditCardValidator.”

Requirements:

• When submitting an Android project, you have to set the build configuration to “Release” and the project as a startup:

• When submitting an iOS project, you have to set the build configuration to “Debug” and the project as a startup:

• Right click on the UITests in “Tests” pad and select “Run in Test Cloud”:

There are two possible options. The first one is connected with choosing an app package for tests. In this case, we will use Android project—in the dialog find release .APK file in “Release” folder:

Click “Upload and Run.”

Here you have to know that NUnit NuGet package version 2.6.4 is required—all of the above are not yet ready.

The second option enables you to choose which project you would like to test directly. Right click on “Test Apps” on the right and select “Add App Project”:

Now with a right click on Tests you can choose the app project for upload:

Dialog will appear, and you should sign in with your account:

Now you can choose on which devices you would like to test your app:

You can also check the specification of each model—that is great because you are becoming familiar with different devices:

Once you select devices, you have to choose a series of the test (in our case it is master) and system language:

You can see that the app is being processed and tested:

Once tests are completed, you should be able to see the full report:

There is information about the number of tests (with results also) and devices on which these tests were launched.

During tests, it is possible to take screenshots:

There are also Device Logs and Test Failures so you can see exactly what happened and why the test failed.

In this case, I used test cases for iOS, so it is obvious that they were not passed (there is no UINavigationBar control):

That’s it!

I hope that I helped you understand possible options for testing using Xamarin Platform.

Now you know what are the frameworks, how to use the REPL console, and how to write tests and deploy them to Test Cloud.

Unfortunately, I could not include everything (different cases), but I hope that this series will encourage you to broaden your knowledge about UI Tests.

This article originally appeared on my blog.