CTSTestProcedure

Compatibility Test Suite

I have seen that people have many doubts over CTS test like-

What is a CTS test?

What does CTS stand for in CTS test?

Do we need to run all test cases once or we can conduct testing one by one?

How to handle CTS test failures?

How to setup my machine for CTS?

How to setup my device for CTS test?

Which system should I use to do CTS test?

The purpose of this post is to give users a good idea of what CTS test is all about and make conducting CTS testing easier and to reduce the failure while doing it( obviously you need to have a good code to pass the test :D) and how to overcome it.

• CTS is an automated testing harness that tests the proper working  all the APIs provided by google before the model is google certified.
• There are two softwares involved in the test, one runs on our Linux PC and other on the mobile device.

• One can run a CTS test on an emulator or on device.
• CTS test can be run for all the APIs together (more than 18k) or for individual test cases.

• After the test/s is/are done a report is generated stating whether the test has passed or failed and also a log is generated which can be used to analyse the reason of failure, if it occurs.

REQUIREMENTS TO PERFORM CTS TESTING

• Linux PC
• Mobile device/emulator
• adb(To install adb, download Android SDK tools , and set up an existing IDE:

http://developer.android.com/sdk/index.html#ExistingIDE
http://developer.android.com/sdk/installing/index.html)

• CTS Verifier.apk for the version of Android under test:

http://source.android.com/compatibility/downloads.html

• user build binary in the device
• in developer option click stay awake option
• connection to a wifi network is recommended ( i guess that's for wifi testing).

PROCEDURE for CTS testing

• Download the CTS and CTS media files.
• Connect device (or emulator) to your machine
•  Ensure 'adb' is in your system path( host machine)
• export PATH=$PATH:/home/myuser/android­sdk­linux_x86/platform­tool
• For CTS 2.1 R2 and beyond, setup your device (or emulator) to run the accessibility tests:
  adb install -r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk
• On the device, enable Settings > Accessibility > Accessibility > Delegating Accessibility Service
• For CTS 2.3 R4 and beyond, setup your device to run the device administration tests:
  adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk
• Install CTS Verifier.apk to the Device Under Test (DUT)

          adb install ­r CTS Verifier.apk

 

• On the device, enable all the android.deviceadmin.cts.* device administrators under Settings > Location & security > Select device administrators
• .Make sure the device is at the home screen at the start of CTS (by pressing the homebutton).
• While a device is running tests, it must not be used for any other tasks and must bekept in a stationary position (to avoid triggering sensor activity).
• Do not press any keys on the device while CTS is running. Pressing keys or touchingthe screen of a test device will interfere with the running tests and may lead to testfailures.
• For CTS 2.3 R12 and beyond, the CTS media files must be copied to the device's external storage. Check section 4.2 of the latest CTS manual for further details on copying these files:
• Download the android-cts-media-X.Y.zip file from    http://source.android.com/compatibility/downloads.html and unzip it.
• Connect the device to the computer and check that adb can connect to it.
• Navigate (cd) to the unzipped folder.
• Use ‘chmod’ to change the file permissions like so:chmod 544 copy_media.sh

Run copy_media.sh like so:

○ To copy clips for just the default resolutions, run:

./copy_media.sh

○ To copy clips up to a resolution of 720x480, run:

./copy_media.sh 720x480

○ If you are not sure about the maximum resolution, try 1920x1080 so that all

files are copied.

○ If there are multiple devices under adb, add the -s (serial) option to the end.

For example, to copy up to 720x480 to the device with serial 1234567, run:

./copy_media.sh 720x480 ­s 1234567

•  Launch the CTS console byrunning the cts-tradefed script from the folder where the CTS package has been unzipped, e.g. $ ./android­cts/tools/cts­tradefed.
• You may start the default test plan (containing all of the test packages) by typing run cts --plan CTS. This will kick off all the CTS tests required for compatibility.
• Alternately, you can just run a CTS plan from the command line using cts-tradefed run cts --plan <plan_name>   eg. run cts --class abc --method xyz

• Once all the tests are executed, you can view the test results in your browser and use the results to adjust your design(use debugging by giving logs).
• When you are ready, you can submit the report generated by the CTS to cts@android.com. The report is a .zip archived file that contains XML results and supplemental information such as screen captures.

following test plans are available:

1. CTS - all tests and will run ~18,000 tests on your device. These tests are required for

compatibility. At this point performance tests are not part of this plan (this will

change for future CTS releases).

2. Signature - the signature verification of all public APIs

3. Android - tests for the android APIs

4. Java - tests for the Java core library

5. VM - tests for the Dalvik virtual machine

6. Performance - performance tests for your implementation.

These can be executed with the run cts command as mentioned earlier

Reference- https://source.android.com/compatibility/cts-intro.html,
 http://source.android.com/compatibility/cts-development.html, http://static.googleusercontent.com/media/source.android.com/en//compatibility/android-cts-manual.pdf