Mozilla Mobile QA (Firefox for Android) and AppThwack

Here at Mozilla, we’re always looking and investing in different ways that our mobile related testing efforts and strategies can evolve and expand into new avenues as Firefox for Android mature as a product. We’re always engaged in seeking out different yet worthwhile services and tools that can integrate into our regular QA process.

Whilst we continue to operate as a small team (Mobile QA for Firefox on Android is about two-full time employees); we’re seeking all the help we can get whether it be from community or from those who have created services and utilities to assist mobile QA efforts.

We react in seeking solutions by continuing to identify and respond to problems. In identifying our areas of contention, we still see Android device fragmentation and access on-demand to Android devices as major paint-points of Android testing. To administer any type of soluton would involve boasting and utilizing the services and assistance from others. Currently we use DeviceAnywhere for one-off direct device testing; but we desired a service that targets automation.  

A couple months ago we stumbled upon AppThwack, a service that manages mobile application and fragmentation issues by deploying uploaded Android Package (APK) files and deploying (installation) on real Android based phones and tablets. The devices are hosted and interfaced through web front-end that makes it easy to view and extract results ran from automated tests (currently self-written and provided MonkeyRunner tests are available, and on-track towards deploying Robotium and possibly in the near future Mozilla based Robocop tests). Alongside, screen-shots are provided from multiple device orientations which make it usful for identifying any user-interface issues. A test-run takes a couple minutes and any data collected may be bundled and downloaded together to deploy towards any new Bugzilla bug for track-keeping.

Currently, we are utilizing their service for testing Firefox for Android builds during our rapid release Beta cycle and are looking at different strategies of integration for the other channels within our cycle.

If you would like to see what a test-run looks like click here.

Site Compatibility Testing for Firefox for Android

The Mozilla Developer Network has recently accepted an excellent informational article written by Jason Smith, Lawrence Mandel, and Janet Swisher essentially entailing the largely problematic issue of browser and content compatibility with the mobile web. If you are interested in site testing for Firefox for Android, please give the article a read; and help us improve the web. Let us know if you have any questions about compatibility testing or want to become involved in Mozilla’s mobile web compatibility effort, please sign up for the compatibility@lists.mozilla.org mailing list.

https://developer.mozilla.org/en-US/docs/Mozilla/Firefox_for_Android/Compatibility_Testing

Debug preference for testing H.264/MPEG-4 AVC in Firefox for Android

Nightly for Android has recently adapted a new about:config preference for using and more-so testing Android H.264/MPEG-4 AVC decoding. The preference is media.stagefright.omxcodec.flags

Utimately, this preference allows one to override whether Firefox uses Android’s H.264/MPEG-4 AVC software or hardware decoder. As Chris Peterson mentions in the same announcement, this is useful for debugging different configurations and conveinent for testing.

Flags values to note:

 

  • 0 – Android will use either hardware or software decoding (default)
  • 8 – Force software decoding
  • 16 – Force hardware decoding

 

Reference: OMXCodec::CreationFlags

Discover and Help Test Web Apps in Firefox for Android on Friday August 24th, 2012

This upcoming Friday, we would love for you to solicit your support towards testing and contributing to the process of qualifying a new alpha-release version of Firefox for Android on the Aurora channel as part of our rapid release testing cycle. We will be focusing on the process of discovering, installing, managing and using web applications inside of Firefox for Android.

This event is open to all those interested: newcomers, experienced testers, developers, and anyone interested in testing and using Firefox on Android.

We would like for you to use the new version of Firefox on your Android phone, and take a close look at the latest Aurora product in order to assist us in identifying any noticeably major issues found, and ensure that all feature functionality that is included in this upcoming release is on its way to a feature and testing complete state with the added emphasis on discovering and using web applications.

To cover the work throughout the whole day we have created an EtherPad test-plan. Please feel free to read and use it as a companion during the testday event. There will be moderators at hand in the IRC channel: #testday, to answer any questions you have.

Together with your help we want to make this testday a success and ensure the high quality of Firefox for Android for mobile devices for all of our users world-wide. If you have time on Friday August 24th, 2012, please join us on IRC, we will have Mozilla community and testers on hand to help answer any of your questions.

Your testing and feedback is highly valuable, and we hope to see you attend our test day event.

Firefox for Android – Website Compatibility Testing

Firefox for Android Testers,

Website compatibility testing will be a covered and focused area as part of the release testing process for Firefox for Android 15.0, and we want you to help us during our testing cycle.

We want you to use Firefox on Android (on the Aurora) channel to assist us in identifying any noticeably major issues found in websites. Issues can range from pages that do not look right due to layout and positioning problems to interactivity and font related issues. We are also looking for testing around top-sites to identify areas where content served to Firefox differs from other mobile browsers.

How you can help us?

Bringing a better Firefox to Android will guide us all towards fulfilling Mozilla’s mission of providing access to one Web that everyone can access, regardless of device or location.

 

Android MonkeyRunner Automation

A few folks have asked me what’s involved with my simple automation script that ultimately produces screenshots of running browsers on Android. It’s rather uncomplicated, I am using the monkeyrunner tool that is bundled with the Android SDK. As the Android Developers guide indicates:

The monkeyrunner tool provides an API for writing programs that control an Android device or emulator from outside of Android code. With monkeyrunner, you can write a Python program that installs an Android application or test package, runs it, sends keystrokes to it, takes screenshots of its user interface, and stores screenshots on the workstation. The monkeyrunner tool is primarily designed to test applications and devices at the functional/framework level and for running unit test suites, but you are free to use it for other purposes.

Intended mainly as a tool for synthesizing input on Android, one can also easily get a capture of the Android screen with its Python/Jython API. The code below listens and connects to an Android device, loops through a list of provided Android browser package/activities, visits a site (provided by a CSV), waits a couple seconds for page-load (Hack! How can I go about better doing this?), snaps a picture and writes it out to a directory provided by argument.

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice
import sys, csv

browsers = ['com.android.chrome/.Main',
                'org.mozilla.fennec/.App']

# Connects to the current device, returning a MonkeyDevice object
device = MonkeyRunner.waitForConnection()

#Start each browser activity with the provided URL
for run, browser in enumerate(browsers):

    # Visit each site
    for visit, site in enumerate(csv.reader(open(sys.argv[1]).readlines())):

        # Start the activity with the provided site
        device.startActivity(component=browser, uri=site[0])

        # Wait for page load timeout
MonkeyRunner.sleep(20)
        # Snap a screenshot of the running activity
        device.takeSnapshot().writeToFile("%s%s-%s.png" % (sys.argv[2], visit, run), 'png')

Call for Help: Testing Firefox on your Android Device

As part of Mozilla Mobile QA’s release criteria for continually testing Firefox on Android, we want to again extend our efforts in leveraging the community for assistance in areas of device.

We want you to test and provide feedback in regards to how Firefox operates on your Android device. 

A problematic challenge we face is the ability to truly test and qualify the ever increasing and vast assortment of Android based devices (phone, tablet and soon to be more) out there in the wild;  the nature of which the carriers and combination of an open platform presents us with. With so many devices available, it is difficult to to ultimately attest and standby the truth in acknowledging if a device will be truly compatible with Firefox or not based on a range of criteria. Compatibility on each device can also very well be a subjective opinion based on overall experience, fulfillment and value. As each device has different hardware and different processes and configurations, it is challenging to produce an appropriate label and or level of compatibility for a device. This is where we want your help.

To gather information from community based testing, we have created a simple survey that can assist in defining and painting the overall picture of a device at hand. We currently have a backlog of untested devices that we not have access to, and we could use your help in testing!

How to Help

  • First, we would like for you to download the new Firefox on Android (currently on the Nightly channel) available here and install it onto your Android device
  • Complete the survey available here: http://bit.ly/firefox-android-survey

  • Create a Bugzilla account in our bug tracker so that you can file bugs you find against Firefox on Android in Bugzilla. Reminder, effective bug reports are likely to be fixed. There are excellent bug writing guidelines for your pleasure to read here. Detail the issues you find to significantly increase the chances that a developer can act on the bug thus increasing the likelihood of positive resolution — provide in-browser Android screenshots, website URL, browser build information, and steps to reproduce the issues
Join and chat with us on IRC on the server irc.mozilla.org in the rooms #mobile and #qa to talk with Mobile QA and mobile developers.

A few minutes of your time in completing the survey will be largely valuable, and you will be contributing towards the testing efforts of Firefox.

Follow

Get every new post delivered to your Inbox.