Q&A: Andre Jay Meissner on the Importance of Mobile Device Testing

By James Knutila on Friday, March 21, 2014

Q&A: Andre Jay Meissner on the Importance of Mobile Device Testing
Andre Jay Meissner is a passionate advocate of the Open Device Lab movement. He’s also an in-demand conference speaker and a business development and developer relations specialist at Adobe. 

We had the chance to talk with him about how ODLs work, the importance of real mobile device testing, and how to participate in this brilliant development in quality assurance. 

What exactly is an Open Device Lab? How does it work?


Jay: Whenever you’re creating content that’s going to be consumed on an ever-growing variety of Internet-connected devices, you face the same problem: Do you own and manage an insane amount of devices, or do you settle for suboptimal quality assurance (or no QA at all)? In other words, you find that if you do not want to or cannot afford to amass a bunch of hardware, you are not doing your job right. Bummer! 

This is where Open Device Labs (#ODL) come to the rescue. As a grassroots community movement, ODLs collect a vast selection of devices and make them available for everybody to use, for free. Some choose to contribute. Everybody wins. It is as simple as that.

Testing on real devices can be time consuming. Why is it worth the effort?

Jay: Because it’s the only way to ensure that your stuff is performing as intended. Of course there are alternatives, such as remote testing (where devices are remote controlled and filmed with a camera, resulting in a movie and some telemetry data) and simulators or emulators. But these can never replace running a real-life test, on a real cell phone tower, in a real network, under real circumstances. 

Your users are not consuming your content in simulators or on remote services - so why should you assume that could do it for quality assurance?

I am pretty thankful that the car industry runs tests on real cars. These tests are important—they make sure that the designers’ work is free of flaws and that their ideas work well in practice as well as on paper. Developers should follow this example: You can begin by simulating and refine via remote testing, but you’ve got to do your final testing on real devices.

How should designers and developers come up with a list of essential devices to test on?

Jay: There is no standard list that would be valid for every project, in every region, in every situation. How do you actually define the test bed? This depends on many factors. For example:

  • If your project is a 3D first-person shooter app that’s limited to certain OS versions or hardware, you will probably not test against low-end feature phones.
  • Even if your customers are expected to be in a certain industry, as would be the case with any internal corporate application, you shouldn’t make too many assumptions about their usage—they might be using your product on unexpected devices.  
  • If your application is only intended to ship in a certain part of the world, you would probably need to test on completely different devices than the ones you’d use to test a web project, which is by definition a global project.

Leave your assumptions at the door. Use statistics and construct concrete requirements for your test bed. Think of your project’s universal accessibility as part of your contract with your customers. There’s more to consider than just OS and device models: Think about network status, latency and other factors as well. And expect to modify the test bed over time—testing should continue after deployment.

With so much fragmentation of devices, browsers, operating systems, etc., do you have any advice on creating websites that are accessible to all users?

Jay: Personally, I believe in the concept of progressive enhancement: Focus on creating a minimal version that truly works across all platforms; then add optional stuff. There are a number of good articles and talks on the web about this topic.

It’s worth noting that progressive enhancement is nothing new—it’s just a name for a basic idea that people can and should apply to their work. It is bad that by making assumptions about available browser or hardware support, by choosing to support only a limited selection of cutting-edge browsers, by focusing on the shiny stuff and forgetting about core functionality, content creators further this fragmentation. 

How can people find ODLs near them? How can they contribute?

Jay: Go to OpenDeviceLab.com. It’s a geolocation-enabled website that displays the ODLs nearest you. You’ll also find a lot of useful content, such as information about how to contribute to individual ODLs and the overall ODL project. 

Any advice for those not located near an ODL? What resources and tools do they need to start their own?

Jay: It's all listed on LabUp!’s resources page. If you find something missing, please let me know. And if you want to open an ODL and have specific questions, you can contact us via that same webpage.

What inspired you to get involved with Lab-Up.org and Open Device Labs?

Jay: My goal is to advance user experience on high-tech media, so when I first heard about ODLs, the idea fascinated me. I did some research and found it hard to locate ODLs on the web, so I created a list on my blog

From my work in developer relations, I knew some networking and lobbying would be needed to attract sponsors and promote the idea to a larger audience, so I started LabUp! to help establish and promote ODLs on a nonprofit basis. We created the first conference ODL and we run a quarterly virtual ODL at our meet-ups in Berlin.

Have device manufacturers shown interest in getting involved or donating devices?

Jay: A lot of manufacturers are interested in working with and supporting Open Device Labs. Manufacturers need to get their product in the field in order to cooperate with developers and ultimately advance their markets. Their developer relations teams are using OpenDeviceLab.com as a tool to check the status and popularity of certain ODLs (based on comments and ratings from ODL users). 

Investing in ODLs is a smart move for manufacturers, and a good addition to their usual focus on comparably expensive individual device seedings.

Have you seen a lot of interest and participation in ODLs from the developer community?

Jay: Oh, definitely! During the conferences I spoke at over the last 15 months, I communicated with hundreds of enthusiastic developers. Everybody who’s been presented with the idea has been impressed by its simplicity and logic, and many have expressed interest in using an ODL or contributing to the project.

Regardless, I believe we are far from seeing a broad awareness of the necessity of real-device testing and ODL availability, so I am thankful that Moboom is supporting the mission by running an ODL in San Francisco and helping to spread the word about the idea—thanks, guys!

There is also another player in this: the customer. You know, the people saying, "Why the big testing budget? Can't you just code it right?" In my 17 years as a software and web developer, I’ve had these discussions as well. We all need to work on solving that issue by proving the value of testing to our customers. I’m currently working on another project that will deal with this problem, so stay tuned!

Visit us in San Francisco


As you already know, we’re big proponents of mobile device testing, and we’re grateful to Jay and the other founders of the Open Device Lab movement for their work building and promoting these resources around the world. They’ve been a great inspiration to us as we set up our own ODL here in San Francisco

If you’re in the area, come by and check it out. If not, take a look at OpenDeviceLab.com and find a lab near you. These resources are tremendously valuable and absolutely free, and you can’t afford not to test your application on real devices.  

photo via Fabian Tempel

DevKit Templates

Find the website template you want in the Moboom Market