As you have probably heard, Apple has announced the closure of the TestFlight website and is recommending iOS developers move beta testing builds to the new native TestFlight service available in iTunes Connect and the App Store.
There have been a lot of opinionated debates within the community about this migration and as a result it has left a lot of people confused and unsure of what to think and do.
We’ve written this as a means to help other developers who may be in a similar situation. We hope this will help alleviate some confusion and better inform developers about the changes to TestFlight. This article touches upon some pros and cons of the change, shows examples of how the community is handling the change, and offers some tips on migrating services. We’ve done a brief Q&A with a few developers and designers who have already been using the new TestFlight as a means to beta test their apps. You’ll get an exclusive view into their thoughts and opinions on TestFlight and the overall process.
Pros and Cons
At a high level the following looks at the pros and cons of the new TestFlight Beta Testing service. These have been pulled from interviews with other developers, searching Twitter for opinions as well as some in house thoughts and final judgements.
Pros:
- No longer need to add tester’s device UDIDs to provisioning profiles.
- Easily invite testers via their Apple ID.
- No longer limited to 100 devices. (Including devices used for internal development and testing.)
- Avoid discrepancies between release candidate test builds and final App Store builds by testing the same build that you later release.
- In-app purchases are free in beta apps, and don’t require manual setup and maintenance of iTunes test users.
Cons:
- Apple TestFlight is only available on iOS 8+, leaving out the 25% still running iOS 7.
- Apps go through a “beta review” before they’re available to external testers.
- Regardless of whether or not an external or internal build has been uploaded to iTunes Connect, there is some processing time (hours in our experience) before a build is available.
Competitor Features
Other beta testing competitors such as Microsoft’s HockeyApp or Crashlytics Beta offer features that Apple TestFlight doesn’t currently support. One of the big things aggravating the community is the lack of crash reporting and analytics. iTunes Connect has supported crash logs for quite some time but other beta testing services offer a better integration of crash reporting. Crashlytics for example is an extremely powerful yet lightweight crash reporting system that creates very detailed real-time reports of your apps status. HockeyApp does a great job at consolidating cross platform apps into a single online account, great for developers with a diverse portfolio of apps on multiple platforms.
Community Opinions
There has been lots of debate within the community recently about the changes to TestFlight. Here are just a few examples of how other developers and app makers are dealing with the changes to TestFlight.
Negative
@testflightapp Going to miss you guys! TestFlight was much easier before Apple changed it! Now all us Enterprise devs are out of luck! :'(
— Chris Emerson (@emerson_chris) February 19, 2015
Apple closing #TestFlight February. Annoying. You have to have Apple binary review for external testers. Dislike. http://t.co/vWNc3nsSdz
— Sebastian Wehner (@redseb) January 26, 2015
Positive
Apple’s #TestFlight really is a great addition to iTunes Connect – finally things catch up behind the polished exterior! @testflightapp
— Paul Evans (@pedesign) February 17, 2015
So my first Apple TestFlight beta testing submission was approved in about 2hrs – so not the bottleneck I thought it might be.
— Graham Dawson (@gpdawson) February 14, 2015
Interviews
We also reached out to a few developers and designers who have recently used Apple TestFlight to distribute beta builds and here is what they had to say about the process.
Alexander Münch – Designer of Short, A Reading List for Busy People
“It’s easier to have an overview of the status of your app. Adding testers is very easy, and easy to navigate… Overall it was a good experience.”
When asked to give advice for designers and developers in regards to TestFlight Alex said “Do it. It’s the easiest way to invite beta testers. TestFlight is an Apple app so you just download it, sign in with your apple ID and send your ID to the app maker. They invite you by pasting in your email to iTunes Connect, and you’re all set. Super convenient.”
Alex said he’s previously used HockeyApp but switched to TestFlight once it was acquired by Apple due to the simplicity of the native process.
Jake Peters – Lead Developer at Ehko
“It enables rapid iteration and testing with hundreds of users. Now we can push out different updates twice daily if we want and people can update with no hassle.”
Jake agrees with us that it’s irritating having to wait for approval on the first build of each version making rapid prototyping a little difficult.
“A massive problem we’ve encountered is that invites can only be accepted in the iOS Mail app, not the Gmail app, so some of our users haven’t been able to install.”
When asked to give advice for designers and developers in regards to TestFlight Jake said
“Do it! Easy, and a great way to get people using your app. Invites send out quickly and it’s super quick to add new testers. Now you can group them too, which is a big plus.”
Jake and Ehko plan on using TestFlight for all future iOS beta testing. And when asked if other tools to distribute beta builds have been used in the past Jake said “I’ve used the original TestFlight (pre-Apple) and also just manually distributed with Ad-Hoc certificates. Code signing is a hassle, and getting people’s UDIDs for ad-hoc distribution is even worse. It’s definitely easier with Apple’s version.
Jaison Vieira – Developer at Senior Solution
Jason confirms the simplicity of the TestFlight process we highlight below, and says “The only problem is when you send builds to external beta testers it needs an approval from Apple’s engineers. The first approval took about 4 hours but the following builds were very instantly approved.” When asked if Jaison was planning on using Apple TestFlight in the future for other projects he said the plan is to use it on all iOS projects moving forward.
TestFlight Features
While TestFlight is still fairly limited in its integration with iTunes Connect, there are some pretty amazing features that Apple is giving us for free.
Groups
It’s now possible to organize testers into groups to send different builds. Groups also allow developers to provide separate instructions on where to focus, which is great for crowdsourced QA. * Groups were available on the old TestFlight and it’s nice to see Apple quickly re-implementing the functionality.
1,000 Testers
With the 100 tester limitations no longer an issue, developers will no longer need to worry about cleaning up old registered devices no longer being used in Apple Developer. Up to 1,000 testers can test up to 10 apps.
Internal Testers
Beta builds can be shared with up to 25 internal testers. Each tester can install the build on up to 10 devices.
External Testers
It’s possible to invite up to 1,000 external testers outside of your development team to test an app intended for public release. The 10 device install limit also exists here.
Test Information
When submitting the beta information in iTunes Connect, it’s possible to upload instructions to inform testers where to focus their attention. *The old TestFlight and Crashlytics Beta also have notes sections that are presented to testers when they install.
Simple Registration
There is no need to worry about managing UDIDs for testers. All you need to send a test build to a tester is their Apple ID.
Importing Contacts
Already have a large database of beta testers contact info? No worries, you can import a CSV file into iTunes Connect to populate an app with testers very quickly. *The old TestFlight and Crashlytics Beta also have a CSV upload option.
No Additional Cost
The best feature is the lack of price tag associated with Apple TestFlight Beta Testing. *The old TestFlight and Crashlytics Beta also have free beta testing services.
Apple TestFlight Beta Testing offers lots of nice features right out of the gate, and it’s only going to get better.
The Simplicity of TestFlight Beta Testing
Let’s turn our attention to the process involved in integrating TestFlight Beta Testing into your development cycle. This is purely a high level overview to show you the simplicity that Apple is trying to achieve.
Once the build has been uploaded to iTunes Connect it’s possible to add testers after enabling TestFlight Beta Testing for the app. Once the build has been reviewed and approved by Apple’s Beta Review team any testers that you’ve added will receive an email with a link to download the beta build via the TestFlight iOS app. Each time you invite a tester, they will automatically receive an invitation email to join your group of beta testers and will be prompted to install the free TestFlight app from the App Store. Once a beta app is installed, TestFlight will notify testers each time a new build is available.
There you have it: The TestFlight Beta Testing flow in a nutshell.
Parting Thoughts
Ultimately, TestFlight as a native solution to beta testing looks very promising. Even though there has been lots of debate in the community about the efficacy of the solution, it seems to be a great option for non-enterprise related beta testing. The acquisition of Burstly (previous TestFlight owner) for Apple is going to take some time before the kinks are worked out and TestFlight runs as smoothly as everyone wants it to. There are a few features missing like in-depth crash reporting and analytics but it’s just a matter of time (hopefully) before we see these things. Apple has already begun adding new features to TestFlight as it has just introduced Groups which we covered briefly earlier in the article.
If you are a developer using TestFlight let us know what you think of the new TestFlight Beta Testing program.
Links to official Apple Documentation for further reading on TestFlight:
Tips and Tricks
While perusing through Twitter to better understand how other developers are handling the change, we came across some interesting handy tips and tricks we’d like to share. Remember, these are unofficial tools and continued support for them is not guaranteed. We are listing them here because they caught our eye and show the brilliance of the innovative minds in our industry.
Hacking TestFlight Invites
Brian Donohue, CEO of Instapaper, released this handy tutorial on creating a form that allows testers to join your beta program themselves. Here’s a link to the guide on Medium the codie is also available on GitHub. Thanks Brian!
BetaFill
BetaFill allows you to save “Beta Information” and fill out the Beta App Review Information with a single click. This is great because every time you submit a new beta for review there is a “Beta Information” section that needs to be filled out. Here’s a link to download the extension.
Deliver
Deliver is an unofficial command line tool to upload builds to Apple TestFlight since it doesn’t support continuous integration. The code is available on GitHub.
