Getting Ready for Apple's Next OS Updates: How to Create a Test Plan

Last week, we talked about why it’s important to test prerelease operating systems. The reasoning is straightforward: You need to test new operating systems before deploying them widely—ideally before they’re officially released—to see how they get along with your existing technical infrastructure. We also talked recently about where and how to download beta OSes.

But once you’ve downloaded the software, what’s next? What’s the best way to test a prerelease OS? Of course, the specific answers depend entirely on your particular circumstances—the kinds and quantity of Apple hardware you manage, the apps and services your organization uses, how much time you have, and so on. But here are some general guidelines for thinking about testing macOS Monterey and iOS/iPadOS 15 now.

Apple’s Own Test Plans

One of the best resources for beta testers is Apple itself. The company supplies sample test plans as part of its AppleSeed for IT program. These plans cover a wide range of topics, including Microsoft Exchange, managed software updates, data separation, and more. They're also incredibly thorough.

So, for example, this year Apple is providing a sample plan for testing macOS on Apple silicon. It covers testing of your MDM solution, your network infrastructure, and apps, and it steps you through a variety of test cases.

You will likely find items in Apple’s plans you’ve never even considered testing. Even if you don’t need to test the specific scenarios or conditions covered by Apple’s test plans, they’re excellent templates to use in planning tests of your own.

Developing a Test Plan of Your Own

If you can't find an Apple plan for your needs—particularly the third-party apps that matter most to your organization—you’ll need to develop one of your own. As we said, we aren't going to provide specific test plans, since every app and environment is different, but here are some general approaches we’d recommend.

1) Use Multiple Apple Platforms

Before you do any testing, you need to settle on a test platform—or, more accurately, platforms. Our recommendation: Do it in stages.

Start by installing the new OS on a clean, wiped Mac, iPhone, or iPad. These systems should be dedicated to testing, not actually in production use. As much as possible, configure the test platform as you would any new production computer or device, with your standard Wi-Fi and other settings in place, then run it through your suite of tests.

That done, try upgrading a test system that’s currently running the last generation OS: from macOS Big Sur to macOS Monterey on the Mac, from iOS/iPadOS 14 to 15 on iPhone or iPad. Once again, configure these devices as you would a production system, then run through your test suite.

You might think to start your testing on a virtual machine, but be warned: Because of the complications of VMs, they have their limitations as testing platforms and may return false positives and negatives. They can be an OK place to start, but they do not deliver conclusive results. 

2) Start with the OS Itself

For each platform, you want to take a tiered approach, starting with the new operating system itself. Does it install? Does it work? Can you connect to local networks (wired and wireless) and the internet? Does Bluetooth work?

If the new OS doesn’t work on a clean Mac, there might be something broken on the Apple side. Check the release notes on AppleSeed for IT to see if there are known issues that might explain the problem and provide feedback (see below).

3) Test Your Apps

Assuming you can get the beta operating system to install and work, turn next to applications. What are your five most important user-facing tools? Focus on productivity apps—Microsoft Office or the Adobe Creative suite, for example—on which your organization relies. Avoid email and calendaring for now, as they depend on back-end infrastructure that you’ll test next. 

Install these apps on your test hardware and run them through their paces. Does the app install and launch under the new OS? Does it provide the functionality you expect? If any of those answers are no, contact the developer of the app in question and let them know. File a report with Apple, as well.

4) Check Your Infrastructure

Next, test the more complex apps and services that your organization relies on. 

Can the test system connect successfully to your email services? Make sure Exchange functions as expected, if that’s what you use. What about other cloud services, such as your VPN and your identity provider? Do your security tools work? (Security apps in particular frequently break with new operating systems.)

One special component of that infrastructure that you should test: Your MDM solution. Does the test device with the beta OS enroll correctly? Try deploying a current configuration. Does it all work? Next, can you use your MDM to deploy the beta OS to the test hardware? Again, try it first on a freshly wiped system, then one that you’re upgrading from a previous OS.

How Much Testing Should You Do?

Finally, there’s the question of how extensive your testing should be. Not surprisingly, that depends entirely on your available time and resources. 

If your time or resources are limited, keep your plan simple: Test the new OS to see whether it installs correctly on your hardware. (If you have a mix of Intel-based and Apple silicon Mac computers, test the new OS on both.) See if you can create a new user and enroll in your MDM solution. If you can take care of those basics, that might be enough for now.

We can’t stress enough how important it is to do as much testing as you can before you roll out a new OS to your users. But if you absolutely don’t have the time or resources for even the most minimal testing program, the fallback is to wait for the OS to be released to the public. When it is, install it on one machine, and see what (if anything) breaks. In the meantime, keep an eye on public forums (such as the MacAdmins Slack channel) to see what your peers are saying. 

Providing Feedback to Apple

While the main goal of prerelease testing is to check the forthcoming operating systems against your existing hardware and software, there’s another crucial goal: Providing feedback to Apple, app developers, and service providers. For the first of those, your best bet is AppleSeed for IT. 

When to submit feedback? Whenever an app or service you’ve used in the past doesn’t work or behaves oddly—crashing, hanging, or being so slow as to be unusable—under the new OS.  

Submit that feedback either through the Feedback Assistant app (on Mac or iOS/iPadOS) or on the AppleSeed site. Make sure you log in using the same Apple ID you used to enroll in AppleSeed for IT. 

When you file feedback, it's important to make it useful. Don't just say, "It doesn't work". The more information you can provide, the more likely your bug will get fixed fast and won’t require a lot of follow-up. At a minimum, include a detailed, descriptive title when filing the report, following the rubrics “[[service or app]] [[is doing this weird thing]]” or “[[can’t do this]] in [[service or app]].” Screenshots are also really helpful.

Whichever way you submit feedback, it will be routed to a specific AppleSeed for IT queue. That means your bug submissions and enhancement requests will go to people at Apple who can actually do something about them. 

If you do submit an issue, note the Feedback ID number you receive; it will be used in any follow-up between you and Apple. You can review the feedback you’ve filed or drafted in Feedback Assistant and on the AppleSeed website.

About Kandji

By testing betas and leveraging an MDM solution, you can make sure the transition is smooth. Here at Kandji, we’re making sure our MDM will be ready on release day so you can get the most out of your fleet of Apple devices. With powerful features like zero-touch deployment, one-click compliance, and offline remediation, Kandji has everything you need to enroll, configure, and secure your devices.