Synthetic Transaction Monitoring for Office 365

Welcome to the first post from our blog series covering digital experience monitoring, or DEM, for Office 365.

People who have seen me present at conferences or webinars are likely aware that I have been quite a fan of synthetic transactions or user simulations in the past. However, as both Office 365, and the companies using Office 365 have matured, so have my views. As such simulations and synthetic transaction monitoring (STM) make the perfect starter topic.

Let’s jump right in!

Synthetic transaction monitoring uses artificial or simulated connections to Office 365. (Disclaimer: You will find that I use synthetic transaction monitoring and simulations as interchangeable terms, even though industry analysts may separate these terms slightly). The simulations are most commonly executed from a dedicated physical device or virtual machine that has the vendor’s product installed. Such simulation applications go by many names – probes, robots, intelligent agents, (insert any missing marketing term here). Regardless of any applied marketing fluff, they all serve roughly the same purpose. STM products are designed to programmatically and repeatably run tests against Office 365.

A few years ago I was doing migrations to Microsoft’s early cloud, before it was named Office 365. At this time monitoring using user simulations was literally the only tool I needed. Why? The cloud was new, and fear was high. (Not to mention things could go wrong in the early cloud.) Being able to prove to a customer that BPOS (The Business Productivity Online Suite) was operating as expected met the primary need for these nervous trailblazers. Monitoring using simulations was a perfect fit back then…but BPOS has been gone for a while.

The upgrade for BPOS was given the name Office 365. And Office 365 went into beta in 2010, entering production in 2011. In other words, synthetic transaction monitoring and simulated user activity was the most valuable methodology nearly a decade ago. Think about how much has changed at Microsoft and in their offerings in the last decade. I write all of our product code, well the parts I do code lol, using my Mac. And Microsoft makes all of the software solutions that I use on my Mac. By comparison, when I attended my MCM/MCSM (Microsoft Certified Master / Microsoft Certified Solutions Master) rotation in Redmond, I thought the Windows Police might escort me out of the building for carrying a Mac. Times have really changed. And the needs of companies using Office 365 have changed too.

Synthetic transactions no longer deliver enough value to be the primary digital experience monitoring tool when it comes to Office 365.

Some vendors will challenge my opinion. They continue to double down on old concepts, claiming that the accuracy and advanced capabilities of their synthetic probes provide the most important detail needed to support your entire Office 365 user population. We will come back to accuracy claims in a bit. For now let’s evaluate how well such a claim lines up with reality. Indulge me for a moment and let’s look at my well crafted fable:

An employee named Juliette is having a “bad user experience” with Office 365 today. She asks a more-technical coworker, Romeo, if everything is working properly for him. Romeo doesn’t see any issues. But as fate would have it, Romeo is hopelessly in love with Juliette. And he does not like to see her sad over her user experience. Romeo tells Juliette something is obviously wrong with her account and he encourages Juliette to call the help desk. Romeo is the most tech savvy person in the Late Renaissance history department and she finds his knowledge Enlightening. Juliette contacts the help desk to report a problem with her perceived user experience in Office 365, a problem she now believes to be in some way specific to her.

Can you help Juliette when she calls ? Or more generally does your Office 365 user experience monitoring product give you the ability to support individual end users through actionable data on their personal user experience?

Does your Office 365 user experience monitoring product give you the ability to support individual end users through actionable data on their personal user experience? This question is the litmus test for any digital experience monitoring solution in 2020.

My fable is designed to highlight the fact that user experience is a combination of true technical issues and non-technical (human) issues. (Possibly created when one coworker inadvertently leads another down the wrong path. It happens, a lot.) You must be prepared to support users with real technical issues that exist in Office 365, in their network or only present on their devices. But you also need to be prepared to help users get back to the correct path when the issue is human in nature. Regardless of the fable I, or any vendor, prepares, the user and their devices are key components of their user experience. Fact or fiction, your monitoring strategy cannot ignore the fact these components now take the priority.

Synthetic transactions can still bring value

Specialized products such as dedicated unified communications products bring tremendous value. They are not what I think of as a simulation or synthetic transaction vendor. They are specialized monitoring vendors using simulations as just one part of their solution, to expand their value. How about products that do not provide simulations as part of their offering? Should you purchase another product?

A clean view from a probe can provide “nice to have” secondary information. And I still like synthetic transaction monitoring’s ability to help provide an early indication of an issue if no user devices are in the office. It’s the the ole 4:00 (AM) on a Sunday scenario. It doesn’t matter if it’s advanced, you just need something watching to get you out of bed before it’s too late. It’s rudimentary, but it definitely has some value. It’s a pretty common use case, one that reminds me of how canaries were used to provide early warnings many years ago.

Interesting side note – canaries were in use in mines until 1986. That feels surprisingly recent. Here’s the article from the Smithsonian if you are interested.

Keeping with that theme we built a “canary mode” into our product that allows IT to perform user independent synthetic transactions from VMs or small hardware. It meets that primal need for safety while you sleep. How much value does such a product deliver? That’s a hard question. Whether speaking of money or labor overhead, determining acceptable value has a lot to do with your larger need. While we built a canary mode to give the around the clock peace of mind for the truly rare outage, we feel that synthetic transactions running on a dedicated computer should not cost our customers anything additional.

Why are we making it free? Any DEM product should include an early warning ability out of the box. I see it like anti-lock brakes on a passenger vehicle. In the 1950s this was a high-flying feature, literally. Anti-lock brakes (ABS) moved from airplanes only to non-flying vehicles in the 1950s. Of course having airplane technology in your car commanded a premium price. But ABS became a baseline feature in the 1990s. It’s part of every vehicle. You might not even notice it was part of the car…unless a manufacturer tries to charge you. That’s negative attention that makes no sense.

Extending my analogy above I will tell you that you can purchase upgraded anti-locking braking kits for racing. Highly specialized, highly accurate and therefore it costs money. Is this a worthwhile investment? On the family minivan, probably not. On the Porsche you take to the track every weekend, maybe. But anyone’s first question will be how much better are these? So Let’s go to that accuracy topic

Synthetic Transaction Accuracy

There are some technologically great simulation products in the market, and there are some technologically poor ones. My primary advice: Do your research, ask questions, and see everything working before you decide on the value a product brings. If you are going to pay a premium for software or technology, be certain that it can do what it claims.

For example, more than one simulation focused product claims to be highly in tune with your user behavior. Therefore performing the best simulations possible. How did they become so attuned to user experience (while simultaneously not monitoring a single user)? Ask. I can tell you that software vendors shouldn’t mind. Products I built a few years ago have features that I built in direct to questions I received. Perform your due diligence, and you’ll quickly spot the vendors that are not up to their claims. A few questions that come to mind:

  • Simulations run code with no one around. The security of the vendor matters. Ask to see their SDLC (software development lifecycle) policy. Ask for the reports from their code vulnerability scanning tools.
  • Ask for detail on their update and release cadence, including detail on how you will be notified for security issues – minor and major.
  • I have had to rebuild more than one Exchange environment due to assumptions and bad claims about the actual user patterns. So back to my previous example of the amazingly tuned software: How does the product gather the metrics used to so precisely tune the software?
  • If a product uses Outlook or OneDrive or Teams, etc., be certain that you are provided exact versions of the client applications in use. You may need to calculate any variance from versions being used in your company to properly use the data.
  • You may also need to apply security patches. So you must identify how to install, update and/or upgrade these apps.
  • And Since we are speaking Office 365 be certain they are actually using the client software.

To expand on the last point, Office 365 has a lot of APIs to offer customers extensibility. APIs make it easy to work around using the real program. Even if a client application such as Outlook uses an API for some tasks, API-only simulations produce results that are drastically different from the real application. If a simulation based monitoring product for Office 365 is not using actual client applications, your job of evaluating the value is quite difficult. Why? More than any other platform, Office 365 users have a heavy dependency on local applications. If a specialized simulation product is not using the actual applications, not monitoring the user devices and not monitoring the users, what is left? Can you ignore all of these components and justify a cost as an add-on product? I suppose such a product would need to deliver a fair amount of non-Office 365 value to justify such a purchase.

But if it is value proposition is non-Office 365 value, it’s not really a digital experience monitoring product for Office 365 now is it?

Leave a Reply

Your email address will not be published. Required fields are marked *