Digital Edition

SYS-CON.TV
Web Service Monitoring 101: Bad Deployments | @DevOpsSummit [#DevOps]
We recently moved some of our systems between two of our data centers – even moving some components to the public cloud

Web Service Monitoring 101: Identifying Bad Deployments

Have you ever deployed a change to production and thought "All went well - Systems are operating as expected!" but then you had to deal with users complaining that they keep running into errors?

When deployments fail you don't want your users to be the first to tell you about it: Sit down with the Business and Dev to define how and what to monitor

We recently moved some of our systems between two of our data centers - even moving some components to the public cloud. Everything was prepared well, system monitoring was set up and everyone gave the thumbs up to execute the move. Immediately following, our Operations dashboards continued to show green. Soon thereafter I received a complaint from a colleague who reported that he couldn't use one of the migrated services (our free dynaTrace AJAX Edition) anymore as the authentication web service seemed to fail. The questions we asked ourselves were:

  • Impact: Was this a problem related to his account only or did it impact more users?
  • Root Cause: What is the root cause and how was this problem introduced?
  • Alerting: Why don't our Ops monitoring dashboards show any failed web service calls?

It turned out that the problem was in fact

  • Caused by an outdated configuration file deployment
  • It only impacted employees whose accounts were handled by a different authentication back-end service
  • Didn't show up in Ops dashboards because the used SOAP Framework always return HTTP 200 transporting any success/failure information in the response body which doesn't show up in any web server log file

In this blog I give you a little more insight on how we triaged the problem and some best practices we derived from that incident in order to level-up technical implementations and production monitoring. Only if you monitor all your system components and correlate the results with deployment tasks will you be able to deploy with more confidence without disrupting your business.

Bad Monitoring: When Your End Users become your Alerting System
So - when I got a note from a colleague that he could no longer use dynaTrace AJAX Edition to analyze the web site performance of a particular web site, I launched my copy to verify this behavior. It failed with my credentials, which proved that it was not a local problem on my colleague's machine:

Business Problem: Our end users can't use our free product due to failing authentication service

Asking our Ops Team that manages and monitors these web services resulted in the following response:

"We do not see any errors on the Web Server nor do we have any reported availability problems on our authentication service. It's all green on our infrastructure dashboards as can be seen on the following screenshot:"

Infrastructure is all green: No HTTP-based errors or SLA problems based on IIS log or on any of the resources on the host

Web Server Log Monitoring Is Not Enough
As mentioned in the initial paragraph, it turned out that our SOAP Framework always returns HTTP 200 with the actual error in the response body. This is not an uncommon "Best (or worst) Practice" as you can see for instance on the following discussion on GitHub.

The problem with that approach though is that "traditional" operations monitoring based on web server log files will not detect any of these "logical/business" problems. As you don't want to wait until your users start complaining, it's time to level-up your monitoring approach. How can this be done? Those developing and those monitoring the system need to sit down and figure out a way how to monitor the usage of these services and need to talk with business to figure out which level of detail to report and alert on.

How can you find out if your current monitoring approach works? Start by looking more closely at problems reported by your users but that you don't get any automatic alerts on. Then, talk with engineers and see whether they use frameworks like mentioned here.

For further insight, and for lessons learned, click here for the full article.

About Andreas Grabner
Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

In order to post a comment you need to be registered and logged in.

Register | Sign-in

Reader Feedback: Page 1 of 1



ADS BY GOOGLE
Subscribe to the World's Most Powerful Newsletters

ADS BY GOOGLE

The graph represents a network of 1,329 Twitter users whose recent tweets contained "#DevOps", or wh...
For enterprises to maintain business competitiveness in the digital economy, IT modernization is req...
"Calligo is a cloud service provider with data privacy at the heart of what we do. We are a typical ...
Cloud-Native thinking and Serverless Computing are now the norm in financial services, manufacturing...
Public clouds dominate IT conversations but the next phase of cloud evolutions are "multi" hybrid cl...
Data center, on-premise, public-cloud, private-cloud, multi-cloud, hybrid-cloud, IoT, AI, edge, SaaS...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web...
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with exp...
Every organization is facing their own Digital Transformation as they attempt to stay ahead of the c...
Most modern computer languages embed a lot of metadata in their application. We show how this goldmi...
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the U...
Moving to Azure is the path to digital transformation, but not every journey is effective. Organizat...
Data center, on-premise, public-cloud, private-cloud, multi-cloud, hybrid-cloud, IoT, AI, edge, SaaS...
DevOps has long focused on reinventing the SDLC (e.g. with CI/CD, ARA, pipeline automation etc.), wh...
Atmosera delivers modern cloud services that maximize the advantages of cloud-based infrastructures....
Intel is an American multinational corporation and technology company headquartered in Santa Clara, ...
Now is the time for a truly global DX event, to bring together the leading minds from the technology...
Artifex Software began 25-years ago with Ghostscript, a page description language (PDL) interpreter ...
On-premise or off, you have powerful tools available to maximize the value of your infrastructure an...
In an age of borderless networks, security for the cloud and security for the corporate network can ...