January 11, 2018 - Update for this weekend

Last Updated: Jan 16, 2018 11:49AM CST

As outlined in the status update on Sunday, January 7, our development team has continued to dig into the issues that contributed to last Sunday’s outage.

In short, this was the issue: a spike in response time due to gathering and reporting Event metrics and overloaded back-end queries for processing Event time data caused the outage.

This status update is an overview of what we’ve accomplished this week to create stability and speed in the platform this weekend, and what we’ll do to continue to improve the platform you rely on for #churchonline.

What improvements were deployed this week?

  1. Implemented new indexes to improve database performance.
     
  2. Moved Event metrics architecture to a dedicated database to improve stability of the primary database.
     
  3. Purchased additional RAM and CPUs to support the overall platform. 
     
  4. Reduced the amount of database connections allowed per server to more evenly distribute the load during peak times. 

When will metrics be available?

To restore service last Sunday, metrics reporting was temporarily suspended. Metrics email reports have been reinstituted as of this evening, Thursday, January 11, and you’ll get normal reports moving forward. Thank you for your patience with this pause in reporting.  

We are confident the fixes implemented this week will prevent the issues that impacted platform stability last week. We'll be in the office to provide steady updates on platform performance this Sunday. As we move into a new weekend, we’re praying for your church services and for many people to enter into a relationship with Jesus.