Down Time

AppFirst was down for approximately 10 minutes today. Our tenant database encountered performance issues. They were severe enough to cause requests to time out between the back end and front end. Please note we don’t manage data streamed from your servers in a database, we use flat files. The database is used to manage tenant details as well as state information for remote collectors.

Our monitoring showed us that response times between the front end and client browsers was slowing quickly. We followed the response times to the connection between database and front end. A detailed look at the database application using Data Insight showed that the aggregate of the four Postgres processes were using a lot more memory than was normal. We also saw that page faults for the four Postgres processes were high and growing.

A look at the database made it clear we needed to vacuum it. We had vacuumed the database three months ago and at that time enabled auto vacuum. It’s clear we need to understand exactly what auto vacuum is enabling.

We apologize for any inconvenience this outage may have caused.

This entry was posted on Monday, May 17th, 2010 at 10:01 pm and is filed under Company News. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.