90% Reduction in IT Incidents

Background

This client provides residential mortgages to customers across a wide financial risk spectrum. Over many years of heavy customisation their core lending platform was prone to failures. Issues affecting business operations were occurring daily. Too much time was spent by IT department staff dealing with these issues leading to significant delays in important projects. Data integrity had become a major concern, with system crashes often leaving data in an incomplete, invalid state. Many operations staff were continuously involved in repairing data and IT team members were in a constant cycle of responding to or reporting on issues.

The systems had become fragile. The four core databases contained over 1400 customised stored procedures and thousands of added tables with inconsistent naming conventions. No-one had a clear picture of the entire database and changes were difficult to implement and often led to major incidents. These incidents were taking several days to identify and repair, and each one was handled in isolation - one-off reactive fixes were applied rather than looking for a systemic solution to the underlying causes.

The Solution

The first step was to eliminate unnecessary complexity. A rigorous examination of all tables, functions and stored procedures in the database identified several thousand tables and stored procedures that could be safely eliminated. This made it easier to identify problems more quickly.

Then the patterns in the system that were causing failures were identified. Where possible, these were fixed at the source, but in many cases the problems would have to wait. A lot of work went into automating repairs so that if certain problems did occur an automated means of repairing the damage quickly and reliably could be implemented. What had previously taken a team of eight or more developers several days could now be achieved by one person in a matter of hours, with more reliable results.

Once this was under control development practices were overhauled. This encompassed:

The way technical resources were hired,
Change approval processes
Code review processes
Coding standards and conventions
Improve coding and systems documentation

The Result

90% reduction in the number of issues affecting business operations
35% reduction in database complexity
30% faster turnaround of coding changes
Repairs to major data incidents reduced from several days to hours in most cases
Increased staff morale

Technology Used

Microsoft SQL Server

What had previously taken a team of eight or more developers several days could now be achieved by one person in a matter of hours, with more reliable results