Case study: In search of system security

Norton Rose is moving head office in May 2007. A host of new IT services require new automated systems-management solutions, and risk awareness led the firm to adopt a proactive service assurance model to reduce the danger of downtime.

By Malcolm Todd, head of systems delivery, Norton Rose

As an organisation Norton Rose is traditionally risk-averse. We stake our reputation on our reliability and professionalism, and our clients entrust us with very sensitive information that we have a legal duty to protect. If the privacy of client information were compromised in any way, we would be liable for heavy damages and our reputation would be in tatters. Continuous, round-the-clock service is essential for our users. If our systems are down, our lawyers cannot prepare cases, communicate with clients or bill their time, so there is a direct correlation between billings and system availability. For this reason IT-system reliability and security are of paramount importance.

Another factor influencing corporate thinking in recent years has been the threat of terrorist activities. Our London offices were devastated by the IRA Broadgate car bomb in 1993. The 7 July bombs in 2005 led to an office closure when the Aldgate area was cordoned off. So another key planning consideration was to provide a business-continuity and disaster-recover plan that would stand us in good stead should disaster strike again.

ISO certification

To underline this focus on security and ensure internal practices were of the highest order, in 2006 we embarked on a programme to become one of the first legal companies to achieve the ISO27001 Information Security Certification. A key component is to be able to provide regular reporting and analysis on risk levels as well as submitting six monthly external audits to verify that standards are being maintained.

The firm’s board had also previously endorsed a major initiative to overhaul the company’s IT infrastructure and equip it with automated systems management to mitigate global operational risk. The project was intended to move the company from a reactive service management approach to a proactive service assurance model, reducing operational risk and absorbing a 20 per cent increase in servers without increasing support manpower.

Business drivers for the project

The IT infrastructure has experienced incredible growth in recent years, from around 120 servers in 2002 to 500 today, 40 per cent of which are spread around the 19 international offices and are supported centrally from the UK. The processes for managing and monitoring remained event-driven, however. We relied on manual/scheduled checks – and on our users actually reporting faults. The process of finding and fixing these faults involved the manual routing of electronic problem tickets to various internal support groups. Unsurprisingly, this process was labour intensive and inefficient, as well as being particular sensitive around the time of major deal closures. The process of reviewing over 44,000 security alerts and un-prioritised e-mail messages each month – requiring 31 man hours every day – was unsustainable. As with any manual system, some issues were also inevitably overlooked.

The key project goals were:

Reduction of risk and exposure from any unplanned downtime and the minimising of
outages through quick, focused, automated resolution;
Ability to quickly view faults to enable an informed, proactive dialogue with users, combined with a guaranteed consistency of response to service issues worldwide;
Maintenance of current staffing support levels while increasing server population by
20 per cent and focusing staff on more productive duties than repetitive monitoring;
Ability to offer tangible service availability and capacity statistics (by office, service and platform) to provide trend information, focusing on the key issues and enabling effective capacity and availability management.

Key challenges

To compound the challenge facing the IT group, in May 2007 we will be moving to a new London head office between London Bridge and Tower Bridge on the South Bank of the River Thames. This move involves closing down six separate offices and relocating to the new larger premises in the More London Riverside development. The plans for the new office include rolling out new IT services, including collaborative tools such as Voice over IP (VoIP), Cisco’s new Meeting Place desk-to-desk videoconferencing services, knowledge-management systems and a new contact-management system – all potentially placing additional strain on our IT support group.

Production services were also migrated to a new data centre in Uxbridge in October 2006, supported by our current facilities in London for disaster recovery and testing. The new office required us to set up 100 new servers with 50 terabytes of data, housing all the production processes and replicating all data to comply with the Law Society guidelines on holding unalterable copies of data for any financial transactions.

Production was very successfully cut-over to Uxbridge in January 2007, with replication of data re-established in reverse back to the London facilities. This ensured that the move to the new London office was accomplished without service disruption to clients. Future plans include triangulating all international traffic via the new London office and keeping a replicated copy of all international data in the UK. Should an international disaster occur, Norton Rose would still have its data intact, and be able to transfer operations to the Uxbridge centre within an eight-hour window. A UK disaster-scenario fail-over has successfully been shown to be possible within five hours. It will be retested in April 2007 (the fourth test so far).

We decided to approach the service improvement and efficiency tasks set in two phases. Phase one began in the fourth quarter of 2006, involving a focus on proactive service management and reducing operational risk. The second, which commenced in 2007, extended the scope of the project to include security and vulnerability monitoring – and the ability to devolve more operational user-administration functions (such as adding, transferring and deleting staff joiners, leavers
and transfers) to the business-support teams worldwide.

The search for an IT partner

We already had a variety of point solutions, such as Compaq Insight Manager and several bespoke in-house tools, in place in various parts of the organisation. These solutions were all in different formats, however, and the IT team had to run multiple logs, which was extremely manpower-intensive.

We therefore set out to find an automated server-monitoring solution that would allow us to improve system reliability at the same time as growing the IT footprint. We recognised the importance of having a comprehensive and integrated IT-management system that would not only handle the servers, but would also address key issues such as security and operational change.

After evaluating a number of alternative products, we decided to roll out an integrated suite of system-management solutions from NetIQ, a division of Attachmate. The selection was based on a number of criteria, including cost-effectiveness and ease of management.

Implementation

The first phase involved rolling out the performance-management solution, NetIQ AppManager, which provided the capability to proactively monitor and analyse system availability across the 250 core servers and associated applications, in 20 countries in Europe, the Middle East and the Far East. This phase was completed successfully (including the integration with existing tools) within three months. The resultant system enabled support teams to have an operational overview of the entire global network. Each support team is responsible for setting its own parameters and alerts, with access to an ‘analysis centre’ enabling them to isolate and drill down to the cause of a fault. The system can apply automatic known-fixes and notify support staff of the alert on a 24x7 basis.

The system allows us to be much more proactive in our approach. We can solve problems before they materialise. Within two weeks of going live, we identified and were able to avert two major systems errors that had the potential to seriously impact the business. Now, when we want to measure server availability, for example, we only need to flick a switch to see accurate, up-to-date data. We can monitor service levels, conduct problem solving and compare and contrast performance across different servers and applications automatically. We are now entering a phase of further tuning these tools to release greater levels of efficiency, tied into the ‘bread and butter’ operational processes, including roll-out of service views to our international and local IT business-support units (first line staff who sit in the business areas).

The benefits to our users

As a result of the performance-management system we can now move forward to measure the availability and performance of applications from the perspective of our users. This means we are equipped with valid measurement metrics for availability, and response times that ensure we comply with service-level agreements. It also gives us the ability to understand the impact and urgency of infrastructure incidents. By measuring application availability and response time from various user locations the IT department can quickly determine the locations that are affected by performance degradations and outages, and therefore respond accordingly. In short we are much better equipped to provide the optimum level of service to internal customers, leading to a reliable and secure service for external customers. What’s more, within months of implementing the NetIQ performance-monitoring system, we were able to increase the number of servers by 30 per cent, without needing to increase headcount in the IT team.

The second phase

Security is the focus for the second phase of the deployment, which began in October 2006. We have selected NetIQ Security Manager to help proactively manage and resolve vulnerabilities and security threats and meet compliance guidelines. We have also deployed a number of their security administration and management tools, including the Group Product Administrator, Directory and Resource Administrator, Secure Password Administrator and Change Guardian for Active Directory. These tools will allow us to increase the effectiveness of our risk and security management by automating and streamlining many processes, and thereby minimising the impact on technical support resource. What’s more, because the different products all have the same look and feel, all support technicians can access the complete range via one user-friendly console.

Finally, we are now approaching the opening of our new London head office in May 2007 and the deployment of a new Voice over IP system and video-conferencing solution. Mindful of the need to ensure complete security for the system, we have put in place stringent physical security measure such as firewalls and VLANs. Around 300 people will move into the new offices each weekend, with 1,200 employees needing to be fully operational by early May when the office is officially opened.

NetIQ AppManager for VoIP combined with Change Guardian for Active Directory allows a smooth transition to a new IP telephony system. As with other IT services, the VoIP-management system will
allow us to monitor security and performance availability to ensure employees have a consistently high quality of service experience.

We are now confidently approaching our application for ISO27001 certification armed with the evidence of a proactive system-monitoring system. In the last 18 months we have gone a long way towards realising our original goal of moving from a reactive-service management approach to a proactive-service assurance model that enables an excellent IT service for internal and external clients. We have also turned technology from a mere business cost to a positive asset and competitive differentiator.

As with all such initiatives this is a continuous learning process, however, and we are evolving and tailoring the solution, alongside our operational processes, to achieve optimum efficiency and return on investment.

Malcolm Todd is head of systems delivery at Norton Rose. He can be contacted at malcolm.todd@nortonrose.co.uk