Preparing for the worst

With security incidents and technology failures increasingly hitting the headlines, the corporate value of a dependable disaster recovery and business-continuity plan should be clear. Law firms cannot afford to be complacent, however, and effective firm-wide communication is key.

By Alistair Roberts, Patrick Stone Associates

The CBI’s new-year resolution this year was to increase awareness of the issue of business continuity. There have been some highly publicised and very obvious terrorist threats to businesses and business people in London over the past nine months, but the Hemel Hempstead oil-terminal explosion also came as a severe reminder that disaster can strike anywhere and often from an unlikely direction. Over 100 businesses were displaced by the fire, many permanently.

These occasions are more frequent than one might think – there was a significant power cut in the East of England last year; a serious flood in London and middle England; a rat infestation and a mini hurricane, both closing businesses in Birmingham for a ‘disastrously’ long time; and, perhaps most likely, a catastrophic and lengthy IT failure brought about by a sophisticated virus. What about the threat of bird flu in your overseas offices or another serious health threat?

Around a quarter of the $40bn lost by New York in the aftermath of 11 September 2001 was attributed to businesses that failed to continue functioning or operate fully within five working days.

As disasters of various types have been in the news so much recently, law-firm clients are asking increasingly penetrating questions. Soon it may become a condition of appointment: not just a question of whether you have a plan, but also what provisions you are taking to ensure documents are secure, and communication and pending transactions can continue. All firms are likely to have felt the increased demands from their insurers.

Phil Woolas, the minister for civil resilience, has launched reforms of emergency procedures. He is determined that businesses should prepare for such catastrophes.

Does your plan work?

Most law firms have carried out their risk assessments and now have a business continuity plan (BCP). However, all too often it has not been updated or tested since its inception and much of the essential information required to ensure the firm can react properly in a disaster situation is out-of-date. For example:

People move house or leave the firm;
Mobile numbers change, phones get lost or upgraded with new numbers and networks;
People forget to take their copy of the plan home or it gets mislaid;
Computer equipment, systems and networks are upgraded;
Most common of all, stand-ins for key managers when away are insufficiently briefed.

Firms also tend to concentrate on testing the IT aspects of their plan, but other key aspects are seldom considered until it is too late. For example:Maintaining contact with key managers and staff;

Space planning for potential alternative sites;
Client contact details;
Alternative financial arrangements and controls;
HR policies for the staff to work elsewhere (perhaps upsetting their personal commuting arrangements);
HR policy for those staff and families directly affected by the disaster.

To the credit of senior IT managers, it is probably because they are so aware of the potential pitfalls that they constantly re-test their systems. However, a recent simple communications exercise undertaken by a law firm showed the alternate site exchange would not accept calls from certain mobile networks, which was a core part of their plan. Certainly, many lessons were learnt on 7 July 2005 about relying on the effectiveness of mobile-phone networks.

The challenges of an effective BCP

One of the main problems firms face in implementing an adequate BCP is failing to test frequency:

The main reason is that there is always something far more important to do than plan and manage disaster-recovery exercises, and it is always the busy people who head up the business-continuity team – the managing partner, possibly the practice-group heads and the service directors (admin, finance, HR, IT
and marketing);
There is also a common belief that testing is very disruptive and will cost the firm time and therefore money but, of course, not half as much as a prolonged disruption of business;
There is quite a strong feeling, particularly among those firms not situated in city centres, that, ‘it won’t happen to us’. It may not, but when you look at the range of events that can bring a modern high-tech firm to a shuddering halt, this is a far from wise approach;
Some firms believe that because they have several offices or buildings, finding alternative accommodation will not be a problem. This may be true, but detailed plans are still required to deal with the contingency, particularly the IT and accommodation aspects, which need regular updating and testing if a smooth and timely transition is to succeed. The plan could also be vulnerable if the document management and IT systems don’t allow you to operate and gain access remotely. A complete bar on entry to a key office can wreck the best-laid IT contingency plans;
Other firms have a fully serviced alternative site where the contractor has agreed to provide the basic communications and accommodation for the firm to continue to operate
and plan its recovery. This is fine as long as the key managers know
exactly what needs to be provided, and exactly which members of the firm will work from the centre during the initial and fraught period of any disaster. It is very unlikely that the whole firm will be able to move in, in which case the initial occupation and plans for the others will need detailed planning.

Putting the plan into action

Assuming the firm already has a viable BCP and has kept it reasonably up-to-date, proceeding to test that plan shouldn’t be too time-consuming or difficult, particularly given the vital role it could play in the future of the firm. What’s more, carried out sympathetically, the plan could revitalise interest and enthusiasm for disaster recovery.

The key stages in planning a simple straightforward exercise with minimum disruption are:

Agreeing exercise aims and objectives at the board level. Outside advice is also often useful at this stage;
Analysing existing BCPs in consultation with key managers (operations/IT/HR/finance) and insurers if required;
Drafting and agreeing a realistic exercise scenario and methodology;
Initiating and coordinating the exercise;
Running the exercise;
Facilitating the after-action review, assisting with evaluation and making recommendations;
Updating the BCP to incorporate lessons learnt, in conjunction with key managers;
Amending and agreeing BCPs.

Practical considerations

The most effective exercises are those that are conducted slowly but methodically and with the full
knowledge of all concerned. By doing it this way, you can minimise disruption to clients and staff, and achieve the greatest buy-in from managers. However, the test must be sufficient to reveal any shortfalls in recovery procedures, assess the board’s ability to manage the firm from an alternative location and reassure staff that their lives and livelihood are in safe hands.

With careful preparation, the exercise can be completed in just one day, although time must also be set aside to debrief key staff and report to the board.

Once the existing BCP has been evaluated and amended where necessary, the board may later want to conduct a confirmatory exercise under more realistic conditions to include the loss of key systems, occupation of the alternative control location and calling in of the disaster-recovery team.

In summary, BCPs drafted in peace should stand the test of an unexpected crisis, whatever the cause.

Case study

In early 2003, a major City law firm decided that a deteriorating security situation in the Gulf region was sufficient cause to test its business continuity plan (BCP). While it was unlikely that the anticipated intervention in Iraq would affect the firm’s main hub in London, collateral disruption was expected to the Middle East offices and possibly elsewhere in the global network. Although the test exercise would be confined to London, most lessons learnt would be applicable worldwide and regional staff would be reassured to know that the firm’s senior management was taking the threat of a confrontation seriously.

Although the impending conflict provided the impetus and justification to test the BCP, at that time the Home Office did not actually believe the overall threat level had exceeded that following the events of 9/11. The firm was therefore more concerned with what is sometimes referred to as a ‘quiet catastrophe’ – an internal and unexpected routine problem, which might have a wholly disproportionate effect on
day-to-day business.

In recent years, rapid organic growth had resulted in the firm operating from a number of dispersed London premises. As the firm grew exponentially, so did the complexity of and reliance on its IT systems, which were co-located primarily at the head office. Had a quiet catastrophe such as a localised flood or power failure brought down the main servers, there could have been serious knock-on effects around the network. A number of contingency plans were considered, ranging from a dedicated ‘shadow’ site to renting space at disaster-recovery centres. As most options were either extremely expensive or lacked the essential ring of confidence, it was fortunate that one of the firm’s premises was big enough and sufficiently distant from the head office to be pressed into service as a disaster-recovery location. Over a period of time, this building came to house an emergency switchboard, several standby servers for key IT applications and a conference room earmarked as ‘The Bridge’. This was equipped with additional workstations and the consumables, instructions and contact lists necessary to maintain an outward appearance of ‘business as usual’.

Crucially, disaster-recovery issues were routinely represented at board level and monitored personally by the chief executive of the firm. Thus, while being wary of disrupting clients, alarming staff or discovering embarrassing shortfalls, the board was equally aware of the potential risks and determined not to be caught out by the unexpected. A low level exercise was duly sanctioned.

Participants in the one-day exercise included the CEO, London managing partner, COO and senior personnel from IT, operations and HR. All formed part of the firm’s disaster recovery team (DRT). The ‘players’ had previously been briefed as to the background security situation, which involved an incident on a train at Liverpool Street. All staff were also kept informed to avoid an inadvertent War of the Worlds-style panic. On the day of the exercise, the opening scenario was e-mailed to the DRT, who were required to make their way to the disaster-recovery location independently, with the ‘situation’ rapidly deteriorating and police cordons quickly expanding. Once assembled, the team members were updated on unfolding events by an exercise director and invited to consider how to regain control of the firm and restore order.

The scenario continued to unfold, with the police ordering the immediate evacuation of all premises within a 400m radius of Liverpool Street and preventing all staff from re-entering buildings. A number of police were also spotted wearing respirators and several panic-stricken individuals were seen to breakout of the immediate cordon from the direction of the station. Garbled explanations suggested some sort of incident that had frightened passengers, and increasingly nervous staff held at assembly points began to thin out spontaneously in the direction of the river. Realistically, mobile-phone networks were failing and communication with employees was quickly being lost.

These and subsequent incidents provided a backdrop against which to run through the recovery process and create a number of realistic ‘what-ifs’, enabling the DRT to tease out any shortfalls in the BCP. Among others, these included evacuation drills, first-aid arrangements, communications
with staff and clients, business prioritisation and the recovery of IT programmes.

On this occasion, a relatively minor incident had triggered a full-scale alert, which might have had a more serious impact on the business community than a temporary interruption. In this firm, end of exercise (ENDEX) and an after-action review saw a wiser and better prepared DRT return to head office. The BCP was reworked accordingly, with the CEO remaining fully engaged in the process until all lessons learnt had been implemented.

Providing a workable BCP already exists, this is the minimum necessary to reveal any major discrepancies, to assess the board’s ability to manage the firm from an alternative location and to reassure staff that their lives and livelihood are in safe hands. At a later stage, the board might have wished to conduct a confirmatory exercise from a cold start, under more testing conditions. This could include role playing the loss of key buildings, evacuation drills, communications transfer, news management and switching
IT servers.

The cost of drafting a simple but workable plan and running modest exercises is minimal, compared to the confidence which they provide to firms and, increasingly, to their insurers. Keeping your fingers firmly crossed is a less satisfactory alternative. 

Alistair Roberts is a member of Patrick Stone Associates, which provides advice on business-continuity planning and exercising existing plans. He can be contacted at alistair.roberts5@btinternet.com