Crisis Management

With evolving broadcast technology, disaster recovery

has taken the software approach and broadcasters are investing

in multi-site operations for data to be synchronised across the globe

We’ve all heard the phrase Disaster Recovery (DR), but what does it actually mean for broadcasters and content owners, and what constitutes a disaster?

DR is a broad term that encompasses a range of scenarios, from catastrophic disaster (for instance, the complete destruction of a whole facility) to operational disaster, such as a transmission server failing. The ideal strategy in the event of a disaster is the seamless continuity of business under all circumstances with no assets being lost.

In the past, DR strategies have involved staff picking up boxes of tapes and equipment, jumping in a car, driving to another facility and getting back on air as quickly as possible. These days being off-air for longer than a few seconds, or a minute at most, is a disaster in itself and with content now being delivered globally, broadcasters have even greater responsibility to ensure that channels stay on air. They often also face 99.999% contracts with their channel partners with reference to airtime, and there are rules on compensation for lost or clipped ads. So there are also strong financial drivers for staying on air. If a catastrophe occurs at a facility in one part of the world, a DR strategy that will automatically kick in from another needs to be in place. With increased availability of wide area bandwidth and dark fibre, it’s much easier to share content globally but it is still not cheap.

So how do broadcasters ensure that their DR strategies are secure enough to continue broadcasting in any situation? The DR utopia includes multi-layered safeguards against the unexpected, using automated content replication systems to provide synchronised, mirrored or like-for-like asset duplication, across the same site or at geographically disparate locations.

At its most straightforward, this can be accomplished by duplicating tapes in the main archive and then moving those tapes to remote DR storage. LTFS works well in this environment, as any LTFS-capable system can read a tape created by any other, and can identify and retrieve the files stored on it. This means there is no requirement for a second archive system to simply read those files.

At the other end of the functionality scale, a fully automated DR-configured archive can be connected to a remote facility with either a robotic tape or disk storage. In this configuration, media assets can be automatically copied across the network and synchronised with the remote site. This model is ideal for broadcasters whose main and DR archives are separated by many hundreds of kilometres.

As we can see, automated site redundancy is an important factor for broadcasters and can be achieved by using rules-based implementations, providing fully automated data duplication across multiple storage layers and locations. Disaster Recovery systems enable multi-site operations to be mirrored and data synchronised across the globe. If one site becomes inoperative, it can be rebuilt entirely from data that has been replicated to other sites.

The more sophisticated archive management systems are able to offer completely customisable rules-based data duplication, through which content can be automatically copied as it is archived across disk and tape layers and, where required, different locations. In single-site scenarios, duplicate tapes can be easily externalised from the storage system, singly or in content-based groups, and removed to safe locations. Once a DR strategy is in place, it’s also important to periodically practise scenarios and test equipment.

One very effective way to ensure archives are DR capable is to implement a clustered system of multiple servers, or nodes, each in constant communication. Each cluster node has identical software installed, and each is connected via fibre channel into the archive devices – generally disk storage and one or more tape libraries. A Microsoft SQL database, at the heart of the cluster installed across two servers running a Microsoft cluster, creates an automatic failover mechanism.

Successful DR projects around the world will encourage more multi-site operations to adopt a distributed approach to their content management. By ensuring sufficient time is spent on collaborative design, an efficient DR strategy can be successfully achieved, ensuring that a channel can continue transmission regardless of any disaster without losing a single frame.

Quite simply, assets equal value, which equals revenue. Can you afford to lose them?