Disruptive innovators

Building a broadcasting solution with a multi-million dollar budget, a systems integrator and tried-and-tested technology is easy. What happens when you have none of the above? In an exclusive interview with Vijaya Cherian, Sky News Arabia’s CTO Dominic Baillie and his team talk about how they designed and engineered a solution that has never been attempted before

Sky News Arabia (SNA) embarked on a cutting-edge project last year that was finally completed last month and will be rolled out to its bureaux this quarter with the intention of truly empowering its journalists to provide content anytime, from anywhere to the channel’s viewers. To enable this, the technical team at SNA, headed by CTO Dominic Baillie, engineered a futuristic system in-house that serves several objectives, one of which is to provide its journalists with the ability to access feeds over a public internet or private network and mix a live show from any part of the world remotely with the help of a laptop, a broadband connection and a USB key embedded with software, designed by the team.

SNA is still mulling a name to encapsulate the power and potential of this solution that few may have been able to fathom and fewer still may have been able to design within the meagre budget that Baillie and his team utilised. For now, the broadcaster is looking at a working title such as The Content Hub (TCH) or SkyNet – the project codename.

Significantly, this system was designed on the grounds of SNA at its Abu Dhabi facility by its own team with no external support from systems integrators.

“As part of our efforts to be closer to where the news was happening, we have been doing a lot of external studio shows and OBs and needed to access different types of live feeds within certain bureaux that were only available at our headquarters,” explains Baillie.

“The obvious answer was to select one or two sites and enhance their capabilities within the existing infrastructure.”

With this in mind, an initial budget of USD 250,000 was set for this project. It turned out, however, that the SNA team was unable to maximise its technical and operational objectives with these existing facilities and would instead have to devise a system from scratch.

Although the basic premise of the project had now changed, Baillie, who has a reputation in the local market for thinking out of the box, decided to take a fresh look at the concept and set his team the challenge of designing and building a system in-house with the same investment. The task did not just demand engineering skills; it required extensive research into alternative technologies to replace more traditional and expensive ones, and called for a software-based approach to the normal method of relying on expensive and limiting equipment. The end result immediately puts both SNA and Abu Dhabi on the map for being the place, where such a solution has been conceived and brought to fruition.

“TCH is an innovative solution to a problem SNA had with ensuring our news content was available wherever and whenever we needed it,” explains Baillie, as he switches on his laptop to demonstrate the power of the app he and his team have engineered.

“SNA has its broadcast headquarters in Abu Dhabi (AUH), which houses our main studios and the infrastructure that supports the HQ and its vast newsgathering infrastructure. All the content from the field, be it packages or live feeds come into the AUH site, where it is edited, packaged and used in our 24/7 breaking news channel. In addition, we have more than 17 bureaux around the Middle East and the wider market. These bureaux vary in their technical capabilities and connectivity from basic live positions to very capable studios. We wanted all of these bureaux to have the ability to access the same vast array of news sources as AUH. The solution was TCH,” he says, and goes on to demonstrate what it can do.

He inserts a key into the laptop, connects to his VPN and we instantly have access to the different feeds that are coming into SNA’s network. He then switches between the different feeds and in five minutes, we manage to simulate a show between live feeds and the studio, add a ticker, the CG and bingo, we were ready to go on air.

“Everything you need to broadcast our channel is in TCH and everything you need to access it is on this key,” explains Baillie.

“Of course, we have built in various levels of security before you can gain control of the system. You need an internet connection, either wired in the office/bureau or you can carry a 4G key with you. You can use any laptop; everything you require in terms of software and documentation is on this key and essentially, if you have the permissions, this allows you to sit anywhere and craft a show for broadcast, live. We have made the user interface really simple and to avoid anyone having to carry around extra hardware, we have made every possible solution using only software,” he explains.

The TCH itself isn’t clever, according to Baillie.

“Anyone can design or build a system that takes signals in, routes them and spits them out again. The clever part is the resilience and the flexibility, making it simple, and channeling this flexibility by developing the control software and making it possible to operate it remotely, over the internet from anywhere in the world. The biggest challenge here was the streaming part. We had to find a way of streaming multiple feeds, both over our private network and public internet in high enough quality but in real-time. If we used standard web encoders, we’d be looking at a delay of multiple seconds, which would make it impossible to vision mix in real-time. Professional encoders would still be over a second encode and decode. We had to find encoders that could encode an HD stream within a few frames. We then had to find a way to decode it quickly. We have deployed hardware decoders in key locations but we wanted to be able to receive these streams anywhere by anyone so we had to find a way using a software decoder.

“We had to find a compromise between latency, ease of use and reliability. Streaming real-time HD video over the internet is not easy; it breaks up constantly due to dropped packets. We found a company called Zixi, which adds a management layer to transcend these unreliable networks. Testing this took a lot of R&D, but TCH has acted as a test bed and incubator for a lot of new technologies. We have since implemented what we learnt in Islamabad and Ramallah, instead of MPLS to deliver real-time high-res HD video (over 20mbps); we use public internet. The latency is equal if not better than MPLS (Multiprotocol Label Switching) and the quality is comparable. TCH is effectively an internet-connected studio. To make it work, we had to make it react superfast and that too, over the internet – not a LAN or WAN. That is the clever bit,” he explains.

“Device control is managed locally within TCH to avoid any risk of packet loss or timeouts. We wrote the software as client server and even wrote our own control protocol to work over the internet to remote clients,” he added.

TCH itself consists of four main functions; acquisition, playout, production and transmission.

Project Manager Dave Mace, who headed the team that delivered this project, explains that theystarted with “the concept of a pool of resources that could be dynamically allocated over SNA’s MPLS network to any bureau”.

“These resources included the equipment needed to receive and process feeds from satellites, MPLS/internet-connected bureaux, cellular bonding equipment, 4G equipment and other internet sources. We then needed the ability to remotely switch any of these resources. For instance, a bureau may only have enough bandwidth to have three HD streams at any one time but during the course of a show, its studio may need access to more,” he explains.

“Using TCH, the operator – who can be either in HQ, a bureau or even in the field – can set up all of the feeds they will need for the duration of the show or segment and view all available feeds on a remote multi-viewer. Bandwidth is allocated by default within the system; we have known paths to different places and the operator would just have to select a path and a source. They can then switch the feeds they need at the right time,” he points out.

The team then added the ability to record and play out the content.

“We can use this for store and forward; we can also record a feed, clip it and send it either as a video stream or file to any of our bureaux for playout or editing. Again, the whole process can be monitored and controlled remotely so for a bureau which does not have a video server, the operator can use the one in TCH to play out content that they need for their show.”

Once the original objective was undertaken, the team decided to push the boundaries even further and wanted to add in the capability to remotely produce a show.

“It’s the next logical step,” chips in Baillie.

“We had all of that content and capability already in TCH. All we needed to do was add a vision mixer and CG with the ability to monitor and control them remotely. This actually gives us a really powerful asset. Not only can we create a full studio show from a bureau that only has cameras, we can utilise this in the field. We can effectively produce a show in real-time anywhere in the world. All we need is relatively good internet connectivity and some cameras and lights. We are already testing the ability to stream directly from cameras to TCH over the internet,” he explains.

The final component to complete TCH was transmission.

Here, the team built in the capability to deliver content wherever it was needed, be it file-based over the internet or SNA’s MPLS network, or delivery via satellite.

“In essence, there are lots of ways to get the video out — over our private network, the internet or as files delivered to wherever it is required,” adds Mace.

Of course, all of this can be done but never at the price at which Baillie and his team achieved it. In fact, that was perhaps the biggest challenge.

“We negotiated very hard with our partners and chose our components carefully. Of course, we had to keep TCH in line with the technologies already utilised at the AUH site so some of the choices were already made. Here, we pushed really hard to get the right deal. Some vendors weren’t interested but there were others, who were keen to be part of a revolutionary project and bent over backwards to help.

“There was a clear distinction between those who wanted to work as partners and those that merely wanted to serve as suppliers. We borrowed and tested a lot of equipment before placing orders. Peyman Dadpanah at Medicast deserves special mention for supporting us extensively as we tested various Blackmagic routers and vision mixers. Nevion and Haivision also supported our R&D efforts. Viz and Zixi were also amazing. Where we had choices, we went for the best combination of cost and fit. We also borrowed whatever we could from existing systems.”

Furthermore, SNA built this at Du’s datamena facility in Dubai because of the locality, and the accessibility to connectivity.

“We receive RF (satellite) feeds from their dish farm; we can send SDI for uplink, we also have access to the internet and our private network. All the TCH functionality works because of this level of connectivity. Again, if we had to build out every bureau, it would mean building a dish farm for each.”

Inherent to the TCH design was redundancy as a breaking news channel needs a reliable solution for its live broadcasts.

The CTO says the team “completely over engineered it” to ensure this objective was met.

“One of the unseen benefits of using cost-effective components is that you can buy more than you need.

We have built so much redundancy into the system that we can survive multiple failures and still broadcast. We have mirrored the main and monitoring routers (two of each), mirrored vision mixers and mirrored presentation switchers (for graphics). We also have redundant networks and redundant links (MPLS, Internet and SDI). We built the system in two halves that are effectively mirrored so we will always have a set of resources of each type available. We can lose a router, both vision mixers and half of the sources and still package and deliver a show without the operator noticing,” Baillie proudly points out.

Creating a simple interface for the operators that did not demand a steep learning curve meant that the tech team also had to develop a fairly complicated design on the backend, explains Baillie.

“By definition, flexibility and redundancy bring complexity to a system. The goal was for the operator to use familiar interfaces or one that is so intuitive that they could operate them without training. Our software development team helped to build our own software to set up and control the TCH. By designing it ourselves, we were able to tailor it to the requirements we knew were relevant,” Baillie explains.

The CTO adds that the team had to design the whole workflow, develop apps and integrate all of the elements in-house because SNA’s requirements were very unique.

“This whole thing is bespoke and primarily software driven. If we bought this, it would have blown the budget many times over and done a small portion of what we have planned. In-house software development is the future of our designs; it allows us to fully recognise the flexibility of the system. We have built into it different modes which we will add over time. For example, it is possible for us to use half the system as a ‘virtual’ studio for broadcasting in the field and simultaneously, use the other half to provide sources to another of our bureaux.

“There is so much flexibility and redundancy built into the system. I can configure it to be redundant or I can use components separately to double my capability. I can feed up to 16 individual streams to end users – whoever or wherever they are. I can also use the mirrored VMs separately to do two separate shows at once. In fact, I can do four if I wanted to because I can control 4 MEs separately. I can even make one big VM with 8MEs. Such is the beauty of the software. I don’t think even Blackmagic envisaged how much we could get out of their products through software. Their SDK (Software Development Kit) was a major selling point for us.”

This project was ridden with challenges. Perhaps one of the biggest, other than budget, was helping both technical and non-technical staff understand what the tech team were trying to put in place, what it would entail in terms of technologies and also, the possibilities it would open up for the channel.

“We drew what we wanted and then tried to find the pieces that fit it for the money. A lot of people didn’t get it at all, nobody else thought of doing it this way and a lot of people really didn’t understand or believe in it until they saw it. Even we were unsure of whether we could make it work as we wanted it to when we set out. We had to find the pieces to realise the full potential of the vision,” Baillie says.

There were other challenges. Making it intuitive took a lot of brainstorming and there were instances, where flexibility had to be sacrificed for usability. There were also other concerns to address such as latency and monitoring.

“We had to keep latency in mind during the design and development process. Managing it was key to the success of the project. We needed to ensure that the delay incurred in TCH to receive process and transmit feeds was tolerable for live interviews. We set the benchmark as better than satellite with the target of better than MPLS.

“We also had to ensure that the control and monitoring aspect of TCH was useable. We needed to get as close to real-time as technically possible but there is a fine line between usable and too latent. It’s that pause between pushing a button and seeing it take effect on a monitor. You need it to be fast enough for the operator not to question whether it worked. We added some fast acting tallys to help with this perception. We think we’ve managed to build something that’s acceptable although I’m challenging the team to further reduce the latency, which is currently between half- and one second depending on where you are in the world and the kind of decoder you use. A hardware decoder, of course, can be even better than this.”

One of the instances where SNA realised how cost-effective such a solution could have been was at a recent event that needed to be covered in Europe. The SNA team carried around 200kg of excess baggage and had to rent a lot of equipment. If TCH was available at the time, the team would have only required local internet access, a few laptops and some cameras.

“It would have been quick, easy and cheap. Think of it as an extremely well equipped OB truck with very long cables to the cameras and control/monitoring positions that can be instantly available anywhere in the world and even in several places at once,” explains Baillie.

The project, however, is far from over. The team is continuously refining the solution.

“The next step for us is to remove all of the baseband video from the system and just process streams in the IP domain. We’ve already started work on this. This is especially important since most of the signals are IP anyway, but imagine if we do everything in software. I can bring up TCH in any datacentre close to where I’m working for the period I need it and pay only operational expenditure. This would be a location close by to help with latency.”

Simple as it sounds, this project took a good year to get to this stage and involved a lot of learning within the engineering team.

“When some elements were missing or things were not quite right, we went back to the drawing board, reworked our designs, got the developers to address some of the code to ensure that the front end would be simpler to use.

Baillie says: “TCH serves as a prototype to prove the technology and the concept”.

“We will now refine this down to a software-only system; of course, carefully selecting our partners.”