In two years, a lot has changed within the world of entertainment, with the onset of Covid leading to companies adopting different strategies for survival and growth.
While several players have entered the streaming market, few are able to distinguish themselves by providing a seamless viewer experience with a well-planned technology strategy on the backend. Discussing this within the context of sport at the recent IP Studios Future of Sports Broadcasting Summit were Faraz Arshad, CTO of StarzPlay; Floris Molijn, CEO & Managing Director, Oranda Private Limited; Micky Edwards, Director of Sales – EMEA at TAG Video Systems; and Peter Van Dam, CEO of AV Consulting LLC. Charles Kraus, Senior Product Marketing Manager of Limelight, moderated.
The impact of Covid on tech operations
In two years, a lot has changed within the world of entertainment, with the onset of Covid leading to companies adopting different strategies for survival and growth. Arshad from StarzPlay called the period productive but also challenging, “given the huge surge in demand in the region” and StarzPlay’s ambitious entry into sports.
“We started off with UFC (MMA fights) and then moved to football. In October last year, we streamed the T20 Cricket World Cup. So it helped us not only in scaling the platform, but to come up with different mechanisms to support the burst of traffic that accompanies sports … We scaled on the CDN side as well as the application infrastructure side. It was a great learning experience accommodating both sides of OTT demand (SVOD and sports).”
Molijn, who is based in Singapore, called it a “roller-coaster ride” for Oranda, which is involved in content distribution, contribution, satellite delivery and broadcast production globally.
“Our company has been managing the transmission side of things for several sports federations. In the last two years, every event was scrapped. Then when things eased, everyone took off at lightning speed at the same time. There has been an increased demand for sports content, but after the pandemic situation we now see a political situation with Russia and Ukraine, and the climate is continuously changing. So delivering for various events at short notice has become a common phenomenon but challenging.”
Micky Edwards says that as a solutions provider, TAG was able to help its clients with some changes in the way outputs from the system were addressed, so that people could pick up the mosaic feeds and analytics at home.
“We created HLS outputs so that people either working on a tablet or a laptop, or even a mobile phone, were able to pick up the services and still continue their work, because they may have been given very short notice that they couldn’t come into office anymore. We very quickly adapted our systems to make these changes and help our clients as best we could.”
As a CDN provider that also felt the shockwaves of Covid, Charles Kraus pointed out that Limelight’s biggest challenge during the pandemic was scaling infrastructure.
“When Covid hit, there were considerable supply chain issues around the world. So even when we had the ability to upgrade and add capacity, one of the biggest bottlenecks was actually getting hardware to do it. And then being able to schedule getting into the various POPs physically and installing the equipment was tough. It was quite a challenge to keep up in some parts of the world with the demand, but we have pretty much caught up and we’re finding that most end users prefer a multi-CDN approach, which helps a lot with capacity.”
Accelerated adoption of cloud technologies and addressing latency
Following the challenges around Covid, cloud adoption gained a lot of traction. Peter Van Dam has been a strong advocate for cloud solutions, and especially lauded the low latency he was able to ensure working within AWS with delivery via SRT for his clients in Saudi Arabia.
With latency a game changer within sport, Kraus queried how forgiving viewers were within the real-time streaming environment and how much latency was considered permissible. He added that although Limelight had a workaround that offered sub-second latency, its efficacy in a sports streaming environment where traffic could scale very quickly was not yet fully confirmed.
He listed some of the challenges around prioritising low latency, such as sacrificing quality and the difficulty of inserting watermarks, as any enhancements add to the delay. While auction and gambling probably had a use case for such low latency, sports didn’t really need it, Kraus added. Nevertheless, he pointed out that Limelight had closely explored CMAP and chunk transfer and coding – essentially micro segments which match broadcast.
“What we’re finding for most people is that the demand really is for just matching the broadcast latency and not necessarily beating it,” he commented. Van Dam agreed that the latency afforded through satellite for broadcast served as the benchmark “and today, we can go the same or faster than satellite”.
Micky Edwards interjected that TAG monitors latency at the production level, “which helps further downstream so you can see what’s happening when you’re getting to the last mile … We’ve put a lot of effort into keeping latency low on the production side. So then it can follow through when it’s going through to the later stages of the workflow, and we have a lot of happy clients using that. We’ve also been looking at 5G networks and using JPEG-XS over CDI. So that’s proving useful to get large amounts of live sports data across the country, and that can be applied to any of the 5G networks.”
Molijn added that no one in the market has asked for latency under a second, not even betting agencies.
“The only people that have asked us about latency down to sub-seconds is actually ourselves, which refers to what Edwards mentioned, where he can bring in the feeds and if there’s something wrong, we would like to see it before our client does. But there’s no client on the planet that will say it needs to be less than a second. Currently, three seconds is the benchmark in the market. And as Peter mentioned with SRT, you are able to do quick delivery so SRT is quickly becoming the benchmark.”
Combating latency in sports streaming
Latency has particularly become a concern with sport, the panellists agreed. Arshad remarked that StarzPlay adopted a number of measures to offer low latency streaming down to four seconds, ingesting feeds from studios and broadcasters into live workflow using SRT & Zixi, with some tweaks on the fragment sizes to deliver dash, smooth streaming and HLS formats.
“Along with the tweaks on the live workflow, we also took an advantage of our hybrid CDN approach, where we were serving traffic using a private CDN with ISPs directly connected over local internet exchanges (IX) and the public CDNs. Delivering sports using a private CDN is something new we tried. When we pushed our feeds over an IX, we were connected with three different exchanges and that helped bring the latency down to four seconds and offered a smoother screening and more consistent experience on all the devices.”
He also emphasised application scaling.
“No matter how much you scale, if your application is not optimised, you will not be able to acquire customers or retain them. When you get significant bursts, the load often goes towards the payment gateways and billing engines, and you don’t want new customers to be impacted. To handle these bursts, you need to plan properly, and that goes all the way from your subscription pages on the apps to the billing engines in order to ensure that the internal infrastructure can be scaled up instantly.
“On the other side, you also have to deal with multiple payment gateway providers. Rather than going with one payment card and payment gateway, we used two or three to ensure the load was shared and you could scale the transactions between these two. Besides this, we also handled subscriptions using in-app methods of payment (like Google and Apple) and a telco direct carrier billing, which helped customers pay through different payment channels.
“So we gave a heads-up to one of our major telco partners in the region that they were going to see significant traffic. We were also continuously making recommendations before, during and after the matches to our sports fans, to attract them to the other products on our platform. This is how we managed to deliver the load across the platform.”
Asked if StarzPlay would consider streaming sports in 4K, Arshad said there would be considerable challenges in the last mile that would compromise the user experience. “I don’t mind offering sports in 4K provided that the last-mile bandwidth is capable of handling 4K bitrates, but in reality not all customers will be able to consume it. On average, we are serving up to 2.5MB and the maximum you can go up to is 4.5MB with full HD.”
Molijn questioned the definition of a good product for the end user, apart from the contribution. “What can an end user digest without complaining and still have that satisfactory feeling?”
Arshad commented that this is where the IX plays an important role. “A service provider should be able to manage a mix of private and public CDNs by focusing on direct versus indirect peering, based on latency, throughput and availability. When we deliver a live stream over IX, the handover costs towards telcos are very minimal and they have capacities available. So we have some network peering mutual agreements with major telcos in the region. Our origins are connected to those IXs feeding our edge caches within these telcos.”
Edwards seconded this, saying that an elastic service enables us to “scale and resize according to the load on the system” and adding that TAG was flexible with both cloud and on-prem deployments, ensuring its solutions were agile and adaptive monitoring could be done on three different layers.
“Offering full monitoring as well as light and extra light features reduces the processing overhead so that you can monitor more channels within the same footprint. So it really makes it a cost-effective solution to deploy.”
Kraus then examined how information was being monitored – the tools in the network to monitor and get visibility into what’s going on and how multiple CDNs anchor the whole ecosystem, especially during a game. “What are some of your best practices for monitoring?” he asked.
Arshad explained that detailed monitoring was implemented on the application as well as on the network side and could provide microscopic insights into the customer experience – jitters, buffering, the latency they’ve been connecting with, their join time, the instinct tuners and so on.
“Essentially, the metrics give you an idea of which ASNs are struggling to provide a good experience, so you are immediately able to switch from IX to IP or vice versa.”
It was agreed that the agreements streamers have with CDN providers, telcos and IX players play a big role in ensuring a seamless experience.
Edwards pointed out that multiple monitoring points bring in multiple streams of data.
“TAG has now put all this into a Redis Data pipe and made it available for third parties so that they can create a dashboard and analyse trends and what’s actually happening through the network, and it’s really been effective. You actually have visibility into exactly what’s happening across the network, and it gives clients a lot of confidence.”
StarzPlay has gone one step further by integrating different sources of data with its Customer Lifecycle Management (CLM), said Arshad. “Data given by CLM also helps our customer support to pro-actively target those customers that are having a bad service experience so they can send them special offers, such as an additional month at no extra cost and so on, by way of compensation.”
Molijn says telcos are slowly catching on that for broadcasters, the internet line is just as important as the satellite dish on the broadcast compound or the fibre link. “Telcos are becoming vital to the entire contribution part,” he commented.
Panellists were then asked what they would like to see from new technologies.
Molijn remarked that the ability to achieve the same goals at every single location would be great; while Edwards said consider new formats in the future but added that technology partners must work together to ensure systems are easy to deploy, manage and use. Peter Van Dam commented that his main objective has been to make sure broadcast qualities were maintained while the quality of experience did not dip. Arshad pointed out that the game has become complex with the attempt to distribute content across different devices, with everyone having their own codecs. He commented that it was important for a regulatory entity to step up and regularise the protocols so as to simplify things for streamers. Protocols should be adopted across all formats, irrespective of whether it is Android or Microsoft.
