I’m Rowan de Pomerai, a technologist at BBC Research & Development. I’ve recently returned to London having spent around 7 months in our northern lab in Manchester. As Anthony mentioned in his post, the last year has seen BBC R&D’s presence in Manchester grow from a handful of engineers to staff numbers in the double digits, with plans to keep that number rising as more staff move north in advance of the move to MediaCity:UK. Where the first few early movers got by with a small office space and a basic broadband-type connection to the R&D network in the south, that situation was rapidly becoming untenable. Physically we were bursting at the seams, and the lack of facilities was restricting the work we could do to certain types of software development and little else. We needed more space and more facilities if we were to grow a lab which could match the breadth of output provided by the London base.

Clearly we needed more desks and more space, that much I’m sure everyone can understand. But why all the fuss about facilities? What exactly did we need? Well, there’s some staff who do just need a relatively ‘normal’ office setup. That might include those doing certain types of software development, or those that look after our partnerships and external engagements, or some of the people doing audience testing and research. But then there’s people working on bigger IT infrastructure testing, who might need racks of noisy and hot-running equipment, or image processing experts, who are writing software but also need cameras, space to film things, and video connections between rooms. There’s people who need to test hardware with signal generators and oscilloscopes, or need to build electronic circuits and so need soldering facilities. And of course we’re regularly demonstrating our work to others (after all, what use is new technology if nobody outside the lab ever sees or uses it?) so we need space for that.

Shortly before I arrived in Manchester, Adrian Woolard had been working with Michael Sparks to acquire and begin planning the use of some office space on the BBC Manchester site at Oxford Road. I took on managing the fit-out, building on the work that had been started before I arrived and working with BBC Workplace and the IT folks to get the space ready to meet the needs of Research and Development. The space in question was originally built as the base of operations for Radio Outside Broadcasts, and next to the garages for OB trucks is a little building with some office space on the first floor. That floor has become the home of BBC R&D’s North Lab.

The building contains essentials like toilets and a kitchen (meaning we can make our own cups of tea rather than always having to buy them – a big benefit!) as well as two small rooms and a fairly large open plan office. The basic planning of what facilities we would need over the next 2 years led us to a list comprising an apparatus room (where servers and broadcast equipment sit in racks), a laboratory (where hardware work can be undertaken), a space for demonstrations, some meeting and breakout space, and an office area. Some simple maths shows us that there weren’t enough rooms for this, but we managed to make it all fit by agreeing that the meeting, demo and breakout space could be one flexible room, and arranging to re-instate a wall which had previously been removed, dividing the large office into a slightly smaller office and a separate laboratory. Thus was born the room plan. Once we finally got the project started, the installation was relatively quick. The new wall went up, a heavy-duty air conditioning system was installed to keep the apparatus room and laboratory cool, the walls were painted and the carpets cleaned.

But the space is only half the story. The other investment was in connectivity. Within the building there were nowhere near enough network ports to support 25 engineers, many of whom use multiple computers for their work. So we has a lot of extra networking put in, as well as extra power sockets to supply the computers, cameras, test equipment and more. Audio and video connections now connect the office, laboratory and apparatus room, so noisy equipment can be used remotely from the office, or signals fed from the new satellite dish into the office. The final piece of the puzzle is connecting the north lab to the south. Until the interim lab was connected, a single ADSL connection – much like the one that probably supplies your home broadband – was all that connected the two sites. With more engineers working and high-data applications needed (such as streaming full-quality HD video or moving huge datasets between sites), something better was needed. An interim solution is now in place using existing BBC networking, providing well over 10 times the bandwidth we had before, with gigabit fibre an option we’re perusing if we need it going forwards.

To top it all off we have a few little extras to help us do a broader array of work and have a little fun too. Tony secured use of a disused radio studio in the main New Broadcasting House building, allowing his exciting work on future audio technologies to gather pace. And back in the main office, I was tasked with attempting to divide the space. There is a push in many organisations towards open-plan, and while we like the feeling of openness and community, many of the engineers find this sort of environment too distracting to concentrate on difficult problems. How do you divide the space and reduce distractions without losing openness? Well that’s a question I couldn’t possibly answer fully, as there is no single answer. I’ve written a little more about it on my own blog, but let’s just say we’re experimenting… below you’ll see our blue sound-absorbing blocks, great for sitting on, a little impromptu wall building, or even fort-building when some of our staff get their hands on them.

All of this is a drop in the ocean compared to MediaCity, where we’ll have more engineers doing a wider variety of work. Extra facilities needed there include a usability lab, which looks much like a domestic living room and allows us to study user behaviour when interacting with new technology, as well as viewing and listening rooms for critical audio and video work. But that’s a whole new chapter in the story of R&D’s north lab, and one I don’t have time for here. Suffice to say, the future looks exciting!

The first and largest problem is creating an environment free enough of distraction that engineers can concentrate. We use the idea of flow – a well established psychological concept – to illustrate this. As Wikipedia puts it:

Flow is the mental state of operation in which the person is fully immersed in what he or she is doing by a feeling of energized focus, full involvement, and success in the process of the activity. Proposed by Mihály Csíkszentmihályi, the positive psychology concept has been widely referenced across a variety of fields. Colloquial terms for this or similar mental states include: to be on the ball, in the zone, or in the groove.

We need to balance collaboration and openness with this desire for contemplation and focus, a problem often faced by others, such as web software firm Digg:

I was particularly worried about the open area being too disruptive for productive engineering work and about the difficulty of managing the delicate balance between good communication and an excessively distracting environment. To mitigate this concern, we took inspiration from the “caves and commons” pattern and installed a cave area, for developers to retreat to.
– John Quinn, Digg

One way we’re seriously considering breaking up the space is into “project neighbourhoods”, an idea used by others including design firm IDEO. Tom Kelly has some interesting things to say on the matter in his book, The Art of Innovation:

We believe in the importance of neighbourhoods and community in fostering innovation.
– Tom Kelly, IDEO

Space is the team and the work. If a member wants to jump aboard another project, he or she needs to be able to quickly take off and land [in the new team’s neighbourhood].
– Tom Kelly, IDEO

Finally for this post, I’d like to share thoughts on one of the words I seem to say an awful lot these days; flexibility. With a regularly changing workplan and a remit to develop the next generation of media technologies, whatever they may be, our requirements change. One size does not fit all, and we’re keen to ensure we can adapt. I’ll leave you with my favourite quote, an extract from Workplace By Design by Becker & Steel, which explains that an organisation’s body language (what it does) must match what the organisation says.

An R&D center laid out in rigid ways… is like a lover whispering “I love you” in a bored voice while doing a crossword puzzle.
– Becker & Steele

Please do share your thoughts in the comments!

One senior R&D figure described the basic requirements as basically an empty box with a fat Internet pipe. To an extent, he could well be right. Within that box we need to put some specialist facilities – a listening room of some sort, a lab area including soldering space, a viewing room for large displays, to name a few – but what more do we really need? Well, like anything, it’s not quite that simple. Our equipment racks have massive power requirements, and that leads to extensive cooling. In our interim lab we’re not looking at too much, but in Salford we need to be ready to handle an IPTV future, whatever that looks like. It will certainly involve that fat Internet pipe, and probably a lot of servers, routers and other gear. How much gear? Well, who knows. If peer to peer distribution takes off, then maybe less servers will be required than todays iPlayer and other systems. But then isn’t the hot buzz right now all around cloud computing? And that means more datacentres than ever.

Our fat pipe isn’t just for the Internet of course. As a research department, we’re working collaboratively both with our southern lab in London and with academic institutions and industrial partners. So we need to be connected to the universities via the JANET network, and have our own dedicated bandwidth between R&D sites. That bandwidth will carry IP traffic (computer networks) of course, but also possibly video and audio at very high bit rates. All that means one thing: fibre. And here’s the tricky thing: that fibre, that fat pipe, is one of the single biggest costs we face. Fitting acoustically treated spaces like listening rooms into an office style building is the other big one, and that’s a whole different headache.

But it’s not just about talking to the outside world that we need to think about. How do we get signals between engineers’ desks, laboratories and server & apparatus rooms? We’ve been doing a lot of work on the types of cabling we need to install, balancing cost and complexity against flexibility. We need to send audio, video, radio frequency (or RF, which includes off-air feeds from satellite, freeview and more, not just radio!) around the place as well as data. We’d like anyone to be able to do any type of work in any area of the lab too, but if we install lots of video, lots of audio, lots of RF, lots of data and some fibre to every desk, we end up with an immense amount of cabling which we can neither afford nor practically manage. So how do we create flexibility while remaining realistic? We’re pinning a lot of hopes on putting audio and video signals over Cat7 (next generation network cabling) and/or fibre, but the fact remains that we’ll be having some dedicated video and RF cabling because sometimes we simply need to test things based around industry standard connections.

We’re getting there. I’m still hoping we’ll have the interim lab up and running within 3 months or less, and my colleague Michael and I are pushing ahead with that as fast as we can. And MediaCity, whilst further off, is hitting us with deadline after deadline, as the buildings are up and fit-out will shortly begin. BBC R&D will be unrecognisable in 5 years, and I think it’ll be better than ever.

BBC R&D have been responsible for (or at least involved in) many of the broadcast world’s most influential inventions, from colour TV to Nicam stereo, from Ceefax to Freeview. Going forward we’re working on everything from next-generation digital TV platforms to improved HD production workflows, from the technical challenges of being the host broadcaster of London 2012 to 3D television. The established base in Kingswood Warren will be closing next year, with a move to a new location in the South East in the planning stages. But we have also made a commitment to building a lab in Salford when the MediaCity development opens. However even before then we want to get a lab up-and-running in the North West, so plans are underway to enable some staff to move to the existing BBC Manchester site in addition to the handful of us that are here already.

So just how do you start a new broadcast research facility? What equipment do we need? What type of spaces? What acoustic requirements are there, what cooling is needed, what networking is necessary; the list goes on… More to the point, what will we need in 3 years time? Or 5 or 10? You can be sure that the reasearch we’re doing then won’t be the same as the work we have now. How do we produce a space which fulfils our needs now and yet is flexible enough to sustain our requirements in the future?

So many questions to answer, so little time. It’ll be a fun piece of work, so if you’re interested, join me for the ride…

We’ve worked on making sure that metadata survives our distribution chain, but that’s all a bit dull, so I won’t bore you. If you want an entertaining metadata story, read Andy’s blog on our trial of Reverse Karaeoke. What I will share is what happens when we have to create metadata in the delivery chain; normally for a surround sound programme the sound engineer will create the metadata that goes with it, ensuring that the metadata matches the programme content well and the effects of the metadata in your receiver aid the artistic intent of the mix rather than disrupting it. But stereo programmes are delivered to us without metadata, so we have to make it up. What values should we choose? And if something goes wrong somewhere, one of the first things to die could be the metadata, so if that happens we need to create new metadata. Again, what values to choose? The two metadata sets I’ve described here are referred to internally as “stereo metadata” and “reversion metadata”.

Stereo should be fairly easy. BBC HD uses Dolby Digital to send its audio, but SD channels use MPEG2 audio, and they have no metadata. So programmes mixed for stereo-only are mixed to a standard level – they all sound much the same volume hopefully. Therefore we don’t need to set a different dialnorm for each programme, all we need to do is choose one value for all stereo. We currently use -23dB based on imperical testing, and a consistency with other broadcasters who use the same value. Then there’s DRC, or Dynamic Range Control. This one’s a bit more tricky to explain, but basically it allows your receiver to reduce the dynamic range of the audio, which is the difference in volume between the quietest and loudest parts. So DRC makes the quiet bits a little louder, and the loud bits a little quieter. The idea is that we can broadcast programmes with a nice big dynamic range so that those with a high-end audio system can get cinematic effects, while allowing the decoder to reduce the range for those of you listening on small speakers in your telly for example, which won’t be able to produce such a range of volumes. So far so good, but stereo programmes are generally mixed for compatibility with stereo-only channels (i.e. all BBC channels except BBC HD), so they have a small dynamic range in the first place – they’re designed to work on all TVs and audio systems without dynamic range control. So recently, we switched from using a small amount of DRC in our stereo metadata to using none at all. This should ensure that stereo programmes sound the same on any channel, and we’re watching the results carefully.

OK, so what about reversion? Well this is trickier. Remember that this is what happens if things go badly wrong – not something we want to happen, but something we must prepare for. We have to come up with a set of metadata which works for all programmes as best we can, causing the least degredation to the biggest range of programmes so that if reversion happens, whatever programme we’re broadcasting will sound OK. So question one is this: do we tell your decoder that we’re sending 5.1 or stereo audio? The answer has to be 5.1 – if the metadata says 5.1 and a stereo programme is sent, your decoder will just reproduce the left and right channels in the left and right speakers. Any fancy Dolby Pro-Logic decoding won’t work, but you will hear the basic audio. If we did the opposite and signalled the programme as 2.0 (stereo), a 5.1 programme would be badly degraded, as you wouldn’t hear the centre and rear channels, which would probably mean you wouldn’t hear the dialogue!

The DRC is the next question, and a relatively easy one – we stick with the default setting, which applies quite a lot of dynamic range processing. This will make sure that any 5.1 programming comes out of your speakers in a way that works for all programming and all speakers, even if it doesn’t sound so impressive on high-end systems. And while stereo programming might be affected a bit, it won’t seriously degrade the audio. The final important question is the dialnorm. Whatever happens, if the dialnorm doesn’t match the programme, you’ll hear the sound either too loud or too quiet. Since not all programmes are at the same level, there is no ‘perfect’ value to choose, we have to simply make a best guess. The choice we made is to use a dialnorm of -23dB, the same as for stereo. What this means is that stereo programmes should sound normal, while surround sound programmes will likely sound too quiet (by a varying amount depending on the programme). Again, we based this decision on the least-worst effect it would have; surround being too quiet is less bad than stereo programmes being too loud (which would have been the other option) as people generally find things jumping up in volume more annoying.

So there you go, that’s metadata for you. We think we have a pretty strong system now, so that all surround and stereo programmes reach you with the best metadata settings possible, and even if things go wrong the results should be pretty good. We’ve also used some tricks with metadata to help us identify the source of a problem if one occurs, so as well as sounding better if things do go wrong, we can fix the problem faster. Some of you may be disappointed to hear it, but I don’t think we’ll be having any more Reverse Karaeoke!

So that’s it from me! If anything particularly exciting happens in the world of BBC HD’s sound, I’ll try to let you know. And as I move up north to help develop a new Research and Development lab I’ll try to tell you a bit about that too, as I think it’ll be an exciting journey. Watch out for a new series of posts from me about that on this website, and for updates on BBC HD and the BBC’s wider technology work, keep reading BBC Internet Blog. Cheers!

If you read the comments on this blog, you’ll recall that we’ve been having problems with the lipsync of stereo programmes on BBC HD. Surround sound programmes have been fine since the previous work I’ve done, but stereo was not working quite right. It turns out that our broadcast chain introduces around 15ms of audio delay on stereo programmes, which is normally too little to notice. However if a programme is delivered with audio slightly late, as recently happened with Hustle, the two small and virtually invisible delays add up to create one much larger and very noticeable delay. So the question has been, where is that 15ms coming from, and how can we remove it? I’ve spent a long time looking at this over the last few weeks, and I’ve been tearing my hair out as every bit of testing we’ve done has given results that we couldn’t quite explain, until yesterday when things became clear.

I’ve mentioned before that we use Dolby E to encode audio around our broadcast infrastructure. It’s a technology that allows us to fit up to 8 channels of audio (plus some extra metadata) into the data space which usually fits just 2 audio channels. The difficulty lies in the fact that surround sound programmes arrive at the BBC with Dolby E encoded audio, whereas stereo programmes use uncompressed audio. The same broadcast infrastructure has to handle both types of audio with the same latencies (delays). Dolby E takes time to decode (1 video frame or 40ms to be precise) so we delay the video by the same amount to compensate and ensure that they stay in sync. But the stereo must therefore be delayed by the same amount, otherwise that video delay we’ve added will cause stereo programmes to be out of sync. The question you’re probably asking by now is “so what went wrong on BBC HD?”. It’s a good question, dear reader, and it has a simple and a complex answer. The simple version is this; we thought that the delays through each piece of equipment were the same for stereo and surround programmes, but our testing proved that this wasn’t the case. Finding which device had the problem and why was more challenging, and that forms the long answer…

Before I continue, I’ll tell you what happens now. The details of the problem, and therefore the solution, finally became clear only yesterday. We used a duplicate set of equipment to do tests without affecting broadcast chain. The solution we found requires some pretty major changes, so we’re waiting now on authorisation from the relevant people to allow us to make the changes in the broadcast chain. I’m sure you’ll appreciate that if any old engineer was allowed to change the on-air equipment in fundamental ways, we’d be a bit open to disastrous mistakes; so you’ll have to allow us a couple of days to get the change done. Soon enough though, we will get the changes on air and we should have properly in-sync stereo as well as surround programmes, rather than only “roughly in-sync” stereo. Phew!

Those who just want to know what we’re up to and when things will be better will hopefully be satisfied with what I’ve said so far. The hardcore tech-heads amoung you may want to know more however, so I’ll do my best to explain… When a programme arrives in the mixing part of the signal path (be it from a playout server or from a live contribution), the audio is embedded in the video. In other words, there’s one cable which carries both audio and video in a digital data stream. So the first thing that we do is de-embed the audio. We take the SDI video stream and extract the audio, spewing it out on a separate digital audio interface. This then goes to a Dolby E decoder, which decodes Dolby E if present, or just passes through the audio unchanged if it’s uncompressed. At this point the audio goes through the mixer, then gets Dolby E encoded, embedded again and continues on its way. It’s the de-embedder and the decoder that we’re interested in here, as they make up the bit of the signal path which is able to behave differently depending on whether the incoming audio is PCM (uncompressed) or Dolby E. My initial suspicions were around the decoder, which has a configuration option for how it handles PCM audio. It passes it through unaltered, however it can delay it by one of two amounts; “minimum” or “single frame”. The single frame option is what we want, as it delays the audio by the same amount as a Dolby E stream would be delayed, thus keeping everything in sync. However we have been using “minimum” because even in this mode the PCM ends up being a bit too late, so setting it to “single frame” just makes things worse. But why? I shan’t tell you all the possibilities we checked and the false leads we had, instead I shall cut to the chase… it was the de-embedder’s fault!

Don’t you just hate it when electronic devices are too smart for their own good? When your word processor magically changes (c) to a copyright symbol and actually you just wanted a c in brackets? Well, our de-embedder was being way to smart for its own good. We had been under the impression that it was applying a frame of delay to all audio. This is much more than it needed to, but by selecting a one frame delay it becomes predictable and easy to manage. Excellent, thought I. The problem is that it wasn’t that simple. After a lot of settings checking, head scratching and examination of the somewhat cryptic manual, it turned out that all audio was not being treated equal. We inadvertently had the device configured to delay the PCM audio by 1 frame but not the Dolby E! The bright idea behind this option in the de-embedder is to allow you to compensate for the decode delay of the Dolby E by similarly delaying PCM and/or the video. However, as I previously mentioned, we were already applying a compensating delay in the Dolby decoder itself. So we had 2 sets of delay where we only wanted one! Then even with the Dolby decoder set to apply “minimum” delay to PCM audio, it seems to add about 15ms of delay – a familiar figure! So what we shall be doing is turning off all delay in the de-embedder and setting the “single frame” delay mode in the decoder. Job done… I hope.

Hopefully I’ve given some insight into the problems we’ve had, and – touch wood – we should be all fixed soon. You’ll likely be unable to see the difference when we do make the change, because you should be unable to see the sync error currently due to it being so small. However, next time we have a programme delivered with a tiny lipsync error it should stay tiny and impossible to notice, rather than being amplified by the broadcast chain into a serious and visible error.

A final note just to say that I come to the end of my time working for BBC HD soon, I shall be moving on to another project. I’ll aim to do one more post before I leave to tell you a little about some other things we’ve been doing, so expect a post next week.

The amount of time available to write in a diary/blog is inversely proportional to the amount of things worth writing in it.

In other words, I’ve been really busy. I have all sorts of news about BBC HD Audio, and lots of ideas buzzing around for personal posts too. I promise I’ll update as soon as I can, and that I’ll do my best to make it worth the wait!

Thanks.

I told you in my previous post that we needed to check every element of the signal chain in order to ensure that we broadcast this signal completely in-sync. We started by getting the sync test itself into the promo loop which runs on BBC HD during the day. Here’s a slide from a presentation I have last week on how we did this:

Given that we had the sync test itself on tape and tested in-sync, we then put this into the edit suite. (In the interests of BBC neutrality I ought to point out that this particular suite is an Avid, though we do use other software too! In fact, a Final Cut Pro suite was used extensively in producing the sync test in the first place.) We then took that to the dub, which is the part of the editing process where the audio is tweaked, finished and finalised. This is the only place in the editing chain where we can move the audio by sub-frame (i.e. millisecond) increments. We then went back to tape, having encoded the audio in Dolby E. Back to the edit suite we go (3 floors down), where we can play out the tape and examine the sync signal on a CRT. This is necessary because the projector and other displays in the dubbing suite are not guaranteed to be perfectly in sync. We used a clever little box with a light sensor and an audio input to check what the sync offset is by examining the white flash at the top of the screen.

Armed with a number of milliseconds of sync offset, we headed back to the dub, moved the audio to correct for the offset, and repeated the process. We went round this loop a few times before getting it perfect, and it’s here that I shall hang my head and beg the forgiveness of my mother who’s a mathematician, because at least one extra iteration was required because I messed up the conversion between milliseconds (which the measuring device works in) and 100ths of a frame (which the dubbing suite works in). Oops! However, I eventually came away armed with the December BBC HD promo which included a sync test which we knew was in sync.

The next part of the puzzle was the broadcast chain:

We ingested the tape into our playout servers, run by Red Bee. We then played the clip out and measured the sync on the server’s output. It was within 1ms, which was great news. From there we checked a few more places through the chain, ending up in CCA, the Central Communications Area in Television Centre. This is the last point in the chain before we encode the video for broadcast, and we discovered a delay of no more than 2ms. Great news!

Shortly after, we took the file from the playout server to Kingswood Warren, where we have a duplicate setup of the encoding chain. We ran the test through the encoders and found that they introduce a few milliseconds of audio delay. However, they have a configuration parameter which allows you to delay the incoming video. We already use this parameter to correct for the delay introduced to the audio by Dolby E decoding and Dolby Digital encoding, so all we needed to do was change this number. We sent this change request to our technology partner Siemens, who implemented the change. We were then good to go, and arranged a one-off broadcast of the test signal.

The reason we do this is so we can test that the signal transmitted really is in sync. As Andy explained in his blog, the usual broadcasting process is to test that a signal is “OK leaving me”, in other words it is correct when we send it. However we wanted to go further here and ensure that it is “OK arriving at you”, meaning that when you receive it from a satellite dish the signal is definately in sync. This means you can be sure that any offset you find is introduced by your equipment and not our broadcast. We were very pleased to find that the offset was 0.9ms, which is very tiny indeed. This made one very happy Rowan last week.

However nothing’s ever that simple, is it?! We found that the sync test caused a problem with our output encoders. The eagle eyed amoung you will notice that in my last post the screenshot of the sync test was green, whereas the version that we now transmit is grey. (The white flashes are and have always been white, obviously!). The short version of the story is that a certain part of the green signal caused a problem in the MPEG4 encoder which caused magenta (purple-ish) flashes to appear on the video. This was obviously no good, so we had to ammend the signal. However don’t panic, the new colouring doesn’t affect how you use the signal to line-up your sync!

So how do we check the off-air signal is in sync? After all, if we decoded it and used the normal sync checker to examine a CRT display, we could get an inaccurate measurement, as the decoder could intoroduce a delay all of its own. However, there’s a more reliable method.

• First we capture the bitstream onto a computer.
• We can the use a transport stream analysis tool to decode the individual frames of video, and find the frame with the first white line flash. Each frame in the broadcast stream has a Presentation Timestamp (PTS), so we note the PTS for that frame.
• We then use the analysis tool to decode the Dolby Digital audio to an analogue waveform displayed on screen. Next, find the start of the audio clap in this waveform. Unlike the video frame there will not be a PTS just for this audio sample, the PTS will cover a large number of samples. So we find this PTS and note it down and then count how many audio samples there are from this PTS to the sample at the start of the audio clap – we refer to this as the audio offset.
• By subtracting one of the PTS values from the other and taking into account the audio offset, the A/V timing in the stream can be determined. This is all done by a colleague at Kingswood, but given the amount of arithmetic involved, and the widely know inability of engineers to do simple maths (as proved by me earlier), he always gets the values sanity checked by me or someone else! (If you’re interested, the PTS values are in units of 90kHz and the audio will have been sampled at 48kHz, just to make everything complex!)

So there you go, that’s how we get a signal in sync to arrive at your home. After changing the colours, we of course had to ingest the clip into our playout servers all over again, so we did a new off-air sync test to ensure that nothing went wrong in the process of making the change, and I’m pleased to say everything went well!

So if you have an HD receiver and your sound is connected to a separate amplifier, get yourself on over to Andy’s blog and find out how to test your setup. After Christmas I’ll let you know how we’re getting on with other aspects of my work, like improving the way we test incoming HD signals from outside broadcasts prior to transmission. Have a great Christmas!

Why? Well think back to your GCSE (or O-Level!) Physics lessons. Sound travels much slower than light, so you are used to hearing the sound of an event slightly after seeing it, particularly if it happens far away. Just try going to a gig in a stadium, and you’ll notice that if you’re at the far end of the stadium from the stage, the singer’s lips will be moving well before you hear the sound. So audio being late, whilst undesirable on TV, is much more tolerable than the video being late. The latter scenario looks very unnatural indeed with even quite a small error.

Most HD set-top boxes allow you to adjust the audio/video sync on the outputs, usually in 20ms steps. However just by watching TV programmes, it’s hard to judge when the sync is correct. As such, we wanted to do something to help our viewers to get their own setups correct – some of you have been asking for this in comments on our blog posts, and we suspect more will be interested in trying it when given the opportunity. So we’ve been devising a sync test which will be broadcast a few times during each day on BBC HD, starting in a week or two. The test signal is based on work from Andrew Mason, Oliver Haffenden, David Kirby and Alastair Bruce at BBC R&I, and it should allow you to adjust your set-top box to an acceptable level of sync just by watching the sequence and tweaking your settings.

If you want to get even more precise, you can make your own sync test device! The signal has been designed to be easy to monitor with a simple electronic circuit, using a light probe to sense a flash which appears on the screen once a second and an audio input to listen for a ‘clap’ which happens at the same time. The device can then tell you the time offset between the two and hence you can adjust your set-top box to compensate. Details will be posted on the BBC website. The ‘clap’, incidentally, is a recording of two bits of wood being snapped together; believe it or not we found this at least as effective as more ‘high-tech’ alternatives such as a brief burst of tone.

My job in all this has been to ensure that the test signal is in-sync when it gets broadcast, which is trickier than it sounds. Of course we aim to ensure that everything we transmit is in-sync however it’s absolutely impossible to ensure that sync is perfect, not least because the delays introduced by some equipment vary over time. Within a small tolerance of a few milliseconds, the difference is imperceptible anyway so it’s not a problem. The EBU defines standards for sync that say that when a programme is delivered to a broadcaster like ourselves, audio should be within +10 to -20ms (i.e. no more than 10ms ahead or 20ms behind the video). Of course, to ensure that the signal is within sensible tolerances when a programme actually gets broadcast, each step of the chain has to have sync errors much smaller than this, otherwise the delays could add up to a much larger a total error. When transmitting a signal whose express purpose is to be in-sync we’d like the sync to be even tighter than usual, which means making sure that every step of the chain is as close to ideal as it can be.

The signal started life on an edit suite, and so job number one is to ensure that when it comes off the edit suite and on to tape it stays in sync. Along the way we have the audio encoded to Dolby E, so we’ve already got 3 bits of equipment in play: the edit system, the Dolby encoder and the tape deck. So we do that, check for any offset, correct it in the edit suite and then get back on to tape again. But wait… how do we check the sync? I’m armed with our sync-checker, but in order to use it, we have to decode the Dolby E and play the video out to a monitor, both of which could introduce their own delays! So we have to ensure we know the delays of each component in the chain, isolating them one by one, before we can rely on our measurements. Rule number 1 of testing is that you must ensure that your test equipment isn’t affecting the results, or at least that the effect is known. In this case we can’t test without having an effect, but isolating that effect allows us to get accurate measurements. It means we have to be really careful though – forget one source of sync error and you mess up all your results! That’s my justification for the terrible pun that titles this post…

Then there’s the signal chain to get the test sequence to air. We ingest the tape onto our playout servers (this process could, of course, introduce sync error), then play it out though a presentation mixer and some processing gear then down some fibre optics to Television Centre. From there it goes on through to the Coding and Multiplex centre where the signal is prepared for digital broadcast. All that could introduce a sync error, as could the coding process itself; the audio and video are coded separately and multiplexed back together, then multiplexed in with other channels for broadcast. Argh! So I’m going to work with staff from Red Bee and Siemens (our technology partners in these areas) to run the signal through the off-air chain, a backup set of equipment and connections used if the main signal chain goes down, and also in testing situations like this. We’ll measure the sync error through this whole set of equipment, and if necessary adjust the offset at the encoders to get everything back into sync.

I’m fairly confident that there shouldn’t be a big sync error in this signal chain, as it’s been tested before and we’re broadcasting with it every day, so major errors would be noticeable. However as I mentioned previously, our tolerances for broadcasting this test signal are much tighter than usual, so it will be a great chance to check that everything’s working as well as it can possibly be. Hopefully all will go well, but you never know – I’ll let you know next week!

Last week was a bit of a busy one for me, dashing around between Kingswood Warren (the home of BBC R&I), Sky and a trip to the BBC’s facility in Glasgow (as well as having just returned from a long weekend in Devon) but I just wanted to share with you some of what I got up to last Tuesday. I spent the afternoon and evening with the sound team of Later Live with Jools Holland at Television Centre. I was there to look at their surround sound production and the technology setup they have. It was an interesting insight into the programme maker’s view on all this surround stuff, and while there is considerable enthusiasm from the team, it does add to their workload…To take just one aspect, let’s think about monitoring. This is the process of listening to (or watching) your own output to ensure that it’s as expected. Back in the days of mono, it was easy. Stereo got a bit more tricky, but a pair of speakers or a decent set of headphones and you’re well on your way. Surround makes things rather trickier.

First off, to listen to a surround sound mix, you need a surround sound system. Sounds obvious, but think not only about the equipment cost but also of the space required to construct such a system. Of course, a very large number of listeners will downmix the surround sound to a stereo output because not everyone has a surround setup, so not only must we monitor the 5.1 mix but also the stereo downmix of the 5.1 mix. That in turn is separate to the ‘native’ stereo mix which goes to BBC 2, where no surround is involved.

Then there’s metadata. This ‘data about data’ controls some elements of how your surround sound receiver decodes the audio, so ensuring the correct metadata is vital, as we know only too well. Therefore the monitoring requirements also include 2 metadata components; checking the metadata we’re sending out is as we intended, and checking how the audio sounds after the metadata effects are applied to the decoding. Finally, we really ought to be checking during a live programme that what makes it to your set-top box is what we’re sending out, so we aim to do ‘off-air’ monitoring too.

This means a total of at least 4 audio mixes to listen to as well as metadata checks to be done. You’ll be pleased to hear that Later does indeed do all these checks, and so we can be very confident that the audio you receive as a viewer is as the sound supervisor intended. The Later team were instrumental in getting a surround sound suite set up at Television Centre to allow monitoring and in some cases mixing to be done in a dedicated environment rather than the crowded sound gallery. But all this costs money, needs training for the staff involved and adds complications, so ensuring it’s done right for all shows all the time is a challenge. Imagine doing all this on an outside broadcast, and you’ll see that the challenge there is greater still. It’s one we’re facing head-on however, and the availability of new testing equipment is helping. In Television Centre’s surround room for example they now have a very useful little box which allows us to examine the metadata they’re sending out and be sure it’s correct, rather than assuming that the metadata authoring equipment is doing it’s job. Assumption is, after all, the monther of all mess ups…

It became clear pretty quickly that synchronisation is a big deal in the world of Dolby E. Really there’s 2 issues here:

• Traditional signal synchronisation, whereby video and audio signals’ timing is adjusted to match a common reference, causes big problems in a Dolby E environment.
• Like any coding system, the encoding and decoding of Dolby E has latency involved. Therefore matching the delays between video and audio paths becomes critical to ensuring accurate lip sync.

So right now that’s what I’m focussing on. Already I’ve discovered that we’ve got a number of gremlins in the system, but the folks at Red Bee and Siemens are working hard to make things better. In the mean time I’m getting under their feet asking for piles of information about their equipment setups, but hopefully also contributing in a helpful way too.

Those of you who watch BBC HD will shortly be seeing the fruits of my labours on air in fact. A short while ago I was discussing the signal chain in one of the many technical areas that our signal passes through, and I walked away with some schematics. I took them back to my desk and quickly found a problem… the processing delays in the audio chain weren’t being correctly compensated for in the video chain, meaning that all BBC HD output had its audio a frame behind the video. In other words, the lip sync was off by 40ms. This isn’t exactly what we want! I won’t name and shame, especially because the blame can’t necessarily be attributed to any individual or organisation, but suffice to say that it was more of a paperwork mess-up than a  technical one. It transpires that the technical area in question was working to an old version of our working practices, which specified a frame of offset was to be kept between audio and video until point of transmission. This means that you can whack the audio through a decoder (with 1 frame of delay) and get sound out of it that was in sync with the video. We now work in a different way however; audio and video should always be kept in sync, and the person decoding the audio has the responsibility to delay the video to match. This ought to be less confusing, but it seems it didn’t work out that way!

What made this issue even more tricky to spot previously was that in the early days of BBC HD (remember it started as a trial service on a shoestring budget and only recently became a fully-fledged BBC service) we had problems with video kit introducing unpredictable delays. In particular, equipment working flat out was producing delays which actually varied over time, making accurate measurement and compensation a nightmare. So with video delays being introduced and no real way to be sure of what they were, an extra 40ms delay in the audio chain (as I spotted) was not only difficult to notice, but may have actually been helping us until recently! Andy Quested (Head of technology for BBC HD) commented in a response to his own blog post. Now we’ve upgraded some of the kit, we’re starting to iron out these issues, and hence come across new ones like one I found here. Since it was found, Andy undertook some consultation with colleagues at BBC R&I in Kingswood Warren, and a change to correct this problem has now been agreed and will be on air very soon. Phew! [Update - it's been changed, so the sync should now be improved.]

Meanwhile in our playout centre, I’ve been running around with Red Bee’s engineers trying to understand the timing issues there. They’ve put in a great deal of effort to attempt to make sure everything is OK, and have done lots of good work.  Their synchronisers are mostly a model which performs a “Dolby E re-align” process before synchronising, which makes sure the Dolby audio data is in-sync with the video data before worrying about synchronising the whole lot with an external reference. This ensures that the audio doesn’t get mangled in the process, so there’s a big tick in that box. However they’ve had issues with what happens between programmes when switching sources (i.e. from programme to programme or to/from trailers). If you pay very close attention, you may notice quiet clicks on the audio or brief silences, usually around half a second after we transition from one video source to another. We’re working through these issues together with some input from Dolby, and I’m hoping to get to the bottom of things soon, at which point I’ll explain more.

Beyond simply problem-solving, my task here is really to help prevent future problems and collate some understanding about the use of multichannel audio technologies in order to improve our operations in future. In Red Bee, the lip sync seems to be pretty much spot on after extensive testing that they’ve done. However what’s happened is that they’ve done some tests on how delayed the audio gets compared with the video and then delayed the video to compensate. This works just fine, but it makes part of me a little nervous. It’s here that you’ll see the difference in approach between myself and the playout engineers; they have a service to keep on air and they want it to look and sound as good as possible. They do that well, but my priority is different; where they want to know that things to work, I want to know why they work. (Actually, they want that too, it’s just priority number 2 for them and priority number 1 for me!) I want a diagram with all the bits of equipment and their associated delays, and I want it to add up! I’m in the process of drawing that diagram, but as yet I can’t quite make it add up. I’m catching up with my colleagues there tomorrow to do some arithmetic, and you never know, we might get somewhere.

Of course there’s lots I haven’t told you here, but I don’t want to bore you. More detail will come soon, as well as some information on the other questions I’m trying to answer, such as “just what do you do when you want to put Dolby E into a file-based workflow?”.  Feel free to comment below.

For high definition production, we at the BBC use a technology called Dolby E to squeeze up to 8 channels of audio down a digital connection designed to take 2 channels (AES-EBU). This allows us to incorporate surround sound into our existing stereo workflow with far less investment in new technology, which has to be a good thing. Dolby E uses a virtually lossless codec, which means the degradation to the sound quality should be inaudible (unlike Dolby Digital, the format used to actually transmit the signal to your set top box, which is lossy). So far, so good?

Dolby E and Dolby Digital also carry a bunch of metadata (literally “data about data”) which describes the audio being carried, and importantly controls the output at the end of the chain. So if you’re listening in stereo rather than 5.1 surround, the metadata may have an effect on how the surround is converted to stereo. And have you ever noticed how adverts and trailers on TV are often louder than programmes? Well that’s a contentious issue, and something that we’re trying to avoid. Dolby E/D ideally should help us with that by using… you guessed it… metadata. So you can see that we have to get the metadata right, and that’s a new issue for the TV world, making content generation more complex than it was in the old stereo days.

Then there’s the amount of kit involved in the signal chain which can mess things up. As an example, many consumer decoders (your surround sound amplifier or receiver) have issues when we switch between stereo and 5.1 output. This results in a loud, short ‘splat’ (around 40ms) which won’t be particularly good for your speakers. On the broadcaster’s side of things, there are issues like timing problems to be aware of. When it’s embedded in a video signal, Dolby E is designed such that the audio is inserted at the same frame rate as the video. However if the exact synchronisation of audio and video isn’t correct, equipment can interpret it as standard linear audio instead of a Dolby E stream, which means you get very high values of data being converted into audio which equates to more loud splats.

In these modern days of the BBC, we have outsourced a lot of our infrastructure, so the signal chain is complex. A programme will be originated by an outside broadcast company, BBC Resources or another studio company, then sent to Siemens who run our central communications area. From there it goes to Red Bee who do continuity and so on (“Coming up next on BBC One…”) before heading back to Siemens to be encoded and finally to a transmitter run by someone else. It all works remarkably well, but in a project like mine it means I have a lot of different parties to communicate with. My main focus right now therefore is in communicating with all these parties and finding out what their part of the chain looks like, what issues they have, and what would help them in the future. Later on I’ll be looking into better ways to test our systems, specifically full end-to-end tests that will allow us to check the whole chain and identify problems if they occur.

More detail will come in time, so if you’re interested in the topic you can subscribe to the RSS feed or watch the category page. I’ll be sharing my experiences in addressing the problem and letting you know what we’re doing to avoid problems in the future and fix problems we’ve had in the past. After all, no-one wants a re-occurrence of our experimental reverse karaoke broadcast! Feel free to keep in touch via the comments.