Media Production, Delivery and Interaction for Platform Independent Systems (eBook)

Format-Agnostic Media

Oliver Schreer, Jean-François Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger, Graham Thomas (Herausgeber)

eBook Download: PDF | EPUB

2013
John Wiley & Sons (Verlag)
9781118706336 (ISBN)

Infos

Presents current trends and potential future developments by leading researchers in immersive media production, delivery, rendering and interaction

The underlying audio and video processing technology that is discussed in the book relates to areas such as 3D object extraction, audio event detection; 3D sound rendering and face detection, gesture analysis and tracking using video and depth information. The book will give an insight into current trends and developments of future media production, delivery and reproduction. Consideration of the complete production, processing and distribution chain will allow for a full picture to be presented to the reader. Production developments covered will include integrated workflows developed by researchers and industry practitioners as well as capture of ultra-high resolution panoramic video and 3D object based audio across a range of programme genres. Distribution developments will include script based format agnostic network delivery to a full range of devices from large scale public panoramic displays with wavefield synthesis and ambisonic audio reproduction to ’small screen’ mobile devices. Key developments at the consumer end of the chain apply to both passive and interactive viewing modes and will incorporate user interfaces such as gesture recognition and ‘second screen’ devices to allow manipulation of the audio visual content.

• Presents current trends and potential future developments by leading researchers in immersive media production, delivery, rendering and interaction.
• Considers the complete production, processing and distribution chain illustrating the dependencies and the relationship between different components.
• Proposes that a format-agnostic approach to the production and delivery of broadcast programmes will overcome the problems faced with the steadily growing number of production and delivery formats.
• Explains the fundamentals of media production in addition to the complete production chain, beyond current-state-of-the-art through to presenting novel approaches and technologies for future media production.
• Focuses on the technologies that will allow for the realization of an E2E media platform that supports flexible content representations and interactivity for users.

An essential read for Researchers and developers of audio-visual technology in industry and academia, such as engineers in broadcast technology companies and students working toward a career in the rapidly changing area of broadcast both from a production and an engineering perspective.

Autor

Oliver Schreer, Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany Dr. Schreer is Scientific Project Manager of the "Immersive Media & 3D Video" Group in the Image Processing Department at HHI, Berlin. He is responsible for the lead of European Research projects (e.g. FascinatE project) and coordinating research on 3D video processing and immersive media. Graham Thomas, BBC Research & Development, London, UK Dr. Thomas is Principal Research Engineer and Section Lead of the Production Magic department at BBC R&D. He leads a team of engineers developing 3D image processing and graphics techniques for TV production. His work has led to many commercial products, e.g. the Piero sports graphics system, which won a Queen's Award in 2011. He holds 20 patents and is Visiting Professor at University of Surrey. Ben Shirley, University of Salford, UK Ben Shirley is Senior Lecturer at the University of Salford and Academic Lead for the Digital Innovation Cluster in the University's College of Science & Technology at MediaCityUK. He has been principal investigator for a range of broadcast related research projects including leading Salford's work in the EU FascinatE project. He teaches audio technology and broadcast engineering at undergraduate and postgraduate level. Georg Thallinger, Joanneum Research, Graz, Austria Dr. Thallinger is Head of the research group "Audiovisual Media" at JOANNEUM RESEARCH, Austria. He oversees a group active in the field of digital media with a focus on video and image processing as well as metadata for applications in media production and archiving, film restoration, media monitoring and security. Dr. Thallinger is involved in the EU FascinatE project. Jean-Francois Macq, Alcatel-Lucent, Antwerp, Belgium Dr. Macq is a Senior Research Engineer at Bell Labs, the research organization of Alcatel-Lucent. He has been responsible for several national and international research projects in the field of video coding and delivery, currently including the EU FascinatE project. Javier Ruiz-Hidalgo, Universitat Politècnica de Catalunya (UPC), Barcelona, Spain Dr. Ruiz-Hidalgo is an Associate Professor at the Universitat Politècnica de Catalunya. He has been involved in various European Projects as a principal researcher from the Image Processing Group at UPC, including the EU FascinatE project. Omar Niamut, Netherlands Organization for Applied Scientific Research (TNO), Delft, The Netherlands Dr. Niamut is a Senior Research Scientist in the "Media Networks and Services" Group at TNO. He has advised the European Parliament on the harmonization of mobile TV and was editor of the technology report on mobile TV in the European Commission-assigned European Mobile Broadcast Council. He participated in ETSI TISPAN standardization with over 200 contributions on interactive services for next generation IPTV and advised the Singapore government on the use of IPTV standards. He has been involved in a number of European projects, including the Fascinate project. Dr. Niamut holds 15 patents.

Inhalt

List of Editors and Contributors xiii

List of Abbreviations xvii

Notations xxiii

1 Introduction 1
Oliver Schreer, Jean-Francois Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger and Graham Thomas

2 State-of-the-Art and Challenges in Media Production, Broadcast and Delivery 5
Graham Thomas, Arvid Engstrom, Jean-Francois Macq, Omar Aziz Niamut, Ben Shirley and Richard Salmon

2.1 Introduction 5

2.2 Video Fundamentals and Acquisition Technology 7

2.3 Audio Fundamentals and Acquisition Technology 21

2.4 Live Programme Production 34

2.5 Coding and Delivery 43

2.6 Display Technology 50

2.7 Audio Reproduction Technology 56

2.8 Use of Archive Material 62

2.9.1 Limitations of Current Production and Delivery Approaches 65

2.10 Conclusion 68

3 Video Acquisition 74
Oliver Schreer, Ingo Feldmann, Richard Salmon, Johannes Steurer and Graham Thomas

3.1 Introduction 74

3.2 Ultra-High Definition Panoramic Video Acquisition 75

3.3 Use of Conventional Video Content to Enhance Panoramic Video 94

3.4 High Frame Rate Video 102

3.5 High Dynamic Range Video 112

3.6 Conclusion 125

4 Platform Independent Audio 130
Ben Shirley, Rob Oldfield, Frank Melchior and Johann-Markus Batke

4.1 Introduction 130

4.2 Terms and Definitions 132

4.3 Definition of the Problem Space 135

4.4 Scene Representation 144

4.5 Scene Acquisition 149

4.6 Scene Reproduction 153

4.7 Existing Systems 156

4.8 Conclusion 161

5 Semi-Automatic Content Annotation 166
Werner Bailer, Marco Masetti, Goranka Zoric, Marcus Thaler and Georg Thallinger

5.1 Introduction 166

5.2 Metadata Models and Analysis Architectures 170

5.3 Domain-independent Saliency 177

5.4 Person Detection and Tracking 180

5.5 Online Detection of Concepts and Actions 189

5.6 Supporting Annotation for Automated Production 195

5.7 Conclusion 204

6 Virtual Director 209
Rene Kaiser and Wolfgang Weiss

6.1 Introduction 209

6.2 Implementation Approaches 219

6.3 Example Architecture and Workflow 225

6.4 Virtual Director Subprocesses 230

6.5 Behaviour Engineering: Production Grammar 237

6.6 Virtual Director: Example Prototype 243

6.7 Conclusion 251

7 Scalable Delivery of Navigable and Ultra-High Resolution Video 260
Jean-Francois Macq, Patrice Rondao Alface, Ray van Brandenburg, Omar Aziz Niamut, Martin Prins and Nico Verzijp

7.1 Introduction 260

7.2 Delivery of Format-Agnostic Content: Key Concepts and State-of-the-Art 262

7.3 Spatial Random Access in Video Coding 267

7.4 Models for Adaptive Tile-based Representation and Delivery 276

7.5 Segment-based Adaptive Transport 281

7.6 Conclusion 294

8 Interactive Rendering 298
Javier Ruiz-Hidalgo, Malte Borsum, Axel Kochale and Goranka Zoric

8.1 Introduction 298

8.2 Format-Agnostic Rendering 299

8.3 Device-less Interaction for Rendering Control 311

8.4 Conclusions 331

9 Application Scenarios and Deployment Domains 337
Omar Aziz Niamut, Arvid Engstrom, Axel Kochale, Jean-Francois Macq, Graham Thomas and Goranka Zoric

9.1 Introduction 337

9.2 Application Scenarios 338

9.3 Deployment in the Production Domain 340

9.4 Deployment in the Network Domain 347

9.5 Deployment in the Device Domain 351

9.5.1 Device Capabilities 351

9.6 Deployment in the User Domain 356

9.7 Conclusion 357

References 357

Index 359

Textauszug

1

Introduction

Oliver Schreer1, Jean-François Macq2, Omar Aziz Niamut3, Javier Ruiz-Hidalgo4, Ben Shirley5, Georg Thallinger6 and Graham Thomas7

1Fraunhofer Heinrich Hertz Institute, Berlin, Germany

2Alcatel-Lucent Bell Labs, Antwerp, Belgium

3TNO, Delft, The Netherlands

4Universitat Politècnica de Catalunya (UPC), Barcelona, Spain

5University of Salford, Manchester, United Kingdom

6Joanneum Research, Graz, Austria

7BBC Research & Development, London, United Kingdom

The consumption of audio-visual media has changed rapidly in the past decade. Content is now viewed on a variety of screens ranging from cinema to mobile devices. Even on mobile devices, today's user expects to be able to watch a personal view of a live event, for example, with a level of interactivity similar to that of typical web applications. On the other hand, current video and media production technology has not kept up with these significant changes. If we consider the complete media processing chain, the production of media, the delivery of audio-visual information via different kinds of distribution channels and the display and interaction at the end user's terminal, many challenges have to be addressed. The major challenges are the following.

Due to reuse of video content for different distribution channels, there is a need for conversion and post-production of the content in order to cope with different screen sizes. It is widely accepted that a movie production for cinema is recorded in a significantly different way to that intended for smaller screens. However, production budgets are limited; hence complex and costly re-purposing must be avoided. A good example is the production of 3D movies, where the aim is to develop camera technologies that allow 2D and 3D capture at the same time. Approaches to multiformat production that require parallel shooting or significant manual re-editing are no longer financially viable.

The convergence of broadcast and Internet requires future media production approaches to embrace the changes brought by web-based media. The habits of media consumption have changed drastically, partially due to the availability of user interaction with users freely navigating around web pages and interactively exploring maps and views of the street for example. Hence, future media production and delivery must support interactivity.

Although the overall bandwidth available for media delivery is continuing to increase, future media services will still face limitations, particularly if the end user at home or on-the-go is considered. Hence, new distribution formats are required to allow for the provision of audio-visual media beyond current HDTV formats, to support interactivity by the end user and to support intelligent proxies in the network that are capable of performing processing, which cannot be offered by low capacity devices. First developments towards resolution beyond HD are already appearing commercially, such as 4K camera and display technologies.

In addition, the user wants to decide when, where and on which device to watch audio-visual media as nowadays a variety of devices are available (including mobiles, TV at home and immersive large projection systems in cinemas). All of these devices must be supported by media delivery and rendering. Therefore, a large variety of audio-visual formats must be provided for the full spectrum of terminals and devices taking their special capabilities and limitations into account.

Even in live events, a lot of human operators such as directors or cameramen are involved in content creation and capturing the event from different viewpoints. Due to the increasing number of productions, automated viewpoint selection may be able to make a significant contribution to limiting production costs.

A new concept appearing on the horizon that could provide answers to these issues and challenges is referred to as format-agnostic media production. The basic idea is to define a new approach to media production that supports the necessary flexibility across the whole production, delivery and rendering chain. A key aspect of this approach is to acquire a representation of the whole audio-visual scene at a much higher fidelity than traditional production systems, and to shift closer to the user-end the decision of how the content is experienced. This idea allows end users to experience new forms of immersive and interactive media by giving them access to audio-visual content with the highest fidelity and flexibility possible. This book discusses current challenges, trends and developments along the whole chain of technologies supporting the format-agnostic approach. This approach could lead to a gradual evolution of today's media production, delivery and consumption patterns towards fully interactive and immersive media.

In Chapter 2 “State-of-the-art and Challenges in Media Production, Broadcast and Delivery”, we give an overview on the current situation in audio-visual acquisition, coding and delivery and the evolution of terminal devices at the end-user side in current media production and delivery. Based on the review of the state-of-the-art and a summary of current and upcoming challenges, the format-agnostic concept is explained. This concept offers the capability to deal successfully with the new requirements of current and future media production.

The acquisition and processing of audio-visual media following a format-agnostic approach is discussed in two separate chapters, Chapter 3 and Chapter 4. In Chapter 3 “Video Acquisition”, the three major video format parameters, spatial resolution, temporal resolution and colour depth (i.e., the dynamic range) are investigated with respect to the benefits they offer for future immersive media production. Due to the large variety of future video formats moving towards higher resolution, frame rate and dynamic range, the need for a format-agnostic concept is particularly helpful in supporting media production and rendering independent of the specific format. The composition and merging of visual information from different sensors will lead to more appealing and higher quality images. In Chapter 4 “Platform-Independent Audio”, the current challenges faced in audio broadcast using a channel-based approach and sound scene reproduction techniques such as wave field synthesis are reviewed. The problem of having many competing audio formats is addressed at both the production and reproduction (user) ends. The concept of object-based audio representation is introduced and several example implementations are presented in order to demonstrate how this can be realised.

In Chapter 5 “Semi-automatic Content Annotation”, both manual and automatic content annotation technologies that support format-agnostic media production are discussed. The specific requirements on those tools, in particular under real-time constraints of live scenarios are investigated. Relevant video processing approaches such as detection and tracking of persons as well as action detection are presented. Finally, user interfaces in media production are discussed, which help the production team to perform semi-automatic content annotation.

One of the advanced concepts of media production currently under discussion and development is presented in Chapter 6 “Virtual Director”. This concept builds on various audio-visual processing techniques that allow for automatic shot framing and selection to be used at the production side or by the end user. Approaches are discussed for addressing the semantic gap between data from low-level content analysis and higher-level concepts – a process called Semantic Lifting, finally leading to content and view selection that fulfils the desires of the user.

Chapter 7 “Scalable Delivery of Navigable and Ultra-High Resolution Video” deals with the main challenges in delivering a format-agnostic representation of media. As the final decision on how content will be presented is moved closer to the end user, two factors have a significant impact on delivery: higher data rate at the production side and higher levels of interactivity at the end-user side. The chapter focuses on coding and delivery techniques, which support spatial navigation based on the capture of higher resolution content at the production side. Methods for content representation and coding optimisation are discussed in detail. Finally, architectures for adaptive delivery are presented, showing how ultra-high resolution video can be efficiently distributed to interactive end users.

Chapter 8 “Interactive Rendering” starts with a list of challenges for end user devices resulting from increased interaction with the content supported by the format-agnostic media production and delivery concept. Gesture-based interaction is one of the recent trends in interactive access to media, and this is discussed in detail. A number of technologies already on the market and currently under development are presented. This chapter concludes with user studies of gesture interfaces showing that technology development must coincide with continuous evaluation in order to meet user requirements.

Finally, Chapter 9 “Application Scenarios and Deployment Domains” discusses the format-agnostic concept from an application point of...

Produkt-Details

• Erscheint lt. Verlag: 6.12.2013
• Sprache: englisch
• Themenwelt: Technik ► Elektrotechnik / Energietechnik
• Themenwelt: Technik ► Nachrichtentechnik
• Schlagworte: 3D object extraction • 3D sound rendering and face detection • Audio & Speech Processing & Broadcasting • audio event detection • Audio-, Sprachverarbeitung u. Übertragung • Audio-, Sprachverarbeitung u. Übertragung • Ben Shirley • Electrical & Electronics Engineering • Elektrotechnik u. Elektronik • Georg Thallinger • gesture analysis • graham thomas • Javier Ruiz-Hidalgo • Jean-Francois Macq • Media Production, Delivery and Interaction for Platform Independent Systems: Format-Agnostic Media • Oliver Schreer • Omar Niamut • Signal Processing • Signalverarbeitung • tracking using video and depth information
• ISBN-13: 9781118706336 / 9781118706336