Soundscape Performer
Work in progress.
A first iteration of an exploration where I’m attempting to bring together the objects, textures and soundscapes of a location into a VR environment that also responds to interaction to become a sound performance space. This project further expands on the ideas of Sitting under a highway of attentive exploration of a geographic location through capturing photogrammetry scans, volumetric scans, texture captures and ambisonic soundscap recordings, that are then used as source materials for remixing and creating new virtual environments. In this case the location used was a ravine near the area where I live, explored during the winter.
This VR performance environment could be used both as an experience offered to the general audience in a gallery space, or as a personal sound art production environment. The use of the hand as a scanner is intended to elicit a sense of connection between the performer and the object, and to invite attentive observation and audition of the virtual environment, which is ultimately connected to the original location through the audiovisual elements.
I’m targeting self contained VR headsets (Oculus Quest) due to the portability and accessibility of such systems. The visual assets are optimized to achieve the highest level of photorealism possible with the limited resources that these headsets have. For this project I’m using Unreal Engine for visual rendering and interaction with embedded Csound (as a custom made plugin) for audio synthesis.
The photogrammetry scans are processed to extract PBR material textures with a process I developed using Substance Designer and xNormal, and then painted over meshes using Substance Painter. These textures were used for the floating object that appears in front of the user and show a section of the snowy ground in the ravine area mixed with tree bark and a section of the support beam of a large bridge (with some graffiti on it).
The semi-transparent mesh of the trees was produced by a volumetric point cloud scan done with a system I assembled using a Zed Mini depth camera, an Nivida Jetson Nano mini computer and a touch LCD screen. This system is battery operated so it can be easily used to perform location scans. The point cloud was edited in Cloud Compare and converted into a mesh using MeshLab. I also experimented with a point cloud renderer to visualize the point cloud directly instead of a mesh, but this is not mobile (Android) compatible yet. The Zed camera produces videos that can be converted into side-by-side color and depth so these can also be used as volumetric videos that play inside Unreal Engine (this is yet to be explored further for this particular project and it’s not shown here).
The audio consists of two layers: a second order ambisonic recording done with an OctoMic and decoded to binaural (with head tracking) using Facebook’s Spatial Audio Workstation Rendering SDK, and the performative layer using Csound for realtime audio synthesis.
The sound performance is done by scanning the surface of the floating object with the hand controller (this will be replaced by the hand alone now that the Quest was given hand tracking capabilities). The location on the surface directly under the hand appears highlighted by a small sphere. The size of this sphere is correlated to the distance between the hand and the surface and this distance is also used to control the amplitude of the synthesized sound. This indicator sphere also changes color according to the texture of the surface directly under it and this color information (in hue-saturation-value format) is also used to drive several sound synthesis parameters sent to Csound. Four different audio sources are generated that are then spatialized (using Oculus Audio) around the user following random (for now) flight paths. The sound synthesis is also affected by the trigger and grip buttons of the controller to change the frequency range and randomization of parameters (this randomization will be replaced later by specific rules driven by the color information).
