From: burgess@cc.gatech.edu (David Burgess) Subject: Re: SCI: Sound in Virtual Worlds Date: Fri, 25 Jun 1993 17:54:23 GMT Organization: Georgia Tech Multimedia Computing Group In article <20f3sc$nbp@news.u.washington.edu>, mpelzshe@man104-1.UCSD.EDU (Michael Pelz-Sherman) writes: |> In Article 8893, burgess@cc.gatech.edu (David Burgess) writes: |> |> |> Instead of forcing everyone to degenerate to a lowest common |> |> demonenator, why not try to define the most general type of system that |> |> Hey, I'm not forcing anybody, and there's certainly no guarantee that |> any standard proposed on this newsgroup is ever going to make any real |> impact in the computer software industry. I am, however, trying to |> take into account the state of existing hardware, rather than attempt |> to predict the vagaries of technological "progress". Although I agree |> that a general system is, in theory, the best approach, I fear that |> what we would end up with might look good on paper, but would be |> impossible to implement with present-day hardware. The point is that we don't actually need to be able to implement this with present day hardware, but should define interfaces in such a manner that when better hardware comes along, the applications will take advantage of it immediately. We don't need to predict technological progress to do this either - there is only a finite ammount of functionality that is useful. |> I think the real bugaboo here is not the programming, it's the lack of |> standards among delivery platforms. Take, for example, the |> proliferation of soundfile formats; you have different word sizes (8 I disagree. If you take a serious look at the software design and signal processing issues, you'll see a lot of work waiting to be done - if you want to design the system right. As for sound file formats and sample rates, you're right, a small set of non-proprietary standard formats are needed. (sounds like a bad topic for a long, boring flame war, too.) BUT, I don't think this is the major issue, because it's not that hard to convert formats, and only a few sample rates seem to be in widespread use. |> [two meritous approaches deleted for brevity] |> These are two completely different approaches to the problem, each |> with its own special set of programming requirements. I'm not at all |> convinced that a "general" system could be written that would |> encompass all possible future approaches to the problem. I would even |> go so far as to say that the search for general solutions to computer |> programming problems has yielded surprisingly little in the way of |> truly useful results, at least in my own work. I'm not talking about a general solution to a programming problem. I'm talking about a general model of an acoustic environment the drive the requirements of application-level programming interfaces. I believe such a general model is possible and there have already been some very good attempts in this direction in the computer music community (I have some good papers here by Kendall and Martins, but I can't find the publication info). Just don't have affordable hardware to implement these models doesn't mean we should take than into account when designing audio for VR - because eventually the hardware will be affordable. |> |> I'd suggest the following list of provisions: |> |> |> |> I. Sources (Audio Streams) |> |> |> |> An audio stream can come from a file, a table in memory, an algorithm, |> |> an external device, whatever. The application programmer shouldn't [stuff deleted] |> |> Seriously, these are hardware dependencies you are describing, which |> will shape the very nature of the program being written. Actually, they should not affect the way a VE is designe, and therefore should not affect the top-level interface. Physical sources are lower level issue. To the application programmed the various source types should be nothing more than tags - ideally, all sources should have the same functionality. |> |> A. Beacons - free-running sources. you start them and they run until |> |> B. Acoustic Events - sounds attached to events in the environment. you |> |> This is an important distinction! Not sure I like the term "beacon", |> but I'll be damned if I can come up with a better one at the moment! We've been calling them beacons because they're so useful for navigation. |> |> |> II. Effects (Signal Processing) |> |> |> |> A. Source effects - sound effects due to properties of the source. |> |> For our purposes, Doppler shift would be such an effect. |> |> |> |> I'm not quite sure what you mean by "Source effects"; isn't Doppler |> shift a function of sound location - specifically of the way it |> changes - *relative to the listener* ? (I understand the physical Yes Doppler shift depends on source velocity relative to the listener, but for computational purposes I've called it a source effect because it's not spatialization and it's not environmental. Perhaps this breakdown is better: A. Source Effects (muffling, distortion, emphasis) B. Relative Effects - effects due to relationships between the source and listener, but which are independent of the environment. That means Doppler shift and true 3D audio. C. Environmental Effects |> |> III. Synchronization (the part every[one] always forgets) |> |> |> |> The interface must provide mechanisms for synchronizing the following |> |> within the limits of human perception: |> |> Is the figure you mention above of .1 seconds really within the limits |> of human perception? (I'm asking sincerely; do you have data on this?) |> In our experience, 0.1sec seems to be adequate time resolution. I have few references on this, except for some comments Elizibeth Wenzel made in the fist issue of Presence. |> [stuff deleted] |> |> |> This is what the system should provide. How well it is provided will |> |> depend on the available hardware. The application programmer should |> |> ask for what s/he really WANTS, and the low-level code should figure |> |> out how to provide it as well as possible under the existing |> |> conditions. |> |> Writing a system which lays down a bunch of "standard commands" for |> all these hypothetical functions is next to trivial; someone still has |> to do the hard work of getting the low level code running. All you're |> doing is laying down the law for the poor "low level" programmer to |> follow, no? (Egad - this is starting to sound political! Sorry!) Defining the commands is only trivial once you know what information the acoustic model needs and what sort of synchronization requirements you'll have to deal with. Then the bulk of the work - synchronization and resource management - happens in a middle level. That's the part we're working on right now. For example, how do you deal with the situation of having more sounds than available hardware/CPU cycles? Problems like this must be resolved cleanly to get large auditory VE's working. Actually the low-level code isn't so bad - it's a lot of signal processing calculations (straightforward number crunching) and a little device driver code that will vary from system to system. |> |> We have built working 3d audio on top of DSP chips. For a few hundred |> |> dollars, you could roll your own 3D sound engine with a $12 DSP chip |> |> and some really fast memory. The only hard part is getting the |> |> coefficients for the filter tables. |> |> Do you have any info on how to do this? Why are those coefficients so |> hard to get? The semenal work is Wightman & Kistler, "Headphone Simulation of free-field listening ... (parts I & II)", Journal of the Acoustical Soc. of America, 85(2), 89 Feb, pp 858-878. The computation is a simple convolution with an FIR filter. Getting the data to build the filters is a complicatated process, requiring elaborate facilities. The measurement proceedures can also be dangerous if not performed properly. See the papers. |> The 1.0 spec for MIDI was written in August, 1983. MIDI-controlled |> effects processors didn't really start appearing until the late 80's. |> I see your point - everyone's afraid of obsolescence, but I still |> don't feel that MIDI has been given a fair shake by the academic |> community! Thousands of professional musicians use MIDI every day to I'm not saying MIDI isn't useful. MIDI could be a valuable implementation technology. I am saying that we should be careful not to allow MIDI to become the standard API for 3d audio applications. |> Thanks for a valuable exchange, David! I hope it will continue. Yes, it has been valuable. I didn't even think of musical sources - I'd better allow the proper hooks for them in the interface, even if the music protocol ends up being MIDI ;-). Up until now, we'd be entirely concerned with modelling natural sources, and really wrapped in in environmental acoutics. oops! -David Burgess (burgess@cc.gatech.edu)