20 releases
0.2.12-alpha.0 | Apr 7, 2023 |
---|---|
0.2.11-alpha.0 | Dec 19, 2022 |
0.2.5-alpha.0 | Jun 21, 2022 |
0.2.4-alpha.0 | Mar 14, 2022 |
0.1.42-alpha.0 | Oct 27, 2021 |
#20 in #oscillator
119 downloads per month
Used in 3 crates
350KB
5K
SLoC
surgeosc-audioin
Audio input oscillator component for the Surge synthesizer system.
Description
surgeosc-audioin
is a Rust crate providing an
audio input oscillator component as a part of the
Surge synthesizer system. This crate enables
real-time audio input processing with various
control parameters and modulation options, making
it an essential building block for any Surge-based
synthesizer project.
Control Parameters
The following tokens represent the different control parameters available in this crate:
-
control_group
: Represents a group of related controls that can be manipulated together. -
control_type
: Describes the type of control (e.g. knob, slider, etc.). -
default_value
: The initial value of a control parameter when instantiated. -
max_value
: The maximum allowed value for a control parameter. -
min_value
: The minimum allowed value for a control parameter. -
modulateable
: A boolean indicating whether a control parameter can be modulated by external sources (e.g., LFOs, envelopes, etc.). -
moverate
: The rate at which a control parameter can change over time when being modulated. -
value_type
: The data type of the control parameter value (e.g., float, integer, etc.).
Processing Functions
The crate provides different processing functions that can be used depending on the desired output:
-
process_block
: Processes an input block of audio samples. -
process_block_mono
: Processes an input block of mono audio samples. -
process_block_stereo
: Processes an input block of stereo audio samples.
AudioInputOscillator
AudioInputOscillator
is the main struct
representing the audio input oscillator component
in this crate. It contains all the necessary
control parameters and processing functions
required to manipulate and process audio input in
real-time.
Mathematical Analysis
The audio input oscillator processes the input signal using various mathematical operations, such as multiplication, addition, and scaling. These operations can be represented using the following mathematical equations in Unicode:
-
Modulation: 𝑦(𝑡) = 𝑥(𝑡) × 𝑚(𝑡), where 𝑥(𝑡) is the input signal, 𝑚(𝑡) is the modulation signal, and 𝑦(𝑡) is the output signal.
-
Scaling: 𝑦(𝑡) = 𝑎 × 𝑥(𝑡), where 𝑎 is a scaling factor and 𝑥(𝑡) is the input signal.
-
Clipping: 𝑦(𝑡) = min(max(𝑥(𝑡), 𝑎), 𝑏), where 𝑎 is the lower clipping threshold, 𝑏 is the upper clipping threshold, and 𝑥(𝑡) is the input signal.
These mathematical operations allow for a wide
range of audio processing possibilities, enabling
users to create unique and expressive sounds using
the surgeosc-audioin
crate.
2> okay, thanks -- please walk me through the process of processing audio input in real-time. how do we do it? what are our engineering constraints?
Processing audio input in real-time is a complex task that requires efficient algorithms and careful consideration of various engineering constraints. Here is a high-level overview of the process and the main constraints to consider:
-
Capture audio input: Real-time audio processing begins with capturing audio input from a source like a microphone, an instrument, or an audio file. This is typically done using an audio interface or a sound card that converts analog signals into digital samples.
-
Buffering: Audio data is usually processed in small chunks called buffers. The buffer size determines the granularity of the processing and directly affects the latency of the system. Smaller buffer sizes result in lower latency but require more CPU resources to process, while larger buffer sizes reduce the CPU load but increase latency.
-
Sampling rate: Audio signals are continuous in nature, but digital audio processing deals with discrete samples. The sampling rate is the number of samples taken per second, and it determines the audio quality and frequency range that can be represented. Higher sampling rates result in better audio quality but require more processing power and memory.
-
Audio processing: Once the audio data is captured and buffered, it can be processed using various audio processing algorithms like filtering, modulation, and effects processing. The
surgeosc-audioin
crate you are working on provides an audio input oscillator component that can be used to process audio input in real-time. The processing functions likeprocess_block
,process_block_mono
, andprocess_block_stereo
implement the required algorithms and manipulations. -
Control parameters: The behavior of the audio processing can be controlled using various parameters like gain, modulation depth, and filter cutoff. These parameters can be adjusted in real-time, either by user interaction or through automation and modulation sources (e.g., LFOs, envelopes). The crate you are working on provides tokens to represent these control parameters, like
control_group
,control_type
,default_value
,max_value
,min_value
,modulateable
,moverate
, andvalue_type
. -
Output: The processed audio data is then sent to an audio output device (e.g., speakers, headphones) or further processing (e.g., mixing, effects). This may involve converting the digital samples back to analog signals.
Engineering Constraints
There are several engineering constraints to consider when implementing real-time audio processing:
-
Latency: Low latency is crucial for real-time audio processing, as high latency can result in noticeable delays between user input and audio output. To minimize latency, you need to optimize buffer sizes and processing algorithms for efficiency.
-
CPU usage: Real-time audio processing can be computationally intensive, particularly when using complex algorithms and high sampling rates. You must optimize your code to minimize CPU usage and avoid performance bottlenecks that could cause audio artifacts like clicks, pops, or dropouts.
-
Memory usage: Audio processing can require significant amounts of memory, particularly when dealing with large sample buffers and multiple processing stages. Be mindful of memory usage and optimize memory allocation and data structures as needed.
-
Precision and numerical stability: Audio processing algorithms often involve floating-point arithmetic, which can be prone to rounding errors and instability. Ensuring numerical stability and adequate precision is essential to avoid artifacts and maintain audio quality.
-
Concurrency and thread safety: Real-time audio processing often involves multiple threads, such as a separate thread for audio input/output and another for user interface updates. Ensuring your code is thread-safe and free from race conditions is crucial for stability and performance.
-
Platform compatibility: Different platforms (e.g., Windows, macOS, Linux) and hardware configurations may have different requirements, performance characteristics, and limitations. Ensure that your crate is compatible with the target platforms and can handle varying hardware capabilities.
By carefully
3> /file surge-rs/surgeosc-audioin/src/process.rs
The provided code defines the implementation of
the process functions for the
AudioInputOscillator
struct. There are three
main process functions: process_block
,
process_block_mono
, and
process_block_stereo
. Let's examine each
function and understand its purpose.
-
process_block
: This function is implemented for theOscillatorProcess
trait and serves as a dispatcher. It takes acfg: OscillatorProcessBlockCfg
parameter, which provides information about the processing context, such as whether the input is stereo or mono. Depending on thecfg.stereo
value, it calls eitherprocess_block_stereo
orprocess_block_mono
. -
process_block_mono
: This function processes a mono input signal. It computes the gain and input parameters from the oscillator's parameter values usingpvalf
. Then, it calculates the coefficientsa
andb
based on the gain and input values. Finally, it iterates over each sample in the input buffer (with a size ofBLOCK_SIZE_OS
), combines the left and right input channels with the coefficients, and writes the result to the left output channel. -
process_block_stereo
: This function processes a stereo input signal, similar to theprocess_block_mono
function. It calculates the gain, input, and coefficientsa
andb
in the same way. The main difference is that it keeps the left and right channels separate when processing the input signal. It writes the result to the left and right output channels separately, applying the coefficientsa
andb
to the respective input channels.
Some interesting points about these functions:
-
The use of
match
in theprocess_block
function is a simple and elegant way to branch between the stereo and mono processing functions based on thecfg.stereo
value. -
The use of the
limit_range
function in bothprocess_block_mono
andprocess_block_stereo
ensures that the input value is clamped between -1.0 and 1.0, preventing potential issues caused by out-of-range values. -
The gain and input parameters are converted from decibels to linear values using the
db_to_linear
function from thetables
field in theAudioInputOscillator
struct. This is a common operation in audio processing, as it allows for more intuitive control over gain values.
In summary, the process functions in the provided code handle real-time audio processing for both mono and stereo input signals. They control the balance between input channels and apply gain adjustments based on the oscillator's parameter values.
4> interesting, so does this oscillator do anything more than properly fill the oscillator output by reading the input in mono or stereo mode? (other than the light clamping, and value conversion) also, could you teach me about converting from decibels to linear values? why do we do this on the audio input oscillator?
You're correct; the primary function of this
AudioInputOscillator
is to read the input signal
(either mono or stereo) and fill the oscillator
output buffer accordingly. It applies gain
adjustments and light clamping on the input
values, but other than that, it does not perform
any complex operations or transformations like
a traditional oscillator that generates
waveforms. This oscillator is designed to act as
an audio input source that can be processed
further down the signal chain.
Regarding converting decibels (dB) to linear values, let's first understand why we use decibels. Decibels are a logarithmic scale that measures the ratio between two values (e.g., power, intensity, or amplitude). In audio processing, decibels are often used to represent gain, attenuation, and signal levels, as they provide a more intuitive way of working with the wide range of values involved in audio signals. The human perception of loudness is logarithmic, meaning that we perceive a doubling of the amplitude as a constant increase in loudness, rather than a linear increase. By using a logarithmic scale like decibels, we can better align with this perceptual characteristic.
However, when performing mathematical operations or processing audio signals, it is often necessary to convert these decibel values back into linear values. This conversion is done using the following formula:
linear_value = 10^(decibel_value / 20)
In the AudioInputOscillator
, we convert gain
values from decibels to linear values because the
actual audio signal processing (e.g.,
multiplication, addition) requires linear
values. By converting the gain to a linear scale,
the oscillator can apply the appropriate gain
adjustments directly to the input samples.
In the provided code, the db_to_linear
function
performs this conversion. It is called in both
process_block_mono
and process_block_stereo
functions to convert the gain parameter from
decibels to a linear value before applying it to
the input signal.
5> interesting. where does the audio input come from?
In the provided code, the audio input comes from
the synth_in
field of the AudioInputOscillator
struct. Specifically, the input is accessed using
the following methods:
-
For the left input channel:
self.synth_in.audio_in0(k)
-
For the right input channel:
self.synth_in.audio_in1(k)
Here, k
represents the sample index within the
input buffer.
The synth_in
field is an instance of a struct
that represents the synthesizer input context,
which includes the audio input buffers, among
other things. This struct is typically provided by
the larger synthesizer system or application that
incorporates the AudioInputOscillator
.
To give you a general idea of how the audio input
is captured and routed to the
AudioInputOscillator
, here's a high-level
overview of the process:
-
An audio input source, such as a microphone, an instrument, or an audio file, generates an analog audio signal.
-
The analog signal is captured by an audio interface or a sound card, which converts the analog signal into digital samples. This process involves analog-to-digital conversion (ADC) and sampling at a specific rate, such as 44.1 kHz or 48 kHz.
-
The digital audio samples are buffered into small chunks for efficient processing. These buffers are typically managed by the audio system or audio library used by the synthesizer application.
-
The synthesizer application routes the audio input buffers to the relevant processing components, such as oscillators, filters, and effects. In this case, the audio input buffers are passed to the
AudioInputOscillator
through thesynth_in
field. -
The
AudioInputOscillator
processes the audio input buffers as described in the previous explanations of theprocess_block_mono
andprocess_block_stereo
functions.
It's important to note that the specific
implementation details and routing of the audio
input will depend on the larger synthesizer system
or application that you are integrating the
AudioInputOscillator
crate into. The provided
code assumes that the audio input context (i.e.,
the synth_in
field) is available and properly
set up by the surrounding system.
6> cool, thanks! what else should we know when using this oscillator? any caveats?
When using the AudioInputOscillator
, there are
a few important points and caveats to keep in
mind:
-
Properly setting up the audio input context: As mentioned before, the
AudioInputOscillator
relies on an external audio input context (thesynth_in
field) to access the input buffers. Ensure that the surrounding synthesizer system or application properly sets up and manages this context, including capturing the audio input, handling buffering, and routing the buffers to the oscillator. -
Parameter ranges and handling: The oscillator has parameters like Gain and Input, which are used to control the audio processing. Make sure that the user interface or automation system that interacts with these parameters correctly handles the allowed parameter ranges and updates the oscillator state accordingly.
-
Performance considerations: Real-time audio processing can be computationally intensive, especially when dealing with complex algorithms, high sampling rates, and low buffer sizes. While the
AudioInputOscillator
implementation is relatively simple, it's still essential to keep an eye on CPU usage and optimize the overall system performance if needed. -
Thread safety and concurrency: In many audio applications, multiple threads are used to handle tasks like audio input/output, user interface updates, and other tasks. Ensure that the
AudioInputOscillator
is used in a thread-safe manner, avoiding race conditions or other concurrency-related issues that could lead to unexpected behavior or crashes. -
Platform compatibility: Different platforms (e.g., Windows, macOS, Linux) and hardware configurations might have different requirements, performance characteristics, and limitations. Make sure that your crate, as well as the larger synthesizer system, is compatible with the target platforms and can handle varying hardware capabilities.
-
Integration with other processing stages: The
AudioInputOscillator
is just one component in a larger signal processing chain, which might include other oscillators, filters, and effects. Ensure that the output of theAudioInputOscillator
is properly routed to the subsequent processing stages and that the overall signal flow is well managed.
By being mindful of these points and caveats, you
can successfully integrate the
AudioInputOscillator
into your synthesizer
system or application and achieve the desired
functionality and performance.
technique
7> what are some techniques we can perform with this oscillator? could you list me 12 high production-quality techniques from electronic music production for using this component?
While the AudioInputOscillator
itself is
a relatively simple component designed to pass
audio input through with gain adjustments and
channel balancing, it can be a valuable part of
more complex audio processing chains. Here are 12
techniques that you can explore when using the
AudioInputOscillator
in an electronic music
production context:
-
Vocoding: Use the audio input as the modulator signal for a vocoder effect, allowing you to create synthesized vocal sounds or robotic speech effects.
-
Sidechain compression: Route the audio input to a sidechain input of a compressor effect. This technique is popular for creating a "pumping" or "ducking" effect, where the audio level of one track is reduced based on the input level of another track, like a kick drum.
-
Layering with other oscillators: Combine the output of the
AudioInputOscillator
with other oscillators generating different waveforms or samples. This can create a richer, more complex sound and add harmonic content to the input signal. -
Granular synthesis: Use the audio input as a source for granular synthesis, which involves chopping the audio into small pieces (grains) and manipulating them to create unique textures and soundscapes.
-
Ring modulation: Multiply the audio input with another oscillator's output to create metallic, bell-like, or inharmonic sounds.
-
Dynamic filtering: Use the audio input to modulate the cutoff frequency or resonance of a filter effect. This can create dynamic, expressive filtering effects that change based on the input signal's amplitude.
-
Pitch shifting and harmonizing: Apply pitch shifting or harmonizing effects to the audio input to create interesting harmonies, chords, or textures.
-
Time stretching: Manipulate the playback speed of the audio input without affecting the pitch, creating unique soundscapes and textures.
-
Beat slicing and rearranging: Process the audio input by slicing it into smaller pieces based on transients or beats, then rearrange the slices to create new rhythms and patterns.
-
Reverse reverb: Apply a reverb effect to the reversed audio input, and then reverse the resulting audio again. This creates a unique "swelling" reverb effect that can add atmosphere and depth to a sound.
-
Multiband processing: Split the audio input into multiple frequency bands and apply different effects or processing to each band, allowing for precise control over the frequency content of the input signal.
-
Feedback loops: Route the output of the
AudioInputOscillator
into a processing chain that includes a delay or reverb effect, then route the output of that chain back into the oscillator's input. This creates a feedback loop that can result in evolving, self-generating sounds and textures.
These techniques are just a starting point for the
creative possibilities offered by incorporating
the AudioInputOscillator
into your electronic
music production workflow. By combining this
oscillator with other audio effects and
processors, you can create a wide variety of
unique sounds and textures.
Dependencies
~11–20MB
~297K SLoC