bl00mbox
bl00mbox is an audio synthesis and processing engine. Its primary documentation is hosted here and covers general use cases. This page does not duplicate that documentation, but rather focuses on bl00mbox in the context of flow3r. Feel free to read these documents in any order, we will introduce basic bl00mbox concepts here briefly as needed.
Environment
flow3r is intended as a music toy where multiple applications can make sound together at the same time. Since only one
user interface can be active at a given time, there is a distinction between applications making sound in the foreground
and in the background. These multiple sources are managed by the system mixer. To get a feeling for it, open a music
application, hold the OS button down for a second and select the mixer page: You will see that a corresponding named
channel has appeared in the mixer and that you can control its volume seperately (from media, for example). If you exit
the application, the channel probably has disappeared.
That is, unless you have picked one of the apps that may continue playing in the background, such as gay drums. Try opening it, create a beat and exit while it continues to play - you can now enter another music application and see the two coexisting channels in the system mixer. This is not only useful for relative volume control; if you mute one of these channels in the mixer it stops rendering and thereby using CPU entirely. If you mute a gay drums beat for a tiny bit, you will notice that its sequence has not progressed when you unmute it.
These channels map directly to bl00mbox Channel objects, which are the central interfaces to create, connect and manage
plugins. In almost all cases each music application uses a single channel whose name is identical to the application name.
If you find a good reason to create multiple channels in a single application, their names should make it clear which
application they belong to (and of course also fit in the system mixer channel box).
If you have been around since the early days of flow3r, you may remember that music applications had a tendency to sometimes produce sound unwanted or to idly consume CPU/RAM indefinitely after having been closed. For a user that wishes to use background channels as a session for example without rebooting flow3r all the time this is inconvenient, so we have tried our best to clean up things automatically, but it doesn’t go all the way (RAM, for example). We could enforce this from the OS side by cleaning up resources more aggressively with the tradeoff that misbehaving apps crash more, but we’d prefer to trust application developers to carefully to manage resources.
Please follow the resource management guidelines presented here so that flow3r can grow into a flexible and reliable multitrack music toy!
Make sound
Let’s make a simple application that uses bl00mbox in its most basic form. The block we have seen in the mixer earlier
is a Channel object. It is initialized with a descriptive name (typically the application name) so that the user
knows what is what in the mixer. A channel can create plugins which create or modify sound. It also provides an audio
signal from the line input, as well as another that line output that routes to the mixer.
import bl00mbox
from st3m.application import Application
class Beeper(Application):
def __init__(self, app_ctx):
super().__init__(app_ctx)
# empty slot for the channel
self.blm = None
def on_enter(self, vm):
super().on_enter(vm)
# this method creates a synthesizer and stores it in self.blm
self.build_synth()
def on_exit(self):
super().on_exit()
# important resource management: delete the channel when exiting.
# this also deletes all plugins.
self.blm.delete()
# good practice: also allow the now-"hollow" channel object to be
# garbage collected to save memory.
self.blm = None
def build_synth(self):
# create an empty channel
self.blm = bl00mbox.Channel("Beeper")
# create an internal mixer with 10 inputs. note: this is not the system mixer.
self.mixer = self.blm.new(bl00mbox.plugins.mixer, 10)
# connect the output of the mixer plugin to the line output of the channel
self.blm.signals.line_out << self.mixer.signals.output
for x in range(10):
# mute the x-th mixer input
self.mixer.signals.input_gain[x].mult = 0
# create an oscillator
beep = self.blm.new(bl00mbox.plugins.osc)
# give it a unique pitch
beep.signals.pitch.tone = x
# connect it to the x-th mixer input
beep.signals.output >> self.mixer.signals.input[x]
# note: the oscillator may go out of scope here without being garbage
# collected. any plugin that is connected (directly or via other plugins)
# to the line out is considered reachable by bl00mbox.
def think(self, ins, delta_ms):
super().think(ins, delta_ms)
for x in range(10):
# unmute the x-th mixer input if corresponding petal
# is pressed to play a sound.
volume = 1 if ins.captouch.petals[x].pressed else 0
self.mixer.signals.input_gain[x].mult = volume
# note: if it wasn't for deleting the channel in on_exit() this would just
# continue playing sound if exited while a petal is held.
This app frees all resources that it doesn’t need anymore, simply by calling .delete() on the channel.
Further attempts to interact with that channel and its plugins will result in a ReferenceError, so a new one
must be created when re-entering. This is okay; the OS recognizes the name of the channel and applies all the
previous mixer settings again. A name should therefore not only be descriptive, but also unique. But not to
worry, you don’t need to check every app ever, if your application name is unique in the app store and you use it for
the channel you have done due diligence.
This application is almost well behaved and ready to ship, but there’s one more thing we should do first to make users happy:
Normalize volume
It is desirable that all music applications default to a similar volume level. You might say, why not just the maximum volume without clipping?, but there is this nasty little thing called crest ratio: The maximum peak of an audio signal is very poorly correlated to its volume. The square wave we generated above is very very loud compared to its maximum peak, but a more delicate sound such as an acoustic instrument sample may hopelessly disappear next to it even if it fills all the range. A good default should be allow for a fair amount of wiggle room for all these cases, so we’ve made an arbitrary decision:
flow3r instruments should aim for a typical volume of -15dB rms.
This volume adjustment must be done manually, but worry not, we provide utilities that make this fast and easy. The most universal approach is to tell a channel to keep track of its line out volume and print it to the REPL. This should of course only be temporary during development; measuring volume takes away CPU from the audio engine which could otherwise use to render other channels for example, printing it reduces your think rate. It’s just 2 lines, it’s not a big deal to comment them out. Let’s modify our application:
class Beeper(Application):
def build_synth():
# (same body as before)
# activate volume measurement
self.blm.compute_rms = True
def think(self, ins, delta_ms):
# (same body as before)
# print current volume in decibels
print(self.blm.rms_dB)
The print rate may be very high, you can always add a temporary sleep or counter or close the connection on the host side. We can see that the volume changes with the amount of petals we’re pressing. This begs the next question: What amount of petals do we normalize to? The answer is very unsatisfying: Whatever is typical for that application. In this case you probably would play 1 or 2 notes at the same time normally; if it’s more, it’s a special case and allowed to be louder. That’s just personal intuition and other answers may be justifiable too, but it’s a fairly reasonable guess. If users don’t like it they may fine tune in the mixer after all, you’re just providing some general purpose default setting.
Let’s measure then! With 1 petal pressed we’re getting -34dB, with 2 petals it’s about -31dB, so to reach our target of -15dB we need to increase volume by 17.5dB. Conveniently the channel has a volume control just for that (seperately from the mixer volume control, which this object has no access to). Unforunately it defaults to -12dB, and its maximum level is 0dB, so we can’t increase it enough. Why is our volume so low in the first place? Mixer plugins are initialized so that all inputs can be processed without clipping, which means the output gain of the mixer is set to a multiplier of 0.1, or -20dB. We can get the missing 5.5dB from the mixer plugin (this comes at the cost of clipping when more than 5 voices are playing. We’ll discuss that issue in the Performance section):
class Beeper(Application):
def build_synth():
# (same body as before)
# done with this, remove
# self.blm.compute_rms = True
# apply as much of the volume difference as we can here
self.blm.gain_dB += 12
# put the rest in the mixer plugin
self.mixer.signals.gain.dB += 5.5
def think(self, ins, delta_ms):
# (same body as before)
# done with this, remove
# print(self.blm.rms_dB)
This isn’t all that hard, but there is an even easier way! You might have noticed a peculiar quality: When we go into the system mixer, we actually do not exit the application, it is just suspended! This means if you enter the mixer while holding a petal the sound continues to play indefinitely - is that bad behavior? Should we squash it? Nay, au contraire, it is desirable! Say the user wants to readjust volume, it would be awfully useful to hear your adjustment while in the mixer, right? Let’s keep it! But we can also use it for development: Try it, and you will notice that the mixer activates volume measurement and displays it in the channel. If you look closely, there is a little notch next to it too: This is our normalization notch that you should aim for.
While this method doesn’t give you an absolute value, it is much better at displaying the dynamic behavior; our little Beeper here is fairly static, but if there’s more movement in the volume it might be hard to follow by just reading the printout. In such a dynamic case, you should normalize so that the loud bits linger mostly around the notch. Going above a little bit for a quick peak is okay. It’s somewhat hard to make a static set of rules for this; when in doubt, compare to similar stock applications, and don’t start a loudness war :D!
Run in background
The above example is designed to free all resources when exited, but didn’t we say earlier these could run in the background? Guess what - more rules and best practices first :P! It’s actually pretty simple:
Firstly, you should give users the option to not have your channel run in the background after exiting. Ideally this option should be obvious and the default. An example: gay drums destroys its channel if the sequencer is not running, i.e. the drumbeat is not playing (or the track is empty, so that it is kind of playing but actually not). We can directly adapt this approach to our Beeper; if we hold a petal while exiting, we may continue playing. Didn’t we say earlier that this was bad?
Well, only if it is unintentional - and it only is unintentional if it’s not in the help text! This one can be accessed
right next to the mixer and should ideally contain all there is to know about your application (it needn’t all be in the
same string, remember that .get_help() may change its output depending on application state). Let’s add this to our next
iteration, and we’re golden!
Secondly, what if a user just wants to be done with that background channel without navigating to and through your app, or
if some application has a bug and cannot not play in the background by accident? Remember that blm.delete() method
from earlier - the mixer can call it too. Not on the currently active foreground channel, so if your app doesn’t do
backgrounding you don’t have to worry about it, but if it does it need to check after re-entry if the channel still
exists, else it might crash with a ReferenceError.
One last thing before we write some code: What’s that currently active foreground channel? Well, simply put, only one
channel is in the foreground at any given time. Most interactions with a channel or its plugins set it as the foreground
channel automatically. Exiting an application clears the foreground channel too. If we want to have a channel rendered
that is not currently foregrounded, we must explicitely set the .background_mute_override attribute. As a general rule
of thumb, if a channel does not have this attribute set it should be deleted when exiting the application in order to not
waste RAM. The OS is not automatically doing it. For now :P.
class BackgroundBeeper(Beeper):
def __init__(self, app_ctx):
super().__init__(app_ctx)
self.blm = None
self.any_playing = False
def on_enter(self, vm):
super().on_enter(vm)
if self.blm is not None:
try:
self.blm.foreground = True
except ReferenceError:
self.blm = None
if self.blm is None:
self.build_synth()
def on_exit(self):
super().on_exit()
if self.any_playing:
self.blm.background_mute_override = True
else:
self.blm.delete()
self.blm = None
def think(self, ins, delta_ms):
super().think(ins, delta_ms)
self.any_playing = False
for x in range(10):
if ins.captouch.petals[x].pressed:
self.mixer.signals.input_gain[x].mult = 1
self.any_playing = True
else:
self.mixer.signals.input_gain[x].mult = 0
def get_help(self):
ret = ( "Simple synthesizer, each petal plays a different note. "
"If you exit while holding a petal that note continues "
"playing in the background to allow for drones." )
return ret
But wait, there’s more! The above approach allows us to do anything in the background that the standalone audio engine
can do; we could modulate volume with a low frequency sine wave easily if we wanted to for example, but that’s not really
convenient or appropriate for many things. To make things more flexible, we can also attach a micropython callable to it
which gets called by the OS regularily as long as the channel is rendered (at the end of each main loop to be exact, see
st3m.application). If the channel is muted it is not called. bl00mbox itself doesn’t specify the arguments, but for
flow3r purposes we call it think-like with ins and delta_ms as positional arguments.
This callback can do anything think can do and obviously can be used very irresponsibly. Avoid using this method irresponsibly please. Here’s a couple of rules:
Wouldn’t it be cool if one app set the LEDs in the background and another did something in the foreground with captouch and the display and all? Yes, but you cannot make sure at this point that the foreground app isn’t accessing the LEDs as well, resulting in some middle ground that is unsatisfying in the best case and epilepsy inducing in the worst. Let’s not do this.
We are actively planning to add more background callback options in a future release, which would allow for proper resource locks and an adequate user interface to control these background tasks. Before that, please be patient, restrain yourself and do not use bl00mbox callbacks to change anything except for the corresponding channel. Attempts to hijack these callbacks for any other purpose is considered malicious.
Well, that means we can still use ins to read out captouch and just make our thing playable along with other instruments,
right? Nope, normally no. If you do subtle indirect changes to a modulation, yes, that can make sense, so we’re still passing
the parameter and don’t just downright block it - but consider: If menuing or navigating sound unrelated apps just plays
like a keyboard, that would be pretty annoying. Don’t make your application annoying. The infrastructure is not quite
ready yet (just like the mixer actually can’t call .delete() yet, we were lying), but at some point users will be able
to permanently block channels from running in the background. Avoid getting on that list ideally. Avoid being the person
who motivates us to release this feature sooner than later.
Now that we’ve set the ground rules let’s do something cool: Let’s add a filter hooked up to the accelerometer that updates when you tilt the badge. This interacts with some tilt-based applications but it’s not as obnoxious as retaining captouch behavior, we’d expect it to be cool with many users. And yes: The stock wobbler application is derived from this example.
import bl00mbox
from st3m.ui import widgets
from st3m.application import Application
class WobblingBackgroundBeeper(Application):
def __init__(self, app_ctx):
super().__init__(app_ctx)
self.blm = None
self.any_playing = False
def on_enter(self, vm):
super().on_enter(vm)
if self.blm is not None:
try:
self.blm.foreground = True
except ReferenceError:
self.blm = None
if self.blm is None:
self.build_synth()
def on_exit(self):
super().on_exit()
if self.any_playing:
self.blm.background_mute_override = True
else:
self.blm.delete()
self.blm = None
def build_synth(self):
self.blm = bl00mbox.Channel("Beeper")
self.blm.gain_dB += 18.5
self.mixer = self.blm.new(bl00mbox.plugins.mixer, 10)
# let's add a global filter
self.filter = self.blm.new(bl00mbox.plugins.filter)
self.filter.signals.input << self.mixer.signals.output
self.filter.signals.reso.value = 25000
self.blm.signals.line_out << self.filter.signals.output
for x in range(10):
self.mixer.signals.input_gain[x].mult = 0
beep = self.blm.new(bl00mbox.plugins.osc)
# and make it a bit lower this time
beep.signals.pitch.tone = x - 36
beep.signals.output >> self.mixer.signals.input[x]
self.tilt = widgets.Inclinometer(buffer_len = 8)
self.tilt.on_enter()
def synth_callback(ins, delta_ms):
# note: in python an inner function like this inherit the outer
# scope so that we can still access "self" in here
self.tilt.think(ins, delta_ms)
# note: tilt.pitch describes the aviation angle here, not frequency
# it's a bit silly ^w^.
pitch = self.tilt.pitch
# note: we're not accessing self.tilt.pitch again because it isn't
# cached internally and we should pay attention to making background
# callbacks as fast as possible.
if pitch is not None:
self.filter.signals.cutoff.tone = -pitch * 10 + 10
self.blm.callback = synth_callback
def think(self, ins, delta_ms):
super().think(ins, delta_ms)
self.any_playing = False
for x in range(10):
if ins.captouch.petals[x].pressed:
self.mixer.signals.input_gain[x].mult = 1
self.any_playing = True
else:
self.mixer.signals.input_gain[x].mult = 0
def get_help(self):
ret = ( "Simple synthesizer, each petal plays a different note. "
"Tilt to change filter cutoff frequency. "
"If you exit while holding a petal that note continues "
"playing in the background to allow for drones." )
return ret
Performance
The CPU can only do so much, and this is a hard real time environment: If an audio frame is rendered too slowly, there will be audible glitches. While different channels can be rendered in parallel on different cores (they’re not right now but hopefully soon, optionally - this could starve other tasks though and is not a magic cure-all), there are no plans to make any single channel render its plugins in parallel. This gives us a hard upper limit of using 100% of one core.
We can easily find out how much our application is using in a given state: Go into the System->Settings menu and activate ftop. This prints a CPU load report on the serial console every 5 seconds (while blocking all micropython execution for a noticable time, which is why it’s ideally turned off outside of debugging). If your app is running with no other channel playing in the background, the audio task CPU load directly corresponds to your channel’s performance. Note that this is averaged over the entire 5 second interval, so depending on how much is going on a CPU load of say 70% may already start “crackling” by producing a few dropouts here and there.
Optimizing CPU load of a given sound includes a lot of trial and error: Different plugins of course cause different CPU load, but also the same type in a different configuration will perform differently. If you have a single audio chain and it’s too heavy, there’s often little choice but to simplify it.
One common issue is excessive polyphony, or how many notes are playable at the same time: Above, we have naively
just tied an oscillator to each petal. For the sake of a simple example this was good enough, but we should ask
ourselves: Do we really want to play all of them at the same time? Isn’t it more valuable to have more CPU available
for each note you play, without choking the entire engine including background channels if a user changes hand
position to hold a shoulder button for example? We can easily implement more intelligent system in python that
limits the amount of voices, or, alternatively, use the poly_squeeze plugin, or a mix of both - the Violin
app for example reduces its output to a single voice, which also helps with switching between notes. Furthermore,
playing a lot of notes at the same time demands a high headroom and may result in clipping. It really is for the
best to consider a hard limit there.
Another technique we can apply is paraphony, or sharing common elements between voices. Our wobbler above is actually paraphonic: It has 10 oscillators, but they all go through a shared filter. Say our filter cutoff was controlled by each captouch position instead, so that it would make sense to have 10 filters - the CPU load of our humble application might very well double (guesstimate)! In that case we could ask ourselves, maybe a shared filter with the maximum of all cutoffs can fake the job well enough? Or two in parallel, one with max, one with min?
One extra parameter is render tree topology. See - whenever we set the input_gain of a mixer plugin to 0,
the connected oscillator is no longer being rendered to not waste CPU cycles (this can be overridden with
the .always_render attribute). Plugins that are not reachable from the line output are not rendered at
all. However, this system is not fully automatic: For example, the filter above is always rendered, even
if all inputs of the mixer before it are muted. If this chain was very long we could run into high idle loads,
which creates problems if we have multiple heavy plugin chains and want to only render one at a time: We must
be careful that rendering is properly suppressed for everything we don’t need. bl00mbox will probably at some
point provide better automation there, but for the time being it is a good idea to carefully observe if idle
voices are maybe not rendered to a large degree.
Common issues
Bugs in bl00mbox
bl00mbox has a good amount known of bugs. If something doesn’t work as you’d expect it doesn’t need to be your fault. It might be worthwhile to check out the latest documentation of that feature.
Writing to flash causes audio glitches
This sadly cannot be avoided due to the bus architecture of the ESP32S3 for external RAM, in which bl00mbox plugins generally live. Music applications should only save data on the SD card.
Channel won’t get loud enough or distorts
bl00mbox audio buffers use 16bit data. This is generally a good amount of resolution for a normalized signal, but we’re not playing a CD back here: Synthesis can be sometimes a bit unpredictable in terms of headroom and volume levels. If you only change the volume at the channel output, you might end up distorting earlier in the chain, either by clipping or by being too silent. It is important to keep in mind the intermediate gain levels as well; you can optimize them by plugging an intermediate output directly in the line out for testing purposes. Excessive polyphony can make this harder: 10 full scale voices playing without clipping at the same time means that each may only use 1/10th of the headroom, resulting in a -20dB gain reduction compared to a single voice. This means 10 voice polyphony normalized to our -14dB target requires compression to avoid clipping.
How do I just get a plain piano sound?
bl00mbox is not a soundfont player. It can sort of kind of be squeezed into that role, but it is not its primary focus for the time being. At this stage it is best to look up (analog) synthesis techniques for the timbre you’re looking for and find out what translates well by trial and error. A wavetable synthesizer as used by the Violin app may help to cut that process a bit shorter for very “clean” sounding instruments, but many have noisy or disharmonic textures which are best emulated by experimentally determined types of modulation, it is often wise to look up other people’s work.
All that background synthesis is nice and well, but what if I just wanna record and loop?
Well, technically there is the sampler plugin, which currently only supports a fixed buffer size, so it’s a
tradeoff between max sample length and RAM hogging, which isn’t great. That doesn’t mean we don’t consider this
feature important, but rather that we’re taking our time getting it right: There’s little point in having each
music application implement its own version of this, but rather we can override the volume up/down buttons to
implement a global looper that all applications may use. This is also why we recommend against using
st3m.application.override_os_button_volume for music applications. We hope to finish this feature soon,
but there’s a lot of details to get right so it will take a little longer to be ready for public!
In the example above, what if I want sound to stop when entering the mixer?
Thing is, right now you can’t really do that. If you are familiar with st3m.ui.View you might ask, why not
just call .on_exit() when opening the system menu, but unfortunately these methods are typically used for
opening/closing applications that do not use views. It’s a regrettable situation, and we will rectify it soon
when we have a clear path on how to resolve the general lack of seperation between applications and views; it’s
probably just gonna be some extra methods, but it is gonna take some careful planning to unravel this cleanly.
I don’t like this, can’t I just use <other micropython audio engine> instead?
The backend allows for easily adding extra engines and we’re happy to take a look if you have a concrete proposition. It’s best to start with opening an issue in the firmware repository so that we can have a look before anybody sinks any potentially futile work into hooking it up. If you just wanna do it for yourself regardless, the backend is simple enough to hook extra engines into.