17
submitted
2 days ago* (last edited 2 days ago)
by
ludrol@programming.dev
to
c/linux4noobs@programming.dev
Any linux wizards know why when my sound sources go to background and CPU is under heavy load the audio crackles?
apps tested: minecraft, vlc(flatpak), spotify(flatpak), other fltapaks
crackles on both bluetooth and minijack headphones,
disabling microphone, easyeffects, gpu screen recorder ( gsr-default_output) didn't help
I use blender to put the load on CPU but it also happens when compiling or gaming
I have increased stuff in cat /etc/pipewire/pipewire.conf
[…]
## Properties for the DSP configuration.
default.clock.rate = 192000
#default.clock.allowed-rates = [ 48000 ]
default.clock.quantum = 1024
default.clock.min-quantum = 32
default.clock.max-quantum = 4096
[…]
pw-top -b
S ID QUANT RATE WAIT BUSY W/Q B/Q ERR FORMAT NAME
S 29 0 0
***
***
***
***
0 Dummy-Driver
S 30 0 0
***
***
***
***
0 Freewheel-Driver
S 45 0 0
***
***
***
***
0 Midi-Bridge
S 48 0 0
***
***
***
***
0 bluez_midi.server
R 52 2048 192000 10,6ms 69,7us 0,99 0,01 453 S32LE 2 48000 alsa_output.pci-0000_06_00.6.analog-stereo
R 213 900 48000 229,0us 69,1us 0,02 0,01 0 F32LE 2 48000 + Firefox
R 219 0 0 13,0us 41,1us 0,00 0,00 0 F32P 2 192000 + easyeffects_sink
R 217 0 0 +++ 17,0us +++ 0,00 428 + ee_soe_output_level
R 174 0 0 9,3ms 34,3us 0,88 0,00 5416 + ee_soe_spectrum
R 206 512 48000 300,5us 54,3us 0,03 0,01 0 F32LE 2 48000 + java
R 186 900 48000 359,5us 56,2us 0,03 0,01 0 F32LE 2 48000 + Firefox
R 242 900 48000 417,2us 52,4us 0,04 0,00 0 F32LE 2 48000 + Firefox
R 129 960 48000 152,1us 74,5us 0,01 0,01 0 S16LE 2 48000 + gsr-default_output
R 116 1920 48000 472,4us 62,5us 0,04 0,01 0 F32LE 2 48000 + VLC media player (LibVLC 3.0.23)
I 53 4096 192000 8,9us 3,3us 0,00 0,00 109 S32LE 2 192000 alsa_input.pci-0000_06_00.6.analog-stereo
S 68 0 0
***
***
***
***
0 v4l2_input.pci-0000_06_00.3-usb-0_3_1.0
I 191 0 0 0,0us 0,0us ??? ??? 0 F32LE 2 48000 Firefox
I 216 0 0 0,0us 0,0us ??? ??? 0 F32LE 2 48000 Firefox
I 220 0 0 0,0us 0,0us ??? ??? 0 F32LE 2 48000 Firefox
S 244 0 0
***
***
***
***
0 Blender
I 205 0 0 0,0us 0,0us ??? ??? 0 F32LE 2 44100 spotify
R 249 2048 192000 10,7ms 35,5us 1,00 0,00 1339 S16LE 1 48000 alsa_input.usb-145f_Trust_GXT_242_Microphone-00.mono-fallback
R 120 0 0 12,2us 13,5us 0,00 0,00 7 F32P 2 192000 + easyeffects_source
R 145 0 0 6,2us 8,2us 0,00 0,00 90 + ee_sie_output_level
R 230 0 0 5,7us 7,6us 0,00 0,00 118 + ee_sie_spectrum
R 144 960 48000 18,9us 49,7us 0,00 0,00 0 S16LE 2 48000 + gsr-default_input
R 257 0 0 122,2us +++ 0,01 +++ 1101 + ee_sie_rnnoise
R 106 0 0 44,8us 139,8us 0,00 0,01 236 + ee_sie_stereo_tools
R 84 0 0
***
8,6us
***
0,00 235 + ee_sie_crossfeed
R 158 0 0 5,3us 7,8us 0,00 0,00 30 + ee_sie_reverb
R 171 0 0 5,2us 8,9us 0,00 0,00 4 + ee_sie_equalizer
S 172 0 0
***
***
***
***
0 plasmashell
S 119 0 0
***
***
***
***
0 plasmashell
S 233 0 0
***
***
***
***
0 ee_test_signals
S 193 0 0
***
***
***
***
0 plasmashell
I 175 0 0 0,0us 0,0us ??? ??? 0 F32LE 2 48000 plasmashell
I know to little to dig into the scheduler stuff. systemctl status rtkit-daemon prints about pipewire sucess at priority 20.
I am on KDE/linux (bazzite based on fedora atomic44)
--------------
OS: Bazzite x86_64
Host: 82JU (Legion 5 15ACH6H)
Kernel: Linux 7.1.3-ogc3.4.fc44.x86_64
Uptime: 4 hours, 46 mins
Packages: 2 (appimage), 4 (brew), 1 (brew-cask), 173 (flatpak), 3014 (rpm)
Shell: bash 5.3.9
Display (BOE08E8): 1920x1080 in 16", 120 Hz [Built-in]
DE: KDE Plasma 6.7.2
WM: KWin (Wayland)
WM Theme: leaf-dark-color
Theme: Fusion (LeafDark) [Qt], Vapor [GTK2/3]
Icons: breeze-dark [Qt], breeze-dark [GTK3/4]
Font: Noto Sans (10pt) [Qt], Noto Sans (10pt) [GTK3/4]
Cursor: Teto (30px)
Terminal: konsole 26.4.3
CPU: AMD Ryzen 5 5600H (12) @ 4.28 GHz
GPU 1: NVIDIA GeForce RTX 3060 Mobile / Max-Q [Discrete]
GPU 2: AMD Radeon Vega Series / Radeon Vega Mobile Series [Integrated]
Memory: 12.05 GiB / 13.49 GiB (89%)
Swap: 16.62 GiB / 35.59 GiB (47%)
Disk (/): 49.00 MiB / 49.00 MiB (100%) - overlay [Read-only]
Disk (/etc): 845.89 GiB / 929.93 GiB (91%) - btrfs
Local IP (tun0): 10.96.0.42/16
Battery (L20M4PC0): 100% [AC Connected]
Locale: pl_PL.UTF-8
You're looking in the right places.
You're getting buffer underruns
the software isn't able to fill the buffer fast enough to keep up with the sound hardware; that's what causes crackling.
I believe that the ERR column is a count of underruns (it certainly includes those; dunno if anything else can cause that). I haven't looked at B/Q before, though apparently that can also be useful in diagnosing load. The outputs are getting underruns.
It looks like you've got some sort of software equalizer in use there, easyeffects. I'd probably try flipping it off and seeing whether your crackling disappears. I haven't used it, but it looks like there's a way to flip it off for a given output. I'm also assuming that the easyeffects stuff attached to your mic means that you're running a bunch of software effects on your microphone (reverb, equalizer, etc).
I don't know how to read the
pw-topoutput off-the-cuff to know the "chain" of audio there, or how to attribute a particular "cause" in a chain, but the fact that there are large ERR numbers for easyeffects does look suspicious.If easyeffects is your culprit, then if you don't care about using it, you could just disable it entirely. If you do, I suppose that things to look at:
Whether it's possible to increase the priority of whatever process is running easyeffects. I'm not familiar with how it runs, whether it's a stand-alone process or what, but if it is, maybe try
renice -n -19 pid-of-easyeffects-process, which will temporarily give it maximum non-realtime priority.Whether you want to increase the quantum. It looks like some of that is running at a sample rate of 192000. The buffer is the quantum divided by the sample rate, so a quantum of 2048 (the smallest I see on what I think is the hardware) is about 10 milliseconds. Your floor of 32 in that config file isn't going to affect it. You can temporarily increase the quantum with
pw-metadata -n settings 0 clock.force-quantum 4096where 4096 is the new quantum. I don't know if that affects running streams or notmight need to re-open the program playing the music, but you can look in
pw-topto see if QUANT goes up when you run the command. If increasing that quantum further addresses the issue, that'd be a data point. Note that this will cause audio delay from a game or such, so cranking it higher than it needs to be isn't ideal.I've seen that when, for example, loading models into VRAM on a Radeon video card for AI computation, things can get jittery. Sometimes the kernel will print messages when a driver is blocking things for too long;
sudo journalctl -krbmight be printing messages, be something to glance at.closing easyeffects helped with VLC, but it still crackles occasionally in MPV and frequently in spotify
journalctl has no messages generated when crackling.
edit: reading through the pw-top article and it might give me clue
Well, a large audio buffer size for music is basically irrelevant and for VLC or MPV playing video, I think that they look at the buffer size to compensate for audio/video sync, and can have their buffer size increased (for mpv, it's
--pipewire-buffer=<milliseconds>).But that's not gonna fix it for everything, and you'd rather not have games breaking up either, I'd imagine.
Let's see...what can measure scheduling latency...
If you install the
linux-perfpackage (well...that's what it's called in Debian...looks like it might be justperfin Bazzite) then you'll have theperfcommand.If you run:
That'll sit there for 5 seconds and record all of the times a process was waiting to run and how long it took.
Then you can view that with:
You'll get something like:
That should give you a way to measure how long it's actually taking for a process to run once it wants to (the "Max delay ms" column). The buffer length is gonna need to cover that. Lemme go find something that'll give some idea of what kernel code is actually running...couple utilities that should do that. If it's some driver using a lot of time, might give a hint as to what kernel code is running.
looks
Ah, okay.
Until you kill it with Control-C, it'll sample and record what's running in
perf.data. Then:And you can expand individual categories with "+".
I can't say "look at X, and that's it", but if something's running in the kernel, it should show up there. If it's, say, something with "nvidia" or similar in its name, that could be a hint that it's your video driver.
EDIT: Also, OP, I hate to bail on you, but I have to get some household chores done that I can't put off any longer, so I'm going to have to disappear for now...
I tried to force a clanker to help me, but it can only get me so far.
I have run
sudo perf sched record -- sleep 5with spotify and stress-ng and without easy effects but it didn't really tell about what was blocking the thread, or what could be the problem.https://pastebin.com/nZnGf2fP
clanker has seen
cyclictestandsudo trace-cmd record -e sched -e irq -e timer -e workqueue -e power -e signalso I tried running thatI have extracted the scheduler events between ~8ms delay of 16705 thread, but I don't know how to tackle it.
https://pastebin.com/D4bbzqT7
Hmm.
Those are very high max delay numbers. Like, a hundred plus milliseconds before a process that wanted to run code could run. I'd be expecting single digits at most. You'd expect the breakups
that's not something that you can reasonably deal with by just jacking up buffer size.
That's also not something I'd expect from normal CPU load, not unless another process has elevated priority or you were running extraordinarily high numbers of threads. Yeah, could be kernel code blocking, but those are extreme numbers. Maybe if the system is severely paging, like, out of memory.
Do you mention memory usage anywhere?
goes looking through stuff
Oh. Yeah. You did and I skimmed right past it. Okay, that'd maybe do it. You're deep in swap. Like:
That'd definitely create at least that much delay on rotational disks. I haven't ever tried putting my system under a lot of swap load on SSDs, but I could believe it. And someone else mentioned it too, which I ignored.
You have swap triple your physical memory and you're using using all your memory and using swap equal to time-and-a-half your physical memory.
I'd normally use maybe 100% of my memory in swap and generally try to avoid hitting it. I'm not saying that you couldn't get away with that much in swap, but if you do, most of it would have to be pretty inactive stuff.
Try killing off some of the heavy memory consumers. You can run
topand hit shift-M and it'll sort by memory usage, help identify the heavy users. If the problem goes away, well, that's probably it.If you run
iostat, you should see something like this:If your %iowait number is up there, your system is probably just spending all that time paging like crazy.
So, couple things you can do.
Obviously, not running some of the heavy memory consumers would help, see what you can pare down. Uh. You can probably save a bit by using a lighter desktop environment or something, but I can already tell you that if you're using that much, most of it's gonna be whatever software you're running, not the base system.
If that laptop can take more memory and you can stick bigger DIMMs in
and it looks, from a quick skim, like it doesn't have soldered memory
then it looks like that thing can handle up to 64GB in SODIMMs, though I would confirm that before you go buying memory.
# echo 1 >/sys/module/zswap/parameters/enabled. If it becomes tolerable after thatthe max delay numbers should drop down to single digits
you can make it persistent by adding
zswap.enabled=1to the kernel command line at boot (I'd have to look up how to do that in Bazzite; on Debian, it'd be settingGRUB_CMDLINE_LINUX_DEFAULT="zswap.enabled"in /etc/default/grub and then running$ sudo update-grub).You can maybe do something exotic like using cgroups to try to force a given process (like, not your memory player or any processes involved in playing audio) from using a lot of real memory so that it is the one that takes more of the brunt of paging, but I haven't done that myself.
I have ruled out the RAM issue by running as simplified test as possible: spotify + stress-ng and easy effects not running
I have started to dig more into the
perf,trace-cmdandcyclictestand finally managed to run all of the in the same time to capture the same high delay event. If the arch forum is correct then pipewire usesdata-loop.0process for realtime processing. It got SCHED_RR, so it seems correct as according to man pages it would preempt the SCHED_OTHER that stress-ng runs in.Interesting thing I (and a council of clankers) noticed in trace_cpu_4.txt is that the interrupt didn't fire at specified time, and when it did, scheduler immediately preempted stress-ng to run the cyclictest as it's SHED_FIFO.
My current theory is some sort interrupt problem
trace-cmd report > trace.txtsed -n '2682652,2684299p' trace.txt > trace_extract.txtcat trace_extract.txt | grep -i "\[004\]" > trace_cpu_4.txtlogs:
sudo perf sched timehist > perf_timehist.txtawk '$1 >= 9980.792239 && $1 <= 9980.797096' perf_timehist.txt > perf_extract.txthttps://pastebin.com/iuB98bXgsudo perf sched latency > perf_latency.txthttps://pastebin.com/JP0PfrSU~~After digging a bit it looks like pipewire and spotify don't get realtime shedules only SCHED_OTHER and SCHED_BATCH~~ https://bbs.archlinux.org/viewtopic.php?pid=2279287#p2279287
I have read through the article and I am 99% sure it's schedule issue
after dozens of seconds without load I get this output: