Run it per band, every frame, feeding the previous result back in. A higher fall keeps energy lingering; raise rise to snap on transients.
float smoothBand(float now, float last, float rise, float fall) {
float k = now > last ? rise : fall;
return mix(last, now, k);
}