One problem that seems to crop up often with PulseAudio is its high CPU usage, which occurs when it needs to resample audio from one sample rate to another. CDs and most MP3s are encoded at 44.1 kHz, so it makes sense to pass the signal to the audio card at this sample rate too - in my case this means setting JACK to run at 44.1 kHz, so PulseAudio passes the audio straight through.
The problem comes when you try and play a FLAC file, for example, which is encoded at 96 kHz (which provides more information per second and so can encode higher frequencies). Two things happen: first is that PulseAudio — by virtue of using a high-quality resampling algorithm (speex-float-3) — uses lots of CPU time; second is that you lose all the additional audio information, as the audio is downsampled.
You could get around the loss of information by running JACK at 96 kHz by default, but then the rest of your audio which is at 41.1 kHz (most of it) would have to be upsampled unnecessarily. You could switch between sample rates in JACK manually when needed, but this often means restarting applications. You could set the default resampling algorithm to one that is lower quality (speex-fixed-0) but faster, but this means losing audio quality again.
The best solution is probably to leave JACK as it is (at 41.4kHz), not worry too much about the CPU usage as most files don't need to be resampled, hope that SSE-optimised resampling gets into PulseAudio soon, and switch JACK to a higher sample rate when you really need the extra quality.
Now to find out what damage audio hardware is doing to the signal...