Power steering whine above idle

I'm chasing a noise that is driving me nuts. As the engine returns to idle, there is a power steering pump sounding whine. It may be happening at higher speeds but I can't tell, but for sure at lower speeds and approaching stops. I swapped out the brown-ish fluid for using the baster method, so I doubt air got in the lines, but it didn't help. It may not even be the PS pump at all, but any ideas would be helpful.