to sb.'s annoyance

adv.为难的是,烦恼的是