Pay Attention to How You Pay Attention