Exactly. Any app that pops up and wants access to your contacts is possibly gathering this information. They'll 'help find your friends' on Facebook by matching up phone numbers and contact info they've already collected and have in their DB.