Yeah, sounds likely it was a scam. Scammers use phone number spooofing which is a service they can make whatever phone number show up on your caller id they want. Having the actual Chase Bank phone number show on your phone is part of the scam. Its such a common scam that most banks have a warning about them. See Chase Bank link here

https://www.chase.com/digital/resources/privacy-security/questions/fraud

Call Chase and tell them what happened. Ask if it was them that called. If it wasn't, change your password and pin number immediately. Subscribe to an identity and credit monitoring service like Experian that would notify you if someone applies for credit or uses your personal identification. Also change your Chase notifications to notify you of all transactions. You may get a lot of notifications but you can change it back when you're confortable that you don't see any fraudulent transactions.