It could be the book. I listen to a lot of audiobooks and occasionally I have issues remembering certain books. However this isn't a pervasive issue. I recommend reading some sort of plot summary every so often if you are having difficulties. I have read spark notes along with the book if it is a complex book and that has helped sparked more interest in a book I am struggling with.