Cassian was the most wanted criminal in the empire at that point, and all he wanted to was keep his head down and they wouldn't even let him do that. He was arrested not because he was a Aldgani raider, but just because someone felt like fucking with him. Zero correlation.

Its what made him realize you don't have a choice. You fight and suffer or you submit and suffer.