As others have said, no it is not particularly unique in terms of the amount of species hosts it affects.

Animal cells come in different shapes and sizes but other than that, they are they are functionally and anatomically the exact same across different species. It does not matter much that we humans are closely related to other primates and not closely to birds, sea lions, etc. because we have the same cells either way.

The relevant part in terms of whether the virus can attach or not is due to things like different animo acid / receptors / acids covering the animal cell and preventing the virus from binding, host body temperature, the virus not producing enough of a particular protein to duplicate given the size/ immune system of their host, etc.

For example, rabies does not typically live in squirrels because their blood is not warm enough for the virus to be comfortable (~95 degrees F), but it does in raccoons (101-106 degrees F). The difference is not that significant but is enough to affect who the virus typically host.

Also,

>I know diseases like HIV are thought to come from primates, which makes sense: we're so closely related

Humans are primates :)