It looks like there's no definitive pathophysiology of sundowning, but several evidence-based hypotheses as to why it likely occurs.

TL;DR: parts of the brain effecting circadian rhythms, behavioral regulation, and stress response are changed/damaged. Environment and fatigue can also play a role.

According to Canevelli et al., 2016:

Sundowning could caused by a disruption in circadian rhythm due to alteration of the suprachiasmatic nucleus (SCN), basically the "pacemaker" of circadian rhythyms, located in the hypothalamus (forebrain). This naturally degenerates with age, but damage to this area is seen even more so in people with AD, as noted by neuronal loss and accumulation of neurofibrillary tangles.

In severe AD, the SCN also shows reactive gliosis (the universal response to brain injury) in response to neuronal loss, with an increase in the astrocyte/neuron ratio. Essentially, a physiological response to brain injury in the hypothalamus interfering with regulation of sleep and emotional activity.

Additionally, circadian rhythms are regulated by melatonin, secreted by the pineal gland in response to darkness. The pineal gland's Melatonin functions are regulated by the SCN (which we know is damaged in AD). Melatonin is usually greatly reduced in people with AD, effecting circadian rhythms.

"Another possible cause is the degeneration of the cholinergic system. The SCN receive several cholinergic projections arising from the cholinergic forebrain and brain stem nuclei. Moreover, it is sensitive to cholinergic stimulation as demonstrated by the expression of muscarinic acetylcholine receptors both in SCN neurons and astroglial cells. Thus, it may be hypothesized that the impaired cholinergic transmission may contribute to the disruption of circadian rhythms and the emergence of behavioral disturbances" (Canevelli et al., 2016)

Another hypothesis is disruption/dysregulation of the hypothalamic–pituitary–adrenal (HPA) axis. In several studies, people with AD showing sundowning had significantly higher cortisol levels (stress response, regulated by HPA) than those without sundowning.

Lastly, environmental impacts such as decreased light exposure during the day (in a facility), fewer staff in late afternoon/evening, fatigue, and changes to or absence of daily routine have been associated with an overall worsening of NPS and the emergence of sundown syndrome