Please also remember that we're already doing voodoo fuckery where the "x nm process" is already smaller than the wavelength of the light doing the process.

In order to get much smaller, you're no longer talking about "light" but rather other high frequency EM radiation, which, as you're pointing out, we don't necessarily have the technology to

A) reliably generate that EM radiation precisely

or

B) use that frequency of radiation to achieve the equivalent photolithography.

There's also going to be (if there isn't already) a point at which the insulative properties of the silicon substrate aren't sufficient to keep the electrical signal isolated in the circuits.

There's also one more element at play, and that's the size of the element silicon itself:

A silicon atom is 1.92 Angstroms wide.

1 nm = 10 angstroms.

Silicon's "Lattice Constant" - how the atoms are arranged in a crystal is 5.4 angstroms wide (.54 nm)