Oxygen will diffuse from capillaries through tissues: through cells, dissolved in interstitial fluids, etc. The local amount of oxygen in tissue does fall off with distance from the nearest blood vessel(s). By and large, your body is normally quite good at building a capillary network that ensures adequate oxygenation.

This issue most often comes up these days in the context of cancer biology and solid tumors. Tumors often don't have well-formed networks of blood vessels, and there is consequently a range of oxygen levels through a section of malignant tissue: a well-perfused and highly-oxygenated exterior, a low-oxygen hypoxic middle layer, and an oxygen starved necrotic (nutrient-and oxygen-starved, dying) core.

These different regions can have very different behaviors, metabolically and biochemically speaking, which can be part of why cancer is so hard to treat.