Because the flame doesn't only need oxygen. It also needs fuel to react with that oxygen and it needs the reaction of the fuel with the oxygen to release energy overall.

When you are burning something in the air, you're burning something -- that is, a fuel. The reason all the nearby stuff doesn't catch on fire as soon as you strike a match is that all of the nearby stuff doesn't react with oxygen in the air in a way that releases energy unless it gets hot enough, and it isn't hot enough. Of course, if it's close enough to a flame, then it too can burn because the flame is heating it up to a temperature where it will react with the atmosphere.