Heavy involvement in sports was very much a thing in the Victorian era, it was viewed as healthy and good for the mind (which, surprisingly compared with many Victorian beliefs on other medical issues, is actually correct.)

The thing is that in that era it was a much more local, amateur scene - professional sports weren't really a thing yet.

Another point I should make though is that Victorian society was heavily stratified. Do not assume that what is normal for the upper classes translates to the behaviour of the lower classes. Not everyone saw an education as being worthwhile, many thought their children should focus on practical matters of making enough to put food on the table and a roof over their heads.