I don't see any problem with this, and it's quite common in building new construction to plow under this sort of debris.

If the structure is pre-1970, I would NOT bury any bricks that have been painted, as it is likely lead paint

The thinking is that you don't want to bury things that are actually toxic, or will breakdown and become toxins, or collapse and create a low spot in your garden over time (I buried a hay bale once, I can still see the low spot in the yard). Bricks, mortar, concrete rubble don't have these problems.