I would dig it all out 4” below the drain pipe and gravel, perforate the side of the box drain, then lay the clean concrete sand for those 4” including and envelope around the drain box. Then top the area that had the drain pipe with whatever you want, that is permeable, ie the gravel cleaned out. The sand will carry the water. This is all assuming the drain box goes somewhere with positive drainage.it wouldn’t hit to put sand below the box too, and poke holes in the bottom so there is no standing water.