Think of it like asbestos, it's perfectly safe as long as you don't mess with it.

Don't lick the walls, or demolish one (making dust) and you are fine.

In regards to the solar panels, I'm sure manufacturing is mostly automated and since we know the hazards of lead very well I'm not concerned there.

From a consumer perspective the wafers are behind a sealed glass panel, you'll never touch them and they will never touch rain.

Recycling is same as manufacturing, we know the risks and can manage them well.