Depending on the model, "tracing" the output is certainly possible - for example, in decision trees. As far as confidence is concerned, you might find the recent work in conformal prediction interesting (basically predicting ranges of outputs at a specified confidence level). A really nice tutorial can be found here: https://people.eecs.berkeley.edu/~angelopoulos/publications/downloads/gentle_intro_conformal_dfuq.pdf.