It depends on how large and diverse your dataset is, but in most cases you should. You'd see an even larger difference between the train and test sets.

You can also try to use multiple folds, to train model 5 times for example with the different test set, to check if the test set you selected accidentally contains simpler samples.