Numerous audio signal processing and analysis techniques using non-negative matrix factorization (NMF) have been developed in the past decade, particularly for the task of source separation. NMF-based algorithms iteratively optimize a cost function. However, the correlation between cost functions and application-dependent performance metrics is less known. Furthermore, to the best of our knowledge, no formal heuristic to compute a stopping criterion tailored to a given application exists in the literature. In this paper, we examine this problem for the case of supervised and semi-supervised NMF-based source separation and show that iterating these algorithms to convergence is not optimal for this application. We propose several heuristic stopping criteria that we empirically found to be well correlated with source separation performance. Moreover, our results suggest that simply integrating the learning of an appropriate stopping criterion in a sweep for model size selection could lead to substantial performance improvements with minimal additional effort.