We present a new framework that reconstructs a 3D surface by incorporating a color image into sparse depth points. Assuming that the image intensity change is highly correlated with the scene geometry, we first generate a planar mesh on the image using color variation, and then, we unproject the triangulated image to the world space by integrating the sparse depth points. A quadric error metric-based mesh simplification method is employed for effective image triangulation, and a non-linear optimization is formulated to estimate the length of the projected ray from the camera center to each vertex of the triangulated image by minimizing the errors between the reconstructed surface and the depth points. Our approach can achieve an accurate 3D surface with smooth planar regions and sharp edges on the object boundaries, and it also outperforms other surface reconstruction methods (which use only depth points) in terms of the accuracy.