Multi-class image segmentation (or pixel labeling) is one of the most important and challenging tasks in computer vision. Currently, many different methods for this task can be broadly categorized into two types according to their choice of the partitioning of the image space, i.e., pixels or segments. However, each choice of the two types of methods comes with its share of advantages and disadvantages. In this study, we construct a novel CRF model to integrate features extracted from pixel and segment levels. We exploit segments generated by Constrained Parametric Min Cuts (CPMC) algorithm in the proposed framework, instead of commonly used unsupervised segmentation method (e.g., mean-shift approach). Additionally, the recognition based on these segments is also integrated into the model, which possible corrects classification mistakes caused by the unary term based on information derived from pixel level. We experimentally demonstrate our model’s quantitative and qualitative improvements over the baseline methods.