Why do some collaborations lead to fruitful outcomes and some don’t? What are the differences between effective collaboration and ineffective collaboration? These questions remain unsolved in spite of the great progress in collaboration research. In this study, we propose an assessment framework for evaluating collaborative problem solving (CPS) skills from a theoretical perspective of constructive interaction. The framework comprises observational and analytical methods. The observational method lets children solve knowledge-rich problems both individually and collaboratively and assesses the differences in performance between these two modes and processes. The analytical method analyzes the performance data and the process data from conversational and cognitive analyses. We have collected data from 110 elementary school pupils belonging to five schools. We chose three math problems from the Type B problems of the National Assessment of Academic Ability that are not easy to solve alone. We assigned one problem from the three to each pupil, asking him or her to first solve it individually in 8 minutes, then with the nearest partner in 8 minutes, and finally individually again. The results indicate that the pupils’ performances in the paired phase mostly were enhanced compared to those in the individual phase. However, there were successful pairs in which both members improved (e.g., solved the problem successfully) and unsuccessful pairs in which neither did. The cognitive analysis showed that the successful pairs discussed the meanings of numbers in the problem and tried to connect them with their knowledge or experiences more than the unsuccessful pairs did. In the successful pairs, the different levels of abstraction in the pupils’ ideas prompted them to reconsider their own ideas from different viewpoints, which further caused the members of those pairs to question or challenge each other. We propose that CPS skills should be defined as learners’ persistent endeavor to deepen their understanding in reaction to others’ contribution by tying their experiential and conceptual knowledge.