This study will report on the findings of first-year Japanese university students using ETS's Criterion© Online Writing Evaluation Service. The study focused on two research questions: (1) were there important differences in ETS's computerized feedback and teacher feedback, and (2) was there any improvement in student writing over the school year? For the 2012-2013 academic year, students wrote six compositions. Results indicate that teachers tended to focus on meaning, whereas computerized feedback centered on grammatical and stylistic issues. As for student progress, only four specific areas showed any marginal improvement: confused words, spelling, missing commas, and too many short sentences, indicating that students became slightly better at expressing meaning and in using punctuation more effectively. The data indicate there were some drawbacks with ETS's computerized feedback evaluating meaning and organization of ideas, but that it was helpful in evaluating some aspects of grammar, mechanics, usage, and style.