Beyond the Score: Exploring the Intersection Between Sociodemographics and Linguistic Features in English (L1) Writing Placement

This study builds on prior research into native English (L1) writing proficiency and placement practices [11], focusing on pre-enrollment students from the 2021–2022 and 2023–2024 academic years. In 2023–2024, TCC reported an unduplicated headcount of 14,538 students. As shown in Figure 1, White students made up nearly half the population, followed by Hispanic/Latino, Black or African American, and American Indian or Alaska Native students.

In terms of gender and academic load, as illustrated in Figure 2, approximately two-thirds of students were female and one-third male. Regarding academic load, 69% were enrolled part-time (taking about two courses per semester), while 31% were enrolled full-time (taking four to five courses). The average student age was 23, with 57% aged 24 or younger. Additionally, about 24% was first-generation college students, defined by NCES as those whose parents did not complete a college degree.

Given this demographic profile, writing placement and literacy emerge as central concerns, with nearly 50% of students placed into at least one DevEd course. The average pass rate for DevEd Levels 1 and 2 in 2023–2024 was 56%. Meanwhile, TCC’s retention rate is approximately 65%, and the graduation rate for full-time students stands at 27%. These metrics are critical as institutions work to support student completion and transition into the workforce.

To evaluate the representativeness of our sample relative to the broader Accuplacer test-taking population, as guided by our first research question, we examined institutional data from students who completed the Accuplacer exam during the same time period. Table 1 summarizes this data by gender, race, and native language.

A total of 2,003 students (unduplicated count) completed the Accuplacer test, producing a written sample that was automatically assessed and classified by proficiency level. In terms of gender, the test-taking population was composed of 62% females and 37% males, closely mirroring the broader institutional demographics. Regarding race, approximately 41% of students self-identified as White, followed by Black or African American (15%), American Indian (10%), and Latino/Hispanic (9%) students, proportions that closely align with the overall student population. First-language data further indicated that 69% of test-takers reported English as their native language, comparable to the 66% institutional ratio.

Building on prior investigations⁹⁹9An earlier phase of this research analyzed a random subset of 450 essays (from the 2,003 total); demographic data was not examined as placement was the focus., this study narrows its analysis to the 1,384 participants who self-identified as native English (L1) speakers. Placement-level breakdowns for this group were available by gender only and are summarized in Table 2.

From this table, we observed that approximately 53% of native English speakers were placed into one of the two DevEd levels (Level 1 or Level 2), with a higher concentration in Level 2. Placement into DevEd was slightly more common among female students. Notably, this 53% placement rate closely mirrors the broader institutional trend of approximately 50% outlined in Section 3.1.

To conduct a more detailed analysis, a subsample of 300 essays was randomly selected from the 1,384 native English speakers ($\approx$22%). Within this subsample, complete and consistent demographic information was available for 210 participants, which formed the final dataset for this investigation. Among these participants, three racial groups are represented: American Indian (33; $\approx$16%), Black or African American (51; $\approx$24%), and White (125; $\approx$60%). One additional participant did not disclose their race but self-reported English as their native language (L1). Overall, the racial distribution in this subsample closely reflects both institutional demographics and those of the broader Accuplacer test-taking population.

Using the same placement classification criteria, we categorized these 210 participants into developmental and college-level groups, as summarized in Table 3.

From this table, it is observed that students classified as DevEd (in bold) accounted for approximately 64% of the placements within the subsample (210 participants), while those deemed proficient in writing (College Level; in italics) made up approximately 36%. Although smaller in size, this dataset, anchored in consistent demographic reporting, provides a foundation for the analyses examining the intersections between sociodemographic variables and linguistic features.

It is pertinent to note that all writing samples analyzed in this study were drawn from the institution’s official entrance exam database. The writing task required students to compose an argumentative response (in English) to a standardized prompt encouraging reflection on a social, educational, or ethical issue. Students were instructed to complete the task in a proctored environment, in a single sitting, and without access to dictionaries, the internet, or AI tools.

In addition to the Accuplacer classifications, all 210 essays with complete demographic information were manually annotated and evaluated for skill level by two trained raters following classification guidelines developed by the authors [8, 9]. To mitigate potential bias, multiple safeguards were implemented: all essays were anonymized, raters were blinded to students’ demographic information, and a calibration session was conducted prior to annotation to ensure scoring consistency. In alignment with TCC’s Review Board protocols, raters had no access to race, gender, or other sociodemographic data at any stage. While these measures reduced the risk of bias, we acknowledge that implicit bias may still persist.

The annotation process involved marking each text with a set of 11 Developmental Education-Specific (DES) linguistic features. These features captured both negative patterns (errors and deviations from proficiency) and positive patterns (indicators of advanced language use). They were organized into four clusters – Orthographic (ORT), Grammatical (GRAMM), Lexical and Semantic (LEXSEM), and Discursive (DISC), each reflecting a critical dimension of foundational writing proficiency, as summarized in Table 4.

Beyond the DES features, additional linguistic features were integrated from prior work [11], which included 106 from Coh-Metrix and 328 from CTAP. Although 434 features¹⁰¹⁰10A detailed description of these features can be found in the documentation of these tools. were originally extracted, only a subset was used in this study. Using Information Gain rankings from ML experiments, the most predictive features of placement were selected. Notably, two DES features – EXAMPLE and ORT – ranked 9th and 31st, respectively.

To emphasize the value of human-derived features, the remaining 9 DES features were retained, resulting in a final feature set of 40, as observed in Table 5. These DES features offer a finer-grained assessment of students’ communicative effectiveness in writing prior to beginning their academic programs [20] and have been documented before and associated with “discrete traits” of academic writing quality [29, 43]. Given ongoing concerns about disparities in placement outcomes across demographic lines, integrating these features with the sociodemographic variables identified for this study (gender and race) allows for a more comprehensive understanding of how writing traits intersect with placement decisions.

Three commonly used statistical tests, validated by the literature, were employed in this study: a one-way ANOVA, Tukey’s HSD, Chi-square test of independence.

The one-way ANOVA test was selected to determine whether statistically significant differences existed in linguistic feature means across groups with more than two categories, specifically gender and race, as supported in the literature [36]. For the interpretation of the $p$-values, we used the following thresholds, commonly mentioned in the literature [38]:

indicates highly significant group differences;
indicates statistically significant group differences;
indicates marginally significant group differences;
$p\ge 0.10$ indicates not significant group differences

While a multi-way ANOVA could have been used to test for interaction effects between gender and race, it was not applied here due to sample size limitations across intersecting subgroups (race × gender). To meet the assumptions of normality and homogeneity of variances, which are required for valid interaction testing, we opted for separate one-way ANOVAs to analyze each variable independently.

Following the ANOVA calculations, we conducted Tukey’s HSD tests to perform pairwise comparisons. This post-hoc analysis identified which specific groups differed significantly and gauged the magnitude of those differences. Tukey’s HSD has also been used in prior writing studies examining complexity and accuracy in student writing proficiency [4].

Finally, to examine potential associations between the demographic variables in this study and placement levels, both Accuplacer and human assigned, we used the Chi-square test of independence, appropriate for the categorical nature of placement outcomes.

To address the second research question, namely, to what extent gender and race correlate with the use of linguistic features in student writing, we applied a one-way ANOVA test to the 210 essays completed by native English speakers. The linguistic features described in Section 3.3 were analyzed against the available demographic data for this subset.

Through a one-way ANOVA test, the means of the 40 continuous (ratios) linguistic variables were compared across three gender groups (male, female, other/undisclosed). In preparation for the calculations, we adopted the conventional $\alpha =0.05$ threshold. The degrees of freedom between groups (df_B) were calculated using: ${\text{df}}_B=k-1$, yielding df_B = 2 given the three gender groups. For the degrees of freedom within groups (df_W), the following formula was used: ${\text{df}}_W=N-k$. While we have 210 individual data points, 3 groups’ means are calculated; therefore, df_W = 207, which is the number of degrees of freedom that remain for estimating variability within the groups. For subsequent experiments the same formulas were employed. All statistical calculations in this study were performed using Python.

When the ANOVA calculations were performed, 36 out of the 40 linguistic features showed no statistically significant differences across the three gender groups, as summarized in Table 6. Of the remaining four features, Positive connectives incidence (in bold), which captures words that aid in discourse flow and cohesion (e.g., however, similarly), showed a $p$-value indicating a significant difference across gender groups ($p=0.007$), a textual characteristic previously examined in writing assessment research [37]. The other three features (in italics), Lexical Richness: HDD (excluding punctuation and numbers) ($p$-value = 0.051); Lexical Richness: Type Token Ratio (STTR NGSLeasy Nouns) ($p$-value = 0.094); and Number of POS Feature: Existential there Types ($p$-value = 0.064), indicating marginally significant group variation, which warrants further investigation.

Following the same statistical approach described in Section 4, we examined whether differences emerged in the use of all 40 linguistic features across four racial groups (American Indian, Black or African American, White, Other/Undisclosed). This time df_B = 3 given the four racial groups. While we still have 210 individual data points, 4 groups’ means are calculated; therefore, df_W = 206.

As shown in Table 7, 31 features showed no meaningful differences across racial groups (compared to 36 features in the gender-based analysis).

The remaining nine features, particularly those connected to syntax complexity, discourse markers, and lexical precision, revealed differences among racial groups. Out of the nine, four of them (bolded), namely, Positive Connectives Incidence, PUNCT, WORDOMIT, and PRECISION, yielded $p$-values that indicated significant differences across race groups. Two features (also bolded), Lexical Sophistication (SUBTLEX Logarithmic Word Frequency) and Number of Syntactic Constituents: Prenominal Noun Modifier, produced $p$-values suggesting statistically significant group differences. The remaining three (in italics), WordNet Verb Overlap, ORT, and REASON, were considered marginally significant, indicating that how these groups exhibit (or not) these linguistic features in their written productions requires further investigation.

Five of these features belong to the DES feature set, suggesting that racial group differences may be more pronounced when assessment focuses on more targeted, humanly-devised indicators of developmental writing. Notably, the Positive Connectives Incidence feature emerged as significant in both the gender and race analyses, potentially highlighting its importance in capturing variation in students’ writing patterns. The differences here analyzed were not uniformly distributed across all features, indicating that group identity may influence only specific aspects of writing.

While the one-way ANOVA identified overall group differences by gender and race, Tukey’s HSD post-hoc test was used to determine which specific groups differed from one another.

For the gender analysis, we zeroed in on the statistical differences highlighted in Section 4¹¹¹¹11For brevity, Tukey’s HSD tests were only conducted on this feature set; other features, including those identified for racial groups, will be analyzed in future work.. The sample analyzed consisted of 141 females (F) and 64 males (M), for a total of 205 participants. The “Other/Undisclosed” category, which had only 5 participants, was excluded in an attempt to make comparisons involving the other two groups as statistically stable as possible. Consequently, for the Tukey’s HSD test, the following values were used: $k=2$ (groups: Male, Female); df_W = 203; $\alpha =0.05$; $q_{\text{critical}}=2.788$.

From Table 8, we see that all pairwise comparisons by gender were not statistically significant at $\alpha =0.05$, as the calculated $q$-statistics for all four linguistic features examined fell below the critical value of $q_{\text{critical}}=2.788$. Although some mean differences, particularly for Positive Connectives Incidence were notable, they did not exceed the threshold for significance. Consequently, we cannot reject the null hypothesis, which posits no meaningful differences in the use of linguistic features across gender groups. These findings should be interpreted with caution, as the limited sample size and unequal group distributions could limit the statistical power to detect more subtle effects.

We then turned to the post-hoc analysis for the race groups. For Tukey’s HSD test, the following values were used: $k=3$ [race groups: American Indian (1), Black or African American (2), and White (3)]. The Other/Undisclosed race category previously mentioned included only one student. Due to this small count, this category was excluded from the statistical analysis in an attempt to preserve the reliability of the test results. For df_W = 208; $\alpha =0.05$; $q_{\text{critical}}=2.788$.

As shown in Table 9, statistically significant differences (in bold) were found for WORDOMIT, PUNCT, and PRECISION between Groups 1 and 2 as well as Groups 2 and 3. A borderline yet statistically significant difference in Positive Connectives Incidence was also observed between Groups 1 and 2 only. Based on the individual means already reported in Table 7, students who identified as Black or African American tended to exhibit a higher incidence of omitted or left-out parts of speech compared to both American Indian and White students. These omissions may affect the overall coherence of their writing. In contrast, American Indian and White students showed nearly identical omission ratios.

Students identifying as Black or African American also used punctuation (e.g., for joining clauses) less frequently than their peers, and exhibited a higher rate of imprecise word usage to describe a concept, as captured by the PRECISION feature. On the contrary, Black or African American students showed a higher incidence of Positive Connectives than American Indian students, suggesting greater use of linking expressions such as however or similarly. These differences may reflect dialectal or regional variation rather than deficits in academic ability [34]. However, given that placement decisions are tied to institutional standards that may not fully account for this linguistic diversity, it is essential that pedagogical interventions not be seen as merely corrective measures but as opportunities to support students in expanding their register and linguistic skills to better navigate academic expectations.

It is equally important to acknowledge that these differences were identified from samples produced at a single point in time, and that external factors, such as test anxiety or unfamiliarity with the writing prompt, may have also influenced students’ performance.

To evaluate the possible association between gender and Accuplacer placement levels, in response to our third research question, a Chi-square test of independence was performed as the final step in this analysis. Table 10 presents the observed counts, expected counts under the null hypothesis of independence, and the individual Chi-square contributions ${(O-E)}^2/E$ for each cell.

The expected counts were calculated based on the assumption that gender and placement level are independent, and the ${(O-E)}^2/E$ values represent each cell’s contribution to the overall Chi-square statistic. The total sum of these contributions yielded the total Chi-square value $({\chi }^2=2.467)$. To assess statistical significance, we compared the calculated ${\chi }^2$ value to the critical Chi-square value $({\chi }_{\text{critical}}^2=5.991)$, determined for $df=2$ and $\alpha =0.05$. Alternatively, the associated $p$-value $(p=0.291)$ was considered. Since and $p>0.05$, we cannot reject the null hypothesis and conclude that there is no statistically significant association between gender and the automatically assigned skill levels via Accuplacer.

The same procedure was applied using the human-assigned skill levels, with identical degrees of freedom and significance thresholds. Because human raters assigned different placement levels than Accuplacer, the distribution of texts by levels varies.

The expected counts, as included in Table 11, were calculated based on the same assumption that gender and placement level are independent, and the ${(O-E)}^2/E$ values represent each cell’s contribution to the overall Chi-square statistic. The total sum of these contributions yielded the total Chi-square value $({\chi }^2=0.415)$. The critical Chi-square value was the same $({\chi }_{\text{critical}}^2=5.991)$. Alternatively, the associated $p$-value $(p=0.812)$ was considered. Since and $p>0.05$, we also conclude that there is no statistically significant association between gender and the human-assigned placement levels.

We also examined whether there was an association first between racial groups and the classification by skill level produced by Accuplacer, and, second, with the human ratings. The distribution of placement levels across racial groups, along with their corresponding observed and expected counts and Chi-square contributions, is presented in Table 12.

The degrees of freedom were calculated based on 3 three levels (DevEd Level 1, DevEd Level 2, College Level) and 3 racial groups (American Indian, Black or African American, and White). The computed Chi-square statistic yielded a value of ${\chi }^2=5.952$, which does not exceed the critical value of 9.488 for 4 degrees of freedom at the $\alpha =0.05$ level. The associated $p$-value of $p=0.203$ is above the conventional threshold for statistical significance. Therefore, we cannot reject the null hypothesis, indicating no statistically significant association between race and Accuplacer placement level within this sample.

Finally, the assessment was repeated but with the humanly-assigned skill level. Table 13 summarizes the observed and expected counts and Chi-square contributions.

The computed Chi-square statistic yielded a value of ${\chi }^2=9.588$, which slightly exceeds the critical value of 9.488 for 4 degrees of freedom at the $\alpha =0.05$ level. The associated $p$-value of $p=0.048$ falls just below the conventional threshold for statistical significance. While this suggests a possible association between race and human-assigned placement levels, even though raters did not have access to demographic data, the result is borderline and should be interpreted cautiously, warranting further investigation.

Upon analyzing each individual Chi-square contribution, we focused on values with the greatest discrepancies between the observed and expected counts within the race. This approach enabled us to identify, within the context of this sample, whether certain racial groups were overrepresented or underrepresented at specific placement levels. A higher observed count relative to the expected suggests overrepresentation, with a lower observed count indicating underrepresentation (italicized values).

In the DevEd Level 1 category, students who self-identified as Black or African American appeared to be more frequently than expected placed within this level, which may help explain their comparatively lower representation at the College Level. In contrast, students who identified as White were more frequently placed in DevEd Level 2 and College Level, indicating a higher-than-expected concentration in those categories. The variations between the observed and expected contributions for American Indians, within each placement level, were not as pronounced as those of the other groups.