Exploring the Readability of Assessment Tasks: The Influence of Text and Reader Factors
Keywords:
Downloads
Abstract
Readability is the degree to which a text is matched to its intended and actual reader. The factors influencing readability, both text factors and reader factors, have been widely researched from the standpoint of attempts to maximise reader understanding of texts. The application of understandings in the area has not, however, always been applied systematically to the design and writing of assessment tasks and consequently test items are sometimes less accessible to the intended test takers than they might be.
This paper is an attempt to provide a wide ranging review of literature which bears on the task of the assessment designer in ensuring that assessment items measure what they are supposed to measure, and not just the reading abilities of the test takers.
Downloads
References
Abedi, J., Kao, J., Leon, S., Mastergeorge, A., Sullivan, L., Herman, J. & Pope, R. (2010): Accessibility of Segmented Reading Comprehension Passages for Students with Disabilities, Applied Measurement in Education, 23:2, 168-186
Google Scholar CrossrefAbedi, J., Leon, S., & Kao, J. C. (2008). Examining differential item functioning in reading assessments for students with disabilities. (CRESST Report 744). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
Google Scholar CrossrefAbedi, J., Leon, S., & Mirocha, J. (2003). Impact of student language background on content-based performance: Analyses of extant data (CSE Technical Report 603). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.
Google Scholar CrossrefAbedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurements: Issues and Practice, 19: 3, 16–26
Google Scholar CrossrefAhmed, A., & Pollitt, A. (2007). Improving the quality of contextualized questions: and experimental investigation of focus. Assessment in Education: Principles, Policy and Practice 14: 2, 201-32.
Google Scholar CrossrefAnderson, R., Wilkinson, I. & Mason, M. (1987). Do errors on classroom reading tasks slow growth in reading? Center for the Study of Reading Technical Report No. 404. Champaign: University of Illinois at Urbana-Champaign
Google Scholar CrossrefAssessment of Performance Unit (1985) A Review of Monitoring in Mathematics 1978-82. Slough: NFER
Google Scholar CrossrefBailey, A. (2000) Language Analysis of Standardized Achievement Tests: Considerations in the Assessment of English Language Learners. In Abedi, J., Bailey, A., Butler, F., Castellon-Wellington, M., Leon, S. & Mirocha, J. (Eds) The validity of administering large-scale content assessments to English language learners: An investigation from three perspectives (CSE Report 663) 85–105. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
Google Scholar CrossrefBarshi, I., & Healy, A. (2002). The effects of mental representation on performance in a navigation task. Memory and Cognition, 30, 1189-1203
Google Scholar CrossrefBeck, I., McKeown, M., Sinatra, G. & Loxterman, J. (1991). Revising social studies text from a text-processing perspective: evidence of improved comprehensibility. Reading Research Quarterly, 26, 251-276.
Google Scholar CrossrefBügel, K. & Buunk, B. (1996) Sex Differences in Foreign Language Text Comprehension: The Role of Interests Knowledge. The Modern Language Journal, 80: 1, 15-31
Google Scholar CrossrefButler, F. & Stevens, R. (1997) Accommodation Strategies for English Language Learners on Large-Scale Assessments: Student Characteristics and Other Considerations. (CSE Technical Report 448). National Center for Research on Evaluation, Standards, and Student Testing (CRESST) University of California, Los Angeles
Google Scholar CrossrefChall, J. S. & Conard, S. (1991). Should textbooks challenge students? The case for easier or harder books. New York: Teachers College Press.
Google Scholar CrossrefChall, J. S. & Dale, E. (1995) Readability revisited: the new Dale-Chall readability formula. Cambridge, Mass.: Brookline Books.
Google Scholar CrossrefChan, D., Schmitt, N., DeShon, R., Clause, C. & Delbridge, K. (1997). Reactions to cognitive ability tests: The relationships between race, test performance, face validity perceptions, and test taking motivation. Journal of Applied Psychology, 82, 300-310
Google Scholar CrossrefChapman, L.J. (1987) Reading from 5-11 Years. London: Open University Press.
Google Scholar CrossrefChi, M. T. H. (1985). Interactive roles of knowledge and strategies in the development of organized sorting and recall. In S. F. Chipman, J. W. Segal, & R. Glaser (Eds.), Thinking and learning skills (Vol. 2, pp. 457-484). Hillsdale, NJ: Erlbaum.
Google Scholar CrossrefCole, N. & Zieky, M. (2001) The New Faces of Fairness. Journal of Educational Measurement, 38: 4, 369-382
Google Scholar CrossrefConnor, U. (1996) Contrastive Rhetoric: Cross-cultural aspects on second-language writing. Cambridge: Cambridge University Press
Google Scholar CrossrefCrisp, V. (2011): Exploring features that affect the difficulty and functioning of science exam questions for those with reading difficulties, Irish Educational Studies, 30:3, 323-343
Google Scholar CrossrefDale, E., & Chall, J.S. (1948) A formula for predicting readability. Educational Research Bulletin, 27, 37-54.
Google Scholar CrossrefDowning, J. & Leong, C. (1982) Psychology of Reading. New York: Macmillan.
Google Scholar CrossrefEdwards, R. & Gibbon, V. (1973) Words Your Children Use: A Survey of the Words Used by Children in Infants' Schools with the Resultant Graded Vocabulary. London: Burke
Google Scholar CrossrefEyles, P., Skelly, J., & Lou Schmuck, M. (2003). Evaluating patient choice of typeface style and font size for written health information in an outpatient setting. Clinical Effectiveness in Nursing, 7: 2, 94-98
Google Scholar CrossrefFang, Z. (2006): The Language Demands of Science Reading in Middle School, International Journal of Science Education, 28:5, 491-520
Google Scholar CrossrefFeely, M., Rubin, G. S., Ekstrom, K., & Perera, S. (2005). Investigation into font characteristics for optimum reading fluency in readers with sight problems. International Congress Series, 1282, 530-533.
Google Scholar CrossrefFisher-Hoch, H., S. Hughes, and T. Bramley (1997). What makes GCSE exam questions difficult? Outcomes of manipulating difficulty of GCSE questions. Paper presented at the British Educational Research Association Annual Conference, September 11-14, University of York.
Google Scholar CrossrefFrancis, W. & Kucera, H. (1979). Brown Corpus Manual. Providence, Rhode Island: Brown University. Available at: http://icame.uib.no/brown/bcm.html
Google Scholar CrossrefFry, E. (1977) Elementary reading instruction. New York: McGraw-Hill.
Google Scholar CrossrefFulcher, G. (1989) Cohesion and coherence in theory and reading research. Journal of Research in Reading, 12: 2, 146-163.
Google Scholar CrossrefGambrell, L., Wilson, R. & Gantt, W. (1981) Classroom observations of task-attending behaviors of good and poor readers. Journal of Educational Research, 74: 400–404.
Google Scholar CrossrefGierl, M., Bisanz, J., Bisanz, G. & Boughton, K. (2003) Identifying Content and Cognitive Skills That Produce Gender Differences in Mathematics: A Demonstration of the Multidimensionality-Based DIF Analysis Paradigm. Journal of Educational Measurement, 40: 4, 281-306
Google Scholar CrossrefGierl, M. (2005) Using Dimensionality-Based DIF Analyses to Identify and Interpret Constructs That Elicit Group Differences. Educational Measurement: Issues and Practice, 24: 3-14.
Google Scholar CrossrefGilliland, J. (1975). Readability. London: Hodder and Stoughton.
Google Scholar CrossrefGraesser, A., McNamara, D., & Louwerse, M. (2003). What do readers need to learn in order to process coherence relations in narrative and expository text. In A. P. Sweet, & C. E. Snow (Eds.), Rethinking reading comprehension (pp. 82-98). New York: Guilford.
Google Scholar CrossrefGuthrie, J., & Cox, K. (2001) Classroom Conditions for Motivation and Engagement in Reading. Educational Psychology Review, 13: 3, 283-302.
Google Scholar CrossrefGuthrie, J. T., Solomon, A., & Rinehart, J. M. (1997) Literacy Issues in Focus: Engagement in Reading for Young Adolescents. Journal of Adolescent & Adult Literacy, 40: 6, 438-446.
Google Scholar CrossrefHalliday, M., & Hasan, R. (1976) Cohesion in English. New York: Longman.
Google Scholar CrossrefHamilton, L. (1998) Gender Differences on High School Science Achievement Tests: Do Format and Content Matter? Educational Evaluation and Policy Analysis, 20: 3, 179-195
Google Scholar CrossrefHarris, T. L. & Hodges, R. (1995) The literacy dictionary: the vocabulary of reading and writing. Newark, Del.: International Reading Association.
Google Scholar CrossrefHughes, L.E. & Wilkins, A.J. (2000) Typography in children’s reading schemes may be suboptimal: Evidence from measures of reading rate. Journal of Research in Reading, 23: 3, 314-324.
Google Scholar CrossrefJanan, D., Wray, D. & Pope, M. (2010) Paradigms in Readability Research, International Journal of Arts and Sciences, 3: 17, 19-29.
Google Scholar CrossrefJohnston, P. H. (1984) Prior knowledge and reading comprehension test bias. Reading Research Quarterly, 19: 3, 219-239.
Google Scholar CrossrefJust, M. & Carpenter,P.A. (1987). The psychology of reading and language comprehension. Newton, MA: Allyn & Bacon.
Google Scholar CrossrefJust, M.A., Carpenter, P.A., and Wooley, J.D. (1982) Paradigms and Processes in Reading Comprehension. Journal of Experimental Psychology, 111: 2, 228-238.
Google Scholar CrossrefKieffer, M., Lesaux, N., Rivera, M. & Francis, D. (2009) Accommodations for English Language Learners Taking Large-Scale Assessments: A Meta-Analysis on Effectiveness and Validity. Review of Educational Research, 79: 3, 1168–1201
Google Scholar CrossrefKintsch, W. & Keenan, J. (1973) Reading rate and retention as a function of the number of propositions in the base structure of sentences. Cognitive Psychology 5: 257-274.
Google Scholar CrossrefKintsch, Walter (1974). The representation of meaning in memory. Hillsdale: Lawrence Erlbaum.
Google Scholar CrossrefLouwerse, M.M. & Graesser, A.C. (2004) Coherence in discourse. In Strazny, P. (ed.) Encyclopedia of Linguistics. Chicago: Fitzroy Dearborn.
Google Scholar CrossrefMaccoby, E. (1990). Gender and relationships: A developmental account. American Psychologist, 45; 4, 513-20.
Google Scholar CrossrefMartin, M., Mullis, I & Kennedy, A. (2007) (Eds). PIRLS 2006 Technical Report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
Google Scholar CrossrefMillard, E. (1997) Differently Literate: boys, girls and the schooling of literacy. London: Falmer Press.
Google Scholar CrossrefMilone, M. 2008. The development of ATOS: The Renaissance readability formula. Wisconsin Rapids, WI: Renaissance Learning.
Google Scholar CrossrefMorgan, P., Farkas, G. & Hibel, J. (2008) Matthew Effects for Whom? Learning Disability Quarterly, 31: 4, 187-198
Google Scholar CrossrefNation, P. & Waring, R. (1997). Vocabulary size, text coverage and word lists. In Schmitt, N. and McCarthy, M. (Eds.) Vocabulary: Description, Acquisition and Pedagogy (6-19). Cambridge: Cambridge University Press.
Google Scholar CrossrefNewbold, N., & Gillam, L. (2010). The Linguistics of Readability: The Next Step for Word Processing. Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics and Writing, (June), 65-72. http://aclweb.org/anthology/W/W10/W10-0409.pdf
Google Scholar CrossrefNickson, M. & Green, S. (1996). A study of the Effects of Context in the Assessment of the Mathematical Learning of 10/11 year olds. Paper presented to the 1996 British Educational Research Association. http://www.cambridgeassessment.org.uk/ca/digitalAssets/113794_A_Study_of_the_Effects_of_Context_in_the_Assessment_of_Mathe.pdf
Google Scholar CrossrefNational Institute of Neurological Disorders and Stroke (NINDS) (2011). NINDS Attention Deficit-Hyperactivity Disorder Information Page. Bethesda, MD: National Institutes of Health http://www.ninds.nih.gov/disorders/adhd/adhd.htm
Google Scholar CrossrefO’Reilly, T. & McNamara, D. (2007). Reversing the reverse cohesion effect: good texts can be better for strategic, high-knowledge readers. Discourse Processes, 43, 121-152.
Google Scholar CrossrefOzuru, Y., Dempsey, K. & McNamara, D. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19, 228-242
Google Scholar CrossrefPerera, K. (1980) The assessment of linguistic difficulty in reading material, Educational Review, 32: 2, 151-161.
Google Scholar CrossrefPollitt A (1985) What makes examination questions difficult? Edinburgh: Scottish Academic Press.
Google Scholar CrossrefRasinski, T. V. (1990). The effects of cued phrase boundaries on reading performance: A review. Kent, Ohio: Kent State University, Department of Teacher Development and Curriculum Studies.
Google Scholar CrossrefReid, J. (1972) Children‘s comprehension of syntactic features found in extension readers. In Reid, J. (ed) Reading: Problems and Practices London: Ward Lock, 394-403
Google Scholar CrossrefRoussos, L., & Stout, W. (1996). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20, 355-371.
Google Scholar CrossrefSaarnio, D. A., Oka, E. R., & Paris, S. G. (1990). Developmental predictors of children’s reading comprehension. In T. H. Carr & B. A. Levy (Eds.), Reading and its development: components skills approaches (pp. 57–79). San Diego, CA: Academic Press.
Google Scholar CrossrefSchagen, I. & Sainsbury, M. (1996). Multilevel Analysis of the Key Stage 1 National Curriculum Assessment Data in 1995. Oxford Review of Education, 22: 3, 265-272
Google Scholar CrossrefSchmit, M. & Ryan, A. (1992). Test-taking dispositions: a missing link? Journal of Applied Psychology, 77, 629-637
Google Scholar CrossrefShorrocks-Taylor, D., Curry, J., Swinnerton, B. & Nelson, N. (2003) National Curriculum Mathematics Tests in England at Key Stage 2: Weights and measures? Oxford Review of Education, 29: 1, 51-66
Google Scholar CrossrefSonnleitner, P. (2008). Using the LLTM to evaluate an item generating system for reading comprehension. Psychology Science Quarterly, 50, 345-362
Google Scholar CrossrefSpache, G. (1974) Good reading for poor readers. Champaign, IL: Garrard.
Google Scholar CrossrefStanovich, K. (1986) Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21: 4, 360-407
Google Scholar CrossrefStone, E., Cook, L, Laitusis, C.C., & Cline, F. (2010). Using differential item functioning to investigate the impact of testing accommodations on an English-language arts assessment for students who are blind or visually impaired. Applied Measurement in Education, 23: 2, 132–152.
Google Scholar CrossrefStricker. L. & Emmerich, W. (1999) Possible Determinants of Differential Item Functioning: Familiarity, Interest, and Emotional Reaction Journal of Educational Measurement, 36: 4, 347-366
Google Scholar CrossrefThompson, C., & Shapiro, L. (2007). Complexity in treatment of syntactic deficits. American Journal of Speech-Language Pathology, 18, 30–42.
Google Scholar CrossrefThompson, S. J., Thurlow, M. L., & Malouf, D. (2004). Creating better tests for everyone through universally designed assessments. Journal of Applied Testing Technology. 6: 1. Available at www.testpublishers.org/atp.journal.htm
Google Scholar CrossrefWaller, RHW. (1991) Designing reports and presentations. Redhill: Monotype Desktop Solutions.
Google Scholar CrossrefWilkins, A., Cleave, R., Grayson, N. and Wilson, L. (2009) Typography for children may be inappropriately designed. Journal of Research in Reading, 32: 4, 402-412.
Google Scholar CrossrefZumbo, B. (2007) Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going. Language Assessment Quarterly, 4: 2, 223–233
Google Scholar CrossrefDownloads
Published
Almetric
Dimensions
How to Cite
Issue
Section
License
All articles are published under Creative Commons copyright (CC BY). Authors hold the copyright and retain publishing rights without restrictions, but authors allow anyone to download, reuse, reprint, modify, distribute, and/or copy articles as the original source is cited.
