Papers and Publications

  • Rutkowski, D. & Rutkowski, L. (accepted). Is the test still too difficult? The case of PISA for Development. Comparative Education Review.
  • Svetina, D., Rutkowski, L., & Rutkowski, D. (2020). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling: A Multidisciplinary Journal, 27(1), 111-130.
  • Rutkowski, D., Thompson, G., & Rutkowski, L. (2020). Understanding the Policy Influence of International Large-Scale Assessments in Education. Reliability and Validity of International Large-Scale Assessment, 261.
  • Tijmstra, J., Liaw, Y., Bolsinova, M., Rutkowski, L., & Rutkowski, D. (2020). Sensitivity of the RMSD for detecting item-level misfit in low-performing countries. Journal of Educational Measurement, early view online. https://doi.org/10.1111/jedm.12263
  • Rutkowski, L. & Valdivia Medinaceli, M. (2020). Review of Computerized adaptive and multistage testing with R: Using packages catR and mstR. Journal of Educational and Behavioral Statistics, 45(1), 108-115.
  • Svetina, D., Rutkowski, L., & Rutkowski, D. (2019). Multiple-group invariance with categorical outcomes using updated guidelines: An illustration using Mplus and the lavaan/semTools packages. Structural Equation Modeling, online first, https://doi.org/10.1080/10705511.2019.1602776.
  • Rutkowski, L. & Rutkowski, D. (2019). Methodological challenges to measuring heterogeneous populations internationally. In L. Suter, E. Smith, and B. Denman (Eds.), The Sage Handbook on Comparative Studies (p. 124-138). London: Sage.
  • Svetina, D., Liaw, Y.-L. Rutkowski, L., & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in international large-scale assessments. Journal of Educational Measurement, 56(1), 192-213. doi.org/10.1111/jedm.12206
  • Sandoval-Hernandez, A., Rutkowski, D., Matta, T., & Miranda, D. (2019). Back to the drawing board: Can we compare socioeconomic background scales? Pensémoslo de nuevo:¿ Podemos comparar las escalas de antecedentes socioeconómicos?. Quarterly Journal Starting year: 1952, 383, 164037.
  • Rutkowski, L., Svetina, D., & Liaw, Y.-L. (2019). The Effects of Collapsing Ordered Categorical Variables on Tests of Measurement Invariance. Structural Equation Modeling: A Multidisciplinary Journal. 
  • Rutkowski, L., Rutkowski, D., & Liaw, Y.-L. (2019). The Existence and Impact of Floor Effects for Low-Performing PISA Participants. Assessment in Education: Principles, Policy & Practice.
  • Rutkowski, D., Rutkowski, L., & Liaw, Y.-L. (2018). Measuring Widening Proficiency Differences in International Assessments: Are Current Approaches Enough? Educational Measurement: Issues and Practice, 37(4), 40–48.
  • Engel, L. C., Rutkowski, D., & Thompson, G. (2019). Toward an international measure of global competence? A critical look at the PISA 2018 framework. Globalisation, Societies and Education, 17(2), 117-131.
  • Matta, T. H., Rutkowski, L., Rutkowski, D., & Liaw, Y.-L. (2018). lsasim: an R package for simulating large-scale assessment data. Large-scale Assessments in Education, 6(1), 15.
  • Treviño, E., Sandoval-Hernández, A., Miranda, D., & David, R. (2019). Invariance of socioeconomic status scales in international studies. In Validity of Evaluation Systems in Latin America. Springer.
  • Liaw, Y.-L., Wu, Y., Rutkowski, D., & Rutkowski, L. (2018). Evaluating PISA scales across Chinese economies. Asia Pacific Journal of Education, 38(3), 432-451.
  • Rutkowski, D., Rutkowski, L., & Liaw, Y. (accepted). Tailoring achievement tests to heterogeneous populations: Are latent trait estimates improved? Educational Measurement: Issues and Practice.
  • Rutkowski, D., & Rutkowski, L. (2018). No One Likes a Bully: How Systematic Is International Bullying and What Relationship Does It Have with Mathematics Achievement in 4th Grade? IEA Compass: Briefs in Education. No. 1. International Association for the Evaluation of Educational Achievement.
  • Rutkowski, D., Rutkowski, L., Wild, J., & Burroughs, N. (2018). Poverty and educational achievement in the U.S.: A less biased estimate using PISA 2012 data. Journal of Children and Poverty, 24(1), 47-67.
  • Rutkowski, D. & Rutkowski, L. (2018). No one likes a bully: How systematic is international bullying and what relationship does it have with mathematics achievement in 4th grade? IEA Compass: Briefs in Education, 1.
  • Oliveri, M. E., Rutkowski, D., & Rutkowski, L. (2018). Bridging Validity and Evaluation to Match International Large‐Scale Assessment Claims and Country Aims. ETS Research Report Series.
  • Engel, L. & Rutkowski, D. (2018). Pay to play: What does PISA participation cost in the US? Discourse: Studies in the Cultural Politics of Education. DOI: 10.1080/01596306.2018.1503591
  • Oliveri, M. E., Rutkowski, D., & Rutkowski, L. (2018). Bridging Validity and Evaluation to Match International Large‐Scale Assessment Claims and Country Aims. ETS Research Report Series.
  • Rutkowski, D. (2018). Improving international assessment through evaluation. Assessment in Education: Principles, Policy & Practice. 25(1). 127-136. https://doi.org/10.1080/0969594X.2017.1300572
  • Sellar, S., Takayama, K. & Rutkowski, D. (2018). Student preparation for large-scale assessments: A comparative analysis. In B. Maddox (Ed). International large-scale assessment in education: Insider research perspectives. London: Bloomsbury Publishing.
  • Sandoval-Hernández, A., & Rutkowski, D. (2018). Tailored Background Scales in Large Scale Assessment. In Comparative and International Education Society.
  • Rutkowski, L. & Rutkowski, D. (2017). Improving the comparability of international assessments: A look back and a way forward. Scandinavian Journal of Educational Research. https://dx.doi.org/10.1080/00313831.2016.1261044.
  • Sellar, S., Thompson, G., & Rutkowski, D. (2017). The Global Education Race: Taking the Measure of PISA and International Testing. Calgary: Brush Education.
  • Svetina, D. & Rutkowski, L. (2017). Multidimensional measurement invariance in an international context: Fit measure performance with many groups. Journal of Cross-Cultural Psychology, 48(7), 991-1008. https://doi.org/10.1177/0022022117717028
  • Rutkowski, L., & Svetina, D. (2017). Measurement invariance in international surveys: Categorical indicators and fit measure performance. Applied Measurement in Education, 30(1), 39-51.
  • Rutkowski, L., & Rutkowski, D. (2016). A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher, 45(4), 252-257.

Working Papers and Papers in Progress

  • Rutkowski, L. & Matta, T. (in preparation). Design and treatment of missing auxiliary data in large-scale assessments.
  • Rutkowski, L. & Rutkowski, D. (under review). International-large scale assessments: Methods and applications for educational and social researchers. New York: Guilford Press.
  • Rutkowski, L. & Rutkowski, D. (under review). Multistage test design considerations in international large-scale assessments of educational achievement. In L. Khorramdel, M. von Davier, & K. Yamamoto (Eds.), Innovative Computer-Based International Large-Scale Assessments. London: Springer.
  • Rutkowski, L. Rutkowski, D., Svetina, D., & Liaw, Y.L. (under review). Multistage test design considerations for measuring heterogeneous populations. Manuscript submitted to Applied Psychological Measurement.
  • Rutkowski, L., Zhou, Y., & Matta, T. (under review). One approach for designing and treating missing data in latent regression covariates. Manuscript submitted to Journal Educational and Behavioral Statistics.
  • Bolsinova, M., Tjimstra, J., Rutkowski, D., & Rutkowski, L. (in preparation). Generalizability of cross-cultural measurement differences: A permutation test method.
  • Rutkowski, D., Rutkowski, L., Svetina, D., & Withrow, A. (in preparation). School-level inferences: What can an adaptive test achieve?
  • Rutkowski, L., Withrow, A., Rutkowski, D., & Svetina, D. (in preparation). Survival analysis: A model for understanding test-quitting behavior cross-nationally.
  • Rutkowski, L. & Rutkowski, D. (in preparation). Introduction to international large-scale assessments. NCME ITEMS module.
  • Bolsinova, M., Rutkowski, L., Tijmstra, J., & Rutkowski, D. (in preparation). Explaining differences in test-taking time: An empirical example of Simpson’s paradox.
  • Valdivia, M., Rutkowski, L., Svetina, D., & Rutkowski, D. (in preparation). Differential item functioning under a multistage test design.

Presentations

Highlights of 2018 Frontiers in Educational Measurement (FREMO) conference in Oslo, Norway

  • Rutkowski, L., Withrow, A., Rutkowski, D., & Svetina, D. (2020). Survival models in international assessment: A model for understanding quitting behavior. 2020 International Meeting of the Psychometric Society, online.
  • Svetina, D., Rutkowski, L., Liaw, Y.-L., Rutkowski, D. (2019). Module length and routing methods in a multistage test design: Design and implementation considerations. 2019 IEA Research Conference, Copenhagen, Denmark.
  • Liaw, Y.-L., Rutkowski, L., & Rutkowski, D. (2019). Effects of aberrant responding in multistage testing and the changes of country rankings. 2019 IEA Research Conference, Copenhagen, Denmark.
  • Validivia, M., Rutkowski, L., Svetina, D., & Rutkowski, D. (2019). Differential item functioning in multistage testing. 2019 IEA Research Conference, Copenhagen, Denmark.
  • Rutkowski, L., Svetina, D., Liaw, Y.-L., & Rutkowski, D. (2019). Parameter estimation stability and probabilistic routing in multistage testing. 2019 IEA Research Conference, Copenhagen, Denmark.
  • Rutkowski, L. & Rutkowski, D. (2019). No one likes a bully: How systematic is international bullying and what relationship does it have with mathematics achievement in 4th grade? 2019 IEA Research Conference, Copenhagen, Denmark.
  • Liaw, Y.L., Bolsinova, M., Rutkowski, D., Rutkowski, L. & Tijmstra, J. (2019) RMSD: Limitations to DIF detection in low-performing populations. 2019 IEA Research Conference, Copenhagen, Denmark.
  • Liaw, Y.L., Bolsinova, M., Rutkowski, D., Rutkowski, L. & Tijmstra, J. (2019) RMSD: Limitations to DIF detection in low-performing populations. 2019 National Council on Measurement in Education Meeting: Toronto, Canada.
  • Svetina, D., Liaw, Y.L., Rutkowski, L., & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in international large-scale assessments. 2019 National Council on Measurement in Education Meeting: Toronto, Canada.
  • Liaw, Y (2018, September). Peculiar subgroup’s aberrance response behavior in multistage adaptive testing: A simulation study. 2018 Frontiers in Educational Measurement: Oslo, Norway.
  • Rutkowski, L. (2018, September). Can multistage testing bridge the cultural measurement divide? 2018 Frontiers in Educational Measurement: Oslo, Norway.
  • Mughogho, K. (2018, September). IRT model specification’s impact on subscale score estimation in international large scale assessments. Poster session presented at the 2018 Frontiers in Educational Measurement: Oslo, Norway.
  • Rutkowski, L. (2018, July). Increased heterogeneity in international assessments and associated measurement challenges. Invited keynote talk at 2018 International Test Commission annual meeting: Montreal, Quebec.
  • Rutkoski, L., Rutkowski, D., & Liaw, Y. L., (2018, April). Tailored booklets: Improved estimates of latent traits in large-scale assessment? 2018 National Council on Measurement in Education Meeting: Ney York, NY.
  • Rutkowski, L. von Davier, M., & Rutkowski, D. (2018). Advances in measurement and research methodology of large-scale assessments (AERA Division D Awardee Presentation). 2018 American Educational Research Association Meeting: New York, NY.
  • Rutkowski, L. (2018, April). Comments on International Education Assessments: Cautions, Conundrums, and Common Sense. 2018 National Academy of Education Methods and Policy Uses of International Large-Scale Assessments Meeting: Washington, DC.
  • Rutkowski, D. (2018, April). Assessing global competency with diverse populations. ULead Conference, Banff, Canada.
  • Rutkowski, D. (2018, April). International assessment and personal learning. Twin Peaks Conference, Banff, Canada.
  • Rutkowski, D. (2018, April). The international math wars. Calgary public teachers. Calgary, Canada.
  • Rutkowski, L. (2018). Comments on International Education Assessments: Cautions, Conundrums, and Common Sense. 2018 National Academy of Education Methods and Policy Uses of International Large-Scale Assessments Meeting: Washington, DC.
  • Rutkowski, D. (2018, April). International assessment and personal learning. Twin Peaks Conference, Banff, Canada.
  • Rutkowski, D. (2018, April). The international math wars. Calgary public teachers. Calgary, Canada.
  • Rutkowski, D. (2018, March). The impact of floor effects in PISA for low-performing countries. CIES 2018. Mexico City, Mexico.
  • Rutkowski, D. (2018, February). Primer on standardized assessment and why this is a contested space. Keynote session: School Leadership: Supporting Sound Assessment in the era of Datafication, Red Deer, Alberta Canada.
  • Rutkowski, D. (2017, December). The global education race. Norwegian Teacher’s Union, Oslo, Norway.
  • Rutkowski, D. (2017 July). A call for a more measured approach to reporting and interpreting PISA results.  University of Bath, UK.
  • Rutkowski, L. (2017, March). Methodological challenges to measuring heterogeneous populations internationally. 2017 Comparative and International Education Society Conference: Atlanta, Georgia.
  • Rutkowski, D (2017, February). TIMSS and PISA: Different tests, different purposes. Same policies? Seminar on Large Scale Assessment. Educational Research Institute. Ljublijana, Slovenia.
  • Rutkowski, D. (2016, November). Problematizing PISA: Widening the debate about international large-scale assessments. Canadian Teacher’s Association Annual Meeting.

Conferences

8th IEA International Research Conference, 2019 – Copenhagen, Denmark (https://www.iea.nl/news-events/irc/8th-international-research-conference)

  • The future of international assessment? The promise and challenge of a multistage design in IEA studies, Chair: Leslie Rutkowski

Rutkowski, L., Svetina, D., Liaw, Y.-L., & Rutkowski, D. (2019). Parameter estimation stability and probabilistic routing in multistage testing. 2019 IEA Research Conference, Copenhagen, Denmark.

Svetina, D., Rutkowski, L., Liaw, Y.-L., Rutkowski, D. (2019). Module length and routing methods in a multistage test design: Design and implementation considerations. 2019 IEA Research Conference, Copenhagen, Denmark.

Valdivia, M., Rutkowski, L., & Rutkowski, D. (2019). Differential item functioning in multistage testing. 2019 IEA Research Conference, Copenhagen, Denmark.

Liaw, Y.-L., Rutkowski, L., & Rutkowski, D. (2019). Effects of aberrant responding in multistage testing and the changes of country rankings. 2019 IEA Research Conference, Copenhagen, Denmark.

  • Mughogho, K., Rutkowski, L., & Rutkowski, D. (2019). An application of PRMSE to evaluate subscale score value in TIMSS 2015 eighth grade mathematics. A poster. 2019 IEA Research Conference, Copenhagen, Denmark.

American Educational Research Association (AERA)/ National Council on Measurement in Education (NCME), 2018 – New York, New York

  • Rutkowski, L. von Davier, M., & Rutkowski, D. (2018). Advances in measurement and research methodology of large-scale assessments (AERA Division D Awardee Presentation). 2018 American Educational Research Association Meeting: New York, NY.
  • Rutkowski, L., Rutkowski, D., & Liaw, Y.L., (2018, April). Tailored booklets: Improved estimates of latent traits in large-scale assessment? 2018 National Council on Measurement in Education Meeting: New York, NY.

International Meeting of the Psychometric Society, 2017 – Zurich, Switzerland (https://www.psychometricsociety.org/)

  • Missing Data in Large-Scale Educational Assessments, Chair: Tyler Matta

Rutkowski, L., Matta, T. (July, 2017). Design and treatment of missing auxiliary data in large-scale assessments.

Robitzsch, A., Lüdtke, O. (July, 2017). An item response model for omitted responses in performance tests.

Grund, S., Lüdtke, O., Robitzsch, A. (July, 2007). Imputation of missing data at level 2 using plausible values.

Matta, T. (March, 2017). Assessing missing data assumptions using posterior predictive checks. .

7th IEA International Research Conference, 2017 – Prague, Czech Republic (http://www.iea.nl/7th-iea-international-research-conference)

  • Embracing Heterogeneity in International Large Scale Assessments Symposium

Sandoval-Hernandez, A., Rutkowski, D., & Matta, T. (June, 2017). Back To The Drawing Board: Can We Compare Background Scales?

Rutkowski, L., Svetina, D., & Liaw, Y. L. (June, 2017). The Impact of Arbitrary Collapsing Choices in Categorical MG-CFA: A Look at Model Fit and Reliability.

Oliveri, M. E., Rutkowski, D., & Rutkowski, L. (June, 2017). Bridging Validity & Evaluation to Help Understand ILSA Utility, Value, and Meaning For Various Stakeholders.

Matta, T., Rutkowski, L., Rutkowski, D., & Liaw, Y. L. (June, 2017). lsasim: An R Package for Simulating Large-Scale Assessment Data.

Comparative and International Education Society Annual Conference, 2017 – Atlanta, Georgia (http://www.cies2017.org/)

Trevino, E., Inostroza, P., & Sandoval-Hernandez, A. (March, 2017). Evaluating measurement invariance in international large-scale assessments.

Miranda, D., Castillo, J., & Sandoval-Hernandez, A. (March, 2017). Young citizens participation: An emprical test of a conceptual model.

Rutkowski, L. (March, 2017). Methodological challenges to measuring heterogeneous populations internationally.