We’re starting this Q&A section with a great question submitted by Jen on 2011/07/15 at 2:57 pm as a comment on the following post:
http://rateofimprovement.com/roi/2011/02/22/nasp-convention-2011/#comments
I am in charge of RtI for our district and this year we are going from use of AIMSWeb, which made this simple, to use of DRA and Fountes and Pinnell Benchmark Assessment System. These systems, unlike AIMSWeb, only offer guided reading levels, but will not be helpful in progress monitoring more frequently unless we can determine a productive rate of improvement. We are looking to have our students gain 1.5 years in 1.0 year. Any thoughts on how we can create a formula for this for teachers to use so they aren’t just guessing. DRA and F& P by the way do not go by 1 level at a time but more like 2, 4, 6 and they are not the same. I was thinking to look at 1.5 years ahead of whereever the student is currently at and expect that for the entire year and then break that down by the # of weeks in the year to determine where the student should be at each week.
Your thoughts?
We also wanted to figure out a way to calculate in the strategies expected at those levels, such as this book level, but also this many (2) strategies every 2 weeks.
Again, your thoughts?
Thank SO much!
Glad I found this site!
Jen
About Caitlin
Jen,
While the DRA is very popular is some regions and can be useful (in the right hands) as a diagnostic tool, it really does not have the psychometric properties needed to provide any solid insight in terms of student growth – at least as a standalone measure. I would suggest reading the McCarthy & Christ (2010) test review on the DRA2 for the strengths and weakness. That being said, many districts/LEAs use the DRA and when it is used in a way that takes advantage of its strengths, it can provide useful information. Ideally, instead of using only the DRA2 for your assessment tool, you would want to use it as part of a more comprehensive assessment model; a model that would also include measures with established reliability and validity like AIMSweb or other CBM probes. A list of tools and their evidence base can be found at http://www.rti4success.org/progressMonitoringTools. None of these tools are perfect (see Ardoin & Christ, 2009, Ardoin & Christ, 2008, and Christ, 2006); however, when you see a student with low levels and RoI on a CBM AND with low levels and not much change on a DRA AND similar limited results on other evaluations (ranging from State Assessments to classroom/teacher tests & measures, etc.), you can conclude that based on ALL the evidence you have, the student is struggling.
Unfortunately, it sounds like it’s a bit late for debate over what tools should be used, so here are a few things to consider. Because of the statistical weaknesses of the DRA2, any approach to look at RoI is going to be flawed. That doesn’t mean you should give up, but rather, keep in mind that your results stand on rather shaky ground. I have used our RoI via OLS LR approach to look at other assessment results here in PA simply b/c in certain grades we had no other data. The data we got was very weak, psychometrically, but it did allow us to at least start to understand to what degree students were learning. We just had to be very careful not to make high stakes decisions based on the results, something you would want to avoid with the DRA2, as well. This is not the best option, but if you can do nothing else, it’s better than nothing.
To buttress the DRA2 data that you’ll have, you might talk with others in your district about using DIBELS or EASYCBM probes, too. I think you can still access these at no cost if you are willing to manage the data yourself (via Excel is pretty easy, depending on the size of your district). If you could pull this off, you could work toward the multi-measurement model mentioned above. An option here might be to just use the CBM probes for Tier 2 and 3 kids so they can be monitored weekly or every other week.
You’ll see that for the most part I’ve avoided the heart of your question, and that is on purpose. Trying to use the DRA2 in the way in which your district intends is very much like me trying to use my VW Jetta in a car race. From the outside, my car doesn’t look all that different from a race car, 4 wheels, an engine, windows, etc. (good face validity!), but when experts who really know cars open up the hood, the very real differences are exposed. Yes, I can take my car to the track and try to pretend to fit in, but when the green flag is dropped, I will be unable to keep up. I really don’t think you’re going to have too much luck forcing the DRA2 onto the RoI/student growth race track. Instead of faking it, try to use it as it can be used best (again, see McCarthy & Christ (2010) for details) and at the same time, look at how other tools can be brought into your garage. By doing this, you’ll not only be able to race, but also able to drive to work, haul the family around, and even go off-roading every once in a while…
Ardoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an experimental passage set. School Psychology Review, 38(2), 266-283.
Ardoin, S. P. & Christ, T. J. (2008). Evaluating curriculum-based measurement slope estimates using triannual universal screenings. School Psychology Review, 37(1), 109-125.
Christ, T. J. (2006). Short-term estimates of growth using curriculum-based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35(1), 128-133.
McCarthy, A. M. & Christ, T. J. (2010) The developmental reading assessment-second edition (DRA2). Assessment for Effective Intervention, 35 (3) 182-185.