November 23, 2017

Emerging Best Practice for Determining ROI #1

As part of my dissertation, I’ve compiled a list of what I think are emerging best practices for determining rate of improvement.  In other words, what are the necessary conditions for having confidence in the ROI statistic?  The first emerging best practice I’m considering is the use of technically adequate and psychometrically sound measures.  If we don’t have good data, there’s no point in creating a trend line or calculating an ROI statistic.  One of the best resources is the progress monitoring tools chart.  Note that this chart can still be accessed through the National Center on Response to Intervention but is now housed at the National Center on Intensive Intervention.  Not all assessments are located on this chart as they are submitted voluntarily but it’s good place to start if your school team is looking for ideas.

When school-based teams are considering making high stakes decisions such as special education eligibility, the quality of data needed is much higher.  Teams need to consider if the assessments they are using is measuring what they intends to measure (validity) and if they can measure skills consistently (reliability).  Other aspects of assessments to consider in relation to generating a stable trend line are whether the measure can demonstrate small increments of growth and if the assessments can be repeated through alternate forms.  An example of a technically adequate measure that was designed to produce ROI is the computer adaptive tests from Renaissance Learning called STAR Reading and STAR Math.  After only four data points, the system will generate a stable trend line that can be used to interpret student progress.  A non-example would be teacher-made assessments or unit tests.  The latter measures may be helpful for teachers to know how students are performing with concepts they are learning in class, but have not been validated for the purpose of generating trend lines or ROI from the results.

Questions and comments are welcome! What are your school teams using to document student progress?

FacebookTwitterPrintFriendlyPinterestTumblrGoogle+EmailShare

Q&A – Realistic vs. ambitious goals for early literacy skills

ROI6

 

Great question from Beth:

I have seen where there is a chart to show what a realistic vs. ambitious goal for Rate of Improvement is for R-CBM or Oral Reading Fluency but is there a chart like that for Letter Naming Fluency or Letter Sound Fluency?  Or can the ROI charts for R-CBM charts be used for other areas of progress monitoring as well??

Response:

The majority of research studies that I have come across have been related to determining expected growth for oral reading fluency, most likely because it is the most popular form of CBM.  The study below is the study where the growth charts originated from for digits correct and oral reading fluency.

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.

The idea of realistic and ambitious goal setting is also described in more detail in a chapter of Best Practices for School Psychologist, Volume 5.

Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes (Eds.), Best practices in school psychology V.  (Vol. 2, pp. 141-157). Bethesda, MD: National Association of School Psychologists.

The amount of growth you can expect from students is dependent on a number of factors such as the intensity, duration, and frequency of the intervention; the skill being measured, and the fidelity with which the intervention is delivered.  You may want to see if the assessment system you are using includes norms for rate of improvement.  If none are provided, then you could calculate the typical ROI for students based on the expected benchmark scores and use that as a comparison.  Remember that students scoring below benchmark would require an ROI that is more than the ROI of their typical peers in order to close the achievement gap.  There are many dissertations to be had in this area! It would be great to have more published, peer reviewed studies that review the progress students typically make with specific skills given specific interventions.

FacebookTwitterPrintFriendlyPinterestTumblrGoogle+EmailShare

Q&A – Rate of improvement for EasyCBM

ROI6

Great question from Leasha:

Please help! Our new state policy manual has listed your website as the place to go for calculating our rate of learning for the upcoming school year. Pretty much every county in the state uses Dibels so it is set up and ready to go for them. However, my county uses EasyCBM for reading. Is it possible for me to just edit the benchmark goals, and then the Excel would calculate it properly for students entered, or would I need to do some special tweaking?

My Response:

Absolutely! You can adjusts the benchmarks to whatever you need them to be and the Excel sheet should automatically adjust the calculations for ROI.  Good luck!

FacebookTwitterPrintFriendlyPinterestTumblrGoogle+EmailShare

NASP Convention 2012

Learning to Graph Rate of Improvement

Learning to Graph Rate of Improvement

After a successful set of workshops at the 2011 Annual Convention for the National Association of School Psychologists, we were just notified that we are invited back to conduct a workshop during next year’s convention to be held in Philadelphia, PA in February!

Participants will be able to choose one or both workshops. The morning session will be an introductory workshop for SLD identification using RTI by Dr. Joe Kovaleski. The second workshop is for advanced applications of calculating and interpreting rate of improvement by Caitlin Flinn and Andy McCrea (bring your laptops for this one it’s a hands on experience).

We are excited that the conference will be local for us! Visit www.nasponline.org for forthcoming registration information.

FacebookTwitterPrintFriendlyPinterestTumblrGoogle+EmailShare