You know you’re a behaviorist when you hear a song with the following lyrics and get excited….
“You just can’t take the effect and make it the cause”
Thank you, White Stripes! Listen to the song, here.
New Resource for Educators – Education ABA Website
February 22, 2014 By Leave a Comment
My good friend and colleague Kevin Kuhn has recently created a website for educators interested in resources and tools related to applied behavior analysis called Education ABA (also listed on the Resources page of this site). Given the scope my site, I’m a big fan of the page within this site about Graphing Data. Contrasted to the various downloads on my site, Kevin’s has many of his graphing templates available in an online form that can be shared through Google Drive. Below is a preview of the types of graphing templates available on Education ABA.
- Frequency / Rate Graphing Template (with Goal Line & Trendline)
- IEP Goal % Graphing
- ABAB Design Semi-Logarithmic Graph
- RTI and IEP Behavior Progress Monitoring – Rating / Likert Scale
Directions for using the graphs are built into the graphs themselves so educators can easily use and understand the templates. Graphs can be adapted and customized and are intended for use in response to intervention (RTI) frameworks and for monitoring goals in Individualized Education Plans (IEPs).
Emerging Best Practice for Determining ROI #1
February 5, 2014 By 2 Comments
As part of my dissertation, I’ve compiled a list of what I think are emerging best practices for determining rate of improvement. In other words, what are the necessary conditions for having confidence in the ROI statistic? The first emerging best practice I’m considering is the use of technically adequate and psychometrically sound measures. If we don’t have good data, there’s no point in creating a trend line or calculating an ROI statistic. One of the best resources is the progress monitoring tools chart. Note that this chart can still be accessed through the National Center on Response to Intervention but is now housed at the National Center on Intensive Intervention. Not all assessments are located on this chart as they are submitted voluntarily but it’s good place to start if your school team is looking for ideas.
When school-based teams are considering making high stakes decisions such as special education eligibility, the quality of data needed is much higher. Teams need to consider if the assessments they are using is measuring what they intends to measure (validity) and if they can measure skills consistently (reliability). Other aspects of assessments to consider in relation to generating a stable trend line are whether the measure can demonstrate small increments of growth and if the assessments can be repeated through alternate forms. An example of a technically adequate measure that was designed to produce ROI is the computer adaptive tests from Renaissance Learning called STAR Reading and STAR Math. After only four data points, the system will generate a stable trend line that can be used to interpret student progress. A non-example would be teacher-made assessments or unit tests. The latter measures may be helpful for teachers to know how students are performing with concepts they are learning in class, but have not been validated for the purpose of generating trend lines or ROI from the results.
Questions and comments are welcome! What are your school teams using to document student progress?
Q&A – Realistic vs. ambitious goals for early literacy skills
November 27, 2013 By Leave a Comment
Great question from Beth:
I have seen where there is a chart to show what a realistic vs. ambitious goal for Rate of Improvement is for R-CBM or Oral Reading Fluency but is there a chart like that for Letter Naming Fluency or Letter Sound Fluency? Or can the ROI charts for R-CBM charts be used for other areas of progress monitoring as well??
Response:
The majority of research studies that I have come across have been related to determining expected growth for oral reading fluency, most likely because it is the most popular form of CBM. The study below is the study where the growth charts originated from for digits correct and oral reading fluency.
Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48.
The idea of realistic and ambitious goal setting is also described in more detail in a chapter of Best Practices for School Psychologist, Volume 5.
Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes (Eds.), Best practices in school psychology V. (Vol. 2, pp. 141-157). Bethesda, MD: National Association of School Psychologists.
The amount of growth you can expect from students is dependent on a number of factors such as the intensity, duration, and frequency of the intervention; the skill being measured, and the fidelity with which the intervention is delivered. You may want to see if the assessment system you are using includes norms for rate of improvement. If none are provided, then you could calculate the typical ROI for students based on the expected benchmark scores and use that as a comparison. Remember that students scoring below benchmark would require an ROI that is more than the ROI of their typical peers in order to close the achievement gap. There are many dissertations to be had in this area! It would be great to have more published, peer reviewed studies that review the progress students typically make with specific skills given specific interventions.
Q&A – ROI for non-normed measures
November 27, 2013 By Leave a Comment
Great question from Amber:
For the past two years, I have worked in an Educational Life Skills Program for students with significant cognitive disabilities. Last year, we discovered a progress monitoring tool called “Significant Cognitive Disabilities” developed by The Research Institute on Progress Monitoring. (you can go to www.progressmonitoring.org for more information) We thought that this would be an excellent tool to track our students growth since the CBM’s on AIMSweb were too difficult for most of our students. Our special education teachers and aides learned all of the different assessments and began testing our students weekly and tracking their progress in an Excel file. At that time, I had an intern who was great at Excel and created the graphs that we needed. Now with pressure added to demonstrate growth in student performance, and then use that information to guide instruction, we were looking for a better way to show rate of improvement. This is when I happened upon your wonderful webpage! You all provide a lot of useful information, but I have an important question that may just be unique for my program. I need to create a better graph(s) than what we have. I have to show rate of improvement. When I read through your Diego example, I learned that you show ROI by graphing slope. However, slope is determined by benchmark data. We dont have any predetermined benchmark data, just progress monitoring data which varies greatly from student to student and of course by different assessments. The progress monitoring tool that we are using was never normed. We were hoping to norm it using our population of students and then have ROI data. Is it possible to create a graph that will show ROI data per student, per grade, and then program wide? If so, how? Thank you very much for taking the time to read this message.
Response:
A previous Q&A was referring to calculating average ROI for groups of students. The directions on that post are one way of calculating average ROI for groups of students. You may want to review information on how to create local benchmarks; however, if the students’ scores vary considerably, it might not make sense to make local benchmarks.
Stewart, L. H. & Silberglitt, B. (2008). Best practices in developing academic local norms. In A. Thomas and J. Grimes (Eds.) Best practices in school psychology V. (Vol. 2, pp. 225-242). Bethesda, MD: National Association of School Psychologists.
You could certainly set up your Excel spreadsheets so that you have all of your students per grade on one sheet, then choose to graph their data individually to see if growth is occurring. If you organize the students by grade level for the whole program, you could then graph grade-level data as well.
Keep in mind, too, that depending on the type of data you are graphing, sometimes the visual analysis is sufficient and a ROI statistic (i.e., slope) does not always make sense.
Q&A – Calculating average ROI for a group of students
November 27, 2013 By 1 Comment
Great question from Pamela:
| Oregon sped regulations for LD referral through RTI specify that ROI needs to be calculated for the RTI group so that the individual ROI can be compared to the RTI group ROI. How do I set up an Excel spreadsheet for that purpose? Do I average CBM scores across students in the group and then compare the group linear regression statistic to the individual linear regression statistic? How do I get Excel to fill in the average for me? |
Response:
If you used a spreadsheet similar to the ones on this web site, individual student ROI is calculated for you. If you highlighted that column, you could enter a formula that will calculate the average of those ROIs to give you a group comparison. To answer your question directly, yes that is one way you could calculate average group ROI.