Perform an item analysis for 60 MCQs. Create a new datasheet where Item Difficulty and Item Discrimination Index will be stored.

[Simwinga Simwinga]

Hi, I'm working on conducting an item analysis for 60 test items by calculating the item difficulty and item discrimination index. After that, I intend to generate a new dataset with the variables Item_Num, Item Difficulty, and Item Discrimination Index. Below is the code I'm using.

capture program drop item_analysis
program item_analysis
    args total_score item_list

    if "`total_score'" == "" {
        di "You must specify the total score variable."
        exit
    }

    if "`item_list'" == "" {
        di "You must specify the list of item variables."
        exit
    }

    tempfile results
    preserve

    local nitems = `: word count `item_list''
    set obs `nitems'
    gen Item = ""
    gen Item_Difficulty = .
    gen Discrimination_Index = .

    correlate `item_list' `total_score', covariance

    local row = 1
    foreach item of varlist `item_list' {
        di "-----------------------------"
        di "Item Analysis for: `item'"
        di "-----------------------------"

        sum `item'
        local difficulty = r(mean)

        local discrim_index = r(`item'_`total_score') 

        replace Item = "`item'" in `row'
        replace Item_Difficulty = `difficulty' in `row'
        replace Discrimination_Index = `discrim_index' in `row'

        local row = `row' + 1
    }

    save `results', replace
    restore
    append using `results'

    list Item Item_Difficulty Discrimination_Index
end

gen total_score = 0
forval i = 1/60 {
    replace total_score = total_score + Item`i' if !missing(Item`i')
}

* Ensure total_score is an integer
replace total_score = round(total_score)

item_analysis total_score Item1-Item60

And this is the response I am getting


observation number out of range
Observation number must be between 183 and 2,147,483,619. (Observation numbers are typed
without commas.)
r(198);



What could I be doing wrong or how do I go around this?


An extract of my dataset and the desired table are below

 
StudentNumberid
Item1
Item2
Item3
Item4
Item5
Item6
Item7
Item8
Item9
Item10

1
1
1
1
1
1
1
1
1
1
1

2
1
1
1
1
1
1
1
1
1
1

3
1
1
1
1
1
1
1
1
1
1

4
1
1
1
1
1
1
1
1
1
1

5
1
1
1
1
1
1
1
1
1
1

6
1
1
1
1
1
1
1
1
1
1

7
1
1
1
1
1
1
1
1
1
1

8
1
1
1
1
1
1
1
1
1
1

9
1
1
1
1
1
1
1
1
1
1

10
1
1
1
1
1
1
1
1
1
1

11
1
1
1
1
1
1
1
1
1
1

12
1
1
1
1
1
1
1
1
1
1

13
1
1
1
0
1
1
1
1
1
1

14
1
1
0
0
1
1
1
1
1
1

15
1
1
1
1
1
1
1
1
1
1

16
1
1
0
0
1
1
1
1
1
1

17
1
1
1
1
1
1
1
1
1
1

18
1
1
1
1
1
1
1
1
1
1

19
1
1
1
1
1
1
1
1
1
1

20
1
1
1
1
1
1
1
1
1
1

21
1
1
1
1
1
1
1
1
1
1

 
Item
Difficulty Index
Disc. Index


#1
0.19
0.31

#2
0.48
0.65

#3
0.18
0.29

#4
0.31
0.41

#5
0.72
-0.12

#6
0.40
0.54

#7
0.27
-0.01

#8
0.13
0.05

#9
0.13
0.29

#10
0.47
0.60

#11
0.18
0.10

#12
0.22
0.24

#13
0.25
0.33

#14
0.26
0.00

#15
0.44
0.15

#16
0.33
-0.14

#17
0.24
0.45

#18
0.62
0.71

#19
0.41
0.59

#20
0.30
0.51

#21
0.27
0.20

#22
0.27
0.22

[Nick Cox]

You've read in some data, and then you're trying to set the number of observations to the number of items. That will only work if the number of items is larger than the existing number of observations, but evidently it isn't. Otherwise put, you're getting confused between the dataset you have in memory and the dataset of results you're trying to create.

To get something rather than a problem, you could just omit that command and try to create new variables alongside and not aligned with the existing data.

There is at least one other problem. The command

local discrim_index = r(`item'_`total_score')

is legal but unlikely to do anything except create a missing value. Having just run summarize, there is an r(mean) you can access, but no r-class result from summarize has a name anything like that form. It's legal to mention r-class results that don't exist but the result is a missing value.

I don't know anything about this topic to suggest better code.

[Simwinga Simwinga]

Let me give a background to what I am trying to do by defining what Item Difficulty and Discrimination Index are.
Definitions.
1. Item Difficulty (P-value)
Item difficulty refers to how challenging an item is for test takers, represented as the proportion of respondents who answered the item correctly. The item difficulty (P) value ranges from 0 to 1 and It is calculated by subtracting the number of test-takers who answered an item correctly from the total number of test takers then divide by the total number of test-takers.

A value close to 1 indicates that the item is easy (most respondents answered it correctly).
A value close to 0 indicates that the item is difficult (few respondents answered it correctly).
For example, if 80% of respondents answered an item correctly, the item difficulty is 0.80, which means the item is relatively easy.
2. Item Discrimination Index
The discrimination index (DI) measures how discriminating items in a test are – i.e. how well an item can differentiate between high performers and less able ones. For each item it is a measure based on the comparison of performance between stronger and weaker test-takers on the test as a whole. It indicates whether test-takers who performed well on the overall test tended to answer the item correctly, while those who performed poorly on the test tended to answer it incorrectly. The test takers are categorized into two groups; the upper group (top 27% of test scorers) and lower group (bottom 27% of test scorers). The discrimination index value for an item range from -1 to +1 and is calculated by subtracting the proportion of correct responses in the lower group from the proportion of correct responses in the upper group.

A positive value (closer to 1) indicates good discrimination (high scorers tend to get the item correct, while low scorers tend to get it wrong) and a negative value (closer to -1) indicates poor discrimination (low scorers are more likely to answer correctly than high scorers, which is undesirable). Whereas a value close to 0 suggests that the item does not discriminate well between high and low performers.

Having defined what I intend to achieve, I want to do the above calculations on my dataset then, create a datasheet in which I will store these values.

[Nick Cox]

First of all, while the term P-value may for all I know be prominent in item analysis, as far as the rest of statistical science is concerned it has a long established meaning as the observed significance level. But whatever you call it, if you have a variable that is 1 for a correct answer and 0 for an incorrect answer, the measure is just the mean of that variable.

You don't need a program for that: summarize gives you the mean and as you have found out the result is stored temporarily after that command in r(mean).

Your other measure is much more complicated and seems intensely arbitrary. Where does 27% come from? Someone else may be able to point to existing code or provide some.

Each member learning programming follows their own path and I have only some rather obvious advice from my experience. I spent a couple of years writing do-files before I started writing programs, strict sense.