Making non-repeating assignments (Assigning students a set of paper for peer review)

[Lance Erickson]

Hello,

I have a peer review assignment in a class I teach and want to develop some code that will randomly assign three papers to each student to review. I have randomly assigned each student a number so that their papers will be anonymous....no problem there. Where I'm getting hung up and would be interested in some help is how to randomly assign three papers to each of the students. What I'm having a hard time figuring out is how to do this so that a student doesn't get assigned...

1. their own paper
2. the same paper more than once.

I've never tried anything like this in Stata and can't think of any set of commands or approaches that even seems like it would work to me. I've thought of resorting the data based on a new random number and using _n to make the assignments but this, of course, isn't sensitive to my two conditions, nor do I know how to make it sensitive to them. I've looked on the forum for things that might produce pieces of a process that I could put together but I either haven't found them or haven't the imagination to recognize them.

Although the data setup is trivial, some sample data are below in case anyone is inclined to help.

Thanks,
Lance

clear
input float student
1
2
3
4
5
6
7
8
9
10
end

[Clyde Schechter]

clear
input float student
1
2
3
4
5
6
7
8
9
10
end

tempfile copy
save `copy'

rename student reader
cross using `copy'
drop if student == reader
set seed 1234
gen double shuffle = runiform()
by reader (shuffle), sort: keep if _n <= 3
drop shuffle
by reader: gen seq = _n
reshape wide student, i(reader) j(seq)

will do what you ask. But with a small class like this, the random assignment results in some papers being read only once, and others being read many times. A systematic approach might make more sense. With a larger class, things will balance out better.

[Lance Erickson]

Clyde,

Thanks. It hadn't occurred to me to take an approach like this.

Unfortunately, as you point out, this results in some papers getting read only once and others many times. Ideally, I would want each paper to be read three times. I tried playing around with your code to see if I could make that happen but I didn't really make any progress.

Also, as you point out, this is not an unbearable problem to do without a coded approach. The course is larger than 10 students but not so large that it's impractical to do systematically. So, this isn't a pressing problem for me, but I was interested in whether it could be done using code. I would welcome any other ideas if there are any.

Thanks,
Lance

[Lance Erickson]

I continue to stew on this and was able to work through something...

clear
input float reviewer
1
2
3
4
5
6
7
8
9
10
end

gen author1=.
gen author2=.
gen author3=.
forvalues i = 1/`=_N' {
    local a = 1
    forvalues j = 1/3 {
        replace author`j' = `i'+`a' in `i'
        forvalues k = `=_N+1'/`=_N+3' {
            replace author`j'=`k'-`=_N' if author`j'==`k' in `i'
        }
        local ++a
    }
}

The solution is systematic, but I generated the numeric assignments randomly so I'm satisfied there. It's not particularly efficient because it loops through observations and therefore has to do some work that a more efficient solution wouldn't have to. But, this class is never going to be so large that efficiency is much of a problem. The solution is general enough that I could use it regardless of the number of students in the class.

Lance

[Wouter Wakker]

Thank you for the brain teaser. Here is a different approach of random assignment without replacement.

The basic idea is to first create a list of all papers to be read (nr of papers to be read by each student * nr of students), shuffle them randomly, put them in a local macro and assign them one by one to each student in a loop. If one of the conditions is not satisfied when trying to assign the paper, the paper is added to the end of the same list and the next paper is tried.

In some cases, when the loop gets to the last student, the papers that are left cannot satisfy the conditions. In that case the loop breaks, the list is reshuffled and it tries again.

The code has become quite long so I have annotated it in the hope of making it more clear. I suspect there are more elegant and efficient ways to do this but I tested it for 200 students having to read 10 papers and it works well, so it should be okay for this specific purpose.

local seed = 1234 // set seed
local students = 10 // set number of students
local nr_papers = 3 // set amount of papers to read

set seed `seed'

local counter = 1
while `counter' != `= `students' * `nr_papers' + 1 ' { // Only false when succefully assigned papers to all students
    clear
    set obs `= `students' * `nr_papers' ' // Total number of papers to assign to all students

    gen paper_to_read = mod(_n-1, `students') + 1 // Generate papers and shuffle
    gen double shuffle = runiform() 
    sort shuffle 
    
    local paper_pool
    forval i = 1/`=_N'{
        local paper_i = paper_to_read[`i']
        local paper_pool `paper_pool' `paper_i' // Put randomly shuffled papers into local to loop over later
    }

    clear
    set obs `students'
    gen student = _n // Generate student numbers
    
    tempvar inlist_cond_var // Tempvar to generate inlist condition below (condition is different for different number of papers)
    gen `inlist_cond_var' = ""
    forval i = 1/`nr_papers' {
        gen paper`i' = . // Generate empty paper variables (papers to read per student)
        replace `inlist_cond_var' = `inlist_cond_var' + ", paper`i'[\`i']" // Generate inlist condition
    }
    local inlist_cond = `inlist_cond_var'[1] // put condition in local macro
    
    local i = 1 
    local j = 1 
    local counter = 1
    local cond_not_satisf = 0
    foreach paper of local paper_pool {
        if student[`i'] == `paper' | inlist(`paper' `inlist_cond') { // Conditions: Student cannot read own paper or same paper more than once
            local paper_pool `paper_pool' `paper' // Put paper at the end of the pool
            local ++cond_not_satisf // Count times condition not satisfied
            if `cond_not_satisf' == `= `nr_papers' * 2' {
                continue, break // break and reshuffle if condition is repeatedly not satisfied
            }
            continue
        }
        replace paper`j' = `paper' in `i' // Assign paper to student
        local cond_not_satisf = 0 // Reset count condition not satisfied
        local i = ceil((`counter'+1)/`nr_papers') // i = student indicator
        local j = mod(`j', `nr_papers') + 1 // j = paper indicator
        local ++counter
    }
}

// check if equally distributed
* reshape long paper, i(student) j(nr)
* tab paper

[Lance Erickson]

Wouter,

That's fantastic...thanks for taking the time to think through that and come up with a solution. That's exactly the outcome that I was originally thinking of.

I've been playing around with your code, both to better understand it and to make it fit with my particular application. When I do this, I'm always going to start with a list of student names, like this:

clear
input str7 first str6 last
"first1"  "last1"
"first2"  "last2"
"first3"  "last3"
"first4"  "last4"
"first5"  "last5"
"first6"  "last6"
"first7"  "last7"
"first8"  "last8"
"first9"  "last9"
"first10" "last10"
end

Actually, I'll likely be loading in the names from a csv file, but that's not a critical detail. The point is that I want to use the number of names in the list rather than set the number in a macro...just to save a step and to keep from having to change the macro from one semester to the next. That's an easy change to your code...I just read in the data and then set the local students to equal _N. If I do that, your code continues to work right.

However, I also want to keep the names in the dataset. This is purely convenience so that I can have the student number and their name together to make it easier to send the students the reviews they have been assigned. From what I could tell, the first two parts of the while loop, starting with clear were first, getting the list of papers for review (three each for every student in the class) and second, to create the student numbers. I came up some code that puts the list of paper for review into a local and shuffles them without having to clear the original data and therefore keeps the names and numbers intact. Here's the code all together...

clear
input str7 first str6 last
"first1"  "last1"
"first2"  "last2"
"first3"  "last3"
"first4"  "last4"
"first5"  "last5"
"first6"  "last6"
"first7"  "last7"
"first8"  "last8"
"first9"  "last9"
"first10" "last10"
end

gen student=_n

local seed = 1234 // set seed
local students = _N // set number of students
local nr_papers = 3 // set amount of papers to read

set seed `seed'

local counter = 1
while `counter' != `= `students' * `nr_papers' + 1 ' { // Only false when succefully assigned papers to all students

    // get list of all papers to review (i.e., each paper 3 times in list)
    local paper_pool
    quietly valuesof student
    local authors `r(values)' 
    forvalues p = 1/`=`nr_papers'' {
        local paper_pool `paper_pool' `authors'
    }
    // randomly sort list of papers
    mata : st_local("paper_pool", invtokens(jumble(tokens(st_local("paper_pool"))')'))
    
    // Tempvar to generate inlist condition below (condition is different for different number of papers)
    tempvar inlist_cond_var 
    gen `inlist_cond_var' = ""
    forval i = 1/`nr_papers' {
        capture drop paper`i'
        gen paper`i' = . // Generate empty paper variables (papers to read per student)
        replace `inlist_cond_var' = `inlist_cond_var' + ", paper`i'[\`i']" // Generate inlist condition
    }
    local inlist_cond = `inlist_cond_var'[1] // put condition in local macro
    
    local i = 1 
    local j = 1 
    local counter = 1
    local cond_not_satisf = 0
    foreach paper of local paper_pool {
        if student[`i'] == `paper' | inlist(`paper' `inlist_cond') { // Conditions: Student cannot read own paper or same paper more than once
            local paper_pool `paper_pool' `paper' // Put paper at the end of the pool
            local ++cond_not_satisf // Count times condition not satisfied
            if `cond_not_satisf' == `= `nr_papers' * 2' {
                continue, break // break and reshuffle if condition is repeatedly not satisfied
            }
            continue
        }
        replace paper`j' = `paper' in `i' // Assign paper to student
        local cond_not_satisf = 0 // Reset count condition not satisfied
        local i = ceil((`counter'+1)/`nr_papers') // i = student indicator
        local j = mod(`j', `nr_papers') + 1 // j = paper indicator
        di "local i = `i'; local j = `j'"
        local ++counter
    }
}
order first last

One additional thought I had was to use macro list manipulation to identify the inlist conditions rather than create a temporary variable but I'm taking a break for now and I don't know that would ends up being more efficient or not. If I can get back to this in the next week or so and work through my idea I'll post a follow-up.

Thanks again,
Lance

[Wouter Wakker]

Thanks for sharing your updated code.

Creating the list of papers without using variables is a definite improvement, I didn't know you could do it like that.

I agree with you that the use of a temporary variable to create the inlist conditions is not ideal. I tried myself to use locals only but couldn't get it to work. If you do find a solution I would be interested to know about it.

[Lance Erickson]

Wouter,

Here's an update. I was able to get rid of the temporary variable altogether. The code still uses the logic that you had originally developed though. At this point, I don't know that I have anything else to offer this. I also changed some of the names of variables and locals to match my taste.

I really appreciate you taking the time to put it together in the first place.

Best,
Lance

clear
input str7 first str6 last
"first1"  "last1"
"first2"  "last2"
"first3"  "last3"
"first4"  "last4"
"first5"  "last5"
"first6"  "last6"
"first7"  "last7"
"first8"  "last8"
"first9"  "last9"
"first10" "last10"
end

gen reviewer=_n
set seed 1234
local students = _N  // number of students
local  reviews =  3  // number of papers to review

local counter = 1
while `counter' != `= `students' * `reviews' + 1 ' { // Only false when succefully assigned papers to all students

    local papers
    quietly valuesof reviewer
    local authors `r(values)' 
    forvalues p = 1/`=`reviews'' {
        // get list of all papers to review 
        // i.e., each paper as many times as reviews needed
        local papers `papers' `authors'    
        // generate papers to review variables
        capture drop author`p'
        gen author`p' = .
    }
    // randomly sort list of papers
    mata : st_local("papers", invtokens(jumble(tokens(st_local("papers"))')'))

    
    local i = 1 
    local j = 1 
    local counter = 1
    local cond_not_satisf = 0
    foreach paper of local papers {
        // Conditions: reviewer cannot read own paper or same paper more than once
        if [`i'] == `paper' | inlist(`paper',author1[`i'],author2[`i'],author3[`i']) { 
            local papers `papers' `paper' // Put paper at the end of the pool
            local ++cond_not_satisf // Count times condition not satisfied
            if `cond_not_satisf' == `= `reviews' * 2' {
                continue, break // break and reshuffle if condition is repeatedly not satisfied
            }
            continue
        }
        replace author`j' = `paper' in `i' // Assign paper to reviewer
        local cond_not_satisf = 0 // Reset count condition not satisfied
        local i = ceil((`counter'+1)/`reviews') // i = reviewer indicator
        local j = mod(`j', `reviews') + 1 // j = author indicator
        local ++counter
    }
}
order first last

// check if equally distributed
reshape long author, i(reviewer) j(review)
tab author
reshape wide
order first last

[Wouter Wakker]

Lance Erickson Great, I'm happy you could adapt the code into a working solution for your application.

I just have one more thing to add, which may be quite trivial. While the list of papers is shuffled randomly, they are still assigned in a non-random way in the end (starting at student = 1, ending at student = 10). Because papers are added again to the end of the list when one of the conditions fails, the papers that are assigned to the last couple of students are not entirely random: I collected the means of the authors assigned to each reviewer from running the code 500 times with different seeds, and compared it to the expectation for each reviewer, which shows this:

. qui reg mean i.reviewer

. margins i.reviewer, post

Adjusted predictions                            Number of obs     =      5,000
Model VCE    : OLS

Expression   : Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    reviewer |
          1  |   6.068667   .0631669    96.07   0.000     5.944832    6.192501
          2  |   5.939333   .0631669    94.03   0.000     5.815499    6.063168
          3  |   5.863333   .0631669    92.82   0.000     5.739499    5.987168
          4  |   5.783333   .0631669    91.56   0.000     5.659499    5.907168
          5  |   5.650667   .0631669    89.46   0.000     5.526832    5.774501
          6  |   5.494667   .0631669    86.99   0.000     5.370832    5.618501
          7  |   5.248667   .0631669    83.09   0.000     5.124832    5.372501
          8  |      4.932   .0631669    78.08   0.000     4.808165    5.055835
          9  |      4.634   .0631669    73.36   0.000     4.510165    4.757835
         10  |   5.385333   .0631669    85.26   0.000     5.261499    5.509168
------------------------------------------------------------------------------

. 
. forval i = 1/10 {
  2.         local reviewer_expectation = (1+2+3+4+5+6+7+8+9+10-`i')/9
  3.         lincom _b[`i'.reviewer] - `reviewer_expectation'
  4. }

 ( 1)  1bn.reviewer = 6

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0686667   .0631669     1.09   0.277    -.0551681    .1925015
------------------------------------------------------------------------------

 ( 1)  2.reviewer = 5.888889

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0504444   .0631669     0.80   0.425    -.0733903    .1742792
------------------------------------------------------------------------------

 ( 1)  3.reviewer = 5.777778

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0855556   .0631669     1.35   0.176    -.0382792    .2093904
------------------------------------------------------------------------------

 ( 1)  4.reviewer = 5.666667

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .1166667   .0631669     1.85   0.065    -.0071681    .2405015
------------------------------------------------------------------------------

 ( 1)  5.reviewer = 5.555556

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0951111   .0631669     1.51   0.132    -.0287237    .2189459
------------------------------------------------------------------------------

 ( 1)  6.reviewer = 5.444444

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0502222   .0631669     0.80   0.427    -.0736126     .174057
------------------------------------------------------------------------------

 ( 1)  7.reviewer = 5.333333

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.0846667   .0631669    -1.34   0.180    -.2085015    .0391681
------------------------------------------------------------------------------

 ( 1)  8.reviewer = 5.222222

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.2902222   .0631669    -4.59   0.000     -.414057   -.1663874
------------------------------------------------------------------------------

 ( 1)  9.reviewer = 5.111111

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.4771111   .0631669    -7.55   0.000    -.6009459   -.3532763
------------------------------------------------------------------------------

 ( 1)  10.reviewer = 5

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .3853333   .0631669     6.10   0.000     .2614986    .5091681
------------------------------------------------------------------------------

I found a solution for this, which is to create a matrix of all reviewer/author combinations and shuffle it in the same way as the list of papers. Here is an update to your code in #8:

clear
input str7 first str6 last
"first1"  "last1"
"first2"  "last2"
"first3"  "last3"
"first4"  "last4"
"first5"  "last5"
"first6"  "last6"
"first7"  "last7"
"first8"  "last8"
"first9"  "last9"
"first10" "last10"
end

gen reviewer=_n
set seed 1234
local students = _N  // number of students
local  reviews =  3  // number of papers to review

local counter = 1
while `counter' != `= `students' * `reviews' + 1 ' { // Only false when succefully assigned papers to all students

    local papers
    quietly valuesof reviewer
    local authors `r(values)' 
    forvalues p = 1/`=`reviews'' {
        // get list of all papers to review 
        // i.e., each paper as many times as reviews needed
        local papers `papers' `authors'    
        // generate papers to review variables
        capture drop author`p'
        gen author`p' = .
    }
    // randomly sort list of papers
    mata : st_local("papers", invtokens(jumble(tokens(st_local("papers"))')'))
    
    // Create shuffled list of reviewer/author combinations
    mat A = J(`= `students' * `reviews' ', 2, .)
    local row_nr = 1
    forval i = 1/`students' {
        forval j = 1/`reviews' {
            mat A[`row_nr', 1] = `i'
            mat A[`row_nr', 2] = `j'
            local ++row_nr
        }
    }
    mata : B = strofreal(jumble(st_matrix("A")))
    mata : st_local("reviewer_nr", invtokens(B[1...,1]'))
    mata : st_local("author_nr", invtokens(B[1...,2]'))

    
    local counter = 1
    local cond_not_satisf = 0
    foreach paper of local papers {
        local i : word `counter' of `reviewer_nr'
        local j : word `counter' of `author_nr'
        if reviewer[`i'] == `paper' | inlist(`paper', author1[`i'], author2[`i'], author3[`i']) { // Conditions: Student cannot read own paper or same paper more than once
            local papers `papers' `paper' // Add paper to the end of the list
            local ++cond_not_satisf // Count times condition not satisfied
            if `cond_not_satisf' == `= `reviews' * 2' {
                continue, break // break and reshuffle if condition is repeatedly not satisfied
            }
            continue
        }
        replace author`j' = `paper' in `i' // Assign paper to reviewer
        local cond_not_satisf = 0 // Reset count condition not satisfied
        local ++counter
    }
}
order first last

// check if equally distributed
reshape long author, i(reviewer) j(review)
tab author
reshape wide
order first last

Here are the results of comparing means to expectations with the updated code:

. qui reg mean i.reviewer

. margins i.reviewer, post

Adjusted predictions                            Number of obs     =      5,000
Model VCE    : OLS

Expression   : Linear prediction, predict()

------------------------------------------------------------------------------
             |            Delta-method
             |     Margin   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
    reviewer |
          1  |      6.088   .0631005    96.48   0.000     5.964295    6.211705
          2  |   5.905333   .0631005    93.59   0.000     5.781629    6.029038
          3  |      5.772   .0631005    91.47   0.000     5.648295    5.895705
          4  |      5.616   .0631005    89.00   0.000     5.492295    5.739705
          5  |   5.536667   .0631005    87.74   0.000     5.412962    5.660371
          6  |   5.485333   .0631005    86.93   0.000     5.361629    5.609038
          7  |   5.303333   .0631005    84.05   0.000     5.179629    5.427038
          8  |   5.225333   .0631005    82.81   0.000     5.101629    5.349038
          9  |      5.036   .0631005    79.81   0.000     4.912295    5.159705
         10  |      5.032   .0631005    79.75   0.000     4.908295    5.155705
------------------------------------------------------------------------------

. 
. forval i = 1/10 {
  2.         local reviewer_expectation = (1+2+3+4+5+6+7+8+9+10-`i')/9
  3.         lincom _b[`i'.reviewer] - `reviewer_expectation'
  4. }

 ( 1)  1bn.reviewer = 6

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |       .088   .0631005     1.39   0.163    -.0357046    .2117046
------------------------------------------------------------------------------

 ( 1)  2.reviewer = 5.888889

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0164444   .0631005     0.26   0.794    -.1072602    .1401491
------------------------------------------------------------------------------

 ( 1)  3.reviewer = 5.777778

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.0057778   .0631005    -0.09   0.927    -.1294824    .1179268
------------------------------------------------------------------------------

 ( 1)  4.reviewer = 5.666667

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.0506667   .0631005    -0.80   0.422    -.1743713     .073038
------------------------------------------------------------------------------

 ( 1)  5.reviewer = 5.555556

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.0188889   .0631005    -0.30   0.765    -.1425935    .1048157
------------------------------------------------------------------------------

 ( 1)  6.reviewer = 5.444444

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0408889   .0631005     0.65   0.517    -.0828157    .1645935
------------------------------------------------------------------------------

 ( 1)  7.reviewer = 5.333333

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |       -.03   .0631005    -0.48   0.634    -.1537046    .0937046
------------------------------------------------------------------------------

 ( 1)  8.reviewer = 5.222222

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |   .0031111   .0631005     0.05   0.961    -.1205935    .1268157
------------------------------------------------------------------------------

 ( 1)  9.reviewer = 5.111111

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |  -.0751111   .0631005    -1.19   0.234    -.1988157    .0485935
------------------------------------------------------------------------------

 ( 1)  10.reviewer = 5

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         (1) |       .032   .0631005     0.51   0.612    -.0917046    .1557046
------------------------------------------------------------------------------

[Wouter Wakker]

Just updating for anyone coming across this topic. The ideas above are now implemented in a package -peerreview- available on SSC.

See https://www.statalist.org/forums/for...ssc-peerreview