Test/Correct using transitivity

[Nicolas Orgeira]

Hi all.

Enumerators were tasked to review property pictures in pairs and confirm whether the pictures were of the same property (status=1) or not (status=0). If they were not sure (or if the comparison wasn't done), the status would be missing. I'd like to do the following:
a) test whether the reviews make sense using the transitive property of equality. For example, for ID 1 below, if a.jpg=b.jpg and b.jpg!=c.jpg, it should be expected that a.jpg!=c.jpg. ID 3 would need to be further reviewed as g=h, g=i but h!=i.
b) fill in missing information when reviews were either not done, or the enumerator wasn't sure. For example, for ID2, d=e and d=f, so I'd like to replace status for e vs f from missing to 1.

The short dataset below is for illustration. There may more than three pictures per ID (the picture review process is ongoing, the current max number of properties is six, so 21 different pairs).

input id str5 photo1 str5 photo2 status
1 "a.jpg" "b.jpg" 1
1 "a.jpg" "c.jpg" 0
1 "b.jpg" "c.jpg" 0
2 "d.jpg" "e.jpg" 1
2 "d.jpg" "f.jpg" 1
2 "e.jpg" "f.jpg" .
3 "g.jpg" "h.jpg" 1
3 "g.jpg" "i.jpg" 1
3 "h.jpg" "i.jpg" 0
end

Thank you for your help

[Leonardo Guizzetti]

You are verging on the territory of inter-rater agreement. What I’m not clear about is what you intend to get out of this process in the end? Are you looking to test whether raters agree with each other, or something else?

[Nicolas Orgeira]

Thanks for your response Leonardo Guizzetti - I never heard about inter-rater agreement before so thanks for that! I'm just trying to clean the picture review data I have while my enumerators are in the field to determine whether they went to the correct property (comparing property pictures from the baseline vs endline survey). As property pictures were taken from different angles, it's possible enumerators who reviewed the pictures (different from those in the field) were not always able to determine the pictures were of the same property and may have left the status missing. But as some of the pictures were for a given ID were of the same property (for example a.jpg and b.jpg were taken by enumerator A, c.jpg by enumerator B), I'm hoping to use these information to complete any missing data, and identifying errors quickly, to potentially send enumerators back to visit the correct property. I just don't really know how to implement this on Stata in an efficient way

[Leonardo Guizzetti]

I'm not sure what is the point of having two enumerators review pictures in pairs to see if they are the same property though. Based on your explanation, you seem to already know which photos belong to which properties, so why not use that as the truth instead of the (noisy) process of pairwise agreement?

[Nicolas Orgeira]

Sorry for not being clear. As part of an endline survey, a team of enumerators is in charge of visiting properties from our baseline data to administer a survey to its owner. They will also be taking pictures of the property - a team of home-based enumerators will then be comparing baseline and endline property pictures to confirm that the same property was visited. Because two or more pictures were taken during the baseline and endline and properties were taken from different angles, enumerators cannot always confirm the same property was visited when comparing two of the properties' pictures. Moreover, because the property may have changed somewhat (different painting color, etc.), enumerators may first think that for the first pair of pictures are of two different properties, but then the second and third pair are of the same property. The transitivity property would be broken here, and I'd like to easily notice if it's the case, so I can review the pictures manually myself. So while I have information on whether pictures belong to the same property (baseline vs endline), pairwise agreement would help confirm the correct property was visited during the endline survey, and using a single picture from the baseline and endline survey may lead to errors due to possible changes in the property, or pictures taken from different angles.

[Leonardo Guizzetti]

This still seems convoluted to me. I understand that properties can change over time, and certainly you'll never have the same angle for photos, so you rely on visual comparison. But you already *do* know which photos belong with the property, unless you have reason to believe your enumerators are making mistakes about which photos belong to which property, or worse, fraudulently completing surveys. Let' assume we can rule out fraud or human error as these would go undetected without additional information or manual review of all properties.

I don't see how transitivity is helpful here. Rather you should probably screen all photos from a property if there are any disagreements. Here's how I would approach it.

bys id: egen byte disagree = max(cond(status==0 | mi(status), 1, 0))