Possible way to pitch this (potentially easier sell; less perspective-y):

• Motivated in the same way as other taxonomy papers (”we need to understand the similarities and differences between tasks; it’s an important moderator; in many cases they explain more of the variance than other factors”)

  • Taxonomies are hugely influential — we can replace all of them with something better

• Effectively propose another taxonomy

• Rather than Steiner’s tree structure or McGrath’s circumplex structure, we use a high-dimensional representation

  • Allows to embrace all of the dimensions from previous taxonomies
  • Combine their strength, and avoid individual differences
  • Advantages … (in the paper already)
    • “Here are 5 tips that you can use” / summarize the lessons learned, and the process for discovering these tips is SI material
      • Can accomplish a lot by taking some things out and moving to the appendix
    • “Here are some survey questions that you can ask” (we provide an instrument; these are not typically included in other taxonomy papers)

• Here are 70 tasks that we have already classified, and where they fit (we have already done the work for you)

• Then potentially, a demonstration where you pick only 2 faraway tasks and 2 close tasks (3 tasks total)

  • Show that these differences exhibit different outcomes (run this on Empirica)
  • “if you want to understand the effect of team size, etc. you can see this”
  • distances in the task space may not be representative; two things that seem close may be far away based on the dimensions that actually matter; a failure of this exercise is not a failure of the taxonomy
    • Does this cannibalize the other paper / high-throughput experiments? (Duncan is nervous about this)
    • Could we re-analyze previously-published results?
      • Two authors picked tasks from the same quadrant of McGrath and found different results
      • Having a concrete example of some contradiction

• Main changes proposed: results, bit of motivation

• Start more directly: people care a lot about team performance; clearly a factor (acknowledged or not) is the task that the teams work on (lots of people have talked about the importance of the task)

  • It’s important to be able to say whether tasks are similar or different from each other

  • We need a language for expressing differences (again, not a new idea; we have cited other taxonomy papers that have tried to do this)

  • Problems:

    1. The taxonomies are not themselves commensurate (you cannot map one into another) — you have to pick a taxonomy before you know, and you don’t know which one to pick because they don’t agree
       • take all tasks that would fall into one quadrant to McGrath, then compute:
         • Average (within-cluster distance)
         • Randomly-chosen tasks
       • Let’s pick a bunch of tasks that are close according to McGrath, and it turns out they’re different when you look according to Steiner
    2. They don’t actually satisfy the principles of taxonomies (results? intro?)
       • Claim to show that every task belongs exclusively to one category and that everything belongs somewhere
       • We solve both these problems and propose a new way of doing the same thing — research cartography
       • We resolve the commensurability problem and show every task can be mapped to a unique point; it’s meaningful
       • Evidence:
         • All you need is anecdotal evidence (task that sits in multiple places in a taxonomy); you do not have to do it for all 70 tasks
         • Examples can come from Phase 1/Phase 2 learnings
         • We can just describe them as examples
           • “This is the same task for McGrath; it’s 2 different tasks for Steiner”
         • Refer to the SI for all the details about what we did
    3. Each taxonomy has pros and cons (when people criticize them in the literature)
       • We get the “good parts” from Steiner, and we are removing the parts that are incompatible
       • This would be a huge plus for the people who really love and use these taxonomies

  • We can pick some tasks that are similar / different (and do something)

  • Which of these differences turn out to matter (on our high-dimensional space) will be determined by later experiments

    • Also, our framework is extensible to new experiments
    • Steiner, McGrath, have no way to extend taxonomy

  • If all I’m doing is proposing a taxonomy (and I’m not doing meta-analysis or systematic sampling), then the focus is on the taxonomy and not on all the Phase 1 / Phase 2 details

    • “it’s nothing but us trying to provide people with an instrument and a starter set of tasks”
    • 3 things that came up:
      • why these dimensions?
        • slide of 3-4 big taxonomies people use; we use all of them — we use the taxonomies people in this literature use and have used for decades
      • why these tasks?
        • shift the emphasis to ‘we are just mapping tasks and illustrating how to do it’
        • not clear what a “representative sample” is (we have no idea of underlying population
      • why this measure of inter-rater agreement? (huge can of worms)
        • we need some method to assign values to coordinates
        • we defined the average of experts as the ‘truth’ (there no ‘consensus’ and the truth could be 0.5)

  “oh, we don’t believe what you have…”

  • This would mean that they would not believe Steiner and the people who came before
  • All dimensions come from highly regarded taxonomies

  Acknowledge that dimensions of tasks might vary depending on the research questions (e.g., some dimensions matter for some questions versus others)

  • We want a taxonomy that we can use to answer all these different questions — researchers should discover how the weights differ
  • This is why we are not attaching specific weights
  • We want to provide a “map”

  the point of “taxonomies are bad” is very hard for people to buy

  • People will try to nitpick you

  Things to release / put on OSF

  • Questions used (instrument)
  • Task Map

  ---

Model Papers?

• Share a few with Abdullah

Look at Psych Science, Management Science, Org Sci (??)

Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., & Thue, B. (2020). Beyond Western, Educated, Industrial, Rich, and Democratic (WEIRD) Psychology: Measuring and Mapping Scales of Cultural and Psychological Distance. Psychological Science, 31(6), 678–701.

| 12