Gerry's Home Page Perspectives on CSCL Journals & Book Chapters Conferences Dissertations Tech Reports Presentations Proposals Interpretations Ideas Selected Writings

ROLE preproposal reviews

"ROLE Pre-proposal: Research on Collaboration in Learning

and on Collaboration Technology in Education"

Research on Learning and Education (ROLE)

Preliminary Proposal Review and Recommendation

 

Principal Investigator:           Gerry Stahl

Institution:                           U of Colorado Boulder

Email:                                 gerry.stahl@colorado.edu

Proposal Number:                0083440

Number of Reviews:            2

Proposal Title:                     Research on Collaboration in Learning and Collaboration Technology in Education

 

Dear ROLE Applicant:

This letter conveys to you NSF’s reviews and recommendations for the referenced preliminary proposal you have submitted to the ROLE program; NSF has included in a separate email attachment a separate document, The FY2000 Research on Learning and Education (ROLE) Preliminary Proposal Review Process: Overview and Commentary that describes the ROLE review process.  Please note that the Overview and Commentary document, along with subsequent updates on the ROLE competition, appears at http://www.ehr.nsf.gov/ehr/rec/june2000rolecompetition.htm.  

The ROLE Program expects to fund between 10 and 20 awards in the June, 2000 competition.  Preliminary proposal recommendations and reviews are not binding, but should be carefully considered before making the considerable effort required to submit a full proposal.  Please consult the Overview and Commentary for further information.  The four summary recommendation categories and the number of preproposals (out of the 182 that ROLE received) that fell into each category are:

 

Recommendation category   Overall tally (out of 182)

·        Encourage; when fully developed, this proposal is likely to be competitive in the expected June proposal pool. .................. 43

·        Encourage tentatively; this proposal has merit but is not likely to be competitive in the expected June proposal pool.  Consider possible later cycle submission to allow development of deeper, more thorough, and/or more focused full proposal............... 54

·        Discourage; elements of this proposal may be considered, however, for inclusion in a later preliminary proposal submission...................................................................... 51

·        Discourage; proposal either is not competitive or does not fall within ROLE Guidelines................................................... 34

 

The summary recommendation for your preliminary proposal (0083440) is:

Encourage tentatively; this proposal has merit but is not likely to be competitive in the expected June proposal pool.  Consider possible later cycle submission to allow development of deeper, more thorough, and/or more focused full proposal.  

Individual NSF program officer reviews follow:

þ  Encourage with comments               
Encourage with comments; 0083440/Stahl.  I am intrigued by this proposal but believe that it will have to be clearer in several areas if it is to be competitive in a ROLE panel.   My primary question is what is actually going to happen in this research? The kinds of micro analysis of face to face collaboration proposed will produce enormous data.  Based on this prelim, its difficult to understand how it will be reduced to provide guidance for software revision.  I realize this is part of the research, but I suspect the authors have a more developed sense of what types of variables or ideas they are looking at.  There is an implicit assumption that the collaboration seen on the "tapes" etc is good.  How will they know that knowledge building occurs? By their own theory, they are not assessing individual knowledge but something that is shared.  What are some of the criteria that can be used to know that this shared knowledge is developing? I do not know the WEBGUIDE software and reviewers might also be equally unfamiliar with it.  I think there should be greater discussion of why this software has the potential to develop collaborative learning and how potentially it could be modified based on the developing theory of collaboration.  Some reviewers might question whether the bibliography and the write-up reflects a comprehensive understanding of the literature on collaborative learning.

þ  Discourage
Discourage; 0083440/Stahl.  This project is about an interesting and important subject - why and how people learn from collaborative experiences and it proposes to expand our knowledge through computer interaction.  But they are wrong in assuming no one has build a theory around these experiences.  No sociologist is listed in this study.  Sociologists have made a life of such questions.  Try getting material from Elizabeth Cohen and the new book by Angela O'Donnell (Rutgers).  Once the authors have a deeper background in the theory of groups and interaction, this might be worth consideration.  Interesting backgrounds of the prospective PI's.  Talk with others there at Colorado and try again.  

Return to REC home page

The FY 2000 

Research on Learning and Education (ROLE) 

Preliminary Proposal Review Process: 

Overview and Commentary

April 10, 2000

Introduction

Research program staff of the Division of Research, Evaluation and Communication have analyzed the 182 preliminary proposals received under the ROLE Program’s first funding cycle.  ROLE represents a significant shift in REC’s approach to supporting research.  The purpose of this overview and commentary is to communicate to you some of our observations about a very gratifying initial round of preliminary proposals.  This overview and commentary may be especially useful to those whom REC is attempting to draw into its research communities; nearly two-thirds of the submitters have not been PIs in previous REC awards, and more than half have not been funded under previous REC awards.  In keeping with both the spirit of ROLE as an evolving program and the intent of the REC staff to serve new and maturing research communities, we welcome your continued input to any of the program staff (listed on third page).  

Proposal Mix

The multidisciplinary approach of ROLE and the four-quadrant continuum we have used to organize ROLE has resulted in a healthy mix of preproposals.  While reviewer comments focus on areas of improvement, the aggregate set of proposals was of very high quality, especially for the first round of a new research program.  With the potential for new contributors to the research effort and the expectation for strong proposals in June, based on the preproposals in March, the REC staff believe that ROLE is off to a very strong start.  One recurrent question has been the “quadrant mix” we expected with ROLE.  ROLE, of course, welcomes proposals that fall primarily into a specific quadrant along with proposals that bridge two or more quadrants.  

In approximate terms, about 40% of the ROLE preproposals were primarily anchored in Quadrants 1 (about 10%-15%) and 2 (about 25-30%), especially representing perspectives and research that formerly found a home in NSF’s Learning and Intelligent Systems (LIS) program.  About 40% of the proposals were primarily anchored in Quadrant 3, especially representing perspectives and research that formerly found a home in NSF’s Research in Educational Policy and Practice (REPP) program.  About 20% of the proposals were primarily anchored in Quadrant 4, representing parts of both LIS and REPP.  ROLE, however, extends significantly beyond reorganizing previous programs, with an interest on creating bridges and interactions across quadrants and the formulation of new questions and approaches that emerge from the continuum framework.  About 40% of the preliminary proposals conveyed significant cross-quadrant bridging; this suggests that the evolving ROLE portfolio will indeed be able to feature the mix of within- and across-quadrant activities that will give it the distinctive and nationally beneficial character that we envision.

The Preliminary Proposal Review Process

The REC research program officers prepared most of the nearly 500 reviews sent to proposal submitters.  About one-fifth of the proposal reviews, however, were completed by twelve other program officers in the Directorate for Education and Human Resources (EHR).  These program officers volunteered to review proposals on the basis of their interest in advancing a closer articulation of the research program in EHR with the directorate.  Additionally, colleagues from the Directorate for Social, Behavioral, and Economic (SBE) Sciences reviewed several dozen proposals that primarily fell in Quadrants 1 and 2 of the ROLE continuum.  Most of the SBE reviews consist of summary recommendations encouraging or discouraging a formal proposal submission, and are identified as SBE reviews. Finally, a small handful of proposals (14) were reviewed either by external panelists or by collaborators from the United Kingdom’s Teaching and Learning Programme as part of a cooperative activity we are undertaking with them.  These 14 are identified as non-NSF reviews.

With virtually no exception, all proposals received between two and four reviews.  As important as reviewing the diverse set of preliminary proposals proved to be, it was also important for each member of the ROLE program staff to examine and understand all of the reviews that he or she did not respectively prepare, and to consider both the totality of the proposal pool and of colleague responses to it.  That is, the review process was viewed not only as an opportunity to respond to the proposers, but also as an important month-long and intensive learning seminar for the REC staff.  In addition to receiving the individual reviews for their proposals, all submitters also received a summary recommendation from a set of four categories that appears below, along with a list of the number of proposals that were each recommendation.

Recommendation category (Overall tally out of 182)

·             Encourage; when fully developed, this proposal is likely to be competitive in the expected June proposal pool. (43 out of 182)

·             Encourage tentatively; this proposal has merit but is not likely to be competitive in the expected June proposal pool.  Consider possible later cycle submission to allow development of deeper, more thorough, and/or more focused full proposal. (54 out of 182)

·             Discourage; elements of this proposal may be considered, however, for inclusion in a later preliminary proposal submission. (51 out of 182)

·             Discourage; proposal either is not competitive or does not fall within ROLE Guidelines (34 out of 182)

Why Do the NSF Reviews on the Same Proposal Sometimes Differ?

Typically, individual reviewers bring different perspectives on a proposal and thus their reviews may differ.  ROLE’s preproposal pool was no exception.  These varying perspectives strengthen the review process.  Differences that you may see in reviews may help you understand how the proposal was interpreted by impartial readers and suggest different interpretations that members of a formal proposal review panel may take.  There are some general differences between REC reviewers relative to reviewers from other divisions, reflecting difference in priority on research methods, on best approaches to transfer between research and practice, and the mix of what might be considered applied versus basic research.  Furthermore, the short narrative length, while providing ample opportunity to convey the main ideas of the proposed research, left room for varying interpretations of the same text.  (Thus, for example, we expect that the deficiencies in some of the proposals rated in the second or even third category may actually be the result of failure to convey ideas clearly or fully in the preproposal; these proposals may in fact be ready for a June 2000 submission.  That is a decision that the Principal Investigator must make.)  In some cases, (such as criticizing some Quadrant 1 or 2 proposals for lack of specific science or mathematics learning focus), NSF reviewers (outside of REC) are still becoming accustomed to the new program; in fact, Quadrants 1 and 2 allow research on learning in general (while Quadrants 3 and 4 require a focus on SMET learning).  The ROLE program will need to make these issues clear to formal proposal reviewers.  Any full proposal submission should consider the comments of the reviews, but please remember that the proposal will be judged against the ROLE Program Announcement (NSF-0017) by reviewers who do not have access to the preproposal reviews.

What Are The Most Common Problems in the Preproposals?

In reviewing the proposals, we discovered that many proposals had similar deficiencies.  These fall into two general areas:  Research in General and Misinterpreting the Announcement.

Research in General:  Importance of Prior Work and Methodology

·             Failure to situate the study within prior work and literature.  New contributors to education or education research, particularly those coming from other disciplines, often prepare their proposals as if there is either very little no prior work on the topic or that no lessons, positive or negative, can be learned from previous work.  Failure to discuss or to value related literature often leads to a failure to communicate the value added of the proposal.  It immediately raises the question of whether the proposed activity benefits from previously acquired knowledge or analysis of the gaps in that knoweldge that the proposal will address.  This, of course, dramatically reduces competitiveness.  ROLE is not unique in requiring that this analysis be conducted prior to writing the proposal, but the multidisciplinary nature of the program and thus of reviewers makes this requirement paramount.

·             Failure to situate the study with a framwork or theory.  ROLE recognizes that new research methods or approaches, and new areas of research have a different relation to theory than more established fields.  However, failure to talk about the relationship of the work to existing frameworks or to the development of new or hybrid frameworks often leads reviewers to conclude that the proposed work will not generalize beyond the particular project.

·             Weak links between research goals and proposed methodology.  Reviewers must determine if the submission has a chance of success – or of documenting why failures occurred.  Discussion of methodologies lets reviewers know the researcher understands the scope of the project, the potential findings and the limitation of the work.  Education research is often especially prone to an a priori bias that recurred in the preproposal pool and that applicants should convincingly avoid in full proposals.

Misinterpreting the Announcement

·             A common error was listing too many quadrants as relevant or trying to “force” contrived connections between quadrants.  The proposal should be placed in the quadrant or quadrants that correlate with research questions the proposal poses.

·             Proposals that were primarily based in Quadrants 1 or 2 often failed to connect their basic research questions in neuroscience or cognitive science research to educational issues or to identify eventual possible connections.  ROLE will not fund research whose natural home is in the SBE or Biological Sciences (BIO) directorates.  ROLE will support research that integrates critical questions in learning and education to neuroscience and cognitive science research traditions. One critical outcome of such research is the development of a new research capacity in the field; the participation of young researchers will be an important component of competitiveness for proposals in quadrants 1 and 2.

To Whom Should I Direct Questions or Comments?

The Program Announcement identifies the four program officers responsible for ROLE (Anthony Kelly, Nora Sabelli, Larry Suter, or Elizabeth VanderPutten).  They may be reached through the general office number of 703-306-1650 or via electronic mail (aekelly@nsf.gov; nsabelli@nsf.gov; lsuter@nsf.gov; evanderp@nsf.gov).  Questions or comments may also be directed to Acting Division Director Eric Hamilton (ehamilto@nsf.gov).  Any of these can field any questions or comments (whether about a specific proposal submission or about the general REC research program), and will refer to colleagues as necessary.  

Thank you for the very satisfying response so far to this first ROLE competition.

REC Research Program Staff

Return to REC home page

 

Go to top of this page

Return to Gerry Stahl's Home Page

Send email to Gerry.Stahl@drexel.edu

This page last modified on October 27, 2003

 

Go to top of this page

Return to Gerry Stahl's Home Page

Send email to Gerry.Stahl@drexel.edu

This page last modified on October 27, 2003