|
|
|
|
"ROLE Pre-proposal:
Research on Collaboration in Learning and
on Collaboration Technology in Education"
Research on Learning and Education (ROLE) Preliminary Proposal Review and
Recommendation Principal Investigator:
Gerry Stahl Institution:
U of Colorado Boulder Email:
gerry.stahl@colorado.edu Proposal Number:
0083440 Number of Reviews:
2 Proposal
Title:
Research on Collaboration in Learning and Collaboration Technology
in Education Dear ROLE Applicant: This letter conveys to you NSF’s reviews and
recommendations for the referenced preliminary proposal you have submitted to
the ROLE program; NSF has included in a separate email attachment a separate
document, The FY2000 Research on Learning
and Education (ROLE) Preliminary Proposal Review Process: Overview and
Commentary that describes the ROLE review process. Please note that the Overview
and Commentary document, along with subsequent updates on the ROLE
competition, appears at http://www.ehr.nsf.gov/ehr/rec/june2000rolecompetition.htm.
The ROLE Program expects to fund between 10 and 20 awards in
the June, 2000 competition. Preliminary
proposal recommendations and reviews are not binding, but should be carefully
considered before making the considerable effort required to submit a full
proposal. Please consult the Overview
and Commentary for further information.
The four summary recommendation categories and the number of preproposals
(out of the 182 that ROLE received) that fell into each category are: Recommendation
category Overall
tally (out of 182) ·
Encourage;
when fully developed, this proposal is likely to be competitive in the expected
June proposal pool. ..................
43 ·
Encourage
tentatively; this proposal has merit but is not likely to be competitive
in the expected June proposal pool. Consider
possible later cycle submission to allow development of deeper, more thorough,
and/or more focused full proposal............... 54 ·
Discourage;
elements of this proposal may be considered, however, for inclusion in a later
preliminary proposal submission......................................................................
51 ·
Discourage;
proposal either is not competitive or does not fall within ROLE Guidelines...................................................
34 The summary recommendation
for your preliminary proposal (0083440) is: Encourage tentatively; this proposal has merit but is not likely to
be competitive in the expected June proposal pool. Consider possible later cycle submission to allow development
of deeper, more thorough, and/or more focused full proposal. Individual
NSF program officer reviews follow: þ Encourage with comments þ Discourage
The FY 2000 Research on Learning and Education (ROLE) Preliminary Proposal Review Process: Overview and Commentary April 10, 2000
IntroductionResearch program staff of the Division of Research, Evaluation and
Communication have analyzed the 182 preliminary proposals received under the
ROLE Program’s first funding cycle. ROLE
represents a significant shift in REC’s approach to supporting research.
The purpose of this overview and commentary is to communicate to you some
of our observations about a very gratifying initial round of preliminary
proposals. This overview and commentary may be especially useful to
those whom REC is attempting to draw into its research communities; nearly
two-thirds of the submitters have not been PIs in previous REC awards, and more
than half have not been funded under previous REC awards. In keeping with both the spirit of ROLE
as an evolving program and the intent of the REC staff to serve new and maturing
research communities, we welcome your continued input to any of the program
staff (listed on third page). Proposal Mix
The multidisciplinary approach of ROLE and the four-quadrant
continuum we have used to organize ROLE has resulted in a healthy mix of
preproposals. While reviewer
comments focus on areas of improvement, the aggregate set of proposals was of
very high quality, especially for the first round of a new research program.
With the potential for new contributors to the research effort and the
expectation for strong proposals in June, based on the preproposals in March,
the REC staff believe that ROLE is off to a very strong start.
One recurrent question has been the “quadrant mix” we expected with
ROLE. ROLE, of course, welcomes proposals that fall primarily into
a specific quadrant along with proposals that bridge two or more quadrants. In
approximate terms, about 40% of the ROLE preproposals were primarily anchored in
Quadrants 1 (about 10%-15%) and 2 (about 25-30%), especially representing
perspectives and research that formerly found a home in NSF’s Learning and
Intelligent Systems (LIS) program. About
40% of the proposals were primarily anchored in Quadrant 3, especially
representing perspectives and research that formerly found a home in NSF’s
Research in Educational Policy and Practice (REPP) program. About 20% of the proposals were primarily anchored in
Quadrant 4, representing parts of both LIS and REPP.
ROLE, however, extends significantly beyond reorganizing previous
programs, with an interest on creating bridges and interactions across quadrants
and the formulation of new questions and approaches that emerge from the
continuum framework. About 40% of
the preliminary proposals conveyed significant cross-quadrant bridging; this
suggests that the evolving ROLE portfolio will indeed be able to feature the mix
of within- and across-quadrant activities that will give it the distinctive and
nationally beneficial character that we envision. The Preliminary Proposal Review Process
The REC research program officers prepared most of the nearly 500
reviews sent to proposal submitters. About
one-fifth of the proposal reviews, however, were completed by twelve other
program officers in the Directorate for Education and Human Resources (EHR).
These program officers volunteered to review proposals on the basis of
their interest in advancing a closer articulation of the research program in EHR
with the directorate. Additionally,
colleagues from the Directorate for Social, Behavioral, and Economic (SBE)
Sciences reviewed several dozen proposals that primarily fell in Quadrants 1 and
2 of the ROLE continuum. Most of
the SBE reviews consist of summary recommendations encouraging or discouraging a
formal proposal submission, and are identified as SBE reviews. Finally, a small
handful of proposals (14) were reviewed either by external panelists or by
collaborators from the United Kingdom’s Teaching and Learning Programme as
part of a cooperative activity we are undertaking with them.
These 14 are identified as non-NSF reviews. With virtually no exception,
all proposals received between two and four reviews. As important as reviewing the diverse set of preliminary
proposals proved to be, it was also important for each member of the ROLE
program staff to examine and understand all of the reviews that he or she did
not respectively prepare, and to consider both the totality of the proposal pool
and of colleague responses to it. That
is, the review process was viewed not only as an opportunity to respond to the
proposers, but also as an important month-long and intensive learning seminar
for the REC staff. In addition to
receiving the individual reviews for their proposals, all submitters also
received a summary recommendation from a set of four categories that appears
below, along with a list of the number of proposals that were each
recommendation. Recommendation category (Overall
tally out of 182) ·
Encourage;
when fully developed, this proposal is likely to be competitive in the expected
June proposal pool. (43 out of 182) ·
Encourage tentatively; this proposal has merit but is
not likely to be competitive in the expected June proposal pool.
Consider possible later cycle submission to allow development of deeper,
more thorough, and/or more focused full proposal. (54 out of 182) ·
Discourage; elements of this proposal may
be considered, however, for inclusion in a later preliminary proposal
submission. (51 out of 182) ·
Discourage; proposal either is not
competitive or does not fall within ROLE Guidelines (34 out of 182) Why Do the NSF Reviews on the Same Proposal Sometimes Differ?
Typically, individual reviewers
bring different perspectives on a proposal and thus their reviews may differ.
ROLE’s preproposal pool was no exception.
These
varying perspectives strengthen the review process. Differences that you may
see in reviews may help you understand how the proposal was interpreted by impartial
readers and suggest different interpretations that members of a formal proposal
review panel may take. There are
some general differences between REC reviewers relative to reviewers from other
divisions, reflecting difference in priority on research methods, on best
approaches to transfer between research and practice, and the mix of what might
be considered applied versus basic research.
Furthermore, the short narrative length, while providing ample
opportunity to convey the main ideas of the proposed research, left room for
varying interpretations of the same text. (Thus,
for example, we expect that the deficiencies in some of the proposals rated in
the second or even third category may actually be the result of failure to
convey ideas clearly or fully in the preproposal; these proposals may in fact be
ready for a June 2000 submission. That is a decision that the Principal Investigator must
make.) In some cases, (such as
criticizing some Quadrant 1 or 2 proposals for lack of specific science or
mathematics learning focus), NSF reviewers (outside of REC) are still becoming
accustomed to the new program; in fact, Quadrants 1 and 2 allow research on
learning in general (while Quadrants 3 and 4 require a focus on SMET learning).
The ROLE program will need to make these issues clear to formal proposal
reviewers. Any full proposal
submission should consider the comments of the reviews, but please remember that
the proposal will be judged against the ROLE Program Announcement (NSF-0017) by
reviewers who do not have access to the preproposal reviews. What Are The Most Common Problems in the Preproposals?
In reviewing the proposals, we
discovered that many proposals had similar deficiencies.
These fall into two general areas: Research
in General and Misinterpreting the Announcement. Research in General:
Importance of Prior Work and Methodology
·
Failure to situate the study within prior
work and literature.
New contributors to education or education research, particularly those
coming from other disciplines, often prepare their proposals as if there is
either very little no prior work on the topic or that no lessons, positive or
negative, can be learned from previous work.
Failure to discuss or to value related literature often leads to a
failure to communicate the value added of the proposal.
It immediately raises the question of whether the proposed activity
benefits from previously acquired knowledge or analysis of the gaps in that
knoweldge that the proposal will address. This,
of course, dramatically reduces competitiveness.
ROLE is not unique in requiring that this analysis be conducted prior to
writing the proposal, but the multidisciplinary nature of the program and thus
of reviewers makes this requirement paramount. ·
Failure to situate the study with a
framwork or theory.
ROLE recognizes that new research methods or approaches, and new areas of
research have a different relation to theory than more established fields.
However, failure to talk about the relationship of the work to existing
frameworks or to the development of new or hybrid frameworks often leads
reviewers to conclude that the proposed work will not generalize beyond the
particular project. ·
Weak links between research goals and
proposed methodology.
Reviewers must determine if the submission has a chance of success – or
of documenting why failures occurred. Discussion
of methodologies lets reviewers know the researcher understands the scope of the
project, the potential findings and the limitation of the work.
Education research is often especially prone
to an a priori bias that recurred in the preproposal pool and that
applicants should convincingly avoid in full proposals. Misinterpreting
the Announcement
·
A common error was listing too many
quadrants as relevant or trying to “force” contrived connections between
quadrants. The proposal should be
placed in the quadrant or quadrants that correlate with research questions the
proposal poses. ·
Proposals that were primarily based in
Quadrants 1 or 2 often failed to connect their basic research questions in
neuroscience or cognitive science research to educational issues or to identify
eventual possible connections. ROLE will not fund research whose natural home is in the SBE or Biological
Sciences (BIO) directorates. ROLE will
support research that integrates critical questions in learning and education to
neuroscience and cognitive science research traditions. One critical outcome of
such research is the development of a new research capacity in the field; the
participation of young researchers will be an important component of
competitiveness for proposals in quadrants 1 and 2. To Whom Should I Direct Questions or Comments?
The Program Announcement identifies the four
program officers responsible for ROLE (Anthony Kelly, Nora Sabelli, Larry Suter,
or Elizabeth VanderPutten). They
may be reached through the general office number of 703-306-1650 or via
electronic mail (aekelly@nsf.gov; nsabelli@nsf.gov; lsuter@nsf.gov; evanderp@nsf.gov).
Questions or comments may also be directed to Acting Division Director
Eric Hamilton (ehamilto@nsf.gov). Any
of these can field any questions or comments (whether about a specific proposal
submission or about the general REC research program), and will refer to
colleagues as necessary. Thank
you for the very satisfying response so far to this first ROLE competition. REC Research Program Staff
Go to top of this page Return to Gerry Stahl's Home Page Send email to Gerry.Stahl@drexel.edu This page last modified on October 27, 2003
Go to top of this page Return to Gerry Stahl's Home Page Send email to Gerry.Stahl@drexel.edu This page last modified on October 27, 2003 |
