Law Review Innovation: The Peer-Assist System
Back when I first joined this blog in November 2006, one of the first topics I discussed was law review innovation (see here and here). What I didn't disclose back then was that law reviews were on my mind because just five days before I made my first post here, a few friends and I had submitted to Penn Law School a proposal for a new business law journal (in response to an open call for proposals from the administration). Our proposal, for a journal that would have been called the Wharton Law Review, would have differentiated itself in a variety of ways, some of which I blogged about on here. For instance, in our proposal we advocated for, among other things, a form of open admissions membership and running a law review like a business.
Among those other things was a new form of submissions review, which we dubbed the "peer-assist" system. Rather than try to summarize peer-assist, I will reprint the relevant section from our proposal:
C. The Peer-Assist System
The most significant departure from the traditional journal structure that we are proposing is what we call the Peer-Assist system. We recognize that this is a novel and ambitious plan, and we are happy to modify any or all of the Peer-Assist structure to fit within the school’s aims. Additionally, we present three versions of the Peer-Assist program, which represent varying degrees of complexity. We believe that Peer-Assist can be of value to any or all of Penn’s journals, but we present it here in the context of the Wharton Law Review. Lastly, because Peer-Assist represents a departure from traditional journal workings, we decided to seek the opinion of faculty members prior to submitting this proposal. We sought responses from 86 faculty members at law schools around the country, of which 36 responded with comments. Most of the responses were positive; some were negative; all were helpful. A complete list of the comments we received is attached to this document as Appendix B.
Sub-section 1 of this section highlights the deficiencies of both student-editing and peer-review. Sub-section 2 describes the full Peer-Assist system that was sent to professors for feedback. Sub-section 3 describes a localized version of the Peer-Assist system, while sub-section 4 explores a student-only version of Peer-Assist.
1. Problems with Current Systems
We believe that the current system of law journals is designed to serve two basic purposes: the dissemination of scholarly works and the training and education of the students involved. The present structure of legal publication is a cross between the two extremes of traditional student-edited journals and fully peer-reviewed journals. Neither of these systems successfully meets both goals, and we hope Peer-Assist serves as an improvement.
i. Problems with Student Editing
Student-edited journals are, of course, the norm in the legal profession. Student-editing allows for the rapid dissemination of scholarly works without the time costs inherent in a lengthy peer-review process. Authors also have the ability to submit to many journals at once, which increases their likelihood of getting published and in a reasonable amount of time. However, many academics have criticized the student-editing system. Students are often under-qualified, or even un-qualified, to be critical reviewers of cutting-edge academic work in many highly specialized disciplines. Additionally, much of the work that students do on traditional journals consists of mechanical acts such as cite-checking, which does little to further the learning process for the students involved. Student-edited journals leave a lot to be desired.
ii. Problems with Peer Review
Peer-reviewed journals also fail to meet these goals. It can often take much longer to publish in a peer-reviewed journal, since the review process is reliant on the schedules of individual faculty reviewers. Also, peer-reviewed journals do not generally allow for multiple submissions, so if an article is not accepted initially, the author has to start the entire process over again. Lastly, peer-reviewed journals do not give their student members substantive tasks such as selecting articles, if they have student members at all.
2. What is Peer-Assist?
i. Description
Our proposed solution to these problems is the Peer-Assist system. This system seeks to tap into the collective wisdom of academics worldwide prior to selecting articles for publication. A journal using the Peer-Assist method would have the same structure as a traditional student-edited journal; only the process would differ. When an article is submitted to the Wharton Law Review, it will be stripped of its identifying information and posted to a secure, password-protected website. Access to this website will be limited to professors of any discipline at any recognized university who request access, all members of the journal (including associate editors), and alumni of the journal. Readers will have the option of browsing articles by subject matter or by keyword, and also will have the option to receive direct email alerts for new articles in certain fields. All readers may submit comments about the articles, which can be as detailed or as frequent as the reader desires. No anonymous comments will be allowed, as the reviewer’s name will be automatically added to every comment he or she submits. Additionally, the content of each comment will be only accessible by the editorial board of the Wharton Law Review, so incoming readers will not be able to see what else has been written about a pending article. Any author may opt out of the Peer-Assist system at his or her discretion, in which case that author’s submission will be reviewed by the editorial board alone. Ideally the Peer-Assist system will include academics from outside the field of law, so that an expert on, say, economics or philosophy can weigh in on cross-disciplinary articles.
It is important to remember that the comments solicited through the Peer-Assist system are simply a tool to aid the editorial board of the Wharton Law Review in its publication decisions. The students of the editorial board will retain final decision-making power over all articles, and will have the option to accept or reject an article at any time, regardless of the feedback received from Peer-Assist.
ii. Benefits of Peer-Assist
The potential benefits of the Peer-Assist system are enormous. The editorial board of the Wharton Law Review would receive guidance from scholars with more experience and more knowledge about specialized subject areas. This would presumably lead to the journal selecting higher-quality articles than a student-edited system alone, while also educating the editorial board to the responses a particular piece garners from other academics. The comments also would give the editorial board ample fodder for potential response pieces or shorter debates, such as the ones that the University of Pennsylvania Law Review currently publishes on its PENNumbra website.
Associate editors on the Wharton Law Review also would have much to gain from the Peer-Assist system. With access to the Peer-Assist database, associate editors would be exposed to a representative sample of the current state of legal academics. Associate editors would have the same ability to comment on articles that professors enjoy. Additionally, the Wharton Law Review can require associate editors to submit substantive comments (similar to a response paper) on a certain number of articles in the Peer-Assist database. This would expand the role of the associate editor into more substantive fields while also increase morale and feelings of involvement on the journal. These voluntary comments or mandatory response papers facilitate the educational training function of the journal, give the editorial board another perspective on the merits of a given article, and provide an excellent barometer of an associate editor’s willingness and analytic abilities, which will aid in third-year selection for the editorial board.
In effect, Peer-Assist takes the benefits that are usually reserved for Articles Editors (including the critical reading of and exposure to many articles) and makes them available to all members of the Wharton Law Review.
At the same time, we believe Peer-Assist represents the best of both worlds for faculty members. The Wharton Law Review can still afford to make relatively rapid publication decisions, but professors will take some comfort in knowing that their articles were reviewed by more than just the students on the editorial board. We also hope that the adoption of Peer-Assist will encourage academics from disciplines other than law to publish in the Wharton Law Review. These professors expect some measure of peer supervision and we hope that they will consider Peer-Assist to be a sufficient substitute, especially when balanced against the benefit of a substantially quicker turnaround time than traditional peer-review.
We believe that Peer-Assist also can be a boon for Penn Law School. If the system is successful, the law school will be viewed as a trailblazer. Even if Peer-Assist is not successful in the long run, Penn will be viewed as an innovator. Several of the law professors who were kind enough to provide advance feedback were so intrigued with the idea that they sought to mention it to their peers, and others asked to blog about it. Regardless of the outcome, Penn’s attempt to improve the journal structure will be noticed throughout the legal academy.
iii. Potential Problems
We recognize that Peer-Assist represents a large departure from the traditional article selection process, and while most of our commentators agreed that the idea was good in principle, there were several recurring criticisms. The most common was the concern that professors across the country have little incentive to give their time to a seemingly random journal. We believe this incentive problem can be alleviated with the offer of tangible benefits to particularly helpful or prolific commentators. These incentives can include automatic expedited review of articles, off-cycle review of articles, membership on a Board of Advisors or similar honor, free subscriptions to the Wharton Law Review, or, budget permitting, a monetary incentive.
A second common critique concerns the faculty reviewers themselves. Some professors worried that the reviewers would give good reviews of their colleagues’ articles in a quid pro quo arrangement, and that these reviews would unduly influence the article selection process. This problem assumes that the editorial board of the Wharton Law Review would be unable to distinguish legitimate criticism from puffery. It is our belief that the editorial board is capable of making these judgments, as distinguishing a good argument from a bad one is what Articles Editors already do. It is an important part of the learning process, and Peer-Assist will simply create another avenue for that skill to be developed. Additionally, we presume that the editorial board of the Wharton Law Review will naturally give more weight to objective comments (e.g. “This argument is pre-empted by X paper,” or “The mathematics in this piece are flawed”) or well-reasoned subjective comments as opposed to comments offering unsupported praise or criticism.
A third and final common criticism of Peer-Assist is that the readers may not be sufficiently qualified to give informed commentary. Peer-Assist will open up the peer-review process to all academics, not just the pre-selected ones that populate traditional peer-reviewed journals. This does present a greater risk of relatively less-qualified commentators, but we believe that the market effects of the Peer-Assist system will outweigh the risk. It is also quite likely that readers will only comment on articles within their field of interest, whereas the current student-edited system relies on only a handful of third-year law students to critique articles in a wide array of fields. Informed commentary from a professor would be a good addition.
3. Localized Peer-Assist
Many of the potential problems of the full Peer-Assist system can be mitigated by adopting what we refer to as “Localized Peer-Assist.” Instead of granting access to all academics, Localized Peer-Assist would give access to the Peer-Assist database only to professors at Penn, and would allow those professors (or the editorial board of the Wharton Law Review) to nominate professors at other institutions. This would allow the network to grow organically, admitting only academics who are interested in spending time to review articles and who are deemed qualified by the editorial board. We also envision adding functionality to the system to allow any authorized reader to forward a given article to any other academic on an ad hoc basis. For example, if a Penn professor was reading a business article that included statistical analysis, that professor could forward the article to a colleague elsewhere with a technical background and ask him or her to ensure that the methods used were satisfactory.
Localized Peer-Assist solves the major potential problems with Peer-Assist. Reader quality is no longer an issue so long as the Wharton Law Review screens those outside of the Penn community who will be given access. A limited pool such as this also minimizes the potential for reviewer puffery. Lastly, the incentives devised for the complete Peer-Assist program still can be used for Localized Peer-Assist, with the additional hope that professors will be more inclined to participate when it is their own school’s journal requesting their assistance.
In light of the smaller pool of reviewers, it would be possible to restrict usage of Localized Peer-Assist until an article passes an initial screening by the editorial board, who would still retain the ability to make the final decision on publication. As in regular Peer-Assist, students and alumni also would retain access to the database. Localized Peer-Assist foregoes the market effects of full Peer-Assist, but eliminates many of the potential problems. In effect, it simply formalizes a faculty review system that some student-edited law reviews have already adopted, except on a larger scale.
4. Student Peer-Assist
Our third and final conception is what we call “Student Peer-Assist.” This consists of the same structure as regular Peer-Assist, only limited to the student members of the Wharton Law Review. Student Peer-Assist preserves all of the educational advantages of Peer-Assist with virtually no downside. All members of the Wharton Law Review, including associate editors, would have the ability to read and comment on all of the submitted articles, thus exposing them to a broad swath of current legal scholarship. Student Peer-Assist allows for the Wharton Law Review to give a significantly more substantive educational experience to its members, which we believe will be valued by the student editors themselves and by the law school administration. The editorial board also gets the benefit of dozens of additional people reading over articles before they are selected.
Student Peer-Assist adopts a system that has been adopted by a handful of other student-edited law journals, including notably the Harvard Law Review. We believe that Student Peer-Assist offers virtually no downside and has the potential to improve morale and involvement of associate editors on the Wharton Law Review, as well as provide a significant educational benefit to all members of the journal.
You may wonder why I felt the need to discuss a proposal that died almost two years ago when it was rejected by Penn Law's administration. Well, upon reading TaxProf Blog today, it seems that Peer-Assist is not as dead as I thought. From a press release issued yesterday:
I can't say I'm surprised by the popularity of the Criminal Law Conversations project--the comments I received from 36 law professors at a wide variety of institutions in a diverse array of fields made it clear that Peer-Assist would be received well if implemented. Though I am obviously disappointed that my co-authors and I had not been given the opportunity to be the first to implement this system back in 2006, I wish Penn and Professor Robinson all the best with their endeavor, and hope that its success will incentivize the top law reviews (and the schools that support them) to implement a similar system and bring a close to the endless "student-edited vs. peer-reviewed law journals" debate.
Among those other things was a new form of submissions review, which we dubbed the "peer-assist" system. Rather than try to summarize peer-assist, I will reprint the relevant section from our proposal:
The most significant departure from the traditional journal structure that we are proposing is what we call the Peer-Assist system. We recognize that this is a novel and ambitious plan, and we are happy to modify any or all of the Peer-Assist structure to fit within the school’s aims. Additionally, we present three versions of the Peer-Assist program, which represent varying degrees of complexity. We believe that Peer-Assist can be of value to any or all of Penn’s journals, but we present it here in the context of the Wharton Law Review. Lastly, because Peer-Assist represents a departure from traditional journal workings, we decided to seek the opinion of faculty members prior to submitting this proposal. We sought responses from 86 faculty members at law schools around the country, of which 36 responded with comments. Most of the responses were positive; some were negative; all were helpful. A complete list of the comments we received is attached to this document as Appendix B.
Sub-section 1 of this section highlights the deficiencies of both student-editing and peer-review. Sub-section 2 describes the full Peer-Assist system that was sent to professors for feedback. Sub-section 3 describes a localized version of the Peer-Assist system, while sub-section 4 explores a student-only version of Peer-Assist.
1. Problems with Current Systems
We believe that the current system of law journals is designed to serve two basic purposes: the dissemination of scholarly works and the training and education of the students involved. The present structure of legal publication is a cross between the two extremes of traditional student-edited journals and fully peer-reviewed journals. Neither of these systems successfully meets both goals, and we hope Peer-Assist serves as an improvement.
i. Problems with Student Editing
Student-edited journals are, of course, the norm in the legal profession. Student-editing allows for the rapid dissemination of scholarly works without the time costs inherent in a lengthy peer-review process. Authors also have the ability to submit to many journals at once, which increases their likelihood of getting published and in a reasonable amount of time. However, many academics have criticized the student-editing system. Students are often under-qualified, or even un-qualified, to be critical reviewers of cutting-edge academic work in many highly specialized disciplines. Additionally, much of the work that students do on traditional journals consists of mechanical acts such as cite-checking, which does little to further the learning process for the students involved. Student-edited journals leave a lot to be desired.
ii. Problems with Peer Review
Peer-reviewed journals also fail to meet these goals. It can often take much longer to publish in a peer-reviewed journal, since the review process is reliant on the schedules of individual faculty reviewers. Also, peer-reviewed journals do not generally allow for multiple submissions, so if an article is not accepted initially, the author has to start the entire process over again. Lastly, peer-reviewed journals do not give their student members substantive tasks such as selecting articles, if they have student members at all.
2. What is Peer-Assist?
i. Description
Our proposed solution to these problems is the Peer-Assist system. This system seeks to tap into the collective wisdom of academics worldwide prior to selecting articles for publication. A journal using the Peer-Assist method would have the same structure as a traditional student-edited journal; only the process would differ. When an article is submitted to the Wharton Law Review, it will be stripped of its identifying information and posted to a secure, password-protected website. Access to this website will be limited to professors of any discipline at any recognized university who request access, all members of the journal (including associate editors), and alumni of the journal. Readers will have the option of browsing articles by subject matter or by keyword, and also will have the option to receive direct email alerts for new articles in certain fields. All readers may submit comments about the articles, which can be as detailed or as frequent as the reader desires. No anonymous comments will be allowed, as the reviewer’s name will be automatically added to every comment he or she submits. Additionally, the content of each comment will be only accessible by the editorial board of the Wharton Law Review, so incoming readers will not be able to see what else has been written about a pending article. Any author may opt out of the Peer-Assist system at his or her discretion, in which case that author’s submission will be reviewed by the editorial board alone. Ideally the Peer-Assist system will include academics from outside the field of law, so that an expert on, say, economics or philosophy can weigh in on cross-disciplinary articles.
It is important to remember that the comments solicited through the Peer-Assist system are simply a tool to aid the editorial board of the Wharton Law Review in its publication decisions. The students of the editorial board will retain final decision-making power over all articles, and will have the option to accept or reject an article at any time, regardless of the feedback received from Peer-Assist.
ii. Benefits of Peer-Assist
The potential benefits of the Peer-Assist system are enormous. The editorial board of the Wharton Law Review would receive guidance from scholars with more experience and more knowledge about specialized subject areas. This would presumably lead to the journal selecting higher-quality articles than a student-edited system alone, while also educating the editorial board to the responses a particular piece garners from other academics. The comments also would give the editorial board ample fodder for potential response pieces or shorter debates, such as the ones that the University of Pennsylvania Law Review currently publishes on its PENNumbra website.
Associate editors on the Wharton Law Review also would have much to gain from the Peer-Assist system. With access to the Peer-Assist database, associate editors would be exposed to a representative sample of the current state of legal academics. Associate editors would have the same ability to comment on articles that professors enjoy. Additionally, the Wharton Law Review can require associate editors to submit substantive comments (similar to a response paper) on a certain number of articles in the Peer-Assist database. This would expand the role of the associate editor into more substantive fields while also increase morale and feelings of involvement on the journal. These voluntary comments or mandatory response papers facilitate the educational training function of the journal, give the editorial board another perspective on the merits of a given article, and provide an excellent barometer of an associate editor’s willingness and analytic abilities, which will aid in third-year selection for the editorial board.
In effect, Peer-Assist takes the benefits that are usually reserved for Articles Editors (including the critical reading of and exposure to many articles) and makes them available to all members of the Wharton Law Review.
At the same time, we believe Peer-Assist represents the best of both worlds for faculty members. The Wharton Law Review can still afford to make relatively rapid publication decisions, but professors will take some comfort in knowing that their articles were reviewed by more than just the students on the editorial board. We also hope that the adoption of Peer-Assist will encourage academics from disciplines other than law to publish in the Wharton Law Review. These professors expect some measure of peer supervision and we hope that they will consider Peer-Assist to be a sufficient substitute, especially when balanced against the benefit of a substantially quicker turnaround time than traditional peer-review.
We believe that Peer-Assist also can be a boon for Penn Law School. If the system is successful, the law school will be viewed as a trailblazer. Even if Peer-Assist is not successful in the long run, Penn will be viewed as an innovator. Several of the law professors who were kind enough to provide advance feedback were so intrigued with the idea that they sought to mention it to their peers, and others asked to blog about it. Regardless of the outcome, Penn’s attempt to improve the journal structure will be noticed throughout the legal academy.
iii. Potential Problems
We recognize that Peer-Assist represents a large departure from the traditional article selection process, and while most of our commentators agreed that the idea was good in principle, there were several recurring criticisms. The most common was the concern that professors across the country have little incentive to give their time to a seemingly random journal. We believe this incentive problem can be alleviated with the offer of tangible benefits to particularly helpful or prolific commentators. These incentives can include automatic expedited review of articles, off-cycle review of articles, membership on a Board of Advisors or similar honor, free subscriptions to the Wharton Law Review, or, budget permitting, a monetary incentive.
A second common critique concerns the faculty reviewers themselves. Some professors worried that the reviewers would give good reviews of their colleagues’ articles in a quid pro quo arrangement, and that these reviews would unduly influence the article selection process. This problem assumes that the editorial board of the Wharton Law Review would be unable to distinguish legitimate criticism from puffery. It is our belief that the editorial board is capable of making these judgments, as distinguishing a good argument from a bad one is what Articles Editors already do. It is an important part of the learning process, and Peer-Assist will simply create another avenue for that skill to be developed. Additionally, we presume that the editorial board of the Wharton Law Review will naturally give more weight to objective comments (e.g. “This argument is pre-empted by X paper,” or “The mathematics in this piece are flawed”) or well-reasoned subjective comments as opposed to comments offering unsupported praise or criticism.
A third and final common criticism of Peer-Assist is that the readers may not be sufficiently qualified to give informed commentary. Peer-Assist will open up the peer-review process to all academics, not just the pre-selected ones that populate traditional peer-reviewed journals. This does present a greater risk of relatively less-qualified commentators, but we believe that the market effects of the Peer-Assist system will outweigh the risk. It is also quite likely that readers will only comment on articles within their field of interest, whereas the current student-edited system relies on only a handful of third-year law students to critique articles in a wide array of fields. Informed commentary from a professor would be a good addition.
3. Localized Peer-Assist
Many of the potential problems of the full Peer-Assist system can be mitigated by adopting what we refer to as “Localized Peer-Assist.” Instead of granting access to all academics, Localized Peer-Assist would give access to the Peer-Assist database only to professors at Penn, and would allow those professors (or the editorial board of the Wharton Law Review) to nominate professors at other institutions. This would allow the network to grow organically, admitting only academics who are interested in spending time to review articles and who are deemed qualified by the editorial board. We also envision adding functionality to the system to allow any authorized reader to forward a given article to any other academic on an ad hoc basis. For example, if a Penn professor was reading a business article that included statistical analysis, that professor could forward the article to a colleague elsewhere with a technical background and ask him or her to ensure that the methods used were satisfactory.
Localized Peer-Assist solves the major potential problems with Peer-Assist. Reader quality is no longer an issue so long as the Wharton Law Review screens those outside of the Penn community who will be given access. A limited pool such as this also minimizes the potential for reviewer puffery. Lastly, the incentives devised for the complete Peer-Assist program still can be used for Localized Peer-Assist, with the additional hope that professors will be more inclined to participate when it is their own school’s journal requesting their assistance.
In light of the smaller pool of reviewers, it would be possible to restrict usage of Localized Peer-Assist until an article passes an initial screening by the editorial board, who would still retain the ability to make the final decision on publication. As in regular Peer-Assist, students and alumni also would retain access to the database. Localized Peer-Assist foregoes the market effects of full Peer-Assist, but eliminates many of the potential problems. In effect, it simply formalizes a faculty review system that some student-edited law reviews have already adopted, except on a larger scale.
4. Student Peer-Assist
Our third and final conception is what we call “Student Peer-Assist.” This consists of the same structure as regular Peer-Assist, only limited to the student members of the Wharton Law Review. Student Peer-Assist preserves all of the educational advantages of Peer-Assist with virtually no downside. All members of the Wharton Law Review, including associate editors, would have the ability to read and comment on all of the submitted articles, thus exposing them to a broad swath of current legal scholarship. Student Peer-Assist allows for the Wharton Law Review to give a significantly more substantive educational experience to its members, which we believe will be valued by the student editors themselves and by the law school administration. The editorial board also gets the benefit of dozens of additional people reading over articles before they are selected.
Student Peer-Assist adopts a system that has been adopted by a handful of other student-edited law journals, including notably the Harvard Law Review. We believe that Student Peer-Assist offers virtually no downside and has the potential to improve morale and involvement of associate editors on the Wharton Law Review, as well as provide a significant educational benefit to all members of the journal.
You may wonder why I felt the need to discuss a proposal that died almost two years ago when it was rejected by Penn Law's administration. Well, upon reading TaxProf Blog today, it seems that Peer-Assist is not as dead as I thought. From a press release issued yesterday:
In a radically new interactive approach to legal scholarship, more than 100 leading scholars are debating the fundamental questions of modern criminal law through a law professor’s version of the TV show American Idol.
Professor Stephen P. Garvey of Cornell Law School, along with Paul Robinson of Pennsylvania Law School and Kimberly Ferzan, professor and associate dean at Rutgers School of Law-Camden, are the guiding professors in a 10-month online effort to create a new method of processing scholarship. In this new project, called Criminal Law Conversations, authors of the top-rated essays can defend their ideas against criticism from the judges, who are other law professors. The essays that receive too few votes get kicked off the stage, which in reality is the University of Pennsylvania Law School Web site, which hosts the Criminal Law Conversations project.
The selected essays will be included in an Oxford University Press book to be published next year.
“Too often opposing advocates talk past each other,” said Paul Robinson, lead editor of Criminal Law Conversations. “You could say that this brings peer review to legal scholarship but it’s more like peer-in-your-face.”
Robinson with co-editors Ferzan, and Stephen Garvey, are guiding professors in a 10-month online effort in which, so far, 120 scholars are participating. They are nominating several dozen scholarly works for discussion, based on the relevancy and compelling nature of the pieces. The author of a nominated work will produce a 4,000-word core text that summarizes his or her thesis, to which four to 10 scholars will then write 800-word criticisms. The original author will reply to the critiques, with these “conversations” making up the published book. ...
The response has been so positive, Oxford University Press is considering applying this model to other areas of the law and other fields of scholarship.
I can't say I'm surprised by the popularity of the Criminal Law Conversations project--the comments I received from 36 law professors at a wide variety of institutions in a diverse array of fields made it clear that Peer-Assist would be received well if implemented. Though I am obviously disappointed that my co-authors and I had not been given the opportunity to be the first to implement this system back in 2006, I wish Penn and Professor Robinson all the best with their endeavor, and hope that its success will incentivize the top law reviews (and the schools that support them) to implement a similar system and bring a close to the endless "student-edited vs. peer-reviewed law journals" debate.
Labels: academia, Anthony Ciolli, law reviews
posted by Anthony Ciolli at 6:55 PM
0 Comments:
Post a Comment
<< Home