Presbyterian Church (U.S.A.) - Blogs - Prep 4 Minhttps://www.pcusa.org/blogs/prep4min/2018-03-21T14:45:14-04:00Presbyterian Church (U.S.A.)info@pcusa.orghttp://www.pcusa.org/Latest Prep 4 Min posts.Clarification of Evaluation Standards for the Standard Ordination Exams2018-03-21T14:45:14-04:00Tim Cargal/blogs/prep4min/2018/3/21/clarification-evaluation-standards/https://www.pcusa.org/blogs/prep4min/2018/3/21/clarification-evaluation-standards/
<p class="date">March 21, 2018</p>
<div class="intro"><p>At its annual meeting from March 5-10, the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) sought to clarify for both examinees and readers the standards of evaluation for determining whether a particular exam submission satisfies the requirement of G-2.0607d in the respective area of examination (and, thus, receives a “Satisfactory” [S] evaluation). This clarification of the standards was in response to a pattern of increased reliance upon direct citation of source materials since the Church Polity, Theological Competence, and Worship and Sacraments exams joined the Bible Exegesis exam as “open book” tests in the second half of 2014. The new statement of evaluation standards is found on pages 93-94 of the now updated <em>Handbook on Standard Ordination Examinations</em> (release 2.5, March 2018; available for download from a link at <a href="/exams">http://www.pcusa.org/exams</a>). It has also been incorporated into the <em>Reader’s Handbook on Standard Ordination Examinations</em> as well.</p>
<p>The previous explanation of the evaluation standards was presented in two points. The first of those points, “Does the response address all the elements required by the question?”, remains unchanged in both its statement and explanation. The second point, “Does the response demonstrate sufficient understanding of the issues presented and ability to convert that understanding into practical application in ministry as would be expected of someone beginning in the ministry of the Word and Sacrament?”, has been divided into separate points, each with further explanation of what is expected in a “Satisfactory” answer.</p>
<p>The second standard of evaluation now focuses specifically on understanding issues raised in the question: “Does the response demonstrate sufficient understanding of the issues presented in the case study that provides the basis for the questions?” The explanation of this standard clarifies readers will be looking for a “broader knowledge relevant to the situation than might be expected of a general church member or person in the broader community” and reflective of an individual who has received “graduate-level education in [the respective] subject area.”</p>
<p>The final point in the standards of evaluation expands on the concern for demonstration of an ability to make practical application of one’s theological education: “Does the response provide evidence of an ability to combine that understanding with personal gifts and experience in ministry into practical application such as would be expected of someone beginning in the ministry of the Word and Sacrament?” The PCC explains the concerns in this standard as focused on “an ability to respond in pastorally appropriate ways” to the situations raised in the questions. It goes on to state that merely “linking together citations … does not truly demonstrate [this] competency.” Rather, “satisfactory” responses must demonstrate “the ability to summarize, restate, explain, and apply in a ministry context [the candidate’s] preparation” through theological education and reflection on the question. Unless the question specifically calls for only listing or citing of resources, “the response should for the most part be the candidate’s words rather than those of sources.”</p>
<p>All those candidates registered to take the exams in April will receive direct notification about this clarification in the evaluation standards, and it will be incorporated in the training of all readers who will evaluate the upcoming exams.</p>
<p>The full text of the updated section is as follows:</p>
<h4>Basic considerations</h4>
<p>Just because the exams are not “standardized” does not mean, however, that the results are then completely “subjective.” The PCC policies developed for evaluating the exams emphasize three basic considerations.</p>
<p>1) Does the response address all the elements required by the questions?</p>
<p>2) Does the response demonstrate sufficient understanding of the issues presented in the exam that provides the basis for the questions?</p>
<p>3) Does the response provide evidence of an ability to combine that understanding with personal gifts and experience in ministry into a practical application such as would be expected of someone beginning in the ministry of the Word and Sacrament?</p>
<p>To see how these standards of evaluation work, it is useful to think about the nature of the questions that appear on the senior ordination exams. In all areas—including Exegesis—the exams are essentially case studies. Drawing upon actual experiences from the life of the church, the questions call upon the candidates to integrate what they have learned in their academic preparation in seminary with experience gained through the use of their gifts in the supervised practice of ministry. Questions asked about these case studies generally require candidates first to identify and explain the pertinent issues that arise in the scenario and then to describe how they would respond using their gifts and skills for ministry.</p>
<p>The first standard—Does the response address all the elements required by the questions?—assesses whether the candidate can identify the pertinent issues in the situation. Does the response focus on key issues, or does it venture down side roads that diverge from what is really important. Does the answer respond to all parts of the question, or are some aspects of what is being asked for overlooked? The abilities to listen carefully to what is set before you and to respond completely to what is required in a given situation are key pastoral skills.</p>
<p>The second standard—Does the answer show sufficient understanding of the issues identified?—looks at how well the answer demonstrates an ability to apply academic training to the situation described in the question. Is the response theologically sound and does it provide evidence that the candidate has broader knowledge relevant to the situation than might be expected of a general church member or person in the broader community? Does the answer demonstrate an ability to choose appropriate resources and to engage types of resources that require graduate-level education in their subject area? And of course, any sources used or directly quoted must be cited in appropriate academic fashion.</p>
<p>The third and final standard—Does the answer combine theological training with personal gifts for ministry into a practical response?—looks specifically for an ability to respond in pastorally appropriate ways in the ministry context created by the question. Is the response expressed in a way accessible to the people in the scenario, considering such factors as whether they are adults, youth or children, and what is explained of their cultural backgrounds? Does the answer demonstrate an ability to use resources in effective ways? Simply linking together citations from either constitutional documents or other sources does not truly demonstrate competency for ministry. Competency is demonstrated through the ability to summarize, restate, explain, and apply in a ministry context one’s preparation (whether generally through seminary training or specifically in researching the particulars in the question scenario). This work must be done with appropriate pastoral sensitivity and consideration of the life stage and setting of the persons in the ministry context. Unless the specific required responses call for just listing or citing resources, discussions within the response should for the most part be the candidate’s words rather than those of sources.</p>
<p>As readers reflect on the questions raised by these evaluative standards, they are not comparing the specific answers either to models of ideal and less than ideal responses or even to answers given by others taking the same exam. Rather, calling upon their own experience as ruling elders and ministers of the Word and Sacrament and the training received from the PCC about the particular test, they are asking themselves, “Do these examination responses provide evidence that the person who wrote them can apply the intellectual background developed in seminary to pastoral situations with ‘energy, intelligence, imagination, and love’?” as expressed in one of the Constitutional Questions put to all those ordained to ministry in the church (W-4.0404h).</p></div>
BCE Preparation Resource Released2017-06-19T11:13:04-04:00Tim Cargal/blogs/prep4min/2017/6/19/bce-preparation-resource-released/https://www.pcusa.org/blogs/prep4min/2017/6/19/bce-preparation-resource-released/
<p class="date">June 19, 2017</p>
<div class="intro"><p><img style="float: left; margin: 3px;" src="https://a6cehq.bn1304.livefilestore.com/y4mm6_Uls60U1tuNMj8cmPRYAzAnGZpQkJ_XlyldOZv1Ou1SBkEyIAmHJOolBd-d6P98ahFLjsTunq8P5_XLbHECGPwMkTMvTI04PU9f8ISuAvqKQLZqYi2hIvfH_OAAOiFXdppCIKnc5lJ_6IYEY-WVGuplyZibRUs5DchK1RwnMR0Y8LcOqvpkPuYX6GBHXzP1xZtoAv90Z51HXPO7eMqwg?width=302&height=193&cropmode=none" alt="BCE Prep Homepage" width="302" height="193" /></p>
<p>Today the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) has officially launched a new resource to assist in preparing for the Bible Content Examination. All inquirers and candidates registered by their presbyteries with the Office of the General Assembly may use their accounts on the exams website (https://exams.pcusa.org) to access a special course on the PC(USA) Training site.</p>
<p>As initially announced in a <a href="/news/2017/4/6/pcc-concurs-task-force-recommendations-bible-conte/" target="_blank">press release</a> on April 6, this course includes updated guidance from the PCC’s <em>Handbook to the Standard Ordination Examinations</em> regarding the structure of and preparation strategies for taking the BCE, a revised video on these topics, the full-length practice BCE, and an archive of questions developed for the online BCE since the fall of 2009. The practice exam and the archives provide the questions in English as well as Korean and Spanish translations. After new BCE questions are used on the tests, they will be added to these archives.</p>
<p>Inquirers and candidates may access these resources at any time by logging on to their accounts at https://exams.pcusa.org. They will now find a link on their “Profile” page to “Access Moodle Materials.” Clicking that link will open a new tab in their browsers where they will be automatically logged onto the PC(USA) Training “Moodle” website where they can access the “Bible Content Exam Preparation” course. Beginning with the upcoming September 1 administration, the official BCE tests will also be administered using special courses on this “Moodle” site. By utilizing these preparation resources, then, inquirers and candidates will also be familiarizing themselves with the testing platform for the BCE.</p>
<p>The exam question archives are presented in seven canonical divisions utilized in structuring the BCE. Within those divisions, multiple-choice questions are presented in both online and printable-format tables in canonical order according to the primary reference in the respective question. Matching questions (because of their different structure and more limited use) are gathered in a separate division. Along with these tables, each canonical division also includes a quiz incorporating both multiple-choice and matching questions.</p>
<p>While these preparation materials can assist inquirers and candidates in their study and preparation, the questions that appear in the BCE are not limited to topics or verses explicitly treated within these materials. Those preparing for the BCE are encouraged to read widely in the Bible, using these preparation materials as a guide to strategies for study and aids in understanding the types of questions included in the test.</p>
<p>Earlier this month an updated <em>Handbook to the Standard Ordination Examinations</em> was also released (available for download at <a href="/exams" target="_blank">http://www.pcusa.org/exams</a>). This edition incorporates the PCC’s new guidance to inquirers, candidates, and their presbyteries that “the BCE be taken after at least one year of formal theological education including introductions to both testaments of the Bible” (p. 26). The handbook and preparation materials also reflect the PCC’s decision to discontinue the inclusion of an ordering question within the BCE; the test now includes only questions in multiple-choice and matching formats.</p>
<p>I am pleased that we have been able to make this resource available well ahead of the August 1 target originally announced in the press release. Please let me know if you have comments or suggestions about the site that I will pass along to the PCC.</p></div>
PCC Responds to September 2016 BCE2016-09-14T09:36:18-04:00Tim Cargal/blogs/prep4min/2016/9/14/pcc-responds-september-2016-bce/https://www.pcusa.org/blogs/prep4min/2016/9/14/pcc-responds-september-2016-bce/
<p class="date">September 14, 2016</p>
<div class="intro"><p>At a previously scheduled meeting on September 8-9, 2016, the Executive Committee of the Presbyteries' Cooperative Committee on Examinations for Candidates (PCC) had extended discussion about the results of the most recent Bible Content Examination (BCE). The following is their response to concerns about the exam from across the church, issued with the concurrence of the full PCC:</p>
<p>As part of its regularly scheduled fall meeting, members of the Executive Committee of the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) gave extended time and discussion to reviewing the results of the September 2, 2016, Bible Content Examination (BCE) along with concerns about recent changes to the test expressed both formally and informally from across the church.</p>
<p>Foremost among the comments being received is a request to return to the practice prior to 2009 of publicly releasing all questions utilized in BCE administrations. The reasons for this request are many and varied, and the members of the PCC have given them serious consideration. The Executive Committee after this deliberation has decided to maintain its current policy of not releasing questions, a policy which has been in place for the past seven years.</p>
<p>In reaching this decision, the Executive Committee believes it is important to recall that criticism of the BCE did not begin with the dramatic change in results when publicly available questions from pre-2009 exams were “retired” from inclusion in BCE administrations beginning in the summer of 2015. For many years, the BCE had been criticized as being an exercise in memorizing “Bible trivia” rather than a useful tool in assessing biblical literacy among inquirers in the preparation for ministry process with the Presbyterian Church (U.S.A.). There was wide agreement that a major contributor to this perception of the BCE was a broad consensus that the way to prepare for the exam was to memorize as many questions from past exams as possible in the hopes that one would encounter a sufficient number of repeated questions in taking the test to receive the 70% score necessary to satisfy the requirement. Neither inquirers nor the presbytery committees overseeing their preparation tended to focus on anything except that result, since considerable effort would have been required to review the questions returned to test takers along with their answer sheets to discern any patterns among missed questions.</p>
<p>For these reasons, when the PCC moved to online administration of the BCE in the fall of 2009 and other options for reporting results became possible, the committee ended the practice of releasing all the questions and instead provides a report to the inquirers and their respective presbyteries and seminaries that identifies the portions of the canon and specific biblical passages for questions that they answered incorrectly. The point is to focus on the pattern showing which portions of scripture and to a degree which books within those canonical divisions were less familiar to the inquirer so as to identify areas which would benefit from more focused study, both for those who scored high enough to satisfy the requirement and for those whose scores would require repeating the test.</p>
<p>So why not release the question text itself rather than just the biblical passages? Well, as already mentioned, that practice had created the conditions where the primary focus of preparation was not the Bible itself and resources that help a person gain proficiency in its stories, themes, and key passages (the areas of assessment as established by the General Assembly), but rather on the questions from previous exams. Editors of each BCE administration utilize questions from earlier exams with levels of difficulty established by how many test takers answered them correctly. The goal is to achieve some balance in overall difficulty from one BCE to the next. As long as questions that will be re-used are released, people will focus on those questions rather than the scriptures from which those questions were drawn.</p>
<p>It is also important to note that in adopting this approach the PCC is following evolving practice among other professional associations that give tests to assess competency in core areas of knowledge within their fields. A current member of the PCC reported to the committee that a similar situation in the Pathology Board Exam required reworking of that testing system. The Board routinely reused 50% of the exam from the previous cycle with new questions for the remainder of the exam. Over time, the common practice was to recall as many of the questions as possible, compile them, and then make them available to the residents taking the exams in the following years. Test takers are now required to affirm that they will not compile such question troves, with penalties if they have been found to do so. In similar fashion, those taking the BCE are now required to affirm an honor statement that they also will not disclose the content of questions in the test.</p>
<p>The PCC does all of its work, including administration of the BCE, on behalf of the broader church. It regularly participates in self-study for and review by the General Assembly, most recently just this summer by the 222<sup>nd</sup> (2016) General Assembly. There have been, however, some calls in response to the results on the last three BCE administrations for a more focused review of those exams. Because the PCC is deeply invested in creating the best possible examinations that measure what they are supposed to measure, we are more than willing to have the previous exams reviewed. To that end, the Executive Committee approved a motion at our meeting this past week asking the Committee on Theological Education to assist in performing an independent, confidential review of the recent exams in order to provide feedback to the PCC on the suitability of the questions in relation to the stated goals and purposes of the exam. The Executive Committee of COTE will be asked at their meeting later this month to nominate up to four persons to form a task group to review the past three BCE’s, as well as two exams from the time period when exam questions were publicly available. Michelle Bartel, Coordinator for Theological Education and Seminary Relations for COTE will convene and chair this task group; Tim Cargal, Assistant Stated Clerk for Preparation for Ministry and PC(USA) staff resource for the PCC will resource the task group; and, Kathy Riley, Bible Task Group Moderator, and Ken Broman-Fulks, PCC moderator, will be present to observe the process and report its feedback to the PCC. We look forward to hearing from this task group and considering their feedback on the exams.</p>
<p>Specific strategies for preparing to take the BCE are available in the Exams Handbook (<a href="/exams">http://www.pcusa.org/exams</a> ) and in a brief video entitled “Understanding and Preparing for the Bible Content Examination” available on the denominational website (go to <a href="/prep4min">http://www.pcusa.org/prep4min</a> and click “Online Training Opportunities” in the left-column menu options). </p>
<p>The PCC remains committed to fulfilling the General Assembly’s mandate to provide a standard ordination exam assessing inquirers’ knowledge of the stories, themes, and key passages of the scriptures and to supporting the preparation by those taking that test. We are convinced that a solid foundation in the biblical tradition is essential for those who engage in ministry within the context of the Reformed tradition, and we are certain that view is shared by all those who are concerned about the Bible Content Exam.</p></div>
PCC Letter on BCE Preparation2015-10-16T16:04:01-04:00Tim Cargal/blogs/prep4min/2015/10/16/pcc-letter-bce-preparation/https://www.pcusa.org/blogs/prep4min/2015/10/16/pcc-letter-bce-preparation/
<p class="date">October 16, 2015</p>
<div class="intro"><p>Today the Presbyteries' Cooperative Committee on Examinations for Candidates (PCC) sent the following message to all CPM moderators and PC(USA) contact persons at theological seminaries:</p>
<p>Greetings:</p>
<p>The members of the Bible Task Group and the Executive Committee of the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) are writing in response to concerns raised about the most recent Bible Content Exam (BCE), administered in September 2015. We have discussed the exam and related issues and have agreed to send this letter with a reiteration of suggested approaches to preparing for the BCE. We also are offering to assist Presbyteries and other groups with preparation of candidates for the upcoming BCE in February 2016 and for future exams.</p>
<p>The best approach for preparing to take the BCE is to study the Bible itself and to review materials that aid understanding its “stories, themes, and key passages” (the areas to be assessed established by the General Assembly). For several years now the PCC and Office of Preparation for Ministry have offered the following advice through the Exams Handbook and online training resources:</p>
<ul>
<li>Read widely in the Bible (preferably the New Revised Standard Version used for all citations in the English BCE) in order to become familiar with the style and content of its different books. </li>
<li>Review the outlines and introductions in study Bibles and Bible dictionaries, taking time to look up and read verses cited in support of key points about the individual books’ contents and central themes. </li>
<li>Take one or two publicly available exams (including the practice test on the testing system offered to registrants in the month prior to the BCE) to assist in identifying areas within the canon of relative strength and weakness and in order to focus preparation. </li>
</ul>
<p>The key thing to remember is that BCE tests given prior to 2009 are now just like practice tests in ACT/SAT/GRE study guides; they can help someone understand the types of questions and subject areas that will present them with the most difficulty. However, reviewing past tests will generally not be as effective as the methods mentioned above in strengthening one’s breadth of knowledge regarding the specific questions that will be encountered on the BCE.</p>
<p>There are also a few widely available books that provide reviews of the basic content of the Bible that may be useful in preparing for the BCE. We believe all of these resources and techniques are the most efficient ways to prepare for the exam. There may still be individual questions that are more specific than the content gleaned from a general study of the Bible. However, we strive to create every Bible Content Exam in a way that reflects its purpose to examine the stories, themes and key passages of the Bible, maintaining the requirement of a score of 70 out of 100 possible points to achieve a passing score.</p>
<p>All six members of the Bible Task Group and additional members of the PCC along with Tim Cargal, staff support to the PCC, are willing and ready to meet with representatives from presbyteries and seminaries in their area. If you are interested in such assistance, please contact Kathy Riley, Chair of the PCC Bible Task Group at <a href="mailto:kathyriley.523@gmail.com">kathyriley.523@gmail.com</a> or contact Tim Cargal at <a href="mailto:Tim.Cargal@pcusa.org">Tim.Cargal@pcusa.org</a>.</p>
<p> </p>
<p>Yours in Christ,</p>
<p>Presbyteries’ Cooperative Committee on Examinations for Candidates</p>
<p><em>Executive Committee:</em></p>
<p>Teaching Elder Steve Ranney, D.Min., Moderator</p>
<p>Ruling Elder Janis Adams, Secretary</p>
<p>Teaching Elder Ken Broman-Fulks</p>
<p>Teaching Elder Michelle Moe</p>
<p>Teaching Elder Kathy Riley, Ph.D.</p>
<p>Teaching Elder Marnie Silbert</p>
<p> </p>
<p><em>Bible Task Group:</em></p>
<p>Teaching Elder Kathy Riley, Ph.D., Moderator</p>
<p>Teaching Elder Clay Allard</p>
<p>Teaching Elder Margaret Aymer, Ph.D.</p>
<p>Ruling Elder Margaret Cowan, Ph.D.</p>
<p>Teaching Elder Leslie Dobbs-Allsopp</p>
<p>Teaching Elder Pamela Szurek</p>
<p> </p></div>
Bible Content Exam Results2015-09-04T20:08:23-04:00Tim Cargal/blogs/prep4min/2015/9/4/bible-content-exam-results/https://www.pcusa.org/blogs/prep4min/2015/9/4/bible-content-exam-results/
<p class="date">September 4, 2015</p>
<div class="intro"><p>The Bible Content Examination (BCE) was administered today, and the results both in terms of the average score on the exams and the percentage of inquirers and candidates who met the minimum score (70%) required to “Satisfy” this requirement in the preparation for ministry process were below historical averages.</p>
<p>A total of 127 individuals took the BCE, and the average score was 63.5%, which is about 10-15% lower than historical trends. However, since that average score fell below the minimum requirement, only 36 people (28.3%) scored high enough to receive a “Satisfactory” evaluation on the exam. By way of comparison, for the 12 previous BCE administrations since the exam was moved online in Fall 2009 the average percentage of individuals who scored 70% or higher in a testing cycle was 81.7%.</p>
<p>So, what might explain such dramatically different results?</p>
<p>Prior to online BCE testing in 2009, the Presbyteries’ Cooperative Committee on Examinations for Candidates (PCC) had released all BCE questions after each testing cycle just as they do with the other exam areas. Unlike the other exam areas, however, the PCC also continued to use the questions even after they had been released. Historically about 80% of the questions in each test were repeated from previous exams, in part to gauge relative difficulty of the overall exam based on how difficult questions had proven to be in the past. The PCC has long been concerned that these practices, though necessary in the period of “paper testing” to provide useful information about the subject of missed questions, were creating a situation where inquirers and candidates were spending more preparation time with old tests than with the Bible itself and study aids that deal with the “form and content of the Bible” in terms of its “stories, themes and key passages” (as the purpose of the test has been set by the General Assembly).</p>
<p>A different form of results reporting from the online tests made it possible to provide useful information about areas that would benefit from further study without having to release the actual questions. So in 2009 the PCC announced (1) it would no longer release questions from the exams and (2) that it was working toward discontinuing the use of questions that had been publicly released prior to 2009. Following the BCE administration in February of this year, the PCC communicated to CPMs and seminaries that it was preparing to end the use of questions that had appeared on paper-based exams and were publicly available. The PCC also announced that it would be bringing back some “matching” and “ordering” format questions to augment “multiple choice” questions in the exam (a practice that had also been used in the early years of BCE administration).</p>
<p>What, then, was different about today’s BCE as compared to the previous 12 online tests:</p>
<ul>
<li>The exam included three questions in either “ordering” or “matching” format, each with five items. The individual items in those questions counted as one point each (and a point was awarded for each proper match within the question), so there were 88 questions to arrive at the total 100 points available on the exam. (Samples of these new question formats were included in the Exams Handbook and the official “practice BCE” that was available to test takers since August 7.)</li>
<li>Those three questions (accounting for 15 possible points) along with two new multiple choice questions were the only questions that had not previously been used on the BCE since it went online in Fall 2009. Thus, 83% of the content of this test was repeated from previous exams, inline with historical practice by the PCC.</li>
<li>However, none of the 88 questions had ever been publicly released since they had only been used in online exams. Consequently, exam takers had not been able to “practice” with any of these specific questions.</li>
</ul>
<p>Preparation materials related to the BCE have for the past several years advised inquirers and candidates to use publicly available exams from previous years (including the “practice BCE” that the PCC makes available in the same testing system used for the “official” exam) to help them identify portions of the scriptural canon with which they are less familiar so as to know on what parts of the Bible to focus their study. They advised against trying to memorize past questions in hopes of seeing “familiar questions” on the BCE. That approach to preparation is even more important now that all publicly available questions have been retired.</p>
<p><em>(This post was originally sent as an email message to all CPM moderators and theological institution contact persons.)</em></p></div>