Email your comments

Background and Progress
The Trainees' views
Trainees' questions answered

In the last issue of this Newsletter, Zoe Wyrko, Chair of the BGS Trainees’ Committee, outlined the understandable concerns our specialty’s trainees about the introduction of Knowledge Based Assessment (KBA).

This therefore seems an appropriate time to review the background to KBA, update colleagues about progress as well as address trainees’ concerns.

Background
The JCHMT held pilot KBA exams in 2006 in four medical specialties: Geriatric Medicine, Neurology, Cardiology and Dermatology. Our specialty’s pilot exam was held in May 2006. There were 448 participants (402 SpRs, 37 consultants and 9 “others”, presumably associate specialists or staff grade doctors). We had an excellent turnout of SpRs representing 77% of the total number of our trainees registered at the time with the JCHMT, and an excellent spread of participants from the five years of SpR training. The KBA exam consisted of 100 best-of-five questions taken as a paper-based exam over 3 hours.

In all 4 specialties there was an increase in mean scores with increasing seniority from SpRs in year 1 of their training to consultants, although the difference was less pronounced in Geriatric Medicine than in the other 3 pilot specialties.

254/448 participants returned an evaluation questionnaire; 64% felt questions were “about right”; 70% that the time allowed for the exam was “about right”; and 80% were familiar with the question format.

In the free text section, comments were made about the length of questions, their structure (including using negatives and double negatives), that too many were in certain domains (e.g. falls and orthopaedics) and that some were not applicable to all four nations.

However, it must be stated that for the pilot KBA exam in Geriatric Medicine there was only a small core of question writers who had little training in question writing (one half day session) and also, although the other 3 specialties standard-set their exam in advance using the Anghoff method, Geriatric Medicine did not, our specialty set a notional pass mark of 50%.

The 4 specialty pilot KBAs were evaluated and deemed to be successful. Anyone wishing to read more about the pilot exams might wish to read the short editorial in Clinical Medicine “Knowledge-based assessment pilot project” Booth J, 2007 1: 9-11.

Development of KBA
PMETB require all StRs registering with JRCPTB on or after 1st August 2007, who are working to the new curriculum to have their competence assessed using workplace based assessments (mini-CEX, case based discussions, directly observed procedures (DOPS) and multi-source feedback) as well as an assessment of knowledge of the specialty, as there is an understandable view that specialists should be able to demonstrate to their patients and public that they have the knowledge and expertise to practise in their specialty.

A lot of thought was given to the type of KBA format and whether this assessment should be summative or formative, but in the light of the successful evaluation of the pilot KBAs the Federation of Royal Colleges proposed this format for the 13 “major” medical sub-specialties.

The Federation’s proposals were to have KBAs consisting of 200 best-of-five questions taken as 2 papers on the same day, the exam would be undertaken on-line utilising the services of Pearson Vue (who organise the driving theory test for the DVLA). There would be a number of exam centres around the UK and some locations (the same as for MRCP) would be located overseas. Any trainee who for medical reasons is unable to sit in front of a computer monitor for long periods can apply for special dispensation to sit a paper based exam.

The Federation initially proposed that the award conferred for anyone passing the exam, who did not have MRCP (UK), would be Diploma and for those who did have MRCP, would be MRCP (specialty). However, this caused our specialty great concern as we felt it would result in confusion with our existing Diploma of Geriatric Medicine and we also felt that there was no differentiation between trainees on UK based training programmes (undertaking a comprehensive training package including work-place assessments) and those who were not.

The Federation proposed that the costs of establishing KBA were shared in proportion with specialist societies 75% v 25%, and that exam fees should be £800 if the KBA exam is taken in the UK, and £1000 if taken overseas.

The Federation would train specialty question writers, and use their considerable expertise in running the MRCP (UK) exam to ensure that exams were set using a standardised approach for all medical specialties and incorporating robust standard setting measures.

Recent Progress
In response to an advert in the BGS Newsletter March 2007 and a message circulated to regional BGS Chairs, council members and special interests groups, I recruited 25 question writers who have all since had training by the Federation in question writing and who include a good mix of colleagues with special interests, from all four nations, “academics” and “non-academics” and age ranges from those who recently acquired a CCT to old(er) hands.

Questions produced by the writers then get discussed at workshops by a sub-group. Suitable questions (sometimes after modification) are banked, others are rejected outright and others are returned to authors with suggestions to refine them.

A Clinical Examination Board has been established (which I chair) whose role it is to choose questions from the bank to go forward to the next KBA exam, ensuring that the curriculum is adequately covered. The Board also acts as a second chamber so if they feel questions in the bank are inappropriate these can be returned to the writing group.

Questions selected by the Board go to the Standard Setting Group chaired by Dr Richard Fuller. This group has a number of members (including the Chair), who have considerable expertise in the standard setting of medical undergraduate final exams, and will utilise the same standard setting principles.

Earlier this year our specialty was in a position to hold our first KBA exam but we delayed running this until the issues around the awards conferred and costs to examinees had been addressed by the Federation. The BGS had a number of concerns about the Federation’s proposed KBA business model, as well as the cost to trainees of sitting the exam itself, which many in the Society feel is too high.
I am pleased to say that we have been successful in re-negotiating the Federation’s proposals about awards. The Federation now agree that anyone who passes KBA will get a Certificate (not a Diploma, which avoids confusion with the DGM) and that this will convert to MRCP (specialty) at the time an StR is recommended for a CCT. This will ensure trainees on UK based training programmes will have an award which is higher than those who are not.

Although the BGS is now prepared to sign the Memorandum of Understanding (MoU) with the Federation there are ongoing discussions between the BGS and other specialist societies and the Federation regarding some of the financial aspects, and in particular the exam cost to trainees.

To date the only specialty to hold a KBA exam is Gastroenterology, who held their first exam on 24 June 2008, hopefully now the MoU is signed, Geriatric Medicine should be in a position to hold their KBA soon.

I would like to thank all the question writers, board members and “standard setters” for the considerable contribution they have made to the KBA process to date. Your help and support is greatly appreciated.

Oliver J Corrado
Lead Physician, KBA Geriatric Medicine

Trainees’ stand on the KBA

The trainees acknowledge that there will be a knowledge based assessment as part of the suite of new assessments for STRs.

We appreciate that our concerns regarding validity of said assessment have been taken seriously with confirmation that the pilot exam which took place in summer 2006 was a procedural pilot rather than a validity pilot. We understand that the questions in the new exam have been written in association with the experienced question writing boards of the RCP, will only assess one area at a time, and are more robust than the pilot questions.

We accept that validity of the assessment will only be established after it has been ongoing for several diets, and that another single pilot would provide mostly procedural data and little to confirm or refute its robustness.

We agree that it is vital that the BGS remains integral to the exam, and that it must not be run without the Society’s cooperation.

We appreciate the support and understanding from consultant colleagues generally, and specifically those on the various committees of the BGS.

We are however still unable to support the exam due to the magnitude of the fee.

Unlimited re-sits at no further charge could help to ease the burden a little, but we would also urge the BGS to investigate the possibility of setting up a hardship fund to support those trainees unable to afford the exam fee.

Zoe Wyrko
as reported at the meeting
of the UKMC
July 2008

Knowledge Based Assessment - Trainees’ concerns answered

We think it important to state, at the outset that all those connected with training (the SAC Geriatric Medicine and BGS Education and Training Committee) are very sympathetic to the views and concerns raised by the trainees.

After all, this is another exam and none of us, if we were honest, would really want to take another exam if we could avoid having to do so!

However, the driver for KBA is the requirement by PMETB to include an assessment of knowledge within the assessment portfolio of specialty trainees using the new curriculum.

There follows a list of trainees’ questions/concerns as included in a paper put to the UKMC’s May 2008 meeting and our responses:

1. Is there a role/need for an exam to ensure we are properly trained?
KBA is only one form of assessment which complements other forms of workplace-based assessment. PMETB requires a knowledge based assessment for all StRs registering with JRCPTB on or after 1st August 2007. There are some areas of the curriculum e.g. gerontology, ageing physiology, which lend themselves better to a knowledge test than workplace-based assessment and vice versa.

2. The exam is designed to have a high pass rate
As this is a test of core specialty knowledge, we very much hope and anticipate that the pass rate will be high.

3. The pilot exam was statistically invalid and taken by trainees on a different system
As described in the earlier article, the pilot KBA was a pilot, question writers received very little training in question writing and no attempt was made by our specialty to prospectively standard set our pilot exam. Much has been learnt from the pilot, question writers have received much better training, there is good representation from the four nations to avoid questions which are not relevant to a particular nation, and by incorporating an examination board and standard setting group into the KBA process, the exam is much more sound.

Whilst the trainees who took the pilot were working to a slightly different curriculum and one which didn’t incorporate assessment of competence, we believe that the trainee cohort were comparable to current StRs. Furthermore many of the SpRs who took the pilot KBA are still in training.

4. The history of computer systems in the NHS is poor at safeguarding information
There are obvious advantages to an on-line versus paper-based system. Pearson Vue have successfully delivered the DVLA driving theory test in a similar format for some time. Clearly every effort will be made by the company to safeguard candidates’ personal information and confidentiality.

5. The timing of the exam in St4 and variability of regional training days
The exam can be sat from St3 onwards. We anticipate that whilst some StRs may want to sit KBA early, the majority will sit in St5. We anticipate that StRs will be dealing with many elderly patients whilst doing GIM in their earlier years, and therefore will still acquire considerable expertise and knowledge of older people and their problems during this time. We have every expectation that as regional training days reflect the geriatric curriculum, they will be an important source of information for KBA and a stimulus for further reading and other types of self-directed learning. If training days are better structured in some regions than others, KBA should help act as a catalyst to improve teaching standards throughout all regions. This is an area the ETC would be keen to monitor, so your feedback will be important.

6. An extensive industry is likely to be built around KBA
We concede that as with MRCP (Parts 1 and Part 2 written) and PACES, it is likely that an industry of books and courses designed to help doctors get through the KBA exam may develop. These will not be mandatory and it will be entirely up to the individual as to whether they wish to utilise such services or not. We firmly believe that KBA is pitched at a level which should allow trainees to pass it with the level of knowledge acquired during their training, supplemented by regional teaching programmes, standard courses and meetings, and a little book work.

7. Lack of differentiation on the award conferred to trainees on UK based programmes and those who are not
We believe the change we have successfully negotiated with the Federation regarding Certificate and MRCP (Geriatric Medicine) (see above) has addressed this concern.

8. Do SpRs have to sit KBA?
There is no mandatory requirement for SpRs working to the old curriculum, who registered with the JCHMT/JRCPTB before 1st August 2007, to sit KBA. It is entirely up to the individual as to whether they wish to do so or not.

9. Costs of sitting KBA are too high
We too (like other specialist societies) are concerned about the proposed exam fee (£800). We have been repeatedly informed by the Federation that this is appropriate for a “high stakes” exam and the costs associated with establishing KBA justify the proposed fees. However the BGS is continuing to negotiate with the Federation on this, it is vital that fees are set at an appropriate level and that the Federation and societies do not profit at the expense of trainees.

We understand that where cases of genuine hardship exist e.g. single parents with low income, that the BGS is looking at helping out in such situations.

We fully understand and sympathise with the concerns raised by trainees, particularly the exam fee cost. However we are required by PMETB to have a knowledge based assessment in place for StRs and we firmly believe that there are considerable advantages in adopting the Federation’s proposed model of KBA which offers the opportunity of a rigorous and standardised approach to KBA for all medical subspecialties. However the BGS, in concert with other specialist societies, must continue to negotiate with the Federation on the question of costs and we promise to do our best to do so.

Oliver J Corrado
Lead Physician KBA Geriatric Medicine

Rhian Morse
Immediate Past Chair BGS Education and Training Committee

Tahir Masud
Chair BGS Education and Training Committee

BGS Newsletter, Aug 2008
Issue 17 ISSN 1748-6343 17