The evaluation strategy comprised both quantitative and qualitative elements and encompassed the following complementary components.

• Impact on the Student with studies focused on assessment of:
  • students’ first impressions of VERT;
  • the impact of VERT on the development of practical skills and confidence;
  • and the impact of VERT characteristics on students’ knowledge and understanding, performance and application of skills.
• Recruitment and Retention: collection and interpretation of data collected by educational institutions, supplemented by questionnaires/interviews about the impact of VERT. Staff and Student Experience: final reports compiled by educational institutions and clinical departments and an evaluation survey distributed at the end of the VERT Project. 

6.1. Impact on the Student

6.1.1 First Impressions
A questionnaire (see Appendix 1) distributed to all pre-registration students during the first semester of the 2008/9 academic year explored initial perceptions of the impact of VERT on learning and skills development. Students were also asked to indicate:

• any difficulties experienced;
• the amount of time and for what purpose they had been using VERT; and
• what they had liked/disliked about its use.

Findings following analysis of the 184 questionnaires returned from students in 7 educational institutions are outlined below.

6.1.1.1 Students had used VERT for 3 hours and 40 minutes on average by the time they had completed the questionnaires. However, considerable variation in exposure to VERT was noted with a range of 30 minutes to 20 hours reported.

6.1.1.2 The students’ first impressions were generally very positive. There was a perception that use of VERT had a positive impact on: development of their understanding of radiotherapy concepts (82% agreed or agreed strongly); enhancement of practical skills (72% agreed or strongly agreed); and motivation (70%). It was reassuring to find that 90% of respondents agreed or strongly agreed that VERT had contributed to their enjoyment of the learning and teaching scenarios. Very few students identified problems during the early use of VERT with 81% indicating either no, or very few, problems.

6.1.1.3 Table 2 below lists comments students included about features of their early VERT experience they particularly liked or disliked.
 

Liked	Disliked
Most frequently identified: Familiarisation with handset and preparation for practice Less pressure (ability to learn at own pace) Feel more confident OK to learn from mistakes Very interactive Relating the CT data to other tasks Feeling immersed in scene Complex concepts easier to understand through visualisation	Most frequently identified: Not having enough time to use it (40% of respondents reported they would like much more individual time) Headaches and/or eye strain (24% of respondents) especially where wearing 3D glasses for over 30 minutes Not enough plans, CT datasets to view 3D glasses heavy/uncomfortable/difficult to wear over spectacles (15% of respondents)
Less frequently identified: Tracking 3D relational anatomy Practising electron set-ups Potential to reduce errors in real set-ups	Less frequently identified: Disorientation (particularly when other users manipulate the view) Nausea (less than 3% of respondents)

Table 2: Early VERT experience - summary of student likes and dislikes

 

Summary of main findings and recommendations

• Early experiences of using VERT were largely very positive.
• The majority of students indicated that VERT had enhanced their knowledge, understanding and skills.
• The majority of students indicated that they had enjoyed using VERT and found the experience motivating.
• Limited individual hands-on time was identified as the most common problem experienced.
• Headaches and/or eye-strain were reported by one quarter of students with increased incidence when 3D glasses were worn for over 30 minutes.

Accordingly it is recommended that educators:

• integrate VERT into learning and teaching scenarios as early as possible in pre-registration programmes; and
• limit session length where 3D stereoscopy is enabled, and consider disabling 3D stereoscopy where depth perception is not essential.

6.1.2 Skills and Confidence

6.1.2.1 Traditionally during the initial clinical placement students have been unfamiliar with equipment controls; lacked confidence in operating a linear accelerator handset; and simultaneously tried to focus on developing important clinical/patient-oriented skills. All educational institutions planned to use VERT to provide students with an opportunity to develop some confidence in operating the equipment prior to the first clinical placement.

6.1.2.2 Quantitative and qualitative data were collected from students via pre- and post-experience questionnaires (see Appendix 2), supplemented with qualitative data generated through focus groups following the initial clinical placement. The impact of pre-placement VERT experience on the skills and confidence demonstrated by students from the perspective of clinical staff was ascertained using an online questionnaire (copy included as Appendix 3) which was completed by 44 staff from 23 radiotherapy and oncology departments

6.1.2.3 The following key findings emerged from analysis of questionnaires which were returned by 98 students from 5 educational institutions.

• 83% of these students described themselves as ‘confident’ or ‘very confident’ in their use of computer technology. (This was considered to be a potentially important factor in predicting confidence in operating a linear accelerator in both virtual and real environments.)
• Statistically significant positive correlations were identified between confidence in the use of computer technology and both confidence in operating a linear accelerator before pre-placement VERT experience and improvement in confidence after that experience.
•  77% of students had 20 minutes or less individual hands-on experience of operating the virtual machine and only 7% had more than 40 minutes.
• 88% of respondents indicated that they felt the pre-placement VERT experience was both enjoyable and had enhanced their practical skills. The remaining 12% neither agreed nor disagreed.
• Student confidence in using a linear accelerator improved after VERT experience as illustrated in Figures 1 and 2 below.
   

[2]
^{(Click to enlarge)}

 

6.1.2.4 Staff perceptions of student skills and confidence in operating the equipment (hand pendant) on commencing clinical placement following pre-placement VERT experience are illustrated in figures 3 and 4 respectively.

[3]
^{(Click to enlarge)}

Levels of skills and confidence were perceived to be lower by staff than by students. 62.8% agreed that the student’s initial skills and confidence were either ‘better’ or ‘somewhat better’ when compared with those of previous students without pre-placement VERT experience. 48.9% were ‘confident’ or ‘very confident’ that the student was able to operate the equipment safely and confidently following VERT experience. The view of the majority of clinical staff is captured by the following comment from one respondent:

“After repeated sessions on VERT in the department I think the skills, confidence and capability increased more rapidly than perhaps previous students.”

However, increased student confidence can be un-settling:

"Students felt they could get involved more quickly on their first placement as they already knew how to dim the lights, rotate the gantry etc. This sometimes led to frustration however as the staff were less comfortable with this, still relying on their own explanation/observation before letting the students take part."

6.1.3 Staff and Student Experience
Seven major themes emerged from analysis of the qualitative data derived from student and staff questionnaires and the student focus groups.

6.1.3.1 Focus on familiarisation with use of the hand-pendant
All students recognised that their VERT skills development centred on familiarisation with use of the hand-pendant and did not prepare them for interaction with a patient or for the way ‘teams’ were involved in the set-up of patients.  The lack of practice with couch controls; the importance of patient positioning; and knowing ‘who was doing what’ were all issues raised during focus groups and highlighted by this student comment.

“When [the radiographer] passed me the handset for the first time I more or less knew what I was doing (with it) but when they started moving the bed I got thrown a bit.  It took me a while to work out what bits I was doing with the handset.  But, I’m fairly sure that this ... knowing what buttons to press ... meant I could concentrate on the patients and what was going on around me rather than only looking at the thing in my hand.  The student I was working with told me they spent ages getting to grips with the handset.”

Similarly, some staff noted that the focus on the hand-pendant may have been to the detriment of other skills development.

“Some of the first yr students have concentrated on being the first to get hold of the hand pendant - they have … been reluctant to do the patient side of setting treatments up. … They usually can move the bed / gantry and do the auto set-up, but forget about the why they are doing what they are doing.  … Some 1st yrs have had to be ‘weaned off’ the hand pendant and then given objectives regarding set up, patient care and communication.”

6.1.3.2 Reduction in anxiety
A frequent comment by students was that, although still very anxious about their first clinical placement, VERT experience had reduced this considerably. This was not universal, however, and students with a relatively low level of confidence in using computer technology tended to feel the most anxious. Some were reluctant to actively engage in the pre-placement VERT sessions although, arguably, they were the most likely to benefit. 
Staff identified similar issues to those highlighted by students as illustrated by the following comments:

“1st year students lack confidence so they may have some basic skills but they are wary of using them.”

“Confidence is difficult to quantify as the first placement is still daunting no matter what level of training.”

“Their anxiety remained high - but was focused on inter personal areas such as talking/touching patients and fitting in with staff.”

Most frequently noted, however, was the perception that more time for individual practice in VERT would have further reduced anxiety.

6.1.3.3 Insufficient time for practice
It was disappointing to find that only a limited number of the students said that they had been able to take advantage of practice sessions using departmental Seminar VERT systems, although most commented that it would have been useful to have had such VERT sessions integrated into early placements.

A majority of students considered that they had insufficient time in VERT prior to their first placement.  Most clarified that this related to individual, interactive engagement with the handset.

“VERT definitely helped me to grasp some of the basics of different techniques.  I wish I could have had more time using it myself though rather than just watching the lecturer demonstrate things.  I didn’t get enough hands-on experience myself and I think this would help a lot.”

and:

“Visualising stuff was helpful but I got less than 15 minutes using the handset and I needed much more time.  Maybe we could have had drop-in sessions that we could have booked ourselves after the ones already in our timetable.”

Some students commented that they had benefited from a mentoring/‘buddying’ system where more experienced students had assisted and/or facilitated sessions. Other suggestions included: start using VERT earlier in the course (consistent with findings of the ‘early impressions’ survey); and have regular sessions right up to commencing the first clinical placement.

A number of staff also commented on the lack of VERT time as the following examples suggest:

“The students remarked that they had only had one session on VERT and it was quite a long time before moving into the clinical environment.”

“The students at [University] said the VERT experience was in groups, not all used the system and their clinical ability to use the handset one year on is very poor due to high numbers of students on each machine and lack of use on VERT at the hospital or the university. VERT is a good idea which is under utilised.”

“More time pre-clinical VERT experience needed (our students only had very little time on it).”

6.1.3.4 Realism
Students commented that, although VERT was realistic and helped them prepare for exposure to a real linear accelerator, there were still differences between the virtual and real worlds.  Some of these were relatively minor, such as small variations in gantry/couch speed or the previously- highlighted lack of couch controls, but others, especially the lack of a realistic patient, were of greater significance.  However, probing during focus groups confirmed that students did appreciate that the VERT experience was not intended to replace the real environment and that focussing on development of psychomotor and simple practical skills prior to placement would allow more time to develop important ‘clinical’ skills in the real environment.

“I remember during the VERT sessions before placement that I made a comment [on the questionnaire] about not having an actual patient seriously limited how useful it would be but a discussion with my mentor kind of opened my eyes. ... She made me realise that I’d been able to spend my first 2 weeks on a linac doing more than learning how to operate the machine.”

This comment also highlights the importance of awareness by clinical staff of not only the benefits but also limitations of VERT in preparing students for clinical practice and the consequent necessity for some adjustment to student training during early placements.

6.1.3.5 Active learning in a safe environment
A key benefit of the use of a virtual environment for skills development is that it allows learners to make errors, from which they can learn important lessons, but which would not be tolerated in practice. Many students did identify as beneficial the ability to develop knowledge and skills through active participation in sessions and to learn through their mistakes. However it was interesting to note that some students were more cautious about the application of pre-placement VERT experience to the initial placement, identifying, as this student did, a different cause for anxiety.

“It was actually really different when I got on to placement.  I wasn’t so much anxious about not knowing what I was doing because of VERT, but I was really worried that I would make a mistake.  It didn’t matter in VERT but I was terrified of crashing the machine or even pressing the wrong button.  It didn’t help that it felt like everyone was watching me.”

6.1.3.6 Developing strategies for clinically relevant tasks
Students were asked to identify any other ways in which pre-placement VERT sessions had contributed to their preparation for clinical practice. Responses included:

• developing a better understanding of basic concepts such as the isocentre;
• understanding couch shifts and developing a strategy for accurately undertaking them;
• building knowledge and understanding of simple techniques; and
• improving knowledge of anatomy and, in particular, the ability to relate radiographic to surface anatomy.

6.1.3.7 Adverse effects
Incidence and frequency of adverse effects noted by students was consistent with the outcome of the ‘first impressions’ survey.  Approximately one quarter of students reported dizziness, headache, eye strain or disorientation and 2% reported nausea.  Most students commented that the adverse effects were minor or, at most, moderate. However, a very small minority reported that headache and/or visual strain and/or disorientation was so severe that they had to stop participating (see section 6.4.1 'Adverse Effects’ on page 30).

Summary of main findings and recommendations

This study reflects the performance of only one cohort and further evaluation is recommended but the general trend is positive.

• Pre-placement VERT experience enhanced basic practical skills and confidence.
• Students’ confidence in operating a linear accelerator using a hand-pendant was significantly improved after pre-placement VERT experience.
• Skills developed during pre-placement VERT experience, in general, transferred well to the clinical environment.
• Confidence with computer technology correlated positively with confidence in operating a linear accelerator before and after pre-placement VERT experience.
• Pre-placement VERT experience that focuses solely on learning how to operate a linear accelerator with a hand-pendant is insufficient.
• A lack of individual hands-on time was identified as a substantial shortcoming both by students and radiographers.
• Seminar VERT facilities were relatively under-utilised.
• The incidence of adverse effects was consistent with that noted in the ‘first impressions’ survey.

Thus it is recommended that educational institutions:

• continue to use VERT for the development of basic psychomotor/practical skills and to enhance confidence prior to initial clinical placements;
• ensure sessions are interactive and increase the time available for all students to engage individually, where possible;
• include within pre-placement VERT sessions: an introduction to fundamental concepts underpinning clinical practice; essential practical aspects; a focus on anatomy; and practice with simple techniques that facilitate the confident development of psychomotor skills;

and that

• pre-placement VERT sessions in educational institutions are supplemented by seminar VERT sessions during initial clinical placements to further enhance knowledge and skills; and
• following use of VERT for pre-placement preparation, the aims of the initial placement are reviewed and consideration given to a change in emphasis from development of simpler psychomotor/practical skills to clinical and team working skills.

6.1.4 VERT and Skills Development in Skin Apposition Techniques
Skin apposition techniques demand good spatial awareness, psychomotor skills and, ultimately, a large amount of experience. However, students have limited opportunity to practice, or even develop, the requisite skills for clinically acceptable positioning accuracy of the electron treatment technique. The VERT technology enables this to be addressed by providing the opportunity for practice in a safe environment.

6.1.4.1 A stratified randomised controlled trial in which 103 students participated was undertaken to assess the influence of both VERT tracking technology and 3D stereoscopy on performance of skin apposition techniques. Student performance was determined using an objective measures schedule and an accuracy tool integral to the VERT software.  Performance was also correlated with students’ spatial ability, measured using a Mental Rotation Test⁵. A post-experience questionnaire (included as Appendix 4) was used to determine: students’ experiences of using VERT; the extent to which they considered it had enhanced their clinical practice; and any adverse effects. Follow-up interviews after relevant placement experience explored the extent to which practice in VERT was transferrable to the clinical environment.

6.1.4.2 Participating students completed a guided self-directed practical enabling them to practice electron technique set-ups after randomisation into 1 of 3 arms:

• 3D stereoscopy ON, Tracking ON
• 3D stereoscopy ON, Tracking OFF
• 3D stereoscopy OFF (tracking therefore unavailable)

6.1.4.3 A subsequent objective assessment of ability to accurately and efficiently complete a specified pre-determined skin apposition set-up provided within the VERT software was the main outcome measure. Accuracy and efficiency were determined using the following factors based on the work of Srinivasan et al⁶ and Park et al⁷:

• Error score:
  • collisions between equipment and patient;
  • incorrect beam alignment to skin marks.
• Degree of skin apposition - defined as the standard deviation of the distance measured at all four corners of the applicator to the ‘skin’ surface.  This measurement was available via the VERT software.  Related to this measurement is an assessment of the degree of proximity of the applicator to the skin surface (the mean distance of all four corners of the applicator to the skin surface).
• Time taken to achieve an 'acceptable' set-up (established by an experienced radiographer whose score was used as a benchmark against which student performance was normalised).
• Economy of movement - defined as the number of gantry/couch/collimator movements to achieve what the student deemed to be an acceptable set up.

The method of calculating the resultant performance score is detailed in Appendix 5. The findings below are based on the results included in Appendix 6.

6.1.4.4 Qualitative analysis indicated that students in the ‘Tracking ON’ group found completion of electron set-ups more challenging although there was no statistically significant difference in set-up scores between the three arms of the trial.

6.1.4.5 It was noted that, for those students with tracking turned OFF, the view had to be manipulated by an experienced radiographer.  Although no verbal guidance was offered by the person manipulating the view, inevitably they used their own experience to intuitively adjust the view for the student. This guided the student as to where they should be looking and provided clues regarding action required.

6.1.4.6 Follow-up interviews explored the extent to which the type of VERT experience influenced the transfer of skills to real world set-ups. Students from the ‘Tracking ON’ arm were more positive about the speed with which they felt able to put their VERT experience into practice. However, the key theme from the interviews was the concern, expressed by most students, that the experience had not fully prepared them for the real set-ups.

6.1.4.7 All students enjoyed the VERT experience and recognised that it helped them to achieve acceptable electron set-ups but the majority felt that the situation in the real clinical environment was very different. They highlighted daily variations in position and patient breathing as examples of where VERT had not adequately prepared them for the need to adapt. While confidence increased as a result of VERT experience, anxiety in real world situations only lessened through real world practice.
 
6.1.4.8 Students found the objective assessment of their performance particularly useful. It helped to improve their skills substantially whether or not they had experience of clinical electron set-ups.  Many students, but particularly those with some clinical experience of electron techniques, suggested that practice in either Immersive or Seminar VERT facilities before and during relevant placements would be very beneficial.

6.1.4.9 Comments from students with poorer spatial ability about the possible benefit of VERT in developing strategies for electron techniques were pertinent in light of the moderately positive correlation found between spatial ability and set-up score (see Appendix 6). The following comment was typical:

“My spatial awareness is terrible and that probably explains why I’ve shied away from getting more actively involved with electron set-ups.  I just can’t see how gantry and couch need to be moved.  Spending time in VERT has really helped.  I wish it had been there when I was in year 1.”

Summary of main findings and recommendations

• Strategies for achieving good skin-apposition can be effectively learnt in VERT.
• Objective assessment of set-up in VERT can lead to improved students skills.
• 3D stereoscopy and user tracking do not appear to influence student performance or experience.
• Tracking appears to more accurately reflect the actual clinical scenario but may detract from students’ ability to accurately visualise alignment of light beam with skin marks.
• Inherent spatial ability correlates positively with set-up score.

It is recommended, therefore, that:

• VERT use is considered for the development of strategies in spatially complex set-ups at an early stage, particularly for those students with relatively poor spatial ability;
• user tracking and 3D stereoscopy are used with caution, particularly where students are prone to vection-induced simulator sickness, headaches or visual disturbance;
• further research which eliminates the influence of intuitive view manipulation by experienced personnel is undertaken to determine the influence of user tracking on student performance in VERT scenarios; and
• inherent spatial ability of students is assessed to assist identification of individuals who are likely to benefit most from VERT experience.

 

6.2. The Impact of VERT on Recruitment, Retention and Attrition

Reliably assessing the impact of VERT on attrition in pre-registration programmes was clearly unachievable given the duration of the VERT Project.  However, data from the annual monitoring survey submitted by educational institutions to the Society and College of Radiographers⁸; a supplementary survey; site visits; and ‘end of Project reports’ were analysed. The outcomes are outlined below.

6.2.1 Recruitment

6.2.1.1 Centres appeared to be using VERT effectively by integrating demonstrations and interactive sessions into prospective student visits, interview days, recruitment fairs and sixth-form events.   At least two educational institutions had purchased/developed a ‘portable VERT’ system with dedicated hand-pendants for recruitment use. Evaluation forms completed by prospective students in some institutions indicated that the VERT experience was the most enjoyable element of their visit. However, it is too early to determine the impact on application rates.

6.2.1.2 The introduction of VERT was too late to have influenced recruitment in 2008 when more educational institutions met target commissions than in previous years.  However, the continuing upward trend for therapeutic radiography programmes to be undersubscribed is of concern. Applications data for the 2009 intake were incomplete at the time of this Report. However, anecdotal evidence from a supplementary survey distributed to educational institutions seemed to indicate relatively limited impact of VERT on applications to date with some institutions noting an increase in applications from younger prospective students while others had seen little or no difference.

6.2.2 Attrition

Overall attrition in radiotherapy programmes decreased by 5.4% between 2007/8 (42.7%) and 2008/2009 (37.3%).  Although still high, the continuing downward trend (attrition was 48.7% in 2006/7) was reassuring. The impact of VERT on this reduction is unknown but it was interesting to note that, in the annual report to the Society and College of Radiographers⁸, no educational institution explicitly identified use of VERT as contributing to a strategy for reducing attrition.

6.2.3 Retention
Although the attrition data were equivocal, VERT may have a role to play in retention.

6.2.3.1 A small number of informal interviews were held with ‘at-risk’ students with whom VERT had been used to support specific weaknesses in an effort to improve retention. VERT had been used on an individual tutorial basis to help explain concepts and/or facilitate guided practice with these students who had been struggling with either academic content or clinical assessments. The students were very positive that using VERT had been a successful approach while acknowledging that alternative approaches may have achieved a similar outcome.

6.2.3.2 Responses from educational institutions in 'end of Project reports' also identified ways in which VERT was being used to address retention issues.  For example, in one institution a ‘peer-assisted learning scheme’, in which final-year students provided support and guidance within VERT for 1st-year students, was piloted and positively evaluated.

It is recommended that:

• staff consider using VERT to support ‘at-risk’ students where appropriate.

6.3. Overall Evaluation of the Implementation and Impact of VERT

At the end of the Project 9 educational institutions and 25 clinical departments completed an ‘end of Project report’ and 108 individuals responded to an online evaluation survey (copy as Appendix 7). Responses in the final evaluation survey reflected comments in the institutional Project reports and were distributed as follows:

• students  - 64%
• academic staff - 14%
• clinical staff - 22%

The survey results, including additional comments made by staff and students, are included for information in Appendix 8 and the findings from both outlined below.

 

6.3.1 Use of VERT

6.3.1.1 VERT facilities were used almost exclusively by pre-registration radiotherapy students although it was noted that other groups did use the facility.  The majority of respondents (83%) considered that 1st year students benefitted most from VERT.
However, many centres were beginning to realise the potential of the technology for:

• postgraduate students;
• those re-entering the profession;
• training new staff in local techniques;
• staff development;
• evaluation of new or unusual techniques; and
• use with other staff groups.

6.3.1.2 A number of departments indicated that dosimetrists made substantial use of Seminar VERT to augment treatment plan evaluation and, in some centres, integrated student learning within the process.
Many radiotherapy departments used VERT to demonstrate principles of radiotherapy to other health care professionals including medical staff, nurses and other visitors.

6.3.1.3 Some educational institutions had begun to develop applications employing stereoscopic visualisation to run on the available hardware for other student groups.  Examples included:

• an interactive 3D house for use with Occupational Therapy students;
• stereoscopic photos and videos for illustration where depth perception is critical; and
• interactive 3D anatomy materials for use with various health care students.

At least one institution had employed a technical officer to be responsible for stereoscopic software development and to assist the local VERT Lead with management and development of the VERT facility.

6.3.1.4 While widening access to other professional and student groups maximises use of VERT facilities, careful management to minimise the impact on pre-registration radiotherapy students is required.

It is recommended, therefore, that local VERT Leads encourage:

• student engagement in the process where other groups of staff such as dosimetrists utilise Seminar VERT facilities; and
• wider use of the facility while ensuring its availability to radiotherapy staff and students.

6.3.2 Enhancement of Knowledge and Understanding
The general perception was that VERT, currently, had the greatest impact on students' knowledge and understanding of fundamental concepts, simple techniques and anatomy as illustrated by the following student comment:

“The anatomy sessions were brilliant and really helped to gain an in depth understanding of how organs overlapped and sat next to each other.”

while this staff member considered that the first year students:

“… seemed to pick up the basics much more quickly (isocentre etc) and the calculating bed shifts was much less of a struggle than in previous years.”

However, the following student comments indicate the value in other areas such as plan evaluation where use may increase as institutions improve their access to DICOM data.

“Very useful for looking at the plans … seeing where we were treating and what the dose distribution was like. Helps put the theory into practice in a safe way.”

and

“Using VERT to evaluate plans … has helped me to understand the importance of accurate contouring and what the dose distribution looks like in 3-dimensions.  It has made me realise that I have to think in 3D when I'm planning.”

6.3.3 Impact on Clinical Competence

6.3.3.1 The results of the end of Project survey and Reports (Appendix 8) generally reflected those from  the pre-and post experience questionnaires (see 6.1.2 'Skills and Confidence’ p. 18) in that   confidence was perceived to be either ‘much better’ or ‘a little better’ by more than 80% of respondents.

6.3.3.2 VERT was considered to have improved the development of general psychomotor skills, performance with simple set-ups (single fields and parallel opposed) and electron set-ups by over 60% of respondents but by fewer (approximately 45%) in relation to complex set-ups.

6.3.3.3 Filtering responses by user group (student, academic staff and clinical staff) indicated that students regarded VERT to be more effective at enhancing aspects of clinical competence - particularly confidence - than either academic or clinical staff.

6.3.4 VERT Influence on Enjoyment of the Course
VERT influenced student enjoyment of the course considerably more in the educational institution than it did during clinical placement. This may reflect the fact that VERT is relatively under-used in clinical departments but, that in addition, there is no substitute for real clinical experience where it is safe and available as the following comment illustrates:

“There may be limited use on clinical placements as this is the only time that students will have real experience of dealing with patients which is highly valued by them.”

6.3.5 Curriculum Enhancement
Respondents provided additional comments regarding the use of VERT and ways in which integration into radiotherapy curricula could be enhanced. These included references to

1. technical developments such as:

• linking treatment planning systems directly to VERT;
• developments to support knowledge and understanding of 3D CT anatomy; and
• improved dosimetric visualisation and the effect of set up errors on dose volume histograms;

as well as

2. specific educational uses including:

• teaching more complex techniques;
• using VERT for assessment;
• incorporation into more post-registration and CPD programmes;
•  ensuring links between VERT use in the academic and clinical settings; and
• integration of VERT into multidisciplinary training within the clinical department.

However, as one respondent noted, meeting the challenge of a “packed curriculum … requires radical thinking” and for several institutions review and revalidation of their programmes provided the opportunity to build in more fundamental use of VERT across the curriculum.

 

6.4. Barriers to Effective Use

Potential barriers to the effective use of VERT identified during the Project, along with possible solutions, are outlined below.

6.4.1 Adverse Effects

6.4.1.1 Side-effects such as vection-induced simulator sickness, visual disturbances and headaches are associated with the use of virtual environments. The prevalence and severity of such symptoms can be affected by a number of factors including: the degree of immersion (presence); susceptibility to travel sickness; image flicker; misaligned projected images; and concomitant illness⁹.

6.4.1.2 The incidence of adverse effects reported in the studies undertaken was relatively low (see 6.1 [4] and 6.1.3.3 [4]). The most commonly reported symptoms were minor and detracted little from user experience. However, minimising the possibility of occurrence by avoiding excessive manipulation of the scene during user interaction and keeping sessions relatively short seems prudent.

6.4.1.3 Adverse effects were less severe when the stereoscopic 3D feature was turned off and users viewed 2D images without the 3D glasses.  The consequent reduced ‘immersion’ in the virtual environment could be considered an acceptable trade-off where appreciation of depth cues is not vital to students’ understanding or skills development.  Similarly, for those individuals susceptible to motion sickness or who experience disorientation when the view is manipulated by another person, user tracking is a useful alternative.

Consequently it is recommended that:

• educators inform all users of the likelihood of symptoms prior to use; and minimise manipulation of the scene when a user is interacting with it.

 

6.4.2 Staffing and Time
Staffing levels and access to VERT facilities, while variable across sites, were identified as barriers to use, particularly in clinical departments.

6.4.2.1 The implementation of VERT and expectation that it would be widely integrated into the curriculum placed substantial pressure on staff in both educational institutions and radiotherapy and oncology departments. The rapid introduction of the technology complicated timetabling and added to workloads for academic staff who had to reconsider curriculum design and prepare VERT sessions. Clinical staffing levels in many departments limited opportunities for radiographers to use VERT in student teaching.

6.4.2.2 VERT use in both clinical departments and educational institutions was most effective where a VERT Lead had been appointed.  It was vital, particularly in clinical departments, for this member of the team to have protected time to: plan the way VERT would be used; learn how to use the system; train colleagues; gather the necessary resources; and then to plan and deliver training sessions.  There is a risk that VERT facilities will be significantly under-utilised, particularly at busy times, without time allocated to support the role. As in the educational institution, VERT needs to be integrated within, rather than ‘bolted on’ to, departmental activity. 

6.4.2.3 Nominating a clinical VERT Lead who may be primarily available for student teaching in VERT is of advantage but can lead to infrequent use by other clinical staff trained in VERT and consequent de-skilling. Students may also become reliant on one style of teaching and not benefit from exposure to a wider breadth of skill, opinion and knowledge.

6.4.2.4 Problem-based learning sessions that require relatively limited direct supervision by clinical and/or academic staff require significant initial preparation and support but provide students with a valuable learning experience and may reduce radiographer input.

6.4.2.5 Some centres evaluated the use of final-year students to facilitate VERT sessions. Students were taught to use VERT and then had the opportunity to develop their own mentorship and supervision skills. This may be a cost-effective way of addressing the issue of limited clinical staff time. However, care needs to be taken to ensure that it does contribute to the development of the students and is not exploitative, over-burdening students at a stage when they need to focus on honing their own clinical skills and experience.

Consequently it is recommended that:

• centres appoint VERT Leads and offer an appropriate level of protected time/workload allocation to ensure successful implementation and management of VERT;
• in the medium term, more radiographers are trained to use VERT and afforded opportunities to facilitate teaching and learning sessions;
• centres consider the use of problem-based learning in VERT; and
• educational institutions consider adopting a mentoring/’buddying’ scheme whereby pre-placement VERT sessions are facilitated by experienced students.

 

6.4.3 Management of VERT Resources

6.4.3.1 Implementation of the VERT technology was funded to support the education and training of therapy radiographers but has the potential for wider use (see 6.3.1 'Use of VERT’ [5]) in both educational institutions and radiotherapy and oncology departments.

6.4.3.2 Care needs to be taken to ensure that under-utilisation, which provides the opportunity for links and further developments of VERT use within an educational institution or Trust, does not lead to encroachment on the time available for students. Academic and clinical VERT Leads need to maximise use to ensure the access required for radiotherapy education and training purposes is protected and maintained.

6.4.3.3 This may take the form of ‘block-bookings’ used for individually negotiated drop-in sessions in addition to those timetabled. Such an approach also allows educators to tailor the use of VERT to individual needs and, if appropriate, make more effective use of user tracking. Including students in work within the clinical department where VERT is used by others such as dosimetrists provides additional learning opportunities for students as well as developing VERT use.

6.4.3.4 The necessity for educational institutions and their associated placement centres to work together in maximising effective use of VERT cannot be overemphasised.

6.4.3.5 The ongoing costs, including those of hardware and software maintenance/support contracts and projector bulbs, need to be considered. The total cost of replacing both bulbs (approximate life-span 1500 hours) together for the Immersive VERT system is close to £3000 for example and thus wider use of the Immersive VERT facility needs to be appropriate. For example, it would be unwise to use it for simple projection of, say, PowerPoint presentations. Some centres have addressed this by installing a separate projector, and a document is now available to assist users to maximise bulb life. Seminar VERT systems can expect a bulb lifespan of 2000 hours, with bulb replacement costs approximately £400.

6.4.4 Location of VERT

6.4.4.1 VERT has been sited some distance from the main department in many clinical departments and in some educational institutions away from the faculty area. The time taken to get to VERT from the main work area has meant that: using VERT can be seen as a chore; and the staff member involved in the teaching session is not readily available if required urgently in the department.

6.4.4.2 The majority of Seminar VERT installations are in busy shared teaching or seminar rooms and, although advance room-booking is possible in many centres, priority is often given to other staff groups and meetings. Consequently, opportunities for spontaneous learning with VERT are often lost because of rigorous timetabling and/or block-booking.

Therefore, it is recommended that:

• VERT installations are located as close as possible to the main work area; and
• VERT facilities are booked on a regular basis and the time utilised in a meaningful way.

 

6.4.5 Access to Treatment Planning Data
Access to DICOM data and integration with treatment planning systems were identified as significant issues during the Project although, during the Project, sample DICOM data was made available by Vertual Ltd to provide a start for system users.

6.4.5.1 Obtaining data to import into VERT for subsequent use in learning and teaching scenarios was a major problem in many centres. Ensuring the necessary patient consent for use of CT and treatment planning data for teaching purposes was an issue initially, although generally resolved at local level. Some users suggested establishing a repository of shared DICOM plan files but this is unlikely in the immediate future because of similar consent issues. Production of example plans based on the visible human female CT dataset has been explored but there are limitations to the usefulness of this approach.

6.4.5.2 Those educational institutions with in-house treatment planning systems (TPS) and associated CT data appear to be at a significant advantage, particularly in relation to the use of VERT for demonstrating techniques and enhancing plan evaluation (see 6.3.2 'Enhancement of Knowledge and Understanding’ on p.28).

6.4.5.3 Centres with TPS networked to VERT identified several issues with data transfer to VERT in the required DICOM format. Some users reported that the process was excessively time-consuming, taking up to 5 minutes where data were exported to a folder on the TPS server. In addition, the risk of students accessing and inadvertently corrupting other vital data on the TPS server when exporting and importing data themselves needed to be considered.  These are not insurmountable problems and using a portable USB drive for export and import of data provides a simple solution which addresses both issues.

It is strongly recommended that:

• educational institutions seek funding for treatment planning systems for integration with the VERT technology.