International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 3

Number 3

September 2009

269

1 INTRODUCTION

Simulators are widely used for different educational

purposes, such as learning, training, research, forma-

tive and summative assessment and evaluation, not

only in maritime education and training, but also in

aviation, nuclear power plants (Ham et al., 2008),

medicine: surgery (Cosman et al., 2002; Maithel et

al, 2006), nursing (Decker et al., 2008). Simulations

are example of technology enabling learning to be

done more easily and effectively (Biggs & Tang,

2008) in a safe and controlled environment. As tech-

nology is developing rapidly the simulators become

more popular educational mean.

Over last few decades there was a huge growth of

automated navigation systems. More and more sys-

tems making navigator’s job easier came on the sce-

ne. It has begun with global positioning systems

(GPS) and nowadays grown into electronic chart

display and information systems (ECDIS). There is

no doubt that new sophisticated systems facilitate

navigator‘s job and ensure safer navigation. For ex-

ample, the study of Gould et al. (2009) proved that

ECDIS appeared to improve navigation performance

compared to conventional navigation based on paper

charts. Although this fact was proved under normal

conditions and more investigations should be under

high-workload conditions and in the presence of

other stressors. Habit to use sophisticated (automat-

ed) systems sometimes have negative impact to the

qualification of the navigator, as captain can lose his

proficiency in using usual navigational means be-

cause of permanent usage of automated navigation

systems. It is obvious for the experienced navigators

that modern navigation equipment today still is not

perfect.

Refusing the traditional means of navigation,

question of navigational safety may arise in case of

the failure or malfunctions of automated navigation

system, because during initial study stage attention

to the traditional navigational tasks slacks as student

contemplates that use of traditional navigational

tasks is not relevant, but this can be designated as

lack of navigators qualification, where qualifications

are well-defined in STCW (Standards of Training,

Certification and Watch keeping) convention.

Improvement of navigational systems tended to

facilitate safer navigation and load of the additional

New Capabilities of the NTPRO 4000 Full

Mission Ship Handling Simulator in the

Assessment and Evaluation Processes at

Lithuanian Maritime Academy

R. Zazeckis, I. Bartuseviciene & R. Maksimavicius

Lithuanian Maritime Academy, Klaipeda, Lithuania

ABSTRACT: Nowadays the equipment used on ships is becoming more and more sophisticated, safety of

shipping depends heavily not only on trustworthiness of high-tech products, but also on mariners’ compe-

tence. It is necessary to find methods to enable seafarers to keep track with technical developments. Simula-

tors as a tool combined with a properly developed course curriculum and qualified instructors provide an ap-

propriate method not only for training, but also for measuring, assessing and evaluating individual mariner

performance in order to test levels of competency and proficiency. According to Nieri (1995, p. 1/6), “The

development of performance-based assessment tools would utilize the widely-recognized advantages of simu-

lators and part-task trainers to generate a scenario in which the license candidate must demonstrate his

knowledge of system relationships, knowledge of operational procedures, monitor systems and situations, re-

spond to unexpected occurrences, identify and diagnose problems, and manage personnel, in a dynamic

world“. This paper discusses the usage of new capabilities of the NTPRO 4000 full mission ship handling

simulator in the assessment and evaluation processes at Lithuanian Maritime Academy.

270

tasks to the navigator increased requirements to the

qualification of seafarers. As a consequence teaching

process and teaching aids had to be improved. So

navigational simulators able to simulate different

navigational situations and to ensure that student is

able to clarify different solutions of navigational task

and implement the very best solution in certain cir-

cumstances were implemented to education and

training process of navigators.

Usage of simulators is the best solution for the

analysis of the actions taken by the students during

the performed tasks and for the understanding of the

competence of students. The diagnostics and analy-

sis of mistakes will allow avoiding possible mis-

takes, which may arise in different navigational situ-

ations.

During the education and training process most of

the mistakes are analyzed theoretically before the

usage of the simulator. However nowadays the in-

structors are practically able to allow students to

make mistakes using technologies of the modern

simulators and show the possible consequences of

the mistakes to students without real damage to the

environment. Possibility to observe mistakes during

performance of special navigational tasks helps to

make appropriate decision about the competence of

the student not only in routine, but also in emergen-

cy situations.

NTPRO 4000 full mission ship handling simula-

tor is used continuously in the process of education

and training of navigators in Lithuanian Maritime

Academy. Two years ago the new capability in

TRANSAS NTPRO 4000 called the TRANSAS

Evaluation and Assessment System (TEAS) was im-

plemented. The new capability allows assessing ob-

jectively the correctness of an exercise performance

by a trainee on NTPRO 4000 navigation simulator.

This opportunity is used continuously during all

learning process for the formative assessment. Ac-

cording to Brown & Glasner (1999), good assess-

ment (formative or summative) has to be valid, reli-

able, practical, developmental, manageable, cost-

effective, fit for purpose, relevant, authentic, closely

linked to learning outcomes and fair. In formative

assessment, the results are used to improve learning:

students can make mistakes: “the error detection is

the basis for error correction” (Biggs & Tang, 2007,

p. 164). It’s completely different in summative as-

sessment and evaluation: the results of such an as-

sessment are used to grade or certificate students at

the end of a course or program. According to Biggs

& Tang (2007), error no longer is there to instruct,

as in formative assessment: error now signals pun-

ishment. That’s why much more attention should be

paid for the final examination, where complex as-

sessment evaluates not only how each part of

knowledge is soaked up, but also how all knowledge

and skills are implemented in close to real situation

in real time. The TEAS gives this possibility.

For two years the TEAS of NTPRO 4000 full

mission ship handling simulator has been used for

summative assessment and evaluation during final

examination at Lithuanian Maritime Academy. The

authors are still searching the ways to improve ob-

jectivity, validity and reliability of the final evalua-

tion of the students. Some results of this experience

are presented in the article.

2 MANDATORY MINIMUM REQUIREMENTS

FOR CERTIFICATION OF OFFICERS

It is well-know that mandatory minimum require-

ments for certification of officers in charge of navi-

gational watch are well-defined in STCW code. The

code clearly states the standards of competence that

every candidate for certification shall be required to

demonstrate: the competence to undertake at opera-

tional level, the tasks, duties and responsibilities are

listed in column 1 of STCW code table A-II/1; the

minimum knowledge, understanding and proficiency

required for certification is listed in column 2 of

STCW code table A-II/1; the level of knowledge of

subjects listed in column 2 of STCW code table A-

II/1 shall be sufficient for officers of the watch to

carry out their watch keeping duties; every candidate

for certification shall be required to prove evidence

of having achieved the required standard of compe-

tence in accordance with methods for demonstrating

competence tabulated in columns 3 and 4 of STCW

code table A-II/1.

With reference to above mentioned TRANSAS

NTPRO 4000 simulator (especially the TEAS) is an

appropriate solution solving the student’s certifica-

tion problems in LMA.

2.1 Possible competency assessment using NTPRO

4000

Most of the educational programs during the process

of education and training of navigators are accom-

plished with reference to STCW code. The same re-

quirement corresponds to assessment and evaluation

process. Assessment and evaluation in accordance to

STCW code requirements can be improved using the

NTPRO 4000 TEAS because this system allows as-

sessing the navigator’s competences stated in STCW

code (table 1).

271

Table 1. The list of competences, which could be assessed us-

ing NTPRO 4000 TEAS.

__________________________________________________

1. Plan and conduct a passage and determine position:

1.1. Ability to determine the ship‘s position by use of:

1.2. Ability to use navigational charts and publications,

such as sailing directions, tide tables, notices to mari-

ners, radio navigational warnings and ship’s routing

information.(ECDIS systems are considered to be in-

cluded under the term “charts”);

1.3. Ability to determine the ship‘s position by use of

electronic navigational aids;

1.4. Ability to operate echo sounders and apply the in-

formation;

1.5. Ability to determine error of the magnetic and giro

compasses and to allow for such errors;

1.6. Knowledge of steering control systems, operational

procedures and change-over from manual to automat-

ic control and vice-versa. Adjustment of controls for

optimum performance;

1.7. Ability to use and interpret information obtained

from shipborn meteorological instruments;

1.8. Knowledge of the characteristics of the various

weather systems, reporting procedures and recording

systems;

1.9. Ability to apply the meteorological information

available.

___________________________________________________

2. Maintain a safe navigational watch:

2.1. Thorough knowledge of the content, application and

intent of the International Regulations for Preventing

Collisions at Sea;

2.2. Thorough knowledge of the basic principles of keep-

ing a navigational watch;

2.3. Thorough knowledge of effective bridge team work

procedures;

2.4. The use of routeing in accordance with the General

provisions on Ship‘s Routeing.

___________________________________________________

3. Use of Radar and ARPA to maintain safety of navigation:

3.1. Performance including:

3.1.1. Factors effecting performance and accuracy;

3.1.2. Setting up and maintaining displays;

3.1.3. Detection of misrepresentation of information,

false echoes, sea return, etc., racons and

SARTs;

3.2. Use including:

3.2.1. Range and bearing; course and speed of other

ships; time and distance of closest approach of

crossing, meeting overtaking ships;

3.2.2. Identification of critical echoes; detecting

course and speed of other ships; effect of

changes in own ship‘s course or speed or both;

3.2.3. Application of the International Regulations for

Preventing Collisions at Sea;

3.2.4. Plotting techniques and relative and true motion

concepts;

3.2.5. Parallel indexing;

3.3. Principal types of ARPA , their display characteris-

tics, performance standards and the dangers of over

reliance on ARPA;

3.4. Ability to operate and to interpret and analyze in-

formation obtained from ARPA, including:

3.4.1. System performance and accuracy, tracking ca-

pabilities and limitations, and proceeding de-

lays;

3.4.2. Use of operational warnings and system test;

3.4.3. Methods of target acquisition and their limita-

tions;

3.4.4. True and relative vectors, graphic representation

of target information and danger areas;

3.4.5. Deriving and analyzing information, critical

echoes, exclusion areas and trial manoeuvres.

___________________________________________________

4. Respond to emergences

4.1. Emergency procedures:

4.1.1. Precautions for the protection and safety of pas-

sengers in emergency situations;

4.1.2. Initial action to be taken following a collision or

a grounding; initial damage assessment and

control;

4.1.3. Appreciation the procedures to be followed for

rescuing persons from sea, assisting a ship in

distress, responding to emergencies which arise

in port.

___________________________________________________

5. Respond to a distress signal at sea;

5.1. Knowledge of the contents of the IMO Merchant

Ship Search and Rescue Manual (MERSAR).

___________________________________________________

6. Use the IMO Standard Marine Communication Phrases

and use English in written and oral form;

6.1. Adequate knowledge of the English language to ena-

ble officer:

6.1.1. To use charts and other nautical publications;

6.1.2. To understand meteorological information and

messages concerning ship‘s safety and opera-

tion;

6.1.3. To communicate with other ships and coast

stations;

6.1.4. To perform the officer’s duties with a multilin-

gual crew;

6.1.5. To use and understand the IMO Standard Ma-

rine Communication Phrases.

___________________________________________________

7. Transmit and receive information by visual signalling;

7.1. Ability to transmit and receive signals by Morse

light;

7.2. Ability to use the international Code of Signals.

___________________________________________________

8. Manoeuvre the ship;

8.1. Knowledge of ship manoeuvring and handling:

8.1.1. The effects of dead-weight, draught, trim, speed

and under keel clearance on turning circles and

stopping distances;

8.1.2. The effects of wind and current on ship han-

dling;

8.1.3. Manoeuvres and procedures for the rescue of

person overboard;

8.1.4. Squat, shallow water and similar effects;

8.1.5. Proper procedures for anchoring and mooring

__________________________________________________

272

In most cases the basic ordinary training and as-

sessment systems are applied. The process of basic

training and assessment system is presented in fig. 1.

Figure 1. Basic training and assessment system in LMA.

The more detailed description of the certain exer-

cise using NTPRO 4000 will be more complicated

and will look like in fig. 2.

Figure 2. Detailed exercise.

Several exercises according to the number of

competences intended to assess can be included to

overall assessment (e.g. final examination). Evalua-

tion of each exercise has its own particular weight

and influences as a part of the overall assessment

and evaluation the final decision about competency

of the trainee.

The thorough explanation of typical example of

above-mentioned detailed exercise is presented in

fig. 3:

Figure 3. Example of detailed exercise.

It is important to mention that only above men-

tioned competencies (table 1) can be assessed and

evaluated using NTPRO 4000 TEAS in present time,

the other competencies defined in STCW code are

evaluated in particular training during common ex-

aminations.

3 EVALUATION AND ASSESSMENT

PROCESS USING NTPRO 4000 TEAS

Taking into consideration all the mentioned above,

final decision for issuing the Certificate of Compe-

tency is made using the complex assessment.

Complex assessment involves competences men-

tioned in 2.1 (table 1) and additional competencies

which were evaluated in particular training. The ad-

ditional competences of the students, such as use of

ARPA, ECDIS, usually are evaluated before the fi-

nal examination using NTPRO 4000 by Lithuanian

Maritime Safety Administration in accordance with

IMO Model Course 1.07 for ARPA, and IMO Model

Course 1.27 for ECDIS; and the results are the basis

for the diploma of the competency. Although the as-

sessment of mentioned competences is a part of

complex assessment as the student must show his

ability to implement all of competences required by

STCW code, and especially use not only of automat-

ed navigation systems, but also the solutions of tra-

ditional navigational tasks in real time simulation.

The overall complex assessment and weight of

assessed tasks can be presented in fig. 4.

Assessment and evaluation

Assessment and evaluation of target task Overall assessment and evaluation

Practical implementation

Practical study of the task

Practical implementaion of theoretical

knowledge

Theoetical basics

Studing of the suject Analysis of the practical examples

Evaluation and assessment of the task

Correct operaor response

Performance measure

Event

Overall assessment and evaluation

Evaluation and assessment of the actions taken and response in

the particular task(s)

Correct operator response

Position observed in accordance with instructions

Acceptable ranges for position observation: BRG +/- 0.5°;

RNG +/- 0.05nm; Speed +/- 1 knot; Course +/- 4°.

Performance measure

Position observed Ploted and recorded vessel data

Event

Operator is instructed to observe vessel position using

bearing and range

Operator is instructed to plot and record vessel data when

appropriate

273

Figure 4. Overall complex assessment.

During all complex assessment process correct

and incorrect actions and responses are electronical-

ly logged additionally and established paper log

book is filled, as not all competences can be logged

electronically.

It is important that employing TEAS the percent-

age system of evaluation should be applied; it means

that student comes to assessment with the score of

all competencies 100%, and later he gets penalty

points for any failed navigational task. Penalty

points are multiplied by weight of the task evaluated,

so final evaluation result E can be expressed by for-

mula:

∑

−=+++−=

nnnn

wPwPwPwPE 100)...(100

2211

Where: P

– penalty point for appropriate task; w

–

weight of the penalty point.

If there is necessity in more accurate results of

competence evaluation each task can be evaluated

separately with its penalty points (applying the same

formula), influencing final evaluation.

Detailed assessment chart is presented in fig. 5,

extraction of the competence evaluation in overall

complex assessment is presented in fig. 6.

Figure 5. Detailed chart of complex assessment.

Figure 6. Extraction from overall complex assessment for

“Plan and conduct a passage and determine position” compe-

tence.

During evaluation and assessment process all

mentioned competencies in 2.1 (table1) are thor-

oughly checked, especially competencies used to

solve traditional navigational tasks. As TEAS sys-

tem electronic logbook, paper logbook is employed,

the subjective assessors’ opinion is minimized, and

it allows making the very correct and objective deci-

sion about the competency and suitability for appro-

priate position of the student.

In other words the assessment using NTPRO

4000 is based on the on the check of the exercise ful-

filment correctness with regard to the selected set of

criteria. The criteria check consists in comparing the

exercise “assessment parameters” to the set limit

values according to the set “rule”. At each moment

of time a relative error is recorded (relative deviation

of the “assessment parameter” from the limit values)

and penalty points are calculated as a function of the

relative error and the “error weight”.

The overall sum of penalty points in the observa-

tion interval is calculated as a sum of penalties at

each moment of time by each “assessment parame-

ter”.

The trainee competency is assessed in points

(Score %) starting from 100% minus penalty score.

The correctness of the exercise fulfilment can be as-

sessed after the end of the exercise during the play-

back of the exercise log. The Training Report is cre-

ated automatically at the moment when the log file is

loaded. Competency assessment can be obtained in

the process of the exercise and after its end as well.

For the final assessment of the trainee competency,

the “Passing score” is entered.

This process allows making objective judgement

about student’s competency and enable saving pub-

lished the results for longer time.

274

4 PRACTICAL EXAMPLE

Extraction from practical evaluation and assessment

of Use of Radar and ARPA to maintain safety of

navigation is presented below. Ability to operate and

to interpret and analyse information obtained from

the radar, including factors affecting performance

and accuracy of the trainee is tested.

The exercise is set in adverse visual and radar ob-

servation conditions (fog, rough se, rain). The train-

ee is assigned with a task to proceed in restricted

waters with narrow passage making a 90° turn round

the buoy (which becomes a reference point) at a set

distance with a margin equal to the possible radar

range measurement error. Penalty Charge value is

entered.

The trainee must adjust the optimum radar picture

quality, identify the buoy echo among the clutter and

perform the required manoeuvre.

Evaluation is made through the exercise. Perfor-

mance criteria – actual distance to reference point

must be near the limit defined (if the actual distance

exceeds defined limits, penalty charge are imposed,

here in this example-30% if grounding occurs-

70%).

Calculations of the assessment for this particular

task will look as follows:

0)7,0100130(%100)(%100 =⋅+⋅−=+−=

ggllradar

wPwPE

70)7,00130(%100)(%100 =⋅+⋅−=+−=

ggllradar

wPwPE

100)7,0010(%100)(%100 =⋅+⋅−=+−=

ggllradar

wPwPE

Where:

-penalty for exceeded limit,

-penalty

if the grounding occurs,

-weight of the penalty in

particular exercise.

As seen above, depending on the trainee’s ability

to accomplish exercise, he gets final evaluation for

the particular task, which is a part of the complex as-

sessment during final examination.

5 CONCLUSIONS

− During initial stage of study attention to the tradi-

tional navigational tasks slacks as student con-

templates that use of traditional navigational tasks

is not relevant.

− Use of navigation simulators enables to test all

competencies in accordance with STCW code es-

pecially solving traditional navigational tasks, fi-

nal examination using NTPRO 4000 TEAS based

on methods described in the paper may be basis

for certification of competency.

− Using TEAS the percentage system of evaluation

is applied, it is predicted that student gets the

score of 100% at the beginning of the assessment,

and later points for any failed navigational task

multiplied by weight are subtracted from initial

100 % in order to get final evaluation.

− Lithuanian Maritime Academy uses new capabil-

ity of NTPRO 4000 not only during training pro-

cess by performing formative assessment but also

for the summative assessment and evaluation of

students’ competency during final examination.

− Assessors are representatives and persons ap-

proved by Lithuanian Maritime Safety Admin-

istration; this enables getting the Certificate of

Competence after final complex assessment in

Lithuanian Maritime Academy.

REFERENCES

Biggs, J., Tang, C. 2007. Teaching for Quality Learning at

University. Berkshire: Society for Research into Higher

Education & Open University press.

Brown, S., Glasner, A. 1999. Assessment Matters in Higher

Education: Choosing and Using Diverse Approaches.

Buckingham: Open University Press.

Cosman, P.H., Cregan, P.C., Martin, C.J., Cartmill, J.A. 2002.

Virtual Reality Simulators: Current Status in Acquisition

and Assessment of Surgical Skills. ANZ Journal of Surgery

72: 30-34.

Decker, S., Sportsman, S., Puetz, L., Billings, L. 2008. The

Evolution of Simulation and its Contribution to Competen-

cy. The Journal of Continuing Education in Nursing, 39(2):

74-80.

Ham, D., Park, J., Jung, W. 2008. Evaluation of Human Inter-

action with Complex Systems Using a Full-Scope Simula-

tor: Lessons Learned and Methodological Issues. Interna-

tional Journal of Human-Computer Interaction, 24 (4):

361-384.

Gould, K.S., Roed, B.K, Saus, E.R., Koefoed, E.F., Bridger,

R.S., Moen, B.E. 2009. Effects of navigation method on

workload and performance in simulated high-speed ship

navigation. Applied ergonomics 40 (2009): 103-114.

International Maritime Organisation. 2001. International Con-

vention on Standards of Training, Certification and Watch-

keeping for Seafarers, 1978, as amended in 1995, (STCW

95). London: Author.

Maithel, S., Sierra, R., Korndorffer, J., Meumann, P., Dawson,

S., Cellery, M., Jones, D., Scott, D. 2006. Construct and

face validity of MIST-VR, Endotower, and CELTS. Are we

ready for skills assessment using simulators? Surgical En-

doscopy (2006) 20: 104-112.

Matveevskii, A.S., Gravenstein, N. 2008. Role of Simulators,

educational programs, and nontechnical skills in anesthesia

resident selection, education and competency assessment.

Journal of Critical Care (2008) 23: 167-172.

Nieri, D. S. 1995. Certification and licensing of mariners based

on performance assessment. Proceedings of the second in-

ternational conference on engine room simulators ICERS 2,

05-09 June 1995: 1-9. Rimouski: Canada.

Transas. 2005. Navi-Trainer 4000 (v.4.5.1) Instructor manual.

Transas Ltd.