739

1 INTRODUCTION

The creation of a methodology for an evaluation is

process which takes several steps from defining the

problem, developing the solution and finally testing

the operability of the solution [10]. Fully operational

evaluation methodology for the quality of the

Computerized PMS (Planned Maintenance Systems)

for ships has been developed in 2017 [12]. After

development, the methodology has been tested and

released for free use. Since then, quite a large number

of databases has been evaluated using the

methodology, either for scientific or for economic

purposes.

Several unwanted characteristics or flaws were

identified during the use of the methodology. The

process of the evaluation requires an expert with

advanced knowledge of the marine engineering

(senior engineering rank), knowledge of the Company

structure and policies (usually from the company or

very well versed about the Company) and advanced

knowledge of the used computerized PMS system.

Another drawback of the evaluation methodology is

that evaluation takes several hours per database.

Therefore, the evaluation of the quality of the PMS

database become costly operation and shipping

companies quite often accept PMS database without

proper and efficient quality control. Drawbacks in the

methodology and the need for constant improvement

[8] were a driving force for the development of the

new methodology which will be without listed flaws.

1.1 Non-disclosure condition

Shipping companies allowed access to their data

strictly under no disclosure condition. All

identification details (either the ship or the company)

are removed from the article.

Assessment of the Concept of the New Methodology

for the Evaluation of Ship Planned Maintenance System

T. Stanivuk, L. Stazić, F. Vidović & K. Bratić

University of Split, Split, Croatia

ABSTRACT: The paper presents an attempt to develop better and cheaper methodology for evaluation of the

quality of the ships’ computerized Planned Maintenance System databases. A concept for the better

methodology has been developed following many times verified rule that faster and cheaper is better. The

concept of the methodology has been checked in operation, tested on four different databases. Results of the

evaluation performed by the concept are compared with an expert evaluation using old, verified methodology,

performed on same databases. Comparison valuated the concept, bringing the verdict if it is functioning or not,

showing that the old and verified rule is not functioning every time.

http://www.transnav.eu

the International Journal

on Marine Navigation

and Safety of Sea Transportation

Volume 15

Number 4

December 2021

DOI: 10.12716/1001.15.04.03

740

2 THE METHODOLOGY CONCEPT

Development of the solution was the next step after

defining the problem [10]. A concept of the new

evaluation methodology has been created, following

the idea that the simpler is the better [1], with

intention to be simple enough that an ordinary skilled

person can evaluate any PMS database without

preparation or special knowledge. Another concept

design condition was based on the principle that

faster is better [2], the evaluation time of the database

quality is shortened to less than half an hour. Another

condition introduced for the concept was that it

should be operable on any type of computerized PMS,

obtaining same results.

Several researches pointed that most common

deficiency in the computerized databases was missing

information [3, 6]. In line with that idea, the concept

for the new methodology is trying to establish link

between the quality and the quantity of the data in

ships’ computerized PMS databases. To minimize the

time needed for the evaluation, only a randomly

chosen sample of the database should be analysed, not

the whole database [4, 5]. This will ensure that the

time for the evaluation is short and cheap. In this case,

several pumps, randomly chosen should act as a

sample for the testing.

The basic idea of the new concept is very simple, it

is basic. The idea stems from already stated fact that

the most common deficiency in computerized

databases is a missing of information [3, 6]. Therefore,

the concept is derived from the idea that the larger

amount of information in a database means that the

database has fewer flaws. If this is true, the evaluation

of the quality of the ship’s computerized Planned

Maintenance database could be reduced to a mere

counting of the data in the database. This procedure

does not require knowledge of systems and

mechanical engineering and can be performed by

anyone. It would create significant savings in the

process of the control and the development of the

database and reduce the total cost of the system.

Testing the operability of the solution (concept)

was performed in two stages, each stage had separate

research objectives. The objective of the first stage [11]

was to determine the functionality of the concept.

Second stage of the testing had to determine if results

obtained by concept are corresponding to the quality

of the DB established by an expert.

3 THE FIRST STAGE TEST

In order to determine the functionality of the concept,

a randomly chosen evaluator, with average skills,

performed evaluation of a database. Database

evaluation was performed on the Company premises,

by a student of Marine Engineering.

Overall evaluation of the database [11], based on

the concept, was that DB quality is good, with note

that improvement should be performed with spare

parts and equipment details and maintenance plan.

Results of the concept testing were compared with

the expert evaluation, performed in 2017 [13]:

“Database on average is in order, database does

not require immediate action. There are some areas

where action is required:

− some components are without spare parts,

− maintenance plan on some components should be

improved,

− some components do not have defined equipment

details”.

Obtained results were very promising although

conclusion was not formed because “small sample

should not lead to great conclusions” [13].

4 THE SECOND STAGE TEST

Further testing has been performed, hoping that

results will look similar to the results of the initial

database, which will verify functioning of the concept.

The evaluation of three ship databases was divided

into two separate evaluations [9].

The first evaluation was performed using the

concept by an ordinary skilled evaluator. Second

quality evaluation of computerized PMS databases

was performed on same databases by an expert using

old methodology. The evaluation was performed on

the premises of the shipping company. Analysed

vessels are not sisterships, data collected using the

concept is presented in Tables 1, 2, 3.

Table 1. Results of information counting of the database A

_______________________________________________

Equipment name Equip. Spares Spare Purch. Work

details number details data data

_______________________________________________

SW cooling pump 5 47 6 2 3

FO Transfer pump 4 40 5 2 2

Firefighting pump 6 49 6 2 5

Bilge/ball. pump 6 43 6 2 3

Cargo pump 6 117 6 2 2

Em’cy FF pump 6 49 6 2 5

Bilge piston pump 5 40 5 2 2

_______________________________________________

AVERAGE 5.4 55 5.7 2 3.1

_______________________________________________

Table 2. Results of information counting of the database B

_______________________________________________

Equipment name Equip. Spares Spare Purch. Work

details number details data data

_______________________________________________

SW cooling pump 6 42 5 2 4

FO Transfer pump 6 37 6 2 2

Firefighting pump 6 44 6 2 3

Bilge/ball. Pump 4 46 5 2 4

Cargo pump 5 92 5 2 3

Em’cy FF pump 4 39 4 2 3

Bilge piston pump 5 46 4 2 3

_______________________________________________

AVERAGE 5.1 49.4 5 2 3.1

_______________________________________________

Table 3. Results of information counting of the database C

_______________________________________________

Equipment name Equip. Spares Spare Purch. Work

details number details data data

_______________________________________________

SW cooling pump 5 51 4 2 5

FO Transfer pump 6 44 5 2 3

Firefighting pump 5 36 6 2 4

Bilge/ball. pump 5 38 6 2 3

Cargo pump 6 108 6 2 5

Em’cy FF pump 5 42 5 2 4

Bilge piston pump 6 45 4 2 4

_______________________________________________

AVERAGE 5.4 52 5.1 2 4

_______________________________________________

741

The evaluator (with ordinary skills) analysed

obtained results and formed overall evaluation about

PMS databases. The overall evaluation of databases is

summarized:

− DB 1 (database 1) has all chosen components; all of

components have linked equipment details. Spare

parts are linked to all components, maintenance

plan is well designed and tuned. Spare parts

details are present as well as extensive purchase

data. There are no areas for improvement found

during this evaluation.

− DB 2 has all chosen components; all of components

have linked equipment details. Spare parts are

linked to all components, maintenance plan is well

designed and tuned. Spare parts details are present

as well as extensive purchase data. There are no

areas for improvement found during this

evaluation.

− DB 3 has all chosen components; all of components

have linked equipment details. Spare parts are

linked to all components, maintenance plan is well

designed and tuned. Spare parts details are present

as well as extensive purchase data. There are no

areas for improvement found during this

evaluation.

An expert evaluated same databases using the old

methodology and the questionnaire [12]. Results of

those evaluations are almost the same for all three

databases and are presented in the Table 4 (Results are

applicable for all three databases). Only major

deficiencies are collected in the table, i.e. areas marked

with low grades. There are also some other areas

where noted deficiencies are minor and no

improvement was recommended by the expert.

Table 4. Old methodology analysis results for all databases

_______________________________________________

Question Priority Grade

_______________________________________________

16. Is the alarm system and its testing 1 1

program entered in the DB?

17. Is PMS self-improvement program 1 1

inserted into the DB and is there control

mechanism for PMS DB

self-improvement program?

20. Are jobs created and grouped according 2 2

to multiplier principle?

21. Are all the same type jobs, coming from 2 1

different sources, synchronized?

22. Are all the same jobs, resulting from 2 1

different requirements (sources), merged?

_______________________________________________

Overall opinion of the expert, based on the

extensive inspection of all three databases:

− Databases have an average grade of 4.3, which is

relatively good overall evaluation grade. There is

frequent usage of the system by several on-board

users and several office users, which is by itself

good sign.

− Databases appear to have all components, and look

to be in good order. All components are marked

properly and uniquely, according to their

shipboard location and markings. The data about

the manufacturer, the type and the serial number is

entered to all relevant items as required.

Maintenance plan is well designed and tuned, all

jobs required by company policy are included in

the DB as well as all jobs required by flag state

rules and regulations and by the Class society. Fire

detection sensor list has been inserted into the DB

together with the testing plan. Spare parts are

linked to all components, together with purchase

data and details.

− Two areas require immediate attention; PMS self-

improvement program has to be established as

soon as possible in order to report and supervise

DB and its functioning and alarm testing program

needs to be inserted in the DB to enable

supervision of this segment. Another three items

are with intermediate priority, improvement is

also needed but that does not have to be performed

as soon as possible. There is a multitude of

examples in databases where same job is scheduled

twice, for example an overhaul is required

(scheduled) due to manufacturer recommendation,

at the same time there is overhaul scheduled due

to Class Survey.

− Databases also have examples where same job is

inserted twice, for work order for example electric

motor overhaul is linked to pump and to its motor

as well. Also, it is noted that work frequency is not

synchronized, i.e., work orders should be grouped

to avoid unnecessary paperwork.

5 DISCUSSION

The evaluator with ordinary skills using new concept

analysed computerized PMS databases and the

duration of that evaluation was 30 minutes per

database or 90. Duration of the evaluation performed

by expert was much longer, almost four hours per

database. Also, the expert's report and discoveries are

much more extensive and detailed as expected [7],

and it contains a whole series of observations that are

not present in the opinion of an ordinary evaluator.

That is expected because of the difference in expertise

and the difference in the time used for the evaluation.

Both evaluations pronounced all three databases to be

good, expert gave an average grade of 4.3 (out of 5).

Both evaluations concluded that analysed databases

have all inspected components and well-established

maintenance plan. Main differences of two

evaluations are noted in discovered shortcomings of

databases and recommendations for the future

actions. The evaluator with ordinary skills using new

concept concluded that all analysed databases are in

order and that there are no areas for the improvement

found during his evaluation. Evaluation performed by

expert discovered several areas which need the

improvement, and above all, two of them are

classified as serious deficiencies which require

immediate action.

Comparison of two evaluations showed major

discrepancy between evaluation results. Although

overall evaluation of databases matches, the results of

evaluations do not. New concept failed to identify any

deficiencies which expert discovered in the database

and therefore failed in main purpose of the evaluation

which should be to discover problems in the database

and to recommend areas for the improvement.

Although there are significant potential savings

connected with the new concept (first, the duration of

the evaluation is eight times longer, than expert wages

are much higher), noted major discrepancies question

the meaning of this evaluation.

742

6 CONCLUSION

Several conclusions about the concept can be

formulated, despite relatively small number of

evaluated databases. Testing of the concept of on the

different PMS verified that the concept is functioning

and that it can be used on different computerized

systems. Two persons, both fulfilling the condition of

average skills, tried the use of the concept and both

succeeded to evaluate database(s) without any

problem. This is confirming that the concept is indeed

simple enough for the use by everybody, as it was

intended. Average time for the evaluation was 20-30

minutes which is huge improvement from the time

needed for the expert evaluation. Short evaluation

time is confirming that another goal of the concept is

achieved.

Comparison of results of the second stage tests

showed a major deficiency of the concept which was

noted in the second stage test in evaluations of all

three databases. Concept evaluation did not produce

same results as the evaluation performed by the

expert, furthermore, concept failed to detect serious

shortcomings inside databases which were detected

by expert and listed line by line. Although the concept

showed great prospect, and evaluation of the first

database was very promising, the discovery of those

shortcomings in the databases changed the outcome.

The concept failed in the primary goal, to perform

proper evaluation and determined the outcome.

Although databases evaluated in the second stage test

had larger number of information linked to them,

there were shortcomings which ordinary skilled

person did not detect. Despite good initial results, the

concept failed to produce satisfactory results and

further use of the concept is not recommended.

REFERENCES

1. Doremus, J., Helfand, G., Liu, C., Donahue, M., Kahan,

A., Shelby, M.: Simpler is better: Predicting consumer

vehicle purchases in the short run. Energy Policy. 129,

1404–1415 (2019).

https://doi.org/10.1016/j.enpol.2019.02.051.

2. Geurts, W.J.W.: Faster is Better and Cheaper. INCOSE

International Symposium. 26, 1, 1002–1015 (2016).

https://doi.org/10.1002/j.2334-5837.2016.00207.x.

3. Hoxmeier, J.A.: Typology of database quality factors.

Software Quality Journal. 7, 3, 179–193 (1998).

https://doi.org/10.1023/A:1008923120973.

4. Jung, H.-J.: The Quantity Data Estimation for Software

Quality Testing. Journal of the Korea Convergence

Society. 8, 10, 37–43 (2017).

https://doi.org/10.15207/JKCS.2017.8.10.037.

5. Kolossa, A., Kopp, B.: Data quality over data quantity in

computational cognitive neuroscience. Neuroimage. 172,

775–785 (2018).

https://doi.org/10.1016/j.neuroimage.2018.01.005.

6. McMahon, P., Zhang, T., Dwight, R.A.: Approaches to

Dealing With Missing Data in Railway Asset

Management. IEEE Access. 8, 48177–48194 (2020).

https://doi.org/10.1109/ACCESS.2020.2978902.

7. Pérez-Medina, J.-L., Acosta-Vargas, P., Sampedro, C.,

Paredes, A., Carrión, M., Santórum, M., Samaniego-

Santillán, L.-P., Maldonado-Garcés, V.-G., Corrales-

Gaitero, C., Ortiz-Carranco, N.-Y.: Using Expert

Evaluation to Assess the Usability of an Educational

Mobile Game for Cognitive Skills Training. In: Nunes,

I.L. (ed.) Advances in Human Factors and Systems

Interaction. pp. 265–271 Springer International

Publishing, Cham (2020).

8. Petrevska, I., Ilic, I., Petrevska, M., Markovic, B.: Kaizen

as the Element of Overall Management Quality Control.

Int’l J. Econ. & L. 9, 25, 129 (2019).

9. Rossi, P.H., Lipsey, M.W., Freeman, H.E.: Evaluation: A

Systematic Approach. SAGE Publications, Inc;

10. Roy-Hubara, N., Sturm, A.: Design methods for the new

database era: a systematic literature review. Software

and Systems Modeling. 19, 2, 297–312 (2020).

https://doi.org/10.1007/s10270-019-00739-8.

11. Stanivuk, T., Stazić, L., Vidović, F., Čobanov, A.: Ship

Planned Maintenance System Data Analysis.

International Journal for Traffic and Transport

Engineering. 10, 4, 432–436 (2020).

https://doi.org/10.7708/ijtte.2020.10(4).02.

12. Stazić, L., Komar, I., Račić, N.: Evaluation Methodology

for Ship’s Planned Maintenance System Database. Trans.

Marit. Sci. 6, 2, 109–116 (2017).

https://doi.org/10.7225/toms.v06.n02.002.

13. Tversky, A., Kahneman, D.: Belief in the law of small

numbers. Psychological Bulletin. 76, 2, 105–110 (1971).

https://doi.org/10.1037/h0031322.