Information system for academic staff performance evaluation (IS HAP)

The IS HAP is a tool for the human resources quality management at universities.

In the IS HAP, the performance of each member of academic staff is evaluated in both pedagogical and creative (Research and Development) areas. Input data are acquired by a web form which is filled by the data from various part of the university information system and other sources, and subsequently supplemented by the data provided directly by the academic staff members. Each activity is assigned a score according to its importance and/or time-consumption. Both pedagogical and creative areas are assigned standard scores – different for senior assistant professors, associate professors, and professors. For better clarity and easier interpretation, the partial evaluations in both areas are available in graphical and verbal form with the use of linguistic fuzzy scale. A linguistic fuzzy expert system is then used to aggregate the partial evaluations in the pedagogical and creative areas. This type of aggregation is transparent and comprehensible even to a layman as it is described verbally (If-then rules are used). The overall aggregated evaluation is also available in graphical and verbal form. The evaluation model of IS HAP also takes into account the load of managerial activities with each member of academic staff. This way the overall work load of members of academic staff can be calculated and presented in graphical and verbal form.

Based on the experience and positive results achieved at the Faculty of Science of Palacký University Olomouc, the pilot testing of IS HAP has been incorporated into the Individual National Project “Maintaining and assessing quality in tertiary education”.

IS HAP is currently being used at Palacký University Olomouc and six other Czech and Slovak universities:

• University of Defence in Brno
• University of South Bohemia in České Budějovice
• University of West Bohemia in Plzeň
• Jan Evangelista Purkyně University in Ústí nad Labem
• Tomas Bata University in Zlín
• University of Ss. Cyril and Methodius in Trnava

The intention to create a mathematical model for academic staff performance assessment was formulated at Palacký University Olomouc as early as 2006. Since then various academic staff evaluation models currently used on universities both in the Czech Republic and abroad were examined.

These models were subjected to a detailed analysis regarding their practical and mathematical aspects. The analysis resulted in the design of several academic staff evaluation models, differing in how members of academic staff are evaluated in separate areas of their activity and in the aggregation method for these partial evaluations (weighted average, OWA, WOWA). The final model implemented in IS HAP relies on a fuzzy expert system for the aggregation of partial evaluations.

General requirements on the model were as follows: It should (1) include, if possible, every aspect of academic staff activity; (2) use only easy to verify and objective data; and (3) be easy to work with. Other requirements were for the final evaluation: (4) to maximally reflect staff benefit to the university; and (5) not to be a simple average of partial evaluations in separate areas of activity, but to be able to flexibly respond to management needs. The desired output of the model was not to arrange members of academic staff in order of their performance, nor to obtain a single crisp number interpretable only with difficulty. A basic piece of information on both focus and performance of the academic staff was considered sufficient. Such assignment implied the use of linguistic fuzzy modelling – linguistic variables, rule bases, and approximate reasoning (i.e. of fuzzy expert systems).

In the model, the performance of each member of academic staff is evaluated in both pedagogical, and research and development (R&D) areas of activities. Input data are imported from various components of the university information system and other sources into a web-based form, which is subsequently completed by the staff members by pieces of information which are not available in the university information system (yet are still relevant for the evaluation purposes). Each item in the questionnaire (an activity or the output of a staff activity) is assigned a score according to its importance or time-consumption. Three areas are taken into consideration for pedagogical performance evaluation: (a) lecturing, (b) supervising students, and (c) work associated with the development of study programs. The evaluation of research and development activities is based on a methodology, which puts emphasis mainly on papers published in journals indexed in respected databases (WoS, Scopus), and on patents in the applied research area. Other important activities (grant project management, contract research, editorial board memberships, promotion of the field of science etc.) are also reflected in the process of R&D performance evaluation. Both pedagogical and R&D areas are assigned a standard score – different for assistants, senior assistant professors, associate professors, and professors. The number representing a partial evaluation of a member of academic staff in a certain area is determined as a multiple of the respective standard for his or her position. For better clarity and easier interpretation, these numbers are transformed into verbal evaluation using linguistic scales and are presented in graphical and verbal form.

A linguistic fuzzy expert system is therefore used to aggregate both partial evaluations – for pedagogical and R&D areas of activities. The main advantage of this type of aggregation is that it allows to set-up the shape of the aggregation function completely in line with the evaluator’s requirements (e.g. to appreciate excellence achieved in one of the areas). This type of aggregation is transparent and comprehensible even to laymen as it is described in linguistic terms (using linguistic rules). The overall aggregated evaluation is also available as a linguistic expression.

The model also takes into account the load of secretarial and managerial activities with each member of academic staff (understood here as activities draining away from his or her time and thus possibly reducing the performance in each of the two areas of evaluation mentioned above). Another fuzzy expert system is therefore used to adjust the evaluation according to the managerial activities load of the particular academic staff member. The overall work load of members of academic staff is thus described in words.

After its pilot testing on one department in 2009, the described model was presented to the academic senate of the Faculty of Science, which accepted it well. An extensive pilot testing on 14 departments followed in 2010. The results of the pilot testing confirmed that the model meets all the requirements. It was therefore decided to implement the designed methodology of evaluation into an information system for academic staff performance evaluation (IS HAP). Since 2012 IS HAP has become the annual performance evaluation tool for academic staff members of the Faculty of Science of Palacký University Olomouc. In 2016 it has been decided that IS HAP will be used at the whole Palacký University.

IS HAP is a product of an interdisciplinary cooperation (mathematicians, computer scientists, a psychologist, a HR practitioner, and the management of the Faculty of Science participated during the development of the first version of the tool; the management of other faculties and the university as a whole was subsequently also involved in further developments). This makes it a tool reflecting the needs of correctness and soundness of the used mathematical methods, but also the needs of easy interpretability and comprehensiveness. This way IS HAP is an effective support tool for human resource management.

Based on the experience and positive results achieved at the Faculty of Science of Palacký University Olomouc, the pilot testing of IS HAP has been incorporated into the Individual National Project “Maintaining and assessing quality in tertiary education” in 2012-2013.

IS HAP is currently being used at Palacký University Olomouc and six other Czech universities – University of Defence in Brno, University of South Bohemia in České Budějovice, University of West Bohemia in Plzeň, Jan Evangelista Purkyně University in Ústí nad Labem, Tomas Bata University in Zlín and University of Ss. Cyril and Methodius in Trnava.

Characteristics of IS HAP

The IS HAP is a web based application, which is accessible through Internet. Each of members of the academic staff fills in a form summarizing his or her activities in the last year. The form was implemented so that it would be as easy to fill it in as possible. All items of the form are divided into categories (pedagogical activities, research and development activities, managerial activities) and subcategories. A brief help for any of the items is also available.

A part of the form filled in by each staff member.

One of the requirements on the software implementation was that it must be easy to integrate with external databases and other information systems. IS HAP has a build-in support for several information systems used on the Czech universities. However, IS HAP was always designed in a way so that it would be easy to load data from any other source. It provides a simple and well-documented API (Application programming interface) for this purpose. By consuming the API, the university can load any available data from its information systems into the forms for each of the academic staff members in the IS HAP. Furthermore, IS HAP has very powerful functions for import data from Excel. Overall, IS HAP makes it easy to load most of the information on the academic staff members’ activities and results automatically. The rest of them, which is not possible to obtain automatically, can be filled in by the academic staff members manually into their forms.

Evaluation of a particular staff member.

After the form is filled in, the head of the respective department and the dean can see the overview of the resulting evaluations of his/her subordinate academic staff members. Even though the mathematical apparatus used to calculate the evaluations is sophisticated, the results are presented in way comprehendible even to a layman, i.e. verbally and graphically (each of the performance classes is represented by a different colour). This was enabled by the use of linguistic fuzzy modelling and fuzzy classification. All the filled in forms are also available to the heads of departments (can access the forms of the academic staff members of his/her department) and to the dean (can access the forms of all academic staff members of the faculty). All these features make the resulting evaluations very easy to interpret and compare.

Evaluation overview for more academic staff members.

The head of the department is provided with the following data about members of his or her department: (1) the name of the academic staff member and his or her position, (2) evaluation of the pedagogical activities including the information on the type of the activities for which it was acquired, (3) evaluation of the research and development, which is again supplemented by an additional information on type of activities, (4) the overall evaluation in the areas of pedagogical activities and R&D, (5) the list of managerial and secretarial functions, (6) the final evaluation of pedagogical activities and R&D which also takes into the account the working time drained by the previously mentioned functions. Besides the table summarizing the results of the evaluation, the heads of the departments can view the complete forms of the members of their departments and see all of their activities. In the form, the academic staff members also specify their plans in pedagogical area and R&D for the next evaluation period.

The evaluations calculated by the IS HAP are then used by the heads of the departments in the interviews with their subordinates. The final qualitative evaluations of academic staff members by the head of the respective department can be stored in the IS HAP.

Finally, tools for statistical analysis of the stored data (scoped by the faculty or department) are also provided by the IS HAP.

IS HAP represents a unique software tool. It is easy to use for both academic staff members and their superordinates. The sophisticated mathematical background provides evaluations highly-valued for the HR management, which are presented in a well-arranged and comprehensible way.

References

• STOKLASA, J., TALAŠOVÁ, J. & HOLEČEK, P.: Academic staff performance evaluation – variants of models, Acta Polytechnica Hungarica 8 (3), 2011, p. 91 – 111, ISSN 1785-8860. (pdf)
• TALAŠOVÁ, J., STOKLASA, J., HOLEČEK, P. & TALÁŠEK, T.: Mathematical Support for Human Resource Management at Universities. Proceedings of the 35nd International Conference on Mathematical Methods in Economics 2017, September 2017, University of Hradec Králové, Hradec Králové, p. 783–788, ISBN 978-80-7435-678-0. (pdf)