Week 4 report

5.1.3. Module 1: background information on the establishment, selection and characterization of an occupational group

Q100c

The number of employees in the establishment is an important piece of information, among others for the steering of the sample which should be done according to the size information from the questionnaire and not the size information from the address sources. In spite of the clear reference of this question to the local establishment, there are hints from the pilot results that the question was sometimes answered for the entire enterprise with all its local units. Therefore, an even clearer hint on the local unit should be inserted. This can be made by programming and reading out an additional clarifying text in all cases where Q100=2, where the selected unit is one of a number of establishments belonging to a larger enterprise or organization.

Q102

This question aims at verifying the sector of activity. For sector clusters (composed of several single sectors), it is recommended to summarize the names of the different sectors the cluster is composed of for this question instead of asking about all the sectors separately. The most important aspect of this question is whether the attribution to the sector cluster is correct since the occupational groups to be shown in the survey depend on the sector cluster, not the single sectors a cluster is composed of.

Q104

This question on the market for the goods and services of the establishment was designed as a single-punch question in the pilot, with the focus on the main market. If all relevant markets are of interest (and not just the main market), then the question may be designed as a multi-punch question (taking out the reference to the ‘main’ market in the formulation of the question).

Q109

This question requesting the largest occupational group was asked in the pilot if none of the pre-selected groups mentioned in Q107 existed at the establishment. The group named in this open-ended question was then taken as reference group for the further questions. For any replication of the survey, it is recommended to terminate the interview if none of the pre-selected groups exist. Following this recommendation, Q109 can be deleted.

Page 37

User guide to developing an employer survey on skill needs

Q111/Q112

Naming the rough percentage of the workforce that is under 30 years or 50 years or older provided some difficulties particularly for respondents in smaller establishments. They sometimes had problems in expressing the answer in terms of percentages and could usually better provide figures on the absolute number of persons in these age groups. Modifying the question by asking for the number of persons (instead of their percentage share) would help respondents’ answers from small establishments. However, respondents from larger establishments can often better express the answer in terms of percentage categories so that this change should be implemented only if the majority of interviews is to be carried out in small establishments.

Q113

Current hard-to-fill vacancies as asked about in this question were reported by only a small share of establishments in the pilot survey. Moreover, the existence of vacancies seemed to be very much influenced by the economic crises as countries hit particularly severely by the crisis had only very few hard-to-fill vacancies. A broadening of the reference period by for example asking about hard-to-fill vacancies in the past two years instead of ‘currently’ would help to reduce the impact of economic cycle effects in this question. The main aim of the vacancy question in this survey is not a precise mapping of skill needs at the moment of the interviews but the analysis of possible correlations between vacancies, change and innovation in the establishment and changes in the task portfolio. Against this background, adding one or two (open-ended or closed) questions about the type of skills that was lacking in applicants for the vacant positions might be worthwhile, to get some more qualitative insight into the reasons for the vacancies.

Q114-Q116

This set of questions aiming at identifying the level of formal education required from applicants of the selected occupational group caused problems in the national adaptations of the questionnaire and partly also during fieldwork. When replicating the survey, this set of questions should be rethought.

Q116

In the current form, particularly Q116 caused frustration in some countries. This is reflected for example by an average rate of 3% for ‘don’t know/no answer’ responses (unweighted figures). In Poland, as much as 10% of the respondents answered the question in this way (unweighted). These are clear signs for a need to reformulate this question.

Page 38

User guide to developing an employer survey on skill needs

Where there is questionnaire space available, for example in a more focused occupation or sector-specific study an additional question on career management might be considered, given the growth of mentoring activities, work-based learning and career development.

5.1.4. Module 2: generic task questions

The module with the generic task questions as tested in the pilot consists of three types of questions:

(a) Q200, Q202, Q204, ...Q240 ask for the importance of a set of 17 generic

tasks, that is tasks applicable to a broad set of different jobs;

(b) Q201, Q203, Q205, ...Q241 ask for the change in importance of these 17

generic tasks;

(c) Q242_1 to Q242_21 ask for each of the generic tasks reported to be on the

increase whether the employees of the selected group are well prepared for these tasks (that are increasing in importance).

All three types of questions appeared to work well in the pilot. Feedback

from the fieldwork partners did not reveal any general problems of understanding for any of these questions. The rates of ‘don’t know/no answer’ were on average also low for these questions, in particular for the set of importance questions. The rate of ‘don’t know/no answer’ for the change in importance and the preparedness questions was slightly higher, but in none of the questions did it surpass an average of 3.5% (unweighted). Also, each of the generic tasks mapped in the pilot turned out to have at least some relevance for a clear majority of respondents. Though a few of the generic tasks did not apply for a sizable minority of up to about a third of the respondents (26% for manual dexterity, 28% for making speeches and presentations, 34% for reducing the use of raw materials; unweighted figures), the generic task dimensions mapped in the pilot master questionnaire can be considered as relevant enough for being kept in the questionnaire.

This indicates little need for revisions in this module as regards the selection and wording of the generic tasks. An issue of general consideration for any replication of the survey as a cross-national survey is however the use of the importance scale: judging the data based on a general knowledge of skills issues and the vocational training situation in the countries raised some doubts regarding the full international comparability of the questions using an importance scale: ‘How important is task x (very important, fairly important/not important/not applicable). Is the importance of this task staying about the same, increasing or decreasing?’ The pilot data showed some unexpected country results on this set

Page 39

User guide to developing an employer survey on skill needs

of questions which might derive from culturally different interpretations of what is considered important.

Using a less subjective measurement scale might reduce potential country biases. In the preparatory phase, the use of a frequency scale (how often is task xy performed by an occupational group in an average week/month?) instead of the importance scale had been discussed but was finally rejected. The reason for this rejection was that the frequency does not necessarily coincide with the relevance or importance of a task for a specific job. For example, a doctor may do dozens of blood tests per week, but maybe just one operation. Drawing conclusions from the frequency on the importance of a task and thus the respective skills required for it on part of the employee might be misleading as this example shows – the operation skills will not be considerably less important for this doctor than his skills in terms of blood tests.

The pilot results do however reconfirm that for comparability between countries the importance scale seems to be problematic and that a frequency or competence-level scale might be a better choice, at least for some tasks. The number of times something is used is a precise, countable category hardly influenced by any cultural response biases. An importance scale leaves more leeway for interpretation and cultural bias – what in country A might be considered as important might be rated as only fairly important in country B, just because of cultural habits in answering surveys (such as the avoidance of going for extreme poles of a scale such as ‘very’ or ‘not at all’ in some countries) or because of nuanced differences in the translation or translatability of the importance scale into some languages. At the very least some items with frequency scales could be used as anchors for the remaining items that involve an importance scale as could the addition of anchoring vignettes.

If deciding to replicate the survey with generic tasks only (without the occupation-specific questions), there is some room for a few additional generic task dimensions. Since this is a very repetitive part of the survey that is not exactly a pleasure to answer for most respondents, it is strongly recommended to limit the number of additional generic tasks to ask about to a maximum of up to five further tasks.

In principle, any further type of tasks can be added to the generic tasks as long as the task is easily understandable for respondents of different types of workplaces and can be supposed to be relevant for a large part of the targeted universe. It is possible to add new aspects of work that are meant to be of particular relevance for the future, for example similar to the environmental tasks that had been tested in the pilot. For any new tasks to be added, some prior pre- testing is recommended, for example in form of a few cognitive interviews. Care should be taken to minimize the overlap of any new tasks with some of the

Page 40

User guide to developing an employer survey on skill needs

existing task dimensions. Only dimensions really adding new, relevant information should be added.

5.1.5. Module 3: occupation-specific tasks and newly-emerging tasks 5.1.5.1. Module 3a: questions on occupation-specific tasks and newly-emerging

tasks

The module with the occupation-specific task questions exists and has been tested for seven occupational groups only. For any replication of the survey intending to use the occupation-specific task modules, it will therefore be necessary to draft different versions of this module for all occupational groups for which this module has not already been developed and tested in the pilot survey project.

The occupation-specific questions developed for the pilot were closely oriented to ISCO-08. For each occupational group, between four and eight occupation-specific task dimensions were elaborated and asked about in the survey. Some of these questions turned out to be problematic. Some of the task dimensions were so general as to cause frustration with respondents (such as the task ‘constructing, maintaining and repairing buildings and other structures by using traditional or modern building techniques’ asked for the ‘building frame and related trade workers’). In other cases, the ISCO level at which the occupational groups were differentiated caused problems since some of the tasks differ considerably within the ISCO 3-digit groups: the tasks ‘providing care and support to women and newborns following childbirth’ or ‘assessing progress during pregnancy and childbirth’ asked for ‘nursing and midwifery associate professionals’ is for example relevant for a part of this group only, namely for the midwifery associate professionals which at the ISCO 4-digit level form an ISCO- 08 unit group of their own.

For any replication and extension of the occupation-specific questions, it is recommended to use again the ISCO-08 job descriptions as an initial orientation, but to take the time and discuss these intensely with (international) experts familiar with the respective occupational groups. In these expert rounds, the portfolio of occupation-specific tasks to be asked for the group should be discussed. As a result of these discussions, the ISCO job descriptions may be further refined. In the development of questions for the occupation-specific part, care should be taken to keep obvious overlaps with the generic task questions to a minimum.

Page 41

User guide to developing an employer survey on skill needs

5.1.5.2. Module 3b: newly-emerging tasks

The questions on newly-emerging tasks proved to be rewarding in the pilot survey. It is recommended to maintain this part without major changes.

The newly-emerging task questions bring in the element of the future developments which are a core issue for an employer survey on skill needs. Though answers will vary considerably in terms of depth and quality, the open answers to this question will reveal developments relevant for the near future. The open answers can also be used to inform the set of task questions for a next wave of a survey.

5.1.6. Module 4: drivers of changing the tasks: innovations in the establishment

The questions of this last module serve for an analysis of the impact of different types of changes or innovations on the shape of tasks. Questions Q400 to Q403 proved to work well, though there were some complaints reported in the pilot about the (high) degree of abstractness in the formulation of some items. Since the questions have been thoroughly tested in the pilot and in other surveys, we recommend keeping this set of questions as it is. In the case of a sector-specific survey some benefit may arise from providing examples of what constitutes an innovation in each sector (cluster).

Questions Q404 and Q405 asking whether the selected group is the one most affected by changes within the organization can be deleted when replicating the survey. This information mainly had the function of an indicator of the relevance of the survey.

It is possible, instead, to introduce some further questions referring to the whole organization here. If a government is for example interested in some local phenomenon or policy, for example how some policy tool is associated with skill demand, an appropriate question could be added.

In general terms, any issues of particular national interest can in principle be added to the questionnaire, be it in Module 4 or in other parts. In the case of adding specific national parts, it will however be important that these amendments do not have repercussions on the parts designed for cross-national comparison. From this perspective, the best position to place national-specific elements is at the end in Module 4.

Any institution intending to replicate the survey should consider that task- based research is a quite new field of research and not already a fixed, well established science. There is currently a lot of ongoing research in this field, for example at the German Federal Institute for Vocational Education and Training (BiBB), from which new insights can be expected, for example for the interpretation of the task-related questions. Particularly if replicating the survey

Page 42