Annotated Bibliography

Computers & Education 91 (2015) 32e45

Contents lists available at ScienceDirect

Computers & Education

journal homepage: www.elsevier.com/locate/compedu

Using online assessments to stimulate learning strategies and achievement of learning goals

Miran Zlatovi�c, Igor Balaban*, Dragutin Kermek University of Zagreb, Faculty of Organization and Informatics Vara�zdin, Pavlinska 2, 42 000, Varazdin, Croatia

a r t i c l e i n f o

Article history: Received 20 May 2015 Received in revised form 7 September 2015 Accepted 19 September 2015 Available online 25 September 2015

Keywords: Online knowledge assessment Distance education and telelearning Evaluation methodologies Improving classroom teaching Learning strategies and goals

* Corresponding author. E-mail addresses: [email protected] (M. Zlato

http://dx.doi.org/10.1016/j.compedu.2015.09.012 0360-1315/© 2015 Elsevier Ltd. All rights reserved.

a b s t r a c t

The main goals of this research are: (i) to explore the influence that announcement of certain type of online assessment has on students' learning strategies and (ii) to explore the influence of stimulated learning strategies on achievement levels that students exhibit during assessments. Research has been conducted by testing and surveying 351 students from higher education institutions. Results indicate that students' learning strategies can be influenced in a relatively short period of time by announcing various types of online assessments and that steering to more desirable deep learning strategies has positive impact on both formal and perceived levels of success in achieving the desired learning goals. These findings can be used to create a novel adaptive online assessment system that incorporates the elements of adaptivity within a series of assessments and uses post- assessment feedback to steer students’ learning strategies.

© 2015 Elsevier Ltd. All rights reserved.

1. Introduction

The subject of this paper is to study the effects of announcing and solving certain types of online knowledge assessments on stimulation of students’ deep and surface learning and achievement of required learning goals.

Knowledge assessment is an important component in the process of achieving desired learning goals among students. Besides the obvious role of quantifying students’ knowledge, research suggests (Macdonald, 2004) that online knowledge assessments have several additional important roles in the context of online education: (i) to stimulate the learning process in critical places of online courses, usually containing more demanding content (e.g. through post-assessment feedback), and (ii) to help in the gradual development of required skills.

Despite extensive research in the field of online education and online knowledge assessment, the relationships be- tween applications of certain forms of online knowledge assessment and their influence on stimulating learning strategies and achieving various levels of required learning goals (e.g. according to Bloom's Taxonomy) are still insufficiently explored.

Multiple-choice questionnaires are the predominant form for practical applications of online knowledge assessment (Kim, Smith, & Maeng, 2008). Although numerous researches (Nowicki & Jones, 2005; Oliver & Dobele, 2007; Scouller, 1998; Shumway & Harden, 2003) indicate that this form of assessment predominantly tests only lower levels of cognitive skills. The effectiveness of this, easiest to implement, type of online assessment on stimulaton of learning strategies and the

vi�c), [email protected] (I. Balaban), [email protected] (D. Kermek).

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 33

achievement of required learning goals is still insufficiently explored. In addition, there is a question of whether the other methods of online knowledge assessment enable the achievement of higher levels of learning goals and the stimulation of deep learning strategies. As one of the scientific contributions of this paper, this research aims to provide additional insights about their effectiveness.

This paper is structured as follows: in Section 2 we explore research findings and analyze research literature in respect to two sub-domains: (1) Learning strategies and online assessments, and (2) Learning strategies and learning goals. Section 3 shows the hypotheses while Section 4 describes the research sample, procedure and the two questionnaires used. Section 5 reveals research results and consists of two parts that correspond to the hypotheses. Extensive results from previous sections are summarized in Section 6. Section 7 describes the plans for further research having in mind the online system that will be built based on the findings of this research. We conclude in Section 8.

2. Research background

This section describes previous research in related areas (learning strategies, levels of knowledge, learning goals) that have been predominantly conducted in the context of traditional education and e-education in general. Only a few efforts have been directed towards researching their influences within a narrower field of online knowledge assessment.

2.1. Learning strategies and online knowledge assessment

Hartley (1998) defines learning strategies as the different combinations of activities (i.e. 'strategies') students use while learning. He also states that, when compared to learning styles, learning strategies are considerably more conditioned by the task at hand and display greater variability over time. These are the main reasons why this paper consideres learning stra- tegies over learning styles and why this paper is focused on the stimulation of desireable learning strategies in order to achieve required learning goals. Despite playing a very important role in overall achievement of learning goals, staticallity of learning styles makes them non-viable target for manipulation. On the other hand, learning strategies are dynamic in nature and can be manipulated within shorter periods (such as one semester).

Instead of the term “learning strategy”, the replacement term “learning approach” is often used in literature (Entwistle, 1988; Hoeksema, 1995; Marton & S€alj€o, 1976; Sankaran & Bui, 2001). Therefore, in context of this paper “deep learning strategy” can be interpreted as “deep learning approach”, while “surface learning strategy” can be interpreted as “surface learning approach”. According to the above-mentioned authors, these approaches can be described as follows:

� Surface approach e typical in situations where the dominant goal is the reproduction of learning contents (rote memo- rization of facts and isolated sets of data, mechanical substitutions in formulas, etc.). Understanding of learning contents is either very low or non-existent.

� Deep approach e typical in situations where the dominant goal is to understand learning contents (questioning of al- ternatives, raising additional questions, exploration of newly learned contents' application limits, etc.).

Watkins and Hattie (1985) and Gow and Kember (1990) have also identified these two basic strategies. According to these findings Biggs, Kember, and Leung (2001) developed a modified SPQ (Study Process Questionnaire) instrument, named “The revised two-factor Study Process Questionnaire” (R-SPQ-2F), which is based on two factors, i.e. measurement of two learning strategies: deep strategy and surface strategy. This instrument has been used in this paper in order to identify learning strategies that were stimulated by announcing particular type of online knowledge assessement. The proposed instrument is suitable for this purpose because it is (i) in-line with dominant theoretical concepts in the field of learning strategies and (ii) is additionally improved for application within academic environment in its second version, which is actually used in this paper.

Although there are clear indications that the influence of assessment types on appearance of learning strategies has been thoroughly researched within the context of a traditional classroom environment (Anderson, 2003; Black & William, 1998; Entwistle, 2000; Rushton, 2005), the same cannot be said within a context of e-education and online knowledge assess- ment. Only partial research has been done, which opens-up the area for more research to additionally confirm or refute the validity of certain concepts originating from traditional teaching and assessment within new environment, created by e- education (especially in blended form).

With reference to such contexts, Hein and Irvine (1998) have stated that participation in “ … on-line discussions could have been a catalyst to promote deeper learning for participating students … ”. Students that participated in voluntary on-line discussions were able to examine each other's reactions to the discussion topics and work collaboratively on finding solutions to problems, while strengthening their understanding by responding to the others. Shen, Hiltz, and Bieber (2008) have been researching the influence of collaborative (team) online knowledge assessment on the appearance of learning strategies and comparing the results with traditional knowledge assessement. It was their conclusion that collaborative online exams significantly reduce the appearance of surface learning strategy and increase the perceived level of learning. Slack, Beer, Armitt, and Green (2003) presented a case study, researching the influence of synchronous online education on stimula- tion of deep learning. They concluded that using synchronous communications and online meetings as a means of delivering lectures increases the appearance of deep learning. It has to be mentioned that their research was not focused on studying the

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4534

effects of online assessments, but on a broader context of synchronous online education. Siew-Rong (2008) researched the influence of using several modern audio-visual technologies (contents in mp3 and mp4 formats and Podcasting) on a process of learning the English language for scientists (i.e. Scientific English course). His study has shown that usage of these tech- nologies both activated higher-order thinking and cognitive levels and stimulated deep learning. Although it has to be highlighted that, within this study, knowledge assessment is being mentioned only as means to assess required learning goals, rather than as one potential factors which could stimulate the appearance of a particular learning strategy or influence the levels of achievement of particular learning goals. Vos, van der Meijden, and Denessen (2011) reported that educational computer games could be used to facilitate deep learning strategy, especially if students are involved in game's creation, and not just playing it.

It can be seen that the majority of existing research regarding learning strategies is focused on a broader context of e- education. Very few studies have been focused on researching learning strategies within the much narrower context of online knowledge assessment. Therefore, this paper examines how particular types of online knowledge assessment influence (i.e. stimulate) the appearance of deep and surface learning strategies (hypothesis H1).

2.2. Levels of knowledge as learning goals

Given the different approaches to the definition and understanding of learning goals, it is important to mention that this paper considers following definition of Learning Goals (i.e. Learning Objectives): “ … brief, clear statements that describe the desired learning outcomes of instruction; i.e., the specific knowledge, skills, values, and attitudes students should exhibit that reflect the broader goals.” (Soulsby, 2009).

Achievement levels of required learning goals must be assessed. Buzzetto-More and Alade (2006) reflect on the position of learning goals within the process of knowledge assessment and highlight the ongoing consensus that every process of knowledge assessment begins with identification of measurable learning goals, assisted by various learning goals taxonomies. Bloom's Taxonomy in particular has been used successfully within many knowledge assessment systems as a foundation for a balanced selection of appropriate questions and assignments (see e.g. Aller, Phillips, Tsang, Kline, & Aravamuthan, 2003; Aluísio, Aquino, Pizzirani & de Oliveira, Jr., 2003; Conole & Warburton, 2005; Crowe, Dirks, & Wenderoth, 2008; Nicol D., 2007; Valenti, Neri, & Cucchiarelli, 2003).

It is known that not every form of assessment is suitable for the assessment of every learning goal (Aller et al., 2003; Shumway & Harden, 2003; Valenti et al., 2003). Within the context of their “New Taxonomy of Educational Objectives”, Marzano and Kendall (2007) gave one of the most comprehensive overviews of the suitability of using a particular type of knowledge assessment to assess the achievement of particular level of educational goals. Their results indicate that multiple- choice questions (in practice, the most commonly used type of assessment) are suitable for the assessment of a very small range of educational goals, while the largest range of educational goals (all higher level goals and most lower level goals) can be effectively assessed by using essay type of questions.

This paper examines how particular learning strategies, stimulated by announcing and using various types of online knowledge assessment, influence the achievement levels of learning goals (hypothesis H2).

3. Hypotheses

Findings from the previous section reveal that learning strategies have not been explored in the context of online assessment and that the relationship between particular learning strategies and achievement of learning goals has not yet been revealed. With this in mind, this study focuses on examination of the influence of various types of online knowledge assessment on the stimulation of learning strategies and the achievement of required learning goals. Considering the research problem, the following hypotheses have been formulated:

H1. Types of online assessment used in the knowledge assessment process have an influence on appearance of specific learning strategy.

Using online tests with multiple-choice questions should stimulate the appearance of surface learning strategies to a greater extent and, to a lesser extent, the appearance of deep learning. On the other hand, using online assessment in a form of essay should stimulate the appearance of deep learning strategy to a greater extent and, to a lesser extent, the appearance of surface learning.

H2. Learning strategies stimulated by different online knowledge assessment types affect the level of achievement of learning goals.

Learning strategies stimulated by using online tests with multiple-choice questions should to a greater extent lead to achievement of lower levels of knowledge and learning goals (rote memorizing, reproduction, understanding) and, to a lesser extent, the achievement of higher levels of knowledge and learning goals (analysis, synthesis, evaluation). On the other hand, learning strategies stimulated by using online tests in the form of an essay should to a greater extent lead to achievement of higher levels of knowledge and learning goals (analysis, synthesis, evaluation) and, to a lesser extent, the achievement of lower levels of knowledge and learning goals (rote memorizing, reproduction, understanding).

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 35

4. Research design

Research was conducted within two courses at institution X. Students were administered two questionnaires: a pre- liminary questionnaire to collect demographic data, and the main questionnaire, which consisted of two parts (Q1 and Q2), to test H1 and H2. Both questionnaires were not anonymous, so that in the subsequent analysis a relationship between (1) data from questionnaires, and (2) data based on ranking the results of online assessment could be established.

4.1. Sample and courses

Analysis of the preliminary questionnaire revealed the following sample sizes:

� Course 1 - Informatics 1, N ¼ 189 (97 þ 92) � Course 2 - Informatics, N ¼ 162 (69 þ 93)

Basic demographic characteristics are summarized in the following table. Course “Informatics 1” (C1) is held at institution X as a part of the undergraduate curriculum for students working towards

the bachelor's degree in the field of information systems and technology. Course “Informatics” (C2) is held at institution X and organized in cooperation with institution Y. Course C2 is a part of the undergraduate curriculum for students working towards the bachelor's degree in the field of economics.

4.2. Procedure

The use of different types of online assessment was examined in both courses, C1 and C2. In each course, the total population of students was divided alphabetically into two groups by Faculty administration due to the requirements of the Bologna process. Such organization was reused in this research, giving four groups of respondents in total: C1-1, C1-2, C2-1 and C2-2.

1. Only a single type of assessment was announced to each group of students. An essay type of online assessment was announced to groups C1-1 and C2-1 and, according to the literature review in the field of traditional education, essay- oriented learning should stimulate the deep learning strategy (Entwistle, 2000; Scouller, 1998; etc.). Two other groups, C1-2 and C2-2, were announced an on-line test with multiple-choice questions. According to the literature review in the field of traditional education, test-oriented learning should stimulate surface learning strategy (Nowicki & Jones, 2005; Oliver & Dobele, 2007; Scouller, 1998; Shumway & Harden, 2003). All students were informed that the results of this assessment would count towards their final scores in courses C1 and C2 (max. 5 points for both types of assessment).

2. In the implementation phase, the students were first administered the type of online assessment that was announced to them and for which they had prepared. After that they were also administered the other, unannounced type of online assessment for which they had not prepared specifically.

Both types of online assessment (essay and test) were implemented in the LMS Moodle. On-line tests contained multiple- choice questions (with both single and multiple correct answers). On-line essays consisted of several free answers (essay- type) questions. All assessments were conducted in a controlled environment to avoid bias towards higher rankings (Ardid, G�omez-Tejedor, Meseguer-Due~nas, Riera, & Vidaurre, 2015) and were related to the same, pre-announced course content for which the students had 2 weeks to prepare. Also, before completing the online assessment, students had to fill in the pre- liminary questionnaire to collect demographic data (Table 1). Immediately upon completion of online assessment, but before they learned the results, students were administered the main questionnaire (Q1 and Q2, see 4.3).

Table 1 Basic demographic characteristics.

Course 1 (C1): Informatics 1 Course 2 (C2): Informatics

Group C1-1 (N ¼ 97) (Essay-oriented learning) Group C2-1 (N ¼ 69) (Essay-oriented learning) Gender Year of study Age Gender Year of study Age

M F 1. 2. <¼ 19 20e25 M F 1. 2. <¼ 19 20e25 70.1% 29.9% 100% e 79.4% 20.6% 40.6% 59.4% 97.1% 2.9% 72.5% 27.5%

Group C1-2 (N ¼ 92) (Test-oriented learning) Group C2-2 (N ¼ 93) (Test-oriented learning) Gender Year of study Age Gender Year of study Age

M F 1. 2. <¼ 19 20e25 M F 1. 2. <¼ 19 20e25 69.6% 30.4% 100% e 63% 37% 40.9% 59.1% 100% e 63.4% 36.6%

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4536

The experiment was designed to avoid as many sources of bias as possible when it comes to explaining the formal results of the assessments, so that student's knowledge (or lack of it) should be the only relevant factor that explains his/hers final score. One source of such bias may originate from visual and presentational aspects of the assessment items (i.e. questions). To keep the experimental online assessments as close as possible to the purely textual pen-and-paper assessments that were traditionally used in courses C1 and C2 and to which students were already accustomed throughout the semester, assessment items in both types of experimental assessments (multi-choice and essay) were presented as pure textual questions.

Also, to keep the online examination environment as simple and as familiar as possible for students participating in research, experimental assessments were conducted within standard Moodle LMS system. Moodle is the prescribed LMS platform in the institution where the research has been conducted, so students were already accustomed to using Moodle. To keep experimental online assessments as close as possible to the usual pen-and-paper examinations used in courses C1 and C2, none of the advanced assessment features of Moodle (e.g. adaptive testing, multiple attempts per question, immediate feedback per answer, multi-paged and/or conditional test layouts, etc.) were used. All the questions were displayed on a single screen and students were given full freedom to choose the order in which they will answer the questions, just as they were be able to do in traditional exams in courses C1 and C2. Sample questions are available in Appendix C.

4.3. Questionnaires

The main questionnaire consisted of two parts (two questionnaires), marked with Q1 (testing students' perception of stimulated learning strategies) and Q2 (testing students' perception of achieved levels of learning goals).

Q1: Students' perception of stimulated learning strategies

A questionnaire used to assess students' perception of stimulated learning strategies (deep or surface) is entirely based on the revised two-factor Study Process Questionnaire (R-SPQ-2F) developed by Biggs et al. (2001). The questionnaire consists of 20 items, from which 10 items are used to assess the stimulation of deep learning strategy, and other 10 items to assess the stimulation of surface learning strategy. The layout of the original questionnaire was retained (two questions for deep strategy, and two for the surface, and then again two for deep, etc.) to make the respondents more difficult to identify items that relate to the same construct.

Q2: Students' perception of achieved levels of learning goals

This questionnaire assessed students' perceptions of their achieved levels of learning goals based on their preparations for the upcoming announced type of online assessment. It is important to emphasize that the introductory notes in Q2 ques- tionnaire instructed students to give their answers only for the announced type of online assessment and to disregard the second, surprise assessment.

The levels of achievement of learning goals (used as names of scales) have been taken from Bloom's taxonomy (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956). Specific items related to a particular level of achievement were designed based on an extensive literature review of studies that identify common practice in most often-asked questions in the context of assessing certain level of knowledge. The entire questionnaire Q2 can be found in Appendix A.

The data collected underwent a statistical analysis to test H1 and H2. It is also important to note that the achievement of certain levels of learning goals has not been explored only qualitatively, by means of the mentioned questionnaire, but also quantitatively, by analyzing the results of online assessements.

5. Research results

This section reveals the results of the statistical analysis of students' achievement in online assessments, and the results of statistical analysis of the main survey (correlation analysis, factor analysis and significance test for difference between means).

5.1. Stimulated learning strategies (H1)

In this section results that lead to support of hypothesis H1 are shown and discussed. The variables that identify learning strategies from the questionnaire R-SPQ-2F (Biggs et al., 2001), ie. Questionnaire Q1,

are designed in accordance with the instructions given in the instrument. A variable that indicates the level of deep learning strategies (q1_deep) was created by summing the responses to questions 1, 2, 5, 6, 9,10,13,14,17 and 18 (10 items). A variable that indicates the level of surface learning strategies (q1_surf) was created by summing the responses to questions 3, 4, 7, 8,11, 12, 15, 16, 19 and 20 (10 items).

T-tests were used to examine if the type of announced online assessment stimulates deep or sufrace learning strategy. Reliability of individual scales (variables) from the questionnaire Q1 required for this analysis was assessed by Cronbach a.

Reliability of most scales exceed the recommended 0.7 or at least a minimum 0.6, so the input data is sufficiently reliable for further analysis.

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 37

Mean values of the perceived level of individual learning strategies and the results of t-tests were used for the significance test for difference between means (see Table 2).

As seen from the table above, the average indicator of a deep learning strategy on both courses is higher than the average indicator of a surface learning strategy for those groups of students to whom only an essay type of online assessment was announced. In contrast, the average indicator of surface learning strategy at both courses is higher than the average indicator of deep learning strategy for those groups of students to whom only assessment in the form of an online multiple-choice test was announced. Looking only at the differences in average values of indicators of stimulated learning strategies, all the results are in line with theoretical expectations.

The results of t-tests show that the observed difference in the average values of the indicators of surface learning strategy is statistically significant (p < 0.01 and p < 0.05) in both courses, while the difference in the average values of the indicators of deep learning strategy is statistically significant (p < 0.001) only in the course C1. In the course C2, the difference is not statistically significant at p < 0.05, but it is noted that the statistical significance of the t-test (0.065) is very close to the required limit of 0.05. One possible reason for the failure of t-test for an indicator of deep learning strategy in course C2 could be considerably smaller number of respondents who participated in the assessment in a form of previously announced online essays (only 69 students). Based on such rationale, it could be concluded that the observed differences are not coincidental. Another reason for slightly higher than expected self-reported level of deep learning in group which has been preparing for online test only, could be the fact that post-assessment questionnaires were not anonymous. Non-anonymity of the ques- tionnaires may lead to the situation where respondents are inclined to give more neutral and socially more desirable re- sponses (Herbert, Ma, Clemow, Ockene, Saperia, Stanek 3rd et al., 1997).

The results obtained on the basis of such design support the hypothesis H1 indicating that the use of multiple-choice online test will strongly influence the occurrence of surface learning strategy, and the use of essay form of online test will stronger influence the occurrence of deep learning strategy.

Thus, we can conclude that the hypothesis H1 is supported; that the type of online assessment has effect on the occurrence of certain learning strategies.

5.2. Levels of knowledge achieved through individual learning strategies and types of online tests (H2)

In this section the results of statistical analysis are presented and discussed in order to support hypothesis H2. The survey was conducted over a population of students on courses C1 and C2 where analysis described in previous

section showed that the announcement of an essay type of online assessment encouraged deep learning strategies, while the announcement of online multiple choice test encouraged surface learning strategies.

Further steps were taken to verify the impact of such learning strategies on the level of achieving the desired learning goals:

1. It was verified that the data obtained from stimulated learning strategies matches the formal results of the assessment (i.e. with scores achieved on assessments as a quantitative indicator of achieving the required level of knowledge).

2. It was verified that the data obtained from stimulated learning strategies matches the data acquired from students' perception of their own ability to achieve certain level of knowledge required by the assessment (i.e. students' perception of their own success regarding the knowledge levels, but before they found out their results from the assessment).

5.2.1. The influence of stimulated learning strategies on formal results of the assessment The following table summarizes the statistically significant correlation coefficients between learning strategies indicators

and scores on online assessments (absolute scores were converted to relative scores, to facilitate analysis and comparison) (see Table 3).

Table 2 The effect of an announcement of certain types of online assessment on learning strategies e the case in which only one type of assessment is announced, but at the same time the other, unannounced type of assessment is carried out as well.

Strategy Group(preparing for the type of assessment)

Descriptive statistics (strategies) t-test

N Average Std. deviation t Sig. (2-Tailed)

C1, N ¼ 189 (97 þ 92) Deep(q1_deep) Essay (C1-1) 97 33.22 5.553 3.724 0.000***

Test (C1-2) 92 30.15 5.759 Surface(q1_surf) Essay (C1-1) 97 29.05 4.678 �2.353 0.020*

Test (C1-2) 92 30.75 5.241 C2, N¼162 (69 þ 93) Deep(q1_deep) Essay (C2-1) 69 32.65 5.244 1.861 0.065

Test (C2-2) 93 31.17 4.824 Surface(q1_surf) Essay (C2-1) 69 28.80 5.666 �3.064 0.003**

Test (C2-2) 93 31.37 4.967

*p < 0.05; **p < 0.01; ***p < 0.001.

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4538

In the case where essay type of online assessment was announced it is evident that deep learning strategy very weakly affects the score growth in the ‘announced’ essay (r ¼ 0.179; p < 0.05) as well as in the ‘unannounced’ online multiple-choice question test (r ¼ 0.218; p < 0.01). It can also be noted that the surface learning strategy for essay type assessment resulted in a lower level of success in the unannounced multiple-choice question test (r ¼ �0.307; p < 0.001), and did not affect the results of the announced essay online assessment. In addition to the previously perceived slight dominance of deep learning strategies, results were in line with expectations: respondents who had more pronounced usage of deep learning strategy achieved generally somewhat better results in announced online essay, but also in the unannounced online multiple-choice question test. On the other hand, the respondents with a more pronounced surface learning strategy achieved mixed results in the announced online essay, but weaker results in the unannounced multiple-choice question test.

In the case where the online multiple-choice question test was announced it was confirmed that the surface learning strategy negatively affects the reduction of scores in the unannounced essay (r ¼ �0.153; p < 0.05). In addition to the pre- viously perceived slight dominance of surface learning strategies, the presence of any deep learning strategy was not noticed.

By comparing results from online assessments (see Table 4) it is evident that the respondents to whom the online essay was announced (and who showed a more pronounced deep learning strategy) achieved, on average, significantly better results on the essay as compared to the respondents to whom the online essay was unnanounced (t ¼ �4.543, p < 0.001). Such results are in line with expectations about the impact of a deep learning strategy. Comparing the average results in the online test, it can be seen that the respondents who were preparing for the online essay, and not for the online test achieved the better results, but this difference was not statistically significant (t ¼ �0.726; p > 0.4).

It can therefore be concluded that, under the given circumstances, a deep learning strategy has a positive effect on results in both types of online assessment, while a surface learning strategy has a negative impact on results in online essay, and no significant impact on results in the online multiple-choice question test.

Therefore, from the aspect of formal success (level at which learning goals are achieved observed solely through the formal results of assessment), hypothesis H2 was supported - learning strategies stimulated by the use of online assessments affect the level at which learning goals are achieved.

5.2.2. The influence of stimulated learning strategies on students' perception of their ability to achieve the level of knowledge After taking the assessment, and before they got their assessment results, students were administered questionnaire Q2 to

assess the extent to which they perceive they were able to solve tasks at different levels, based on the preparations for the announced type of online assessment. As in the context of supporting the hypothesis H1 it has already been shown that the type of the announced online assessment has an effect on learning strategies, hereby the impact of these strategies on the perception of students' own ability to solve various tasks was indirectly explored. Furthermore, students' own perception of their ability to solve various tasks reflected their ability to achieve different levels of knowledge according to Bloom's taxonomy.

The following table summarizes the statistically significant correlation coefficients between indicators of learning stra- tegies and indicators of perception to achieve different levels of knowledge.

As indicated in Table 5, when online essays were announced, and in the context of previous analysis where respondents predominantly employed a deep learning strategy, it was noted that the latter has a slight increased effect on the perception of achieving all levels of knowledge. Surface preparation has a slightly decreased impact on the perception of achieving almost all levels of knowledge. Furthermore, studies have shown that a deep approach to learning affects the perception of self-confidence and efficiency of the individual (MacFarlane, Markwell, & Date-Huxtable, 2006). Based on this, it can be assumed that due to a higher percentage of respondents with a deep approach to learning, which resulted in better prep- aration for assessment, respondents' views on the possibilities of achieving different levels of knowledge are very clearly expressed. Therefore, the number of observed correlations is significantly higher than with those to whom online multiple- choice question test had been announced.

Table 3 Correlations (Spearman's Rho) between formal results on assessments and learning strategies.

Assessment scores (percentage)

Courses C1 and C2, essay-oriented learning (C1-1 and C2-1, N¼166) Strategya Essay results (announced) Test results (unannounced)

Deep 0.179* 0.218** Surface �0.063 �0.307*** Courses C1 and C2, test-oriented learning (C1-2 and C2-2, N¼185) Strategy Essay results (unannounced) Test results (announced)

Deep 0.131 0.060 Surface �0.153* �0.074

*p < 0.05; **p < 0.01; ***p < 0.001. a The division between ‘Deep’ and ‘Surface’ strategy is identical to division used in Table 2, based on variables ‘q1_deep’ and ‘q1_surf’ (explained in Chapter

5.1).

Table 4 Comparison of result in online assessments.

Preparations for the type of assessment Results (percentage of scores achieved)

Descr. Statistics t-test

N Avg. Std. dev. t Sig. (2-Tail)

Online essay, courses C1 i C2 Essay (announced, C1-1 and C2-1) 166 49.83 22.15 �4.543 0.000*** Essay (unannounced, C1-2 and C2-2) 185 39.73 19.15 Online test, courses C1 i C2 Test (announced, C1-2 and C2-2) 185 53.49 20.44 �0.726 0.469 Test (unannounced, C1-1 and C2-1) 166 55.04 19.46

*p < 0,05; **p < 0,01; ***p < 0.001.

Table 5 Correlations between learning strategies and indicators of perception to achieve different levels of knowledge.s

Levels of knowledge (Bloom)

Remember Understand Apply Analyze Synthesize Evaluate

Strategya Groups: C1-1 þ C2-1, essay-oriented learning (N ¼ 166) Deep 0.203** 0.261** 0.236** 0.238** 0.243** 0.305*** Surface �0.165* �0.180* �0.202** �0.089 �0.225** �0.302*** Strategy Groups: C1-2 þ C2-2, test-oriented learning (N ¼ 185) Deep 0.045 0.226** 0.211** 0.129 0.159* 0.187* Surface 0.092 �0.074 �0.025 0.034 �0.142 �0.048

*p < 0.05; **p < 0.01; ***p < 0.001. a The division between ‘Deep’ and ‘Surface’ strategy is identical to division used in Table 2, based on variables ‘q1_deep’ and ‘q1_surf’ (explained in Chapter

5.1).

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 39

With respondents who had the online multiple-choice question test announced to them, a more pronounced surface learning strategy has already been noted. Therefore, due to more superficial preparation for assessment, their attitudes about their own capabilities are less clear. Although the existing correlations of deep learning strategy refer to the identical levels of knowledge, as they did for respondents who had only the online essay annonuced, and have the same direction, their in- tensity is weaker. Especially interesting is the lack of any influence of surface learning strategies. As the proportion of re- spondents with a pronounced surface learning strategy is larger, in the context presented by MacFarlane et al. (2006), we can assume that these respondents share lower levels of perception of self-confidence and efficiency. Their views on the pos- sibilities of achieving different levels of knowledge are thus much less clear. According to Herbert et al. (1997), we can assume that this group of respondents gave more neutral answers in the assessment of their own ability to achieve certain levels of knowledge, and that these neutral responses disguised the possible negative correlation with surface learning strategy. The fact that post-assessment questionnaires, including Q2, were not anonymous may have also increased the proportion of more neutral, socially desirable answers (Herbert et al., 1997) for respondents with dominant surface preparation.

Exploratory factor analysis (with Principal Component Analysis, PCA) was conducted using data from questionnaire Q2, at first with respondents who had only the online essay announced (N ¼ 166, C1-1 þ C2-1), and then with respondents with only online multiple-choice question test announced (N ¼ 185, C1-2 þ C2-2). The procedure extracted four factors in both groups around which in an identical manner the items from Q2 were grouped (see Table 6).

Correlations between the factors and indicators of stimulated learning strategies (see details in Appendix B) where the online essay was announced revealed the following results:

1. A deep learning strategy has a slight effect on an increased perception of achieving results at the highest levels of knowledge (factor F1, synthesis and evaluation, r ¼ 0.187; p < 0.05), and in a part of the lower levels of knowledge (factor F2, understanding and application, r ¼ 0.168; p < 0.05)

2. A surface learning strategy has a slight effect on a decreased perception of achieving results at the highest levels of knowledge (factor F1, synthesis and evaluation, r ¼ �0.198; p < 0.05).

Table 6 Factors and levels of knowledge identified by questionnaire Q2.

Factor Level of knowledge addressed with items from Q2 Levels according to Bloom

F1 (Creative knowledge application) Synthesize, Evaluate 5 and 6 F2 (Explaining what is learned) Understand, Apply 2 and 3 F3 (Facts retrieval) Remember 1 F4 (Analysis of what is learned) Analyze 4

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4540

In the case where online multiple-choice questions test were announced, observed correlations (details in Appendix B) suggest that:

1. A deep learning strategy has a slight effect on the increase in perception of achieving results in lower levels of knowledge (factor F2, understanding and application, r ¼ 0.198; p < 0.01)

2. A surface learning strategy has a slight effect on the decrease in perception of achieving results at the highest levels of knowledge (factor F1, synthesis and evaluation, r ¼ �0.15; p < 0.05).

Overall, it is shown that:

(1) A deep learning strategy has a positive impact on the overall result in formal assessment, and surface learning strategy a negative effect.

(2) Strategies stimulated by the use of online tests have more affect on the candidates' perception of possibilities to achieve lower levels of knowledge and learning objectives (memorizing, reproduction and understanding of content).

(3) Strategies stimulated by the use of online essays have more affect on perceptions of opportunities to achieve higher levels of knowledge and learning objectives (analysis, synthesis and evaluation of content).

Therefore, we conclude that the hypothesis H2 is supported.

6. Discussion

This research confirmed that types of online assessment have an influence on the appearance of specific learning stra- tegies. Using Biggs et al. (2001) questionnaire it was shown that online tests with multiple-choice questions mainly stimulate the appearance of surface learning strategy. On the other hand, online assessment in a form of an essay mainly stimulates the appearance of deep learning strategy. These findings confirm that identical concepts taken from the traditional classes, which were the subject of a thorough research in the past, are also valid in the context of online knowledge assessment within blended e-education. Our research additionally confirms that learning strategies are variable in time and conditioned by task at hand (Hartley, 1998), as well as the results of Roscoe, Segedy, Sulcer, Jeong, and Biswas (2013), stating that “Although students seemed to acquire one beneficial [learning] strategy, they did so at the cost of other beneficial strategies.”.

It was shown that stimulated learning strategies have impact on the level of achievement of the desired learning goals, extending the findings that suggest that learning strategies may account for a significant portion of an individual's formal success in learning. According to Shih and Gamon (2002), learning strategies explained about 25% of student achievement measured by class grade, while Blom and Severiens (2008) reported that only average grade explained the variance observed in self-regulated deep learning. Kühl, Scheiter, Gerjets, and Gemballa (2011) also report that learning strategies are the differentiating factor when measuring learning outcomes - students that used particular strategies consistently outperformed those that used other less effective strategies.

In addition, it was shown that the data obtained from stimulated learning strategies matched the formal results of the assessment. It was also demonstrated that under given circumstances a deep learning strategy has a positive effect on results in bothtypes of online assessment, while surface learning strategyhas a negative impactonresults in online essay, and no impacton results in online multiple-choice question test. All this was inline with expectations. These results, especially those related to all- round positive effects of deep learning strategy, are complementary with Marzano and Kendall (2007) findings, stating that most levels of educational goals (all higher and most of the lower level goals) can be effectively assessed using essay type approaches.

Considering the fact, that the experiment involved both announced and unannounced knowledge assessment, there may be an issue of students having the feeling of being deceived because they had to participate in an unannounced assessment too. To avoid any bias as much as possible, before they were given post-assessment questionnaires to fill, students were informed that the results of the unannounced assessment would not influence their formal results. They were reassured that only the results of the announced assessment would count towards their final scores and that the results of the unannounced surprise-assessment will be used only for the research purposes.

It was further shown that learning strategies stimulated by using online tests with multiple-choice questions lead to the achievement of lower levels of knowledge and learning goals (memorizing, reproduction, understanding). On the other hand, learning strategies stimulated by using online assessment in a form of essay lead to the achievement of higher levels of knowledge and learning goals (analysis, synthesis, evaluation). Here we also confirmed that the deep approach to learning affects the perception of self-confidence and efficiency of the individual, which in our case was reflected in the fact that students who had better prepared themselves for the assessment very clearly expressed their views on the possibilities of achieving different levels of knowledge. On the other hand, those who exhibited a more pronounced surface learning strategy, demonstrated more superficial preparation for the assessment and less clear attitudes about their own capabilities (which may have been intentionally made more neutral, as suggested by Herbert et al., 1997). Our findings related to the positive influence of deep approach to learning are in line with Shen et al. (2008), stating that absence of surface learning increases individual's perceived level of learning, and with MacFarlane et al. (2006), stating that a deep approach to learning affects the perception of self-confidence and efficiency of the individual.

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 41

7. Research limitations and future research suggestions

One of the limitations of this research is related to the sample of respondents. Research has been conducted solely within a higher education institution in the field of social sciences, with students of an IT and economic orientation. Conducting this research in other fields of higher education (technical, humanistic, natural, biomedical, etc.) or even outside of higher edu- cation (e.g. high-school system, life-long learning centers, workplace education etc.) would encompass a broader population of respondents, with even more variability regarding IT literacy, previous experiences with online education, etc.

Research has been conducted only in the context of blended online education e caution is advised when generalizing results of this research to the environments that practice full online education.

On the other hand, this research opens-up the area for further research. Although tests around the reliability of the scales used within the questionnaires indicate that current scales are sufficiently reliable in general, analysis of itemetotal corre- lations has indicated that there is a further need to improve the instrument to achieve greater levels of reliability and predictability.

Factorial analysis of questionnaire Q2 (students' perceptions of their ability to achieve certain levels of knowledge based on their preparation for an announced type of assessment) has shown that students perceive certain levels of knowledge (according to Bloom's Taxonomy) as very similar, which led twice to the fusion of two of Bloom's original levels into a single factor. This opens the space for further research into the suitability of Bloom's Taxonomy to evaluate students' perception to achieve certain knowledge levels, as well as the potential to propose a more suitable instrument for this purpose.

Sample sizes in this research were limited by actual population sizes for courses C1 and C2. It would be desirable to repeat this research with larger samples, so that the data samples would be large enough to perform confirmatory factorial analysis in order to extend and generalize the results obtained.

Complementary, numerous studies support the idea of innovative ICTapplications (including web-based assessment tools) to facilitate various self-regulatory behaviors in order to improve e-learning effectiveness. Valentín et al. (2013) have shown significant associations of the different uses of ICT with learners’ improved performance and satisfaction, including the importance of learning strategies while using ICT and measuring learning outcomes. Wang (2011) reports that students were more willing to take the peer-driven, Web-based formative assessment instead of more traditional Web-based tests. Same author (Wang, 2010) also describes the importance of having timely feedback within Web-based dynamic assessment systems.

Therefore, results of this research showing that: (i) the announcement of certain type of online assessment has influence on students' learning strategies and (ii) the stimulated learning strategies have influence on achievement levels that students exhibit during assessments, should be the main design principles of an ideal online assessment system that would utilize timely feedback and stimulation of learning strategies to improve students' achievement of required learning goals. Such system should be designed to incorporate the elements of adaptivity within a series of assessments. A teacher should design the first one in a series of assessments manually, while subsequent assessments should be constructed by the system in an adaptive manner. Such design would allow for knowledge assessment that is both continuous and cumulative: (i) earlier course topics are re-assessed in every subsequent test and (ii) earlier course topics are re-assessed in adaptive manner, based on student's learning goals achievements regarding those topics during previous tests. Also, post-assessment feedback should be used to guide each student in preparations for the upcoming tests, by stimulating the emergence of appropriate learning strategies in order to increase levels of success in achieving learning goals. Such system, taking into account the results obtained from this research, would further expand the notion of feedback, as described by e.g. Wang (2010), by taking into account both the learning goals and the results of previous assessments.

8. Conclusion

This paper has shown that types of online assessment used in the assessment process influence the occurence of particular learning strategies. Specifically, an announcement and application of online tests with multiple-chioce questions stimulates the occurrence of a surface learning strategy and to the lesser extent the occurrence of deep learning strategy. An announcement and application of online assessment in a form of an essay stimulates the occurrence of deep learning strategy and to the lesser extent occurrence of surface learning strategy.

Furthermore, it is proven that learning strategies stimulated by particular types of online assessments have influence on the level of learning goals achieved. Roscoe et al. (2013) similarly report that use of the SRL-supportive (Self-Regulated Learning) tools positively correlate with learning goals, due to the promising learning strategy patterns that students have developed when they started to use the tools. However, they also report negative effects of shallow (surface) strategy that began to develop during prolonged usage of those tools. Hartley's (1998) findings that learning strategies are variable in time and conditioned by task at hand additionaly support these conclusions and confirm the validity of the decision to focus this research on learning strategies, instead of learning styles, which are difficult to manipulate. By announcing and administering appropriate types of online knowledge assessment, it is possible to stimulate desirable learning strategies, as well as, consequentially, to influence the rejection of undesireble learning strategies.

The results of this paper can be useful to educational institutions for designing and implementing online classes and knowledge assessments. Research results suggest that improvement in the achievement of required learning goals’ can be influenced by appropriate learning strategies, which are, in turn, stimulated by announcing and conducting appropriate types of online assessments.

Appendix A. Questionnaire Q2

(B1knwl) Based on prepara ons for knowledge assessment using [[test type]] 1, I'm now be er at … Differen a ng true and false statements 2

Recognizing the elements of various classifica ons and defini ons Answering the ques ons where I have been asked to recall the meaning of a term Recognizing the term, given the descrip on of that term Wri ng basic defini on of a term Describing basic characteris cs of a term

(B2cmpr) Based on prepara ons for knowledge assessment using [[test type]], I am now be er at … Answering the ques ons where I have been asked to explain the meaning of a term using my own words Reformula ng given statement so that its meaning is preserved Answering the ques ons where I have been asked to single out statements related to a given term Describing how something works or func ons Explaining the role an element has in a larger whole Briefly retelling the facts relevant to a given problem

(B3appl) Based on prepara ons for knowledge assessment using [[test type]], I am now be er at … Answering the ques ons where I have been asked to solve given problem and write the exact solu on Iden fying or explaining the most important elements related to a given concept or term Explaining the rela onships that exist between some given terms Explaining why certain element is important for func oning of a larger whole Comparing the importance of the different elements of a larger whole Using concrete examples to explain some theore cal concept

(B4anly) Based on prepara ons for knowledge assessment using [[test type]], I am now be er at … Indica ng important details regarding the structure of an object, concept or model Answering the ques ons where I have been required to divide a whole to its cons tuent components Answering the ques ons where I have been asked to compare advantages and disadvantages of something Sketching the structure of an object, concept or model Making classifica ons/categoriza ons of something, according to various criteria Enumera ng the facts or evidences which confirm the given concept

(B5synt) Based on prepara ons for knowledge assessment using [[test type]], I am now be er at … Answering the ques ons where I have been asked to state my assump ons about some problem Reaching my conclusions about some problem Exposing my sugges ons about solving a problem Proposing a new structure of an exis ng system, model etc. Exposing my opinion about the consequences of certain ac vity Designing new „product“ (either in material or non-material form), based on present knowledge and facts

(B6evlt) Based on prepara ons for knowledge assessment using [[test type]], I am now be er at … Answering the ques ons where I have been asked to decide how to solve a given problem Media ng in resolving disagreements and conflicts of opinions about a given problem/topic Explaining why I agree or disagree with a given statement about a given problem/topic Ordering objects, terms and concepts in order of importance and explaining my reasons for such ordering Explaining which criteria I could use to es mate good or bad aspects of an object, phenomenon, and the like Explaining my opinion about some problem to other people

1 Possible test types: Essay type of online assessment or Mul ple choice test type of online assessment 2 Answers to all statements: on 1 to 5 point Likert type scale (1 – I disagree/Completely untrue; 3 – Can't decide/Neither true nor untrue; 5 – I completely agree/Completely true)

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4542

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 43

Appendix B. Factor Analysis of Questionnaire Q2

Table B.1 Correlations between indicators of stimulated learning strategies (R-SPQ-2F) and PCA factors extracted from indicators of perceiving the ability to achieve different levels of knowledge (questionnaire Q2) e courses C1 and C2, N ¼ 166, online essay announced, essay-oriented learning (C1-1 þ C2-1)

Creative knowledge application (F1)

Explaining what is learned (F2)

Facts retrieval (F3) Analysis of what is learned (F4)

q1_deep Spearman rho 0.187* 0.168* 0.143 0.113 Sig. (2-tail) 0.016 0.031 0.065 0.149 N 166 166 166 166

q1_surf Spearman rho ¡0.198* �0.140 �0.040 �0.072 Sig. (2-tail) 0.010 0.072 0.610 0.359 N 166 166 166 166

*p < 0.05; **p < 0.01; ***p < 0.001.

Table B.2 Correlations between indicators of stimulated learning strategies (R-SPQ-2F) and PCA factors extracted from indicators of perceiving the ability to achieve different levels of knowledge (questionnaire Q2) e courses C1 and C2, N ¼ 185, online test announced, test-oriented learning (C1-2 þ C2-2)

Creative knowledge application (F1)

Explaining what is learned (F2)

Facts retrieval (F3) Analysis of what is learned (F4)

q1_deep Spearman rho 0.138 0.198** �0.019 0.100 Sig. (2-tail) 0.060 0.007 0.795 0.177 N 185 185 185 185

q1_surf Spearman rho ¡0.150* �0.047 0.057 0.101 Sig. (2-tail) 0.041 0.526 0.444 0.170 N 185 185 185 185

*p < 0.05; **p < 0.01; ***p < 0.001.

Appendix C. Sample Assessment Items

Following are the sample questions used to assess the course topic related to “Data management”

1. Sample essay ques on

Problem descrip on: Let us imagine two companies:

1. The first company is manufacturing-oriented and needs to develop custom so ware to track its own produc on. This so ware will predominantly have to process highly structured textual and numerical data related to the produc on flow. This so ware should also be able to create and display detailed reports about various aspects of produc on flow.

2. The second company needs to develop mul media portal. Main func onality of this portal is to enable the end-user to search for various low-structured and unstructured mul media content (documents, video clips, audio clips, etc.).

We also know that, given the type of data they store, there are three fundamental types of database systems:

1. Structured databases 2. Unstructured databases 3. Knowledge bases

Task: For both above-men oned companies, recommend the most appropriate type of database system - i.e. the system that will provide the applica ons they have to develop with the best support to handle the data. Back-up your sugges on by highligh ng and describing the key features of recommended database systems, i.e. the features that, in your opinion, are in favor of your recommenda ons.

2. Sample mul -choice test ques ons

What is the name of the a ribute (or the set of the a ributes) that is used to uniquely dis nguish every row in a table within a rela onal database? (an example of ques on with one correct answer)

a. Public key b. Foreign key c. Private key d. Primary key(true) e. External key

Terms ‘physical’ and ‘logical organiza on of records’ are related with which type of files? (an example of ques on with one correct answer)

a. Index files b. Sequen al files (true) c. Unstructured files d. Direct-access files

Which of the following components are considered as parts of the rela onal scheme defini on? (an example of ques on with mul ple correct answers)

a. Name of a database b. Names of a ributes (true) c. Name of an opera on d. Name of a rela on (true)

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e4544

References

Aller, B. M., Phillips, C., Tsang, E., Kline, A. A., & Aravamuthan, R. (2003). An online assessment system to enhance teaching and learning in engineering. In Proceedings of the 33rd Annual Frontiers in Education FIE 2003 (Vol. 1, p. 7).

Aluísio, S. M., Aquino, V. T., Pizzirani, R., & de Oliveira, O. N., Jr. (2003). Assessing high-order skills with partial knowledge evaluation: lessons learned from using a computer-based Proficiency test of English for academic purposes. Journal of Information Technology Education, 2, 185e201.

Anderson, T. (2003). E-Learning in the 21st Century: A Framework for research and practice. USA: Routledge. Ardid, M., G�omez-Tejedor, J. A., Meseguer-Due~nas, J. M., Riera, J., & Vidaurre, A. (2015). Online exams for blended assessment. Study of different application

methodologies. Computers & Education, 81, 296e303. Biggs, J. B., Kember, D., & Leung, D. Y. P. (2001). The revised two-factor study process questionnaire: R-SPQ-2F. British Journal of Educational Psychology, The

British Psychological Society, 71, 133e149. Black, P., & William, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7e73. Blom, S., & Severiens, S. (2008). Engagement in self-regulated deep learning of successful immigrant and non-immigrant students in inner city schools.

European Journal of Psychology of Education, 23(1), 41e58. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W., & Krathwohl, D. R. (1956). Taxonomy of educational objectives, the classification of educational goals,

Handbook I: Cognitive domain. USA: McKay Press. Buzzetto-More, N. A., & Alade, A. J. (2006). Best Practices in e-Assessment. Journal of Information Technology Education, 5, 251e269. Conole, G., & Warburton, B. (2005). A review of computer-assisted assessment. ALT-J, Research in Learning Technology, 13(1), 17e31. Crowe, A., Dirks, C., & Wenderoth, M. P. (2008). Biology in bloom: implementing bloom's taxonomy to enhance student learning in biology. CBE-Life Sciences

Education, 7(4), 368e381. Entwistle, N. (1988). Motivational factors in students' approaches to learning. In R. R. Schmeck (Ed.), Learning strategies and learning styles (pp. 21e52). USA:

Plenum Press. Entwistle, N. (2000). Promoting deep learning through teaching and assessment. In L. Suskie (Ed.), Assessment to promote deep learning: insight from AAHE's

2000 and 1999 assessment conferences (pp. 9e21). American Association for Higher Education (2001). Gow, L., & Kember, D. (1990). Does higher education promote independent learning? Higher Education, 19, 307e322. Hartley, J. (1998). Learning and studying: A research perspective. USA: Routledge. Hein, T., & Irvine, S. E. (1998). Assessment of student understanding using on-line discussion groups. In 28th Annual Frontiers in education Conference [Online]

Available: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp¼&arnumber¼736819 [March, 2015]. Herbert, J. R., Ma, Y., Clemow, L., Ockene, I. S., Saperia, G., Stanek, E. J., III, et al. (1997). Gender differences in social desirability and social approval bias in

dietary self-report. American Journal of Epidemiology, 146(12), 1046e1055. Hoeksema, L. H. (1995). Learning strategy as a guide to career success in organizations. Leiden University, Netherlands: DSWO Press. Kim, N., Smith, M. J., & Maeng, K. (2008). Assessment in online distance education: a comparison of three online programs at a university. Online Journal of

Distance Learning Administration, 11(1) [Online] Available: http://www.westga.edu/~distance/ojdla/spring111/kim111.html [March, 2015]. Kühl, T., Scheiter, K., Gerjets, P., & Gemballa, S. (2011). Can differences in learning strategies explain the benefits of learning from static and dynamic vi-

sualizations? Computers & Education, 56(1), 176e187. Macdonald, J. (2004). Developing competent e-learners: the role of assessment. Assessment & Evaluation in Higher Education, 29(2), 215e226. MacFarlane, G. R., Markwell, K. W., & Date-Huxtable, E. M. (2006). Modelling the research process as a deep learning strategy. Journal of Biological Education,

41(1), 13e20. Marton, F., & S€alj€o, R. (1976). On qualitative differences in learning: I. Outcome and process. British Journal of Educational Psychology, 46, 4e11. Marzano, R. J., & Kendall, J. S. (2007). The new taxonomy of educational objectives (2nd edn). USA: Corwin Press. Nicol, D. (2007). E-assessment by design: using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53e64. Nowicki, C., & Jones, E. L. (2005). A framework for extending online assessment. In Proceedings of the 6th International Conference on information technology

based higher education and Training ITHET 2005, Florida, USA (pp. 12e17).

M. Zlatovi�c et al. / Computers & Education 91 (2015) 32e45 45

Oliver, D., & Dobele, T. (2007). First Year courses in it: a bloom rating. Journal of Information Technology Education, 6, 347e360. Roscoe, R. D., Segedy, J. R., Sulcer, B., Jeong, H., & Biswas, G. (2013). Shallow strategy development in a teachable agent environment designed to support

self-regulated learning. Computers & Education, 62(C), 286e297. Rushton, A. (2005). Formative assessment: a key to deep learning? Medical Teacher, 27(6), 509e513. Sankaran, S. R., & Bui, T. (2001). Impact of learning strategies and motivation on performance: a study in web-based instruction. Journal of Instructional

Psychology, 28(3), 191e198. Scouller, K. (1998). The influence of assessment method on students' learning approaches: multiple choice question examination versus assignment essay.

Higher Education, 35(4), 453e472. Shen, J., Hiltz, S. R., & Bieber, M. (2008). Learning strategies in online collaborative examinations. IEEE Transactions on Professional Communication, 51(1),

63e78. Shih, C.-C., & Gamon, J. A. (2002). Relationships among learning strategies, patterns, styles, and achievement in web-based courses. Journal of Agricultural

Education, 43(4), 1e11 [Online] Available: http://pubs.aged.tamu.edu/jae/pdf/vol43/43-04-01.pdf [May 2015]. Shumway, J. M., & Harden, R. M. (2003). AMEE Guide No. 25: the assessment of learning outcomes for the competent and reflective physician. Medical

Teacher, 25(6), 569e584. Siew-Rong, W. (2008). Evaluation of the learning of scientific English in podcasting PCs, MP3s, and MP4s Scenarios. In Proceedings of the SUTC '08 IEEE

International Conference on Sensor Networks, Ubiquitous and Trustworthy Computing, Taichung, Taiwan (pp. 537e542). Slack, F., Beer, M., Armitt, G., & Green, S. (2003). Assessment and learning outcomes: the evaluation of deep learning in an on-line course. Journal of In-

formation Technology Education, 2, 305e317. Soulsby, E. P. (2009). Assessment notes [Online] Available: http://assessment.uconn.edu/docs/resources/Eric_Soulsby_Assessment_Notes.pdf [March, 2015]. Valenti, S., Neri, F., & Cucchiarelli, A. (2003). An overview of current research on automated essay grading. Journal of Information Technology Education, 2,

319e330. Valentín, A., Mateos, P. M., Gonz�alez-Tablas, M., P�erez, L., L�opez, E., & García, I. (2013). Motivation and learning strategies in the use of ICTs among university

students. Computers & Education, 61, 52e58. Vos, N., van der Meijden, H., & Denessen, E. (2011). Effects of constructing versus playing an educational game on student motivation and deep learning

strategy use. Computers & Education, 56(1), 127e137. Wang, T.-H. (2010). Web-based dynamic assessment: taking assessment as teaching and learning strategy for improving students' e-Learning effectiveness.

Computers & Education, 54(4), 1157e1166. Wang, T.-H. (2011). Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-

Learning environment. Computers & Education, 57(2), 1801e1812. Watkins, D. A., & Hattie, J. (1985). A longitudinal study of the approach to learning of Australian tertiary students. Human Learning, 4(2), 127e142.

Miran Zlatovi�c works as Lecturer at the Department ofs Computing and Technology at the University of Zagreb, Faculty of Organization and Informatics. He published 15 scientific and professional papers. His main areas of interest include e-learning, online knowledge assessment, usage of ICT in education, programming and computer security.

Igor Balaban works as Assistant Professor at the Department of Computing and Technology at the University of Zagreb, Faculty of Organization and Informatics. He published over 30 scientific and professional papers and 4 book chapters. His main areas of interest are Information System success applied to e-learning and user-centered learning technologies.

Dragutin Kermek works as Full Professor at the Department of Theoretical and Applied Foundations of Information Sciences at the University of Zagreb Faculty of Organization and Informatics. He has published over 50 research and professional papers in international and domestic journals, books and conferences. His research interests include software engineering (components, design patterns, integration and interoperability), web engineering (quality, developing methods, security and performance), e-learning (efficiency in ICT based teaching).