How to assess information and communication technology knowledge and skills of the students

Assessment of ICT knowledge and skills is as important as gaining these knowledge and skills. There are two commonly used assessment types as (multiple-choice) Test Based Assessment (TBA) and Practice Based Assessment (PBA) and the question that which one is better still remains controversial. In this research, the focus of the study was examining these factors in relationship with the academic achievement of the students. More specifically, impact of assessing the students with TBA and PBA types on the students' academic achievement scores based on the aforementioned factors was examined. For this purpose, two study groups consisting of 243 students who were assessed with TBA and PBA in a course titled as Computing I were included in the present research. The results showed that there is a significant difference in academic achievement scores of the students and that the students who took TBA are more successful. According to the results, students are more comfortable and more confident with TBA whereas students taking PBA believe that PBA is better to measure ICT knowledge and skills. Therefore, it could be concluded that these two different assessment types are complementary once students’ positive opinions, expectations and feelings are pedagogically secured.


Introduction
Assessment of Information and Communication Technologies (ICT) knowledge and skills is as important as gaining these knowledge and skills since the degree of ICT competency of the students can provide faculty with important information about the strengths and weaknesses of their students' ICT related skills (Shuster & Pearl, 2011).Measurement of teaching and learning processes are important aspects for learning management, however, some teachers delivering ICT courses do not have enough skills about measurement (Suwatthipong, Thangkabutra, & Lawthong, 2015).An acceptable method regarding measurement of ICT skills is still being discussed while usually either a Test -consisting of multiple-choice questions-Based Assessment (TBA) or Practice-Based Assessment (PBA) is conducted.As one of the leading organizations dealing with technology use in education, ISTE publishes standards (ISTE, 2016) that characterise the students as digital citizens, innovative designers and so on through the effective use of technology.From the perspective of such attributions, assessing the students' ICT knowledge and skills with a typical type of assessment would stand contradiction to the expectations from the students and this raises the importance of academic debates over how to assess ICT knowledge and skills.
A further fact in relation to assessment methods concerns the nature of knowledge domain in ICT courses.ICT in general is composed of the procedural knowledge.Procedural knowledge is defined as behavioural performance (Christoph, Goldhammer, Zylka & Hartig, 2015) consisting of sequential steps or actions to reach a certain goal (Rittle-Johnson & Schneider, 2015).As the definition implies, procedural knowledge inherently requires performance based assessment.That being said, in practice, assessment of ICT courses is found challenging because of out-numbered classes (Mustapha, Samsudin, Arbaiy, Mohamed & Hamid, 2016), problem-solving oriented and very practical nature of the course (Wang, Li, Feng, Jiang & Liu, 2012) and technical difficulties in administering PBA (Clough, 2008).Due to these challenges, some educators tend to adopt TBA.Although in behavioural-based learning problems, there are various ways of solving a problem, in TBA, usually the correct answer only lies in the one choice.However, this is problematic because as Mustapha et al (2016) remark "Sometimes students get marked down inappropriately if their solutions do not follow the answer scheme" (53).In the same fashion, procedural knowledge is taught usually by employing teaching strategies such as learning by doing; so, measuring a type of 'knowledge that cannot be verbalized directly' (Rittle-Johnson & Schneider, 2015) with TBA might bring about problems of accuracy.In addition, (Oosterhof, 2011) puts forward that different types of knowledge require a different type of assessment approach.For instance, he suggests that when designing the assessment to measure procedural knowledge, the students need to be asked about new examples as otherwise their declarative type of knowledge would be addressed.To sum up, when taking into account of the importance of designing the assessment based on the type of the knowledge in the ICT domain and given the gap in the literature pointing out the instructors' guidance on how to assess ICT knowledge and skills, there is a need to examine assessment methods of ICT knowledge and skills through different factors to ensure academic success of the students.
On looking at the assessment types from the students' psychological point of view, Bandura refers to two distinct expectancy beliefs about outcome of certain behaviour as 'outcome expectations' and 'efficacy expectation' (Eccles & Wigfield, 2002).Accordingly, although students, for instance, believe that their certain behaviors will lead to certain outcomes (outcome expectations), they may not perform that behaviour (efficacy expectation) while efficacy expectations are the major source of activity choice, willingness to expend effort and so on (Eccles & Wigfield, 2002), (Parmenter, 2009).Drawing on this, it could be assumed that the students' performance could be influenced by their expectation beliefs and on this point, it is important to find out their outcome expectations from PBA and TBA.Given that the students usually prefer TBA to other forms of assessment (Parmenter, 2009), (Yaman, 2011) and their preference could form their expectations, it is worthwhile to examine students' assessment type preference and their performance on each type of the assessment.A further psychological state as a predictor of student performance regards "anxiety".As research has shown (Von der Embse, Schultz & Draughn, 2015), there is a relationship between anxiety and academic achievement.In that sense, it is important to explore with which type of the assessment students feel more comfortable.
Finally, to the best of our knowledge, there has not been any research study dealing with preassessment conditions of the students such as the students' facilities to get ready for the assessment.After all, as (Dodeen, 2008) stresses test scores do not only reflect students' knowledge.Students' preassessment conditions should also be taken into account.This is important in the context of PBA and TBA in ICT courses because it is expected that a student would obtain higher scores if she/he possesses technology or reaches at technology and practice before the assessment compared to a student who is deprived of these facilities.If it is a TBA, then students could equally prepare for the exams merely with course materials (e.g.printed materials).Therefore, it is worthwhile to investigate whether there is an effect of conditions and facilities on academic success of the students.
Against this background, the present study aims to contribute to the literature by revealing comprehensive results through investigating types of assessment for Information and Communication Technology skills and knowledge with the factors influential on the assessment.The remainder of this study will explore the following research questions: 1. Is there any significant difference in students' academic achievement scores who were assessed through PBA and TBA? 2. Is there any significant difference in students' academic achievement scores who could access to the computer and internet and those who could not in order to get ready for their PBA? 3. Is there any significant difference in students' academic achievement scores who could access to the computer and internet and those who could not in order to get ready for their TBA? 4. Is there any significant difference in students' academic achievement scores based on their preference on assessment types as PBA and TBA? 5. What is the average score of the students based on their outcome & efficacy expectation, feelings and opinions about the assessment types?

Methodology
Quantitative method was used in this study.In data analysis, percentage and frequency values as well as Mann Whitney U-Test as non-parametric test were used.
The study group consists of 369 students enrolled in Computing I (Introduction to Information and Communication Technology) course, at a Faculty of Education in a state university in Turkey in 2016.The students who failed in the course according to their grades or those who drop out were excluded in the study due to the fact that those who dropped out did not attend the course, hence they were unable to give accurate feedback and due to the possibility that those who failed might give biased feedback about the design of the assessment.Among 369 students, 243 students who returned the questionnaire and who were assigned with a pass grade were included in the study.As a data collection tool, an online questionnaire was used to find out students' outcome & efficacy expectations, feelings and opinions about the assessment types and also their ICT facilities to get ready for the exam were addressed.At the time of the data collection process, all students were provided with online facilities in the computer labs and in other university facilities to fulfil the questionnaire.Students' academic achievement scores were measured by their exam result.For PBA, a holistic rubric was used by the tutor of the course to measure whether students could perform the given tasks on time, administer the correct commands and degree of successful fulfilment of the tasks in the exam taken place in the computer lab.For the TBA, students' correct answers from the test questions were calculated and scored.Table 1 below shows information about the students who participated in the study based on their department and assessment type.Demographic information of the students is as follows:  The students who took the TBA consist of 11 (12.6%)male and 76 (87.4%) female students.Their average score from the achievement test is 93.85 and standard deviation is 8.48. The students who took the PBA consist of 37 (23.7%)male and 119 (76.3%) female students.
Their average score from the achievement test is 82.79 and standard deviation is 15.24. Among the students who took TBA, 88.5% of them possess a mobile phone, 9.0% of them possess a laptop; 13.8% possess a tablet, 21.8% of them possess a PC. Among the students who took the PBA, 84.6% of them possess a mobile phone, 52.6% of them possess a laptop; 26.9% possess a tablet, 17.9% of them possess a PC.
Overall, it could be seen that in both groups, the students have similar facilities in terms of having technological devices for them to get ready for the exams.In addition to this, the students in both groups have similar opportunities to access to the internet.On a closer look at the students' internet access, most of them access to the internet via their mobile phones (84%), from home (55.1%) and from the wireless network at the school (40.4%), whereas small number of the students use the wireless network at someone else's house (6.9%) or cyber cafes (5.4%).

Findings
2.1.Is there any significant difference in students' academic achievement scores who were assessed through PBA and TBA?
Mann Whitney U test results based on the students' scores taken from the exam in the course was summarised in the Table 2 below.According to this, a significant difference was found between the academic achievement scores of the students who took PBA and TBA (U=10367.50,p<.01).When considering mean rank, it could be seen that academic success of the students is higher in TBA than in PBA.2.2.Is there any significant difference in students' academic achievement scores who could access to the computer and internet and who could not in order to get ready for their PBA?
Mann Whitney U test was used to answer this question.According to the results, it was found that there is no significant difference in academic achievement scores of the students between who possess technological devices and the internet and those who do not (p>.05).

Is there any significant difference in students' academic achievement scores who could access to the computer and internet and those who could not in order to get ready for their TBA?
Mann Whitney U test was used to answer this question.According to the results, it was found that there is no significant difference in academic achievement scores of the students between who possess technological devices and the internet and those who do not (p>.05).

Is there any significant difference in students' academic achievement scores based on their preference on assessment types as PBA and TBA?
Mann Whitney U test was used to answer this question.Results were summarised in the Table 3 below.According to the results, a significant difference was found in academic achievement scores of the students who preferred PBA and TBA (U=6007.00,p<.05).When considering mean ranks, the students who prefer TBA scored higher than the students who prefer PBA.

What is the average score of the students based on their outcome & efficacy expectation, feelings and opinions about the assessment types?
In regard to a question asking students' preference on the assessment type, out of 156 students who were assessed with PBA, 121 (78%) of them prefer to have PBA again, 35 (22%) of them prefer to have TBA.Similarly, out of 87 students who were assessed with TBA, 66 (%76) of them prefer to have TBA again, while 21 (%24) of them prefer to have PBA.In general, it was found that:  students are more comfortable with TBA than PBA  students find TBA as a more objective way of assessing  students can more confidently answer the questions in TBA than PBA (Compared to TBA) The students' responses to the questions about their outcome and efficacy expectations from the assessment and their feelings and opinions about the assessment types are summarised based on the average scores in Table 4 below.

Discussions and Conclusions
This study has attempted to find a suitable assessment type for measuring ICT knowledge and skills of the students drawing on student academic achievement scores and students' expectations, feelings and opinions about the different assessment types.Also, it was aimed to investigate whether there is a significant relationship between the pre-assessment conditions of the students and their academic success.
As a result of the study, it was found that academic achievement scores of students taking TBA are higher than students taking PBA.This finding is consisted with the literature (Heck & Stout, 1998) (Lee, & Weerakoon, 2001) (Russell, 1999).The reason for the emergence of this difference could be stemmed from the students' habits because most of the times students are exposed to TBA throughout their education life including high-stake tests.Also, student answers to the questionnaire demonstrated that they are more comfortable and confident while taking TBA.Consistent with the findings in the literature, it could be assumed that factors which are influential on academic success of the students taking TBA could be related to their positive mentality about TBA.The students taking PBA stated that PBA is a suitable way of monitoring their knowledge and skills in terms of measuring what they have learnt from the course and transferability of the knowledge they learnt.In some studies, similar findings were found (Bugbee, & Bernt, 1990) (Pomplun, Frey, & Becker, 2002).It is noteworthy to observe in the data that in general students believe that PBA is better to measure ICT knowledge and skills while they prefer TBA.This paradox strengthens the view that positive psychological states of the students should be secured when they take PBA.In other words, although the students who preferred and took TBA had higher scores than other students, this result should not be interpreted in a way that TBA is better to assess ICT knowledge and skills of the students since some psychological factors could be influential in getting better assessment results.In addition to the feelings of the students.opinions of the students could also have an influence on the results.In some studies, it was suggested that despite of an overall positive testing effect, students sometimes come to believe that distracting answers are correct and therefore they leave the exam having acquired false knowledge (Roediger & Marsh, 2005) (Toppino & Brochin, 1989).In line with this, in this study, students remarked that when they are assessed with PBA, they learn what they have not learnt during the course once they find out the answers; in that sense PBA could be regarded as a more instructive method since the students promptly get a chance to see results of their attempts to fulfill the given tasks.Also, as emerged from the findings, the students may think that it is difficult to ensure objectivity in PBA (Price, 2007); hence they might prefer TBA.In that case, a holistic rubric for PBA could be formed and its dimensions could be discussed with the students before the assessmenttaking place.
Based on the academic success of the students, given the types of the knowledge in the ICT domain, TBA could be used for measuring conceptual content knowledge of the students while PBA could be used for measuring procedural content knowledge of the students.As a conclusive reflection, it could be assumed that both PBA and TBA as complementary assessment types could be used and in fact, (Stecher, & Hamilton, 2014) state that most of the ICT competencies could be assessed using more than one method.On this point, it is important to pedagogically discuss the assessment methods with the students and ensure their positive psychological states and opinions about assessment that they would receive.A further finding leading us to recommend employing both methods concerns the preassessment conditions of the students.It was reflected that for PBA, the students need to have their own ICT facilities to practice before the assessment whereas it is more convenient for them to get ready for their TBA through printed materials.However, in this study, the findings did not provide any evidence for this assumption.A possible reason could be that of providing students with equal access to the ICT during the class-hours and providing them with flexible schedule to access the facilities out of class hours in case they miss the class.Another reason could be that of the ensuring their learning during the class time so that regardless of the type of the assessment, they could demonstrate successful results without needing any facilities.
PBA helps them recall what they have learnt  PBA measures what they have learnt from the course  PBA sufficiently measures transferability of what they have learnt from the course into the real life problems  when assessed with PBA, they learn what they think they have not learnt in the course once they find out correct answers  PBA is consistent with/serves the aim of the course  PBA gives them an opportunity to see what they have learnt or have not learnt from the course

Table 1 .
Number of the students in the study, their departments and how they were assessed (TBA or PBA)

Table 2 .
Data analysis results based on the assessment types and academic achievement scores

Table 3 .
Data analysis results based on assessment type preference of the students and their academic

Table 4 .
Student responses about their expectations, feelings and opinions about assessment types