Learning Objects Reusability Effectiveness Metric (lorem)

In this research we aim to propose an advanced metric to evaluate the effectiveness of learning objects in order to be reused in new contexts. By the way learning objects reusability is achieving economic benefits from educational technology as it saving time and improving quality, but in case of choosing unsuitable learning object it may be less benefit than creating the learning object from scratch. Actually learning objects reusability can facilitate systems development and adaptation. By surveying the current evaluation metrics, we found that while they cover essential aspects, they enables all reviewers of learning objects to evaluate all criteria without paying attention to their roles in creating the learning object which affect their capability to evaluate specific criteria. Our proposed Approach (LOREM) is evaluating learning objects based on a group of Aspects which measure their level of effectiveness in order to be reused in other contexts. LOREM classifies reviewers into 3 categories; 1. Academic Group: (Subject Expert Matter " SME " and Instructor). 2. Technical Group: (Instructional Designer " ID " , LO Developer and LO Designer). 3. Students group. The authorization of reviewers in these several categories are differentiated according to reviewer's type, e.g., (Instructor, LO Developer) and their area of expert (their expertise subjects) for academic and students reviewers.


INTRODUCTION
The increasing of learning objects reusability popularity became very observable nowadays; it can be evidenced by the large number of learning objects repositories which became available (Falcão de Berredo and Soeiro, 2007).Sampson and Papanikou (2009) defined Learning Objects Reusability as "the extent to which a LO can operate effectively for a variety of users in a variety of learning contexts over time in order to achieve the same or a different objectives from that envisaged by its supplier".
The proposed metric (LOREM) is a system evaluating learning objects according to some criteria, some of these criteria have been used before in other significant metrics like LORI and MERLOT metrics while some other criteria we suggest after researching and conducting meetings with professional education specialists.
This system enables only registered users to evaluate LO based on their areas of expert as they fill in the registration form.In addition to that every question has "not applicable" answer which reviewer can use if he/she isn't sure of answer.
The proposed metric supposed to help educational organizations, trainers, LO developers and any one working in learning objects filed by giving them evaluation to the effectiveness of reusable learning objects to help in the selection of reusable educational materials from repositories on the web and it can be also used in a certain organization to evaluate its created learning objects which saved in its repository to be reused afterwards in other contexts.We expect that this metric would be an approach to make managerial decisions regarding to Learning Objects Reusability.
Learning object evaluation: There are some initiatives approached the evaluation of learning objects to offer an estimation of the guaranteed quality.According to MERLOT (http://www.merlot.org/merlot/index.htm),objects are classified into 7 categories: Arts, Economy, Education, Humanities, Mathematics, Science and Technology.It depends on collection of experts and users evaluation on three dimensions (content quality, usability and effectiveness as a learning tool) on a 1-5 scale, while LORI (www.elera.net)using more extended technology which includes more aspects that it includes 9 aspects, every one of them is assessed on 1-5 scale, using collaborative evaluation scheme depends on evaluation of group of experts (Rodríguez et al., 2008).
Both of MERLOT and LORI will be discussed in more detailed in the following section.

SURVEY Current approaches of measuring learning object reusability effectiveness: Multimedia Education Resource for Learning and Online Teaching (MERLOT):
MERLOT is an openaccess repository which provide user with both of evaluated and unevaluated learning objects, in descending order of quality rating, with unevaluated objects at the end of the list.
MERLOT evaluation criteria: According to MERLOT approach there are 3 categories of evaluation standards: Quality of content: Focus on: • Valid (correct) concepts, models and skills; to evaluate these elements, reviewers depend on their expertise • Educationally significant concepts, models and skills for the discipline to evaluate these element reviewers rely on the following guidelines: o LO covers the essential knowledge of the domain which it specialized in o LO contains data difficult to learn and teach o LO contains data requested as introductory Level of advanced material

Potential effectiveness as a teaching-learning tool:
It's the most difficult element in MERLOT Evaluation criteria.This element requires actual use by real students and teachers.In this element reviewers are asked to judge based on their experience on answering some questions addressed by MERLOT to determine whether the learning object can improve the process of teaching and learning in ways faculty and students can use.
Ease of use: The issue of this aspect is how the evaluated learning object is easy to be used for the first time by students and teachers.

Learning Object Review Instrument (LORI):
According to Nesbit et al. (2004) LORI is an online instrument, has been developed as a service on (www.elera.net)website for evaluating the learning objects in learning objects repositories.
In LORI, reviewers state their rating and comments based on nine items to evaluate learning object: • Content quality: Veracity, accuracy, balanced presentation of ideas and appropriate level of detail The rating scale of every item of these nine items is consisting of 5 levels.If learning object is not relevant to the specific criterion or reviewer are unable to evaluate it according to that criterion, it signed as "not applicable".
Using of LORI: LORI can be used for both of individuals or panel reviewers.When LORI is used by a review panel, it's recommend to use the convergent participation model for collaborative evaluation.Evaluation results supposed to be listed as a set of averaged rating, one per item, while it may be summarized as a single average covering all the items which have been used in the evolution.All comments recorded by reviewers should be reported (Nesbit et al., 2004).
The convergent participation model: This model is mainly depending on LORI that it's done through two stages; at the first stage experts evaluate learning object according to LORI principals.This stage is supposed to take few days.
In the other stage; reviewers are meeting each other in a virtual conference controlled by a moderator where they discusses their evaluation and comments, during this conference objects are discussed in order as objects with more inter-rating variety are discussed before object with less inter-rating variety.
Finally at the last stage, on the fifth day, participant re-evaluate the sets in both of the previous stages and fill a questionnaire asks about participants' opinions on a range of topics related to the research goals of the study (Vargo et al., 2003).

PROPOSED METHODOLOGY
Learning Objects Reusability Effectiveness Metric (LOREM): LOREM is our proposed metric that based on dynamic evaluation application filled by 6 types of reviewers, criteria of evaluation in this questioner is classified in 8 categories and clarified by guidelines gotten through analytical study to the factors which affect reusability of learning objects in order to When these reviewers register in LOREM system they fill their occupation which determine their authorization on the website, that there are some elements must be filled by specialized persons only, for example there are pedagogical elements must be filled by teaching team only, other categories of reviewers can't give accurate judgment.
Evaluation method is differentiated from category to another; most of categories are rating scale consisting of five levels, one other category is check box and the last type of categories is "3 choices radio buttons".Every criterion in the rating scale has minimum acceptable value, when reviewer assigns less value; he/she gets notification message asking him/her to leave comments and suggestions.Table 1 explains criteria in the eight categories, the authorized reviewer for every criterion and the minimum accepted value.

The evaluation criteria:
Retasking and repurposing: Retasking; where the LO is used as it is but in another context for objectives other than the objectives which LO is created for, while in Repurposing; some changes are done to the LO to be used in another context for other objectives.If the LO is qualified for repurposing logically it would be qualified for retaking too.Below are 8 characteristics should be found in LO to can be repurposed.
Evaluation criteria: Rating scale consisting of 5 levels for each element.This category has 6 evaluation points; everyone is evaluated by specific reviewers: • Self-contained: It shouldn't depend on any technical or educational object to work effectively.
(Reviewers; SME, Instructor and ID, Minimum Rating Scale "5"; in this context we focus on the word "any" of Palmer and Richardson (2004) as LO may be unable to work because of depending on very simple external object, so the idol LO should take 5 points to be accepted otherwise reviewer has to mention the needed objects).• Date and time independent: Doesn't depend on special external events.(Reviewers; SME, Instructor and Developer, Minimum Rating Scale "4").• Location independent: Doesn't depend on special location to work effectively (Reviewers; ID, Developer and Designer, Minimum Rating Scale "4").
• Generic: Can be reused in any educational context as is "context free".(Reviewers; SME and Instructor, Minimum Rating Scale "3").
• Differentiated: Can be used for different levels of education (Reviewers; SME, Instructor and ID, Minimum Rating Scale "3").• Modifiable: Has the possibility to be modified according to its new objectives (Reviewers; MSE, Instructor and ID, Minimum Rating Scale "3").

Gender:
The second evaluated aspect is "Gender".As there are essential differences in learning between males and females; from the perspective of performance, participation and outcomes (Collins, Kenway and McLeod (2000)), we are keen to know whether LO is suitable for one gender more than the other or it's suitable for both, (Evaluation Criteria: Radio buttons with three choices; 1. Males, 2. Females 3. Both, Reviewers; Instructor, ID and Student).

Accessibility:
The category "Accessibility" is responsible of determining how it is easy for people even people with disabilities to use the learning object.
We divide the disabilities into 3 types: • People who have sensory or mobility disabilities

Appropriateness:
Evaluation criteria: Rating scale consisting of 5 levels for each element.This category has only 2 evaluation points: • Conformity to its topic (Reviewers: SME, Instructor and ID, Minimum Rating Scale "3").• Suitability to the designed audience (Reviewers; SME, Instructor, ID and Students, Minimum Rating Scale "3") (Rahman and Tech, 2003).
Content quality: Evaluation Criteria: Rating scale consisting of 5 levels for each element.This category has 8 evaluation points: • Overall content quality (Reviewers: all reviewers, Minimum Rating Scale "3")  (Nesbit et al., 2004;Paulsson and Naeve, 2007;Rahman and Tech, 2003) Metadata: Metadata is data about objects.The purpose of metadata is facilitating several processes like; searching using and evaluating LO, it also facilitates sharing and exchange LOs.(IEEE, 2002) and by the way the reusability of learning object is dependent on the quality of its metadata (Garcı´a-Barriocanal et al., 2006) (Evaluation Criteria; rating scale consisting of 5 levels).There is one point only in this aspect; Traceable: Learning object should be well identified by suitable metadata, metadata should include; name, explanation, size, order, example, datatype, knowledge objects, educational objects, knowledge chunks, digital objects and digital educational computer programs (IEEE Review Committee, 2002;Nash, 2005) (Reviewers: All reviewers, Minimum Rating Scale "4").

Motivation:
Evaluation criteria: Rating scale consisting of 5 levels for each element.This category has 5 evaluation points: • Goal achieving: Learning object must meet the goals of students (Reviewers; SME, Instructor and ID, Minimum Rating Scale "4").

RESULTS AND DISCUSSION
Analysis of results: By Evaluating 100 learning object we got some notes about the criteria of evaluation and their relation with the types of learning objects is giving a summary to the results of 100 evaluated  App. Sci. Eng. Technol., 7(12): 2475-2482, 2014 2480 e assigned learning Instructor, ID, Designer and Minimum Rating Scale "2").
Rating scale consisting of 5 levels for each element.This category has 2 learning object: Learning object shouldn't need special software or requirements to be used effectively.
In this point reviewer should evaluate how it is easy and clear for user to use the learning object.The interface should how to interact with the learning All reviewers, Minimum

DISCUSSION
By Evaluating 100 learning object we got some notes about the criteria of evaluation and relation with the types of learning objects.Table 2 is giving a summary to the results of 100 evaluated learning object.Following is analysis to all of evaluation criteria: • Retasking and repurposing: o Self-contained/ (interoperability): 100 LO, we found that all of them are completely self contained as they are essentially created to be used isolated.So all of them got high marks o Date and time independent: learning objects are date and time independent so it's usually expected that this evaluation get high mark, in evaluating the 100 LO, all of highest mark "5" o Location independent: As it's mentioned before for LO to be location independent it should not be tied to a particular place and they were extremely location independent o Generic: This element wouldn't be essential if the learning object is mainly created for speci curriculum of educational stage as it's supposed to be dependent on the curriculum of the previous stage or on other learning objects in a course, however in some cases we found that a learning object in a series can be used isolated or even modified to represented in another context, Most of the 100 evaluate learning object got marks between 4 and 5 Ease of use learning object 3.50 g is analysis to all of (interoperability): By analyzing 100 LO, we found that all of them are completely self contained as they are essentially created to be So all of them got high marks In general most of learning objects are date and time independent so it's usually expected that this evaluation get high mark, in evaluating the 100 LO, all of them got the As it's mentioned before for LO to be location independent it should not be tied to a particular place and they were extremely This element wouldn't be essential if the learning object is mainly created for specific curriculum of educational stage as it's supposed to be dependent on the curriculum of the previous stage or on other learning objects in a course, however in some cases we found that a learning object in a series can be used isolated or even o represented in another context, Most of bject got marks between o Differentiated: As many of the 100 learning objects are mainly target specific education levels so they wouldn't be suitable for different educational levels, it was observed that the learning objects of kids got very low mark as they can't be used for different education levels.We conclude that if the designed audiences are assigned in the description, this evaluation element can be escaped o Modifiable: As learning is presented for evaluation in its final status so it can't be evaluated for technical modification, in LOREM system the capability of LO to be modified is only evaluated from pedagogically perspective.As the topic of any learning object is usually applicable for updating so this evaluation element is accepted to usually get high marks and that exactly what happened for the many of the 100 evaluated LO however some leaning LOs got low marks as their contexts were heritage texts so there were no way to modify them • Gender: After evaluating the 100 learning object, it found that 99 of them are suitable for both of males and females and only one learning object called "Grammar Girl's Quick and Dirty Tips to Better Writing" is more suitable for females than males • Accessibility: o Sensory or mobility disabilities: All of evaluated learning objects are not applicable to "Sensory or mobility disabilities" o Hearing-impaired users: There were 2 learning objects dedicated for hearing-impaired students in the evaluated learning objects, they are called; "Alphabetic for deaf students" and "numbers of deaf students", these 2 LOs got the highest mark "5".Through the evaluations of the chosen learning object, it's found that some learning objects can be suitable for hearing-impaired users even if they aren't mainly created for these category of users, these learning objects can be from the types; animations, assessment, presentation and book.
Other LOs are targeting normal students but can be slightly used by deaf students o Blind or visually impaired users: There is only learning object in the 100 LO is dedicated for blind student, it's called "Teaching the Holy Quran for blind students" took mark "5".Some other learning objects target normal students but have audio description so they would be somewhat useful for blind students This criterion is applicable for all LOs o Ease of use learning object: Results of evaluation for this criterion are differentiated from LO to another one as they are some LOs very easy to be used and navigated while the other learning objects a little bit vague and don't provide user with clear guidelines about using them

CONCLUSION
The main objective of this study is offering a new model as a metric to evaluate the effectiveness of learning objects in order to be reused in other contexts.This research was done though three stages; beginning by searching the current models and analyzing them, followed by assigning the new proposed model with its new criteria of evaluation and finally test the model by evaluating some research samples.
The new proposed model depends on the reviewers' evaluation to the several aspects of learning objects.It classifies reviewers into 3 categories; academic, technical and students.With six types of reviewers; Subject Matter Expert, Instructor, Learning Object Developer, Learning Object Designer, Instructional Designer and student, the authorization for evaluating the several aspects of learning object is differentiated from types of reviewer to another type.
Aspects of evaluation are divided into 8 categories; every category has some internal points; every point has its own evaluation as a ranking from 1:5 and the average of points in every category is represented by stars from 1 to 5. The whole average of all categories is also represented by stars from 1 to 5.

Fig. 1 :
Fig. 1: Hierarchy of evaluation team facilitate the evaluation process for reviewers.Reviewers of this system are divided into three categories: • Academic Group (Subject Expert Matter "SME" and Instructor) • Technical Group (Instructional Designer "ID", LO Developer and LO Designer) • Students group: This group consists of students only (Fig. 1)

Table 1 :
Evaluation criteria in LOREM • Deaf or hearing-impaired users • Blind or visually impaired users Evaluation criteria: Reviewers will determine if the learning object is accessible for people with each type

to blind or visually impaired users:
At this point reviewer evaluate how learning object is providing access to visual aspects of learning technologies to be accessible for blind or visually impaired users.LO should: o Has text descriptions (alternative text or alt-text) to all static images (e.g., pictures, logos, charts, links, other graphics) so the text can then be read by a screen reader or output to a Braille display o Has utilization of the "longdesc" attribute for images that have useful content and require more lengthy descriptions o Has an audio description track for multimedia, describing visual aspects of the content.

Table 2
By revising the comments on this point in the results of the 100 learning objects we found that some of them got low marks and have been described as a very difficult to the designed audience, while other got lower marks.Logically this point should be evaluated taking into account the designed audiences.This criterion is applicable for all LOs o Feedback: By evaluating the 100 learning object, it's found that some of them enable student to answer question cover the lesson and check the answers while other don't give any feedback, so the mark of this criterion ranking from 1:5.This criterion can be applicable for all LOs o Multimedia and graphic usage: The results of evaluation is ranking from low to high marks while there are some learning objects are inapplicable to this criterion, like; image, figure, graph, table and book o Narration: The mark of this criterion is ranking from low to high marks depends on the method of narration in the learning object.Some of learning objects types are inapplicable to this criterion like; assessments, games, puzzles, image, exercise, diagram figure, graph, table, text and exam • Usability: o Easy of playing/viewing learning object: This criterion got high marks in all of the 100 evaluated LOs as the required programs for these LOs to work effectively are easily downloaded and played.