Interactive Videos incorrectly receive 0s and/or score is changed to 0 later after receiving a grade

Submitted by llungingllama on Tue, 09/26/2017 - 21:09
Forums: 
Bugs in content types

Hi,

I have been having a number of students this semester complain about completing videos before they are due and receiving zeroes in the moodle gradebook.  In some cases it appears that they have completed the video and later lose the score.  I have asked my student which browsers they are using and it appears to be a cross-browser problem.

Moodle 3.1.1
H5P 1.4
Interactive Video 1.15.0

Here are the results for one student:

Grade Report -- 0
But they answered the questions correctly:

Questions Answers

These were the completion settings:
Completion settings

 

The course logs confirm they watched it days before:
Course Logs 1


They did however view it again more recently
Course Logs 2


They also complain of submitting answers but no answer being recorded?

I did not have this problem in previous semesters. The change is that I updated H5P/Interactive Video versions over the summer.

Content types: 
Interactive Video
thomasmars's picture

thomasmars

Wed, 09/27/2017 - 13:45

Hi,
I'll try to explain how Interactive Video scoring currently works:

  • First to generate a grade the user must answer the interactions in the video.
  • Second to make this grade stored in the gradebook and for the report the user must "complete" the video. This means that they must watch the last few seconds of the video.
  • Third, if you take Interactive Video at a later time and complete it(watch to the end), the previous grade will be overwritten.

So for the users that have gotten a grade first, then their grade has been removed or changed to 0, this could happen when:

  • They first finish the Interactive Video by taking some interactions and then watching the whole video, which would store the results in the gradebook and in the report.
  • They would then watch the video again, answering the interactions wrong, or skipping the interactions completely, and then finishing the video. This would replace their old grade in the gradebook with a new one that has 0 as the score.

I have no good explanation for why you would get a report that is displaying correct answers for a users and 0 score in the gradebook. Perhaps if you could link an image of the complete report we would be able to make some sense out of it. They should be consistent with eachother. Do you have some steps I can follow to reproduce this consistently ? If so I could look closer into this, for now I am not able to reproduce it.

I'm aware that this behaviour in Interactive Video is far from optimal. I would be happy to take feedback if you have suggestions for how we can improve the process of when answers/grades should be completed and stored, overwritten, etc. that will be reasonable in different contexts of Interactive Video.

I'm sorry that this is making your experience of H5P troublesome. Thanks for the detailed description of this issue,
Thomas

llungingllama

Wed, 01/17/2018 - 10:40

Thank you for your reply. Apologies for my late followup. I thought I had responded.

I was reviewing the issue as it continues to be a problem for the upcoming semester.

I was never able to confirm exactly why sometimes students would answer questions correctly and receive empty marks. I have since launched a fresh server and will see if persists this semester.

As to when grades should be completed/stored/overwritten, last semester I was forced to throw out the moodle video grades and simply give my students 100%s because of the overwriting.

It would be one thing if it was possible to retake the quiz for a higher grade, although that has the problem of students being able to click through mindlessly to get the answers, and then go back and enter whatever was spit out the first time.  But I suppose at least it forces them to interact with the questions. I just wouldn't be able to weight the scores very much.

However, having them able to take their 100%s and later convert them into 0%s accidentally means the grades cannot be used in any gradebook calculation.  I find it especially bad because one of the main reasons for using online lectures is that my students can review the lectures multiple times, in class while they are doing problems, before tests, etc.  It is usually the students who struggle who find this the most helpful. But it is unconscionable to have my struggling students risk zeroing out their grades every time they go back and watch a snippet of lecture.  There is no way I can use the grades in that case.

I would prefer that the score on the first attempt was used and left permanently marked.  Or that subsequent attempts would be marked as "review/practice" with the first score unaffected.

I don't suppose there is any way at present to make the initial scores persistent?

Thanks you very much for you time and your previous answer.

thomasmars's picture

thomasmars

Thu, 01/18/2018 - 14:45

Hi,

Good news is there is already a way to get all results from H5Ps through xAPI events, which H5P facilitates, this way you can see the full history of all scores that a student has received, together with much richer data than just the scores.

The bad news is that this is not available out of the box for H5P. You need to set up an LRS (Learning Record Store) in order to store these values and see them in a structured way. You can read more about how to integrate with LRS in the "Analyzing results and answers" guide.

In the meanwhile we're quite aware that the built-in functionality must be improved for Interactive Video scoring and we'll look into ways of improving this in a reliable way for all contexts that IV can be used in. A potential solution would be an actual "submit" button at the end of the video. For now I hope you'll bear with us until we solve this in a good way.

Thanks again for the feedback. Best regards,
Thomas

akaskens

Mon, 01/28/2019 - 17:40

We have the same problem, but not just with interactive video - this is how the grading behaves for all the H5P activitiy types. If a learner does the activity again, the scrore in the gradebook is overwritten. This makes the gradebook an unreliable source of information for the course instructors. A solution would be to allow a setting (in Moodle) that would allow course developers to select either "best score" or " most recent score". I've made this suggestion in other forum topics here. It would be great if someone could make the request in gitub. thanks! other than that, we love H5P!!

 

dmcwo

Fri, 02/22/2019 - 00:10

+1 for "best score" or "most recent score"

llungingllama

Fri, 01/19/2018 - 23:08

A submit button would be excellent!  I appreciate the consideration.

H5P people has been an excellent tool and well-worth the wrangle: I would say this is my last major hurdle to having it do what I need for assessment purposes.

Thank you for the reply and for the tip on the LRSs, I will definitely see if I can implement a solution that way.

adr1957

Fri, 03/09/2018 - 02:27

Hello,

I am experiencing the same problem on moodle 3.1.1 and after some testing I would like to add some comments.

First of all I could obviously reproduce what Thomas says, i.e. that replaying the video will erase previous scores, so by inspecting the gradebook I could monitor any change taking place.

Erasing answers wouldn't be much of a problem if students were at least given the possibility to provide new answers, the real problem in my opinion is that this is not possible with some types of questions.

My starting situation was with 2 multiple choice and 4 single choice quizzes in one video. This is what I have done:

I played it first time providing all correct answers and got correct score (recorded as printout)

Then I replayed and bypassed questions simply clicking on forward button and everything was overwritten es expected (recorded by printout). Notably, at this point the activity was marked as "completed, passing grade not achieved". This might be relevant if one wanted to set a conditional step on completing the activity instead of needing passing grades.

Then I replayed thinking of providing new answers and discovered that this was only possible for multiple choice questions, while the single choice were taken as completed and were not asked for a second time. Using the verify buttons didn't let the original answers to be reconsidered for recalculating the score.

At thispoint I seemed to be stuck, as the real problem is to be able to provide new answers and force the video to recalculate the score.

I solved this incidentally when, while editing the video and changing a a little detail, such as the time interval set at the beginning of question1, I discovered that after saving it  I got a warning message "the content has changed since last time you viewed the video, SO YOU WILL HAVE TO START OVER"....

I tried to replay and to my wonder it was totally blank: I could provide new correct answers and be graded correctly as if it were the first time.

Conclusion: the real problem is to be able to empty whatever container stores the answers every time the video id played again. It is not really convenient to students, but at least they don't get stuck in a never ending circle.

There must be a way of emptying it at will, since the simple change of a little parameter plus a new save did it, quite serendipitously I should say.

At the moment my temporary solution is to re-edit the movie with meaningless changes and re-save it. The ones who have completed it successfully will keep their grades, that will be reset only upon reopening the video page by those who are not happy with their grades.

My suggestion at this point would be to implement a "reset" button erasing all answers, or incorporating something that changes automatically every time the page is opened (a time-stamp or the like?) and forces the emptying of the cached answers.

I will be looking forward these improvements if possible, as the remaining part of the software is really powerful and very appreciated by my students

Regards

Adriano

BV52's picture

BV52

Fri, 03/09/2018 - 07:21

Hi Adriano,

Thank you for the thorough testing and explanation. We will surely consider this in future updates to the content.

-BV52

adr1957

Sun, 04/08/2018 - 23:04

Hello, I was wondering when would the updates be available and whether is there a specific channel to forward requests for new features. It would be great for example if students had the possibility to choose between running the video without quizzes a first time and then play it back taking the quizzes. Would it be possible to get requests like this to passed one to the developers team? Or where else?

Thanks

Regards

Adriano

BV52's picture

BV52

Mon, 04/09/2018 - 07:17

Hi Adriano,

The update to IV (submit button for activities) is now in the testing phase. If all goes well it should be included in the next release hopefully within the month.

We’re now working on something called the H5P supporter program allowing the H5P community to vote for and fund the top voted H5P features. Also there are developers in the community who every now and then works on a feature they find interesting or useful. In order for your feature request to attract as much interest as possible make sure it follows the below guidelines:

 

  1. It is clear from every perspective how the feature will work. We recommend describing the feature with one or more user stories, for instance “As an author I want it to be possible to pick between different effects for the check answer animation so that the learners will see a variety of effects and also I can adapt the effects to my target audience(I’ll be using pink unicorns which works really well for both my target audience which are 4 year old girls and venture capitalists)”

  2. If the feature can be illustrated with images or videos it always helps

  3. Make it clear what content types this is relevant for, and or if this is a new content type

 

Make sure you post the feature in the Feature Request forum.

 

-BV52

ehcourville

Mon, 10/15/2018 - 15:44

The same issue is happening for me.  The strange part is that some students are graded correctly, and some are not.  This is still occuring even with the submit button addition.  

Attachments: 
Image icon screenshot-learn.granite.edu-2018.10.15-09-41-57.png
Image icon screenshot-learn.granite.edu-2018.10.15-09-42-40.png

adr1957

Mon, 10/15/2018 - 21:28

Hello ehcourville,

I quote you 100%. after the upgrade release everything seemed to be smooth, while the problem was back after a few videos. At this point it might well be concurrent causes, i.e the question type coupled to replaying  the video several times to grab all the details. I also noticed that sometimes the final submit screen is there by default at the very last second, sometimes it isn't. I wonder if an added submit star button is somehow different from the spontaneously apperaing one.

I look forward to the course due in a fortnight with fear, as I will have to change my policy and conditional steps if this feature doesn't work again.

Thanks

Adriano

ajmendoz

Thu, 01/24/2019 - 16:54

I have experienced a similar issue with Interactive Video scores changing or not being scored correctly. There appears an error in the way Multiple Choice questions are scored  when an Interactive Video is re-watched. I have written a bug report document  with screenshots describing various conditions and testing scenarios under which this occurs. H5P files referenced in the document are attached below. I'll offer a condensed version here.

  • Platform: Moodle 3.4

  • Desktop

  • Browsers: Chrome, Safari

  • H5P plugin version 1.14

  • H5P content types: Interactive Video, Multiple Choice (both updated on 01/17/2019)

    • Interactive Video - (1.20.3), previous versions (1.17.5)

    • Multiple Choice - (1.13.2), previous versions (1.12.0), (1.10.5)

 

Summary

The following statements apply to Interactive Video ver. 1.20.3.

 

Multiple Choice questions will be scored as 0 on subsequent Interactive Video attempts under either of the following conditions:

  • The student clicks on the “Submit Answers” button without re-watching the video

  • The student re-watches the video, and clicks only the “Check” and “Continue” buttons, without clicking on any answers

 

Multiple Choice questions will be scored correctly on subsequent Interactive Video attempts only if the student does all of the following:

  • Does not click on the “Submit Answers” button when it shows at the beginning of the subsequent attempt

  • Clicks “Play” to re-watch the video

  • On Multiple Choice questions:

    • Clicks on all selected answers to deselect them

    • Clicks on answers to select them again

    • Clicks on “Check” and “Continue” buttons

  • On the Submit screen after the attempt, clicks on the “Submit Answers” button

 

The grade for the Interactive Video will remain unchanged if a student opens an Interactive Video after the first completed attempt, but does not click on anything and just goes back to the course home page.

 

The major issue is that when an Interactive Video is re-watched, previous answers for Multiple Choice questions are already selected. When a student clicks on “Check” the automatic feedback for the question tells them that they answered correctly. However, the score that is recorded for that question does not match the automatic feedback given. This is extremely confusing for students who want to review the video, having already answered the questions correctly, and receive a changed grade without changing any answers.

 

Suggested solution

If a student sees that their previous correct answers are already selected, and they click “Check” and receive feedback that they (again) answered the question correctly, then the correct score matching the feedback should be recorded for that question.

I hope that this issue gets submitted to the bug report queue soon, as it is potentially affecting many instructors at my institution (NC State University).

Attachments: 
Binary Data updated-interactive-video-1-733.h5p
Binary Data activity-1-orientation-715.h5p
BV52's picture

BV52

Fri, 01/25/2019 - 09:35

Hi ajmendoz,

Thank you for the very detailed report. I filed a bug report and you can follow the progress here.

-BV52

ajmendoz

Fri, 08/30/2019 - 19:32

Has there been any interest in fixing this bug? It continues to adversely affect many instructors and students.

ajmendoz

BV52's picture

BV52

Mon, 09/02/2019 - 18:35

Hi ajmendoz,

I'm afraid I do not have any updates for this.

-BV52

ajmendoz

Wed, 09/09/2020 - 01:13

Instructors and students are completely frustrated with the way scores/grades are changed when re-watching an Interactive Video. Instructors are giving up on H5P altogether because of this issue. Do you have any information on how re-watching videos affects scoring/grading with the H5P integration in Moodle 3.9? I am hoping it resolves these issues.

Realia

Mon, 04/03/2023 - 16:46

Hi !

I am having the same problem on WP 6.1 + IV (last version) + H5P plugin (last version).

The student score is recorded as it should (10/12) the first time, but then if the student watches the video again, the recorded score falls down to 0/12, allthough the displayed score on the summary activity (which can't be retried) is still 10/12. Here are some screenshots :

Score displayed for the summary activity :

Recorded final score

It has changed from 10/12 to 1/12 and then 0/1 when the student watched the video again

 

Is there something to do to prevent this from happening ?

Thanks for any advice or help.

Isabelle

Attachments: 
Image icon Capture d’écran 2023-04-03 à 13.41.05.png
Image icon Capture d’écran 2023-04-03 à 13.41.22.png
BV52's picture

BV52

Mon, 04/03/2023 - 18:37

Hi Isabelle,

As far as I know there are no work arounds. I will follow up with the dev team regarding the bug report.

-BV

Realia

Thu, 04/06/2023 - 10:17

Thanks BV ! I hope that this can be solved :)