dropped frames

[Stephan]

Hi Daniel,

I observe dropped frames at regular intervals (when running at 60fps, there one or more dropped frames every ~75 frames as measured from the sync output; when running at 30fps, there are  several dropped frames every ~148 frames).
It comes at very regular intervals. Have you ever encounter that? Any idea what is going on?

Although there are definitely dropped frames, the software does not detect them and the "dropped frame counter" remains at zero.

Thanks,
Stephan

PS: in the current implementation I did not connected a LED to the camera board. I am running the system with an externally connected LED.

[Daniel Aharoni]

Hi Stephan,
We actually have never used the Sync output outside of some initial testing of the function. I will hook up a system right now and test things out. Can you look at timestamp.dat and see if the timestamps of the frame acquisition match what you see from the sync output.  Feel free to email me (dbaharoni@gmail.com) the timestamp file and I will take a look at it.

I am pretty sure one of the following is happening:

• The part of the firmware code that toggles the sync output has a bug in it. I think this is unlikely but I will look into it.
• The data stream over USB is getting backed up (due to your computer/USB drivers not the DAQ PCB). This could cause the USB host controller on the DAQ PCB to skip a frame acquisition. This would cause the sync output to show a frame was skipped but the DAQ software would just think a frame was never sent over USB.

[Daniel Aharoni]

Hi again Stephen,
So I just tested out the sync output running the Miniscope at 30FPS and 60FPS and everything looks to be functioning properly. I ran it for a few minutes and recorded the sync output on my oscilloscope (here is a screen grab, https://dl.dropboxusercontent.com/u/42465128/IMG_20161019_160111.jpg).

I am pretty sure the source of your issue is due to some slow down of data transmission to your computer. Either it is due to what I said in my last post, that your USB isn't keeping up with the data stream (this is a bit weird since we are sending about 10MBps where as USB3.0 should be able to handle ~500MBps). It could also be possible that your computer is having an issue with writing the imaging data to its harddrive fast enough. When recording, look at the numbers displayed in the middle right of the DAQ software and make sure 'Write Speed fps' is much larger than the actual FPS.

Please update me on any progress. Thanks!

[Stephan]

Hi Daniel,

The sync output does indeed work well. And the issue seems to be related to the writing process.
I changed computer and now use a very recent one (Dell Inspiron, 3 months old) equipped with a SSD drive, and running windows 7.
I run a ~1800 frames movie at 30fps, and observed that the number of missing frames has been reduced by a lot, but unfortunately there are still some (4 in total). 

The writing speed has dropped a few times to 120fps, as indicated by the software.
As mentioned in my previous post, the counter of dropped frames remained at zero, although it should be at 4.
The time stamps data are showing when the problem occurred (calculating the difference between successive time stamps and plotting these values, you see peaks corresponding to the times one or few frames were lost).

Not sure what to try next because this machine is pretty good. Maybe try windows 10? I will check with a Windows 10 laptop, and let you know.

Stephan

PS: Here is the way I am measuring this effect:
In this set-up, no LED is connected to the CMOS board.
I have an arduino which is receiving the Sync output of the camera. The arduino is programmed to turn on a LED for ~10ms once every 2 rising edge of the Sync Signal (once every 4 frames). The LED is directed at the camera. That way, I can check whether and when a frame is lost looking at the movie (I should always see 3 dark frames followed by 1 bright one unless a frame has been lost).
The light of the LED is also reaching a photodiode. Its output, as well as the Sync Output are recorded on a NI-DAQ during the movie acquisition. This is used to make sure the Sync Output is fine and the LED indeed turned on properly.

[Stephan]

Tested with a Dell Laptop running windows 10. It was the worst: 60 frames lost out of 1800.

So far, the best performing machine I tested was a recently purchased Dell Inspiron desktop equipped with a SSD drive: 1 to 4 frames lost out of 1800 acquired at 30fps.

I was originally using the DAQ 2.0. I just switched to the latest model, but that did not make a difference.

[Daniel Aharoni]

Hi Stephan,
Thanks for the update with more information and details.

After thinking an bit more about it, the only explanation for the dropped/missing frames you are seeing is the USB on your computer not being able to keep up with the data streaming out of the DAQ PCB. The frame sync output from the DAQ PCB toggles whenever and 'End of Frame' event occurs inside the DAQ PCB. This 'End of Frame' event is almost completely decoupled from what is happening on the computer side of things except for the case where the computer's USB gets backlogged and can't accept the next USB packet being sent out of the DAQ PCB. Due to the way the imaging data is sent over USB (using USB Video Class (UVC) standards), if a packet is lost, the transfer of the entire frame does not get completed by the DAQ PCB.

The implementation of detecting dropped frames in the DAQ software is looking for frames dropped once the frame has been received by the computer, not frames dropped in the transmission over USB. The DAQ software has a large frame buffer implemented in RAM so dropped frames really never happen once the frame has been acquired by the computer.

We usually record using higher end MacBook Pros running Bootcamp Windows 7 or 10 or average powered desktops. Both these options generally do not have issues keeping up the the USB data rate. A few other things to avoid is running through a USB hub or recording using an encrypted computer.

In the end, the frame sync output and the recorded timestamps in timestamp.dat should still match up with the data stored to the harddrive and a missing frame every so often shouldn't be a problem in regards to analysis. That being said, when I have time I will try to write up a new implementation of dropped frames that detects when a frame is dropped in the process of being transmitted over USB.

[Jesper.vernooij]

Hi Stephan/Daniel,

I have a similar problem when imaging and recording behavior simultaneously at 30 fps both. The laptop We're using is a brand new Dell inspiron 7559 (5400 rpm drive, no ssd). All ports are USB3.

During recording error messages pop up in the dialog box mentioning both scope and behavior frame buffer errors after about 5 seconds of imaging, which looking at the resulting videos yields a mouse movement jump spanning roughly 5 seconds in the worst case to not noticeable in the best case. I've noticed that write speed for both Behavior and Scope fluctuates rapidly between 1 and 200. The fps fluctuates around 28 fps and shoots up randomly occasionally to over 2000, which seems to coincide with the frame buffer errors, but not always.

Any clue as to what might be the issue?

Thanks!

Jesper

[Daniel Aharoni]

Hi Jesper,
The problem you are having is a bit different than what Stephan was describing. Your problem is due to your hard drive not being able to keep up with the acquisition of video data (both scope and behavior). The DAQ software streams and records uncompressed video data to the computer's hard drive and this data stream can exceed 20MBps depending on the resolution of your behavioral camera. While the DAQ software does have a frame buffer to handle short slowdowns in hard drive write speeds, it appears your computer still cannot handle the bandwidth of data being written to the drive.
My suggestions would be to first make sure your computer is not encrypted and nothing else is actively reading/writing to the hard drive. Run the DAQ software without connecting to a behavioral camera and see if it can keep up with the data stream. If you do not run into buffer errors then try connecting a lower resolution behavioral camera and limit its ROI to as small as possible.

If this doesn't resolve the issue switch over to a computer with a quicker hard drive.

[Jesper.vernooij]

Hi Daniel,

Thank you for the detailed information! I have checked to see if reducing the Behavior ROI has effect on the frame buffer errors and it does indeed (smaller ROI == less errors). Unfortunately our setup has some limits regarding the height of the camera, so getting the ROI smaller than it is now is not an option without getting a new camera. The write speed is fine when only imaging neurons, but with the behavior camera (no matter how small the ROI) the hard drive can't manage.

I am thinking of upgrading the computer with an m2.2280 sata SSD and put the savefile folder on that partition, which would increase my write speed to 510 mb/s. I'll let you know if it fixes the problem!

Jesper

[Daniel Aharoni]

It might be the case that just the act of writing to two different video files, one for behavior and one for imaging, could slow down the overall write speed enough to cause buffer errors, despite the actual size of each written frame. This would be due to the two files sitting in different sector locations on the HDD, causing the writing head to scan back and forth between sectors often. An SSD would improve this significantly since they are much quicker at nonconsecutive sector writes.

[Jesper.vernooij]

Hi Daniel, We have just ordered an crucial SSD for the laptop. I will make an update once I have recorded data with the new drive! Thank you for the suggestion.