Author Topic: Memory error after auto segmentation  (Read 3104 times)

Keiland

  • Newbie
  • *
  • Karma: +0/-0
    • View Profile
Memory error after auto segmentation
« on: January 19, 2017, 09:32:23 PM »
So running the code on the test data set (H15_M18_S6) from the google drive, and ran it without error until autosegment2, where I received the error below:

Requested 459x732x5392 (13.5GB) array exceeds maximum array size preference.

it seems fairly self explanatory, except I cannot figure out which array it is referring to. I'm guessing that during the memory allocation phase, it is the "data" on line 56:

data =
uint8(nan(vidObj.alignedHeight(vidObj.selectedAlignment),vidObj.alignedWidth(vidObj.selectedAlignment),numFramesDown));


Any ideas on what's tricking the memory allocation so the file can fit into memory? or maybe minimum memory requirements? I don't need to use the batch analysis on the dataset above do I?.

Thanks!







Daniel Aharoni

  • Administrator
  • Full Member
  • *****
  • Karma: +15/-0
    • View Profile
Re: Memory error after auto segmentation
« Reply #1 on: January 20, 2017, 04:12:47 AM »
Hi kwcooper,
autosegment2 is the only function of our analysis code that loads the entire recording session into memory (RAM) and stores it in the variable 'data'. Generally we use the downsample option to temporally downsample our recordings to around 5 frames per second when running autosegment2. 5FPS is more than enough time points for the algorithm to run properly.

No, you shouldn't use the batch analysis when running on a single data set.

Keiland

  • Newbie
  • *
  • Karma: +0/-0
    • View Profile
Re: Memory error after auto segmentation
« Reply #2 on: January 25, 2017, 09:40:00 PM »
I ran it with 5 and it's still too large, what do you think the lower bounds would be for data retention?

Daniel Aharoni

  • Administrator
  • Full Member
  • *****
  • Karma: +15/-0
    • View Profile
Re: Memory error after auto segmentation
« Reply #3 on: January 25, 2017, 09:59:32 PM »
How much RAM do you have in your computer?

Keiland

  • Newbie
  • *
  • Karma: +0/-0
    • View Profile
Re: Memory error after auto segmentation
« Reply #4 on: January 26, 2017, 08:55:42 PM »
The computer I first ran it on was 8Gb, I set it up on the 32Gb rig which should have no problem. Any thoughts on running it with access to a smaller memory size if needed for whatever reason?

Daniel Aharoni

  • Administrator
  • Full Member
  • *****
  • Karma: +15/-0
    • View Profile
Re: Memory error after auto segmentation
« Reply #5 on: January 26, 2017, 09:51:26 PM »
You could consider a couple options:
1) Spatially downsample your data. You can use imresize() to do this. Generally, downsampling by 2x both horizontally and vertically will be ok.
2) Run the analysis on a smaller time window. You could run it on the first half of the recording then run it again on the second half. This isn't ideal but should work pretty well.
3) You could run the analysis on multiple smaller spatial regions then stitch up the results.