<div dir="ltr"><div>Hi Obaid,</div><div>Thanks for your report. So your volume and your projections should use 10 GB. One thing I recently noticed while working on CUDA 12 compatibility is that the forward projection code uses layered texture and actually doubles the memory of the volume (for scalars, adds one component for a volume of vectors)<br></div><div><a href="https://github.com/RTKConsortium/RTK/blob/master/src/rtkCudaForwardProjectionImageFilter.cu#L203">https://github.com/RTKConsortium/RTK/blob/master/src/rtkCudaForwardProjectionImageFilter.cu#L203</a></div><div><a href="https://github.com/RTKConsortium/RTK/blob/master/src/rtkCudaUtilities.cu#L171">https://github.com/RTKConsortium/RTK/blob/master/src/rtkCudaUtilities.cu#L171</a></div><div>That amounts to 18.8 GB which is still much less than 80 GB. I am probably missing something but I don't know what. I would suggest to add</div><div><div><span style="font-family:monospace"> os.system('<span class="gmail-il">nvidia</span>-<span class="gmail-il">smi</span>')</span></div></div><div>in your python code to check what is the memory status (before ForwardProj.Update()).</div><div>To be honest, the projection images is a minor portion of the volume so the streaming will probably not help, sorry.<br></div><div>Thanks for reporting any additional info on the mailing list,</div><div>Simon<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 26, 2024 at 1:36 PM Rahman, Obaid <<a href="mailto:rahmano@ornl.gov">rahmano@ornl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Hi Simon,<div><br></div><div>Thank you for the suggestion, I will definitely try that.</div><div>Please find attached the output of nvidia-smi.</div><div>Thanks.</div><div><br></div><div>Best,</div><div>Obaid</div><div><br></div><div><img src="cid:ii_18f28682738b1ebe3251" alt="Screenshot 2024-04-26 at 7.32.11 AM.png" width="630"></div><div><br id="m_7748307742549191118lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Apr 26, 2024, at 3:03 AM, Simon Rit <<a href="mailto:simon.rit@creatis.insa-lyon.fr" target="_blank">simon.rit@creatis.insa-lyon.fr</a>> wrote:</div><br><div><div dir="ltr"><div>Hi Obaid,</div><div>If you remove the "ForwardProj.Update()" line and connect instead a <a href="https://urldefense.us/v2/url?u=https-3A__itk.org_Doxygen_html_classitk-5F1-5F1StreamingImageFilter.html&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=irXqD18u-c9Im_eQprxOMZNcAsAeuIrOn7RWNk8LABzPsfkFpG_WuH03l9pRCH7S&s=wSLRSuiTcAhTsEGggRb-clAgYNx67JL22y315GEq6TI&e=" target="_blank">StreamingImageFilter</a>, your projections can be processed piece by piece.</div><div>I'm surprised by your 80 GB memory, can you display the output of nvidia-smi?</div><div>Simon<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 25, 2024 at 11:56 AM Rahman, Obaid <<a href="mailto:rahmano@ornl.gov" target="_blank">rahmano@ornl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Thanks, Nils for the swift response.<div>I have actually 80 GB memory per GPU which should be plenty.</div><div>The image is 8.65 GB (read as float32), and the projection size is 1.5 GB <span>(read as float32).</span></div><div><span>Also, </span>CudaBackProjectionImageFilter works fine for the given projection and image sizes.</div><div><span><br></span></div><div>I have tried other forward projectors (Astra) and they work fine with the given memory.</div><div>I was wondering if my code was suboptimal memory-wise.</div><div>Thanks!</div><div><br></div><div>Best,</div><div>Obaid<br id="m_7748307742549191118m_-2575662995419800001lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Apr 25, 2024, at 3:54 AM, krah <<a href="mailto:nils.krah@creatis.insa-lyon.fr" target="_blank">nils.krah@creatis.insa-lyon.fr</a>> wrote:</div><br><div><font style="font-size:14px;font-family:sans-serif">
<div><font style="font-family:sans-serif;font-size:14px"><font style="font-family:sans-serif">Hey,</font></font></div><div><font style="font-family:sans-serif">without knowing further details nor having verified the code, I guess your image is just too large to fit into the memory of your GPU.</font></div><br><div>Naïvely, I would calculate the size in GBytes of your image and compare with the memory of your GPU. Increasing the spacing means using fewer pixels = less memory. On top of that, I guess there are the projections which require space on the GPU.</div><div></div><div>If you need a small spacing, maybe you can reconstruct in chunks along the rotation axis and merge the chunks later?</div><div>Or the RTK experts know other tricks ...</div><div></div><div>just my non-expert two-cents ...</div><div>Nils</div><div></div><div>On Apr 25 2024, at 3:21 am, Rahman, Obaid <<a href="mailto:rahmano@ornl.gov" target="_blank">rahmano@ornl.gov</a>> wrote:</div><blockquote class="gmail_quote"><div><div>Hello all,</div><div><br></div><div>I am getting “out of memory” error when I run CudaForwardProjectionImageFilter.</div><div>Please refer to the attached screenshot.</div><div><br></div><div>These are the array sizes with which I get the error:</div><div><strong>Projection size: (slice/row, view, column)=(1456,145,1840)</strong></div><div><strong>Recon size: (z, y, x)=(1264, 1356, 1356)</strong></div><div><br></div><div>When I increase the voxel size to twice i.e. <strong>Recon size: (z, y, x)=(632, 678, 678)</strong>, it works fine.</div><div>Maybe I’m not being very efficient with the memory?</div><div><br></div><div>If anyone has suggestion to make it more memory efficient, please let me know.</div><div>Thanks.</div><div><br></div><div>Best,</div><div>Obaid</div><div><br></div><div><strong><u>Here’s what I’m doing:</u></strong></div><div><div><div><br></div><div><span style="color:rgb(180,237,255)">def rtk_projection(recon_arrar, proj_params, miscalib, vol_params, return_itk=False, return_RVC=True):</span></div><div><span style="color:rgb(180,237,255)"> '''</span></div><div><span style="color:rgb(180,237,255)"> inputs:</span></div><div><span style="color:rgb(180,237,255)"> recon_arrar: image numpy array (ZXY format is expected)</span></div><div><span style="color:rgb(180,237,255)"> proj_params: projection parameter dictionary</span></div><div><span style="color:rgb(180,237,255)"> miscalib: miscalibation parameter dictionary</span></div><div><span style="color:rgb(180,237,255)"> vol_params: image volume parameter dictionary</span></div><div><span style="color:rgb(180,237,255)"> return_itk: True if itk image is expected, False if a numpy array is expected</span></div><div><span style="color:rgb(180,237,255)"> output:</span></div><div><span style="color:rgb(180,237,255)"> itk image (c,r,v format internally) or image numpy array (r,v,c format) of projection data</span></div><div><span style="color:rgb(180,237,255)"> '''</span></div><div><span style="color:rgb(180,237,255)"> ImageType = itk.Image[itk.F,3]; CudaImageType = itk.CudaImage[itk.F,3]</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # Convert the recon to itk format</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><span style="color:rgb(180,237,255)"> # proj_data is in (z,x,y); rtk requires numpy array in (x,z,y)</span></div><div><span style="color:rgb(180,237,255)"> recon_arrar = np.transpose(recon_arrar, [1,0,2]).astype('float32') # now in (x,z,y)</span></div><div><span style="color:rgb(180,237,255)"> recon_shape = recon_arrar.shape</span></div><div><span style="color:rgb(180,237,255)"> recon_arrar = itk.GetImageFromArray(recon_arrar) # internally img has format (y,z,x)</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # Center the image around 0 which is the default center of rotation</span></div><div><span style="color:rgb(180,237,255)"> recon_arrar.SetOrigin([-0.5*(recon_arrar.GetLargestPossibleRegion().GetSize()[0]-1)*recon_arrar.GetSpacing()[0],</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(recon_arrar.GetLargestPossibleRegion().GetSize()[1]-1)*recon_arrar.GetSpacing()[1],</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(recon_arrar.GetLargestPossibleRegion().GetSize()[2]-1)*recon_arrar.GetSpacing()[2]])</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # Graft the projections to an itk::CudaImage</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><span style="color:rgb(180,237,255)"> vol_xy = float(vol_params['vox_xy']); vol_z = float(vol_params['vox_z'])</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon = CudaImageType.New()# img will have format crv</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetPixelContainer(recon_arrar.GetPixelContainer()) # img has format crv</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetLargestPossibleRegion(recon_arrar.GetLargestPossibleRegion())</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetBufferedRegion(recon_arrar.GetBufferedRegion())</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetRequestedRegion(recon_arrar.GetRequestedRegion())</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetSpacing([vol_xy, vol_z, vol_xy]) # spacing for xzy</span></div><div><span style="color:rgb(180,237,255)"> rtk_recon.SetOrigin([-0.5*(recon_shape[2]-1)*vol_xy, # origin for x direction</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(recon_shape[1]-1)*vol_z, # origin for z direction</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(recon_shape[0]-1)*vol_xy]) # origin for y direction</span></div><div><span style="color:rgb(180,237,255)"> # print('GPU recon:', rtk_recon)</span></div><div><span style="color:rgb(180,237,255)"> del recon_arrar</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # Defines the RTK geometry object</span></div><div><span style="color:rgb(180,237,255)"> geometry = rtk.ThreeDCircularProjectionGeometry.New()</span></div><div><span style="color:rgb(180,237,255)"> for i in range(len(proj_params['angles'])):</span></div><div><span style="color:rgb(180,237,255)"> angle_deg = proj_params['angles'][i]*180/np.pi # angle in degree</span></div><div><span style="color:rgb(180,237,255)"> geometry.AddProjection(proj_params['cone_params']['src_orig'],</span></div><div><span style="color:rgb(180,237,255)"> proj_params['cone_params']['src_orig']+proj_params['cone_params']['orig_det'],</span></div><div><span style="color:rgb(180,237,255)"> angle_deg, miscalib['delta_u'], miscalib['delta_v'])</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # define some parameters (to make code more readable)</span></div><div><span style="color:rgb(180,237,255)"> det_pix_x = proj_params['cone_params']['pix_x'] # row direction pixel size</span></div><div><span style="color:rgb(180,237,255)"> det_pix_y = proj_params['cone_params']['pix_y'] # channel direction pixel size</span></div><div><span style="color:rgb(180,237,255)"> proj_shape = [int(proj_params['dims'][0]), int(proj_params['dims'][1]), int(proj_params['dims'][2])] # r,v,c</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # define a zero projection (GPU)</span></div><div><span style="color:rgb(180,237,255)"> constantImageSource = rtk.ConstantImageSource[CudaImageType].New()</span></div><div><span style="color:rgb(180,237,255)"> constantImageSource.SetSpacing( [det_pix_y, det_pix_x, 1.] ) # c,r,v</span></div><div><span style="color:rgb(180,237,255)"> constantImageSource.SetSize( [proj_shape[2], proj_shape[0], proj_shape[1]] ) # c,r,v</span></div><div><span style="color:rgb(180,237,255)"> constantImageSource.SetOrigin([ -0.5*(proj_shape[2]-1)*det_pix_y, # origin for channel direction</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(proj_shape[0]-1)*det_pix_x, # origin for row direction</span></div><div><span style="color:rgb(180,237,255)"> -0.5*(proj_shape[1]-1)*1]) # origin for view direction (will be ignored anyway)</span></div><div><span style="color:rgb(180,237,255)"> constantImageSource.SetConstant(0.)</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # forward project the recon to fill out the zero projection image</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj = rtk.CudaForwardProjectionImageFilter[CudaImageType].New()</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj.SetGeometry( geometry )</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj.SetInput(0, constantImageSource.GetOutput()) # projection image</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj.SetInput(1, cpu2gpu(rtk_recon)) # recon volume</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj.SetStepSize(float(vol_params['vox_xy'])/4) # step size along the ray (default is 1mm)</span></div><div><span style="color:rgb(180,237,255)"> ForwardProj.Update()</span></div><div><span style="color:rgb(180,237,255)"> ###########################################################################</span></div><div><br></div><div><span style="color:rgb(180,237,255)"> # graft the projection to CPU / extract the array</span></div><div><span style="color:rgb(180,237,255)"> rtk_reprojection = gpu2cpu(ForwardProj.GetOutput()) # array inside the image has c,r,v format</span></div><div><span style="color:rgb(180,237,255)"> if return_itk:</span></div><div><span style="color:rgb(180,237,255)"> return rtk_reprojection # array inside the image has c,r,v format</span></div><div><span style="color:rgb(180,237,255)"> else:</span></div><div><span style="color:rgb(180,237,255)"> rtk_reprojection = itk.GetArrayViewFromImage(rtk_reprojection) # now v,r,c format</span></div><div><span style="color:rgb(180,237,255)"> if return_RVC:</span></div><div><span style="color:rgb(180,237,255)"> rtk_reprojection = np.transpose(rtk_reprojection, [1,0,2]) # r,v,c format to match Astra projection</span></div><div><span style="color:rgb(180,237,255)"> return rtk_reprojection # numpy array (r,v,c)</span></div></div></div><div><br></div><div><br></div><div><span id="m_7748307742549191118m_-2575662995419800001cid:417B41B3-5736-41A2-AA06-75C877EED5F7"><Screenshot 2024-04-24 at 8.34.20 PM.png></span></div><div><br></div><div><br></div></div><div><div>_______________________________________________</div><div>Rtk-users mailing list</div><div><a href="mailto:rtk-users@openrtk.org" target="_blank">rtk-users@openrtk.org</a></div><div><a href="https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwQFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=mi_1gmtry3HzMfO0caEhCP-0wcNe3su2gocFndAoByaorId18bE57XL9ozhaj01k&s=K4sXSgvZVGEDHgEKoU168ighF9MBDAXWRLef6iJDkrs&e=" target="_blank">https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users</a></div></div></blockquote>
</font><span id="m_7748307742549191118m_-2575662995419800001cid:417B41B3-5736-41A2-AA06-75C877EED5F7"><Screenshot 2024-04-24 at 8.34.20 PM.png></span></div></blockquote></div><br></div></div>_______________________________________________<br>
Rtk-users mailing list<br>
<a href="mailto:rtk-users@openrtk.org" target="_blank">rtk-users@openrtk.org</a><br>
<a href="https://urldefense.us/v2/url?u=https-3A__www.creatis.insa-2Dlyon.fr_mailman_listinfo_rtk-2Dusers&d=DwMFaQ&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=J7uT21mkGp7aMwIrHQkTLGwy72wKx_bOB0IkoGp__bQ&m=irXqD18u-c9Im_eQprxOMZNcAsAeuIrOn7RWNk8LABzPsfkFpG_WuH03l9pRCH7S&s=jxU_OqFdvXhIND4OxiV2Qk7vLRtlZbkMZpGf5fuYKeA&e=" rel="noreferrer" target="_blank">https://www.creatis.insa-lyon.fr/mailman/listinfo/rtk-users</a><br>
</blockquote></div>
</div></blockquote></div><br></div></div></blockquote></div>