Hey Danny. You do not get d from the RGB image. Those are two completely different spaces. The depth matrix has only 1 channel with each entry being a number from 0-2047. That is the disparity, which you use to calculate the actual depth from the camera. I would suggest you simply use the Point Cloud Library, it covers up all this low lever mumbo jumbo and lets you get straight to the point cloud analysis fun. Happy hacking!