[3D CV] Project 2D images to point clouds, depth map backprojection

Backproject depth map (Project 2D images to point clouds)

2D image coordinates에 대한 z value를 알면, 각 픽셀들을 3D로 backproject한 point clouds를 얻을 수 있습니다.

다시말해, detph map이 있으면, 각 픽셀들을 3D로 backproject한 point clouds를 얻을 수 있습니다.

Project 2D images to point clouds github code

...
    # Process each image file
    for k, filename in enumerate(filenames):
        print(f'Processing {k+1}/{len(filenames)}: {filename}')

        # Load the image
        color_image = Image.open(filename).convert('RGB')
        width, height = color_image.size

        # Read the image using OpenCV
        image = cv2.imread(filename)
        pred = depth_anything.infer_image(image, height)

        # Resize depth prediction to match the original image size
        resized_pred = Image.fromarray(pred).resize((width, height), Image.NEAREST)

        # Generate mesh grid and calculate point cloud coordinates
        x, y = np.meshgrid(np.arange(width), np.arange(height))
        x = (x - width / 2) / args.focal_length_x
        y = (y - height / 2) / args.focal_length_y
        z = np.array(resized_pred)
        points = np.stack((np.multiply(x, z), np.multiply(y, z), z), axis=-1).reshape(-1, 3)
        colors = np.array(color_image).reshape(-1, 3) / 255.0

        # Create the point cloud and save it to the output directory
        pcd = o3d.geometry.PointCloud()
        pcd.points = o3d.utility.Vector3dVector(points)
        pcd.colors = o3d.utility.Vector3dVector(colors)
        o3d.io.write_point_cloud(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + ".ply"), pcd)

• meshgrid로 $x_{pixel}, y_{pixel}$의 모든 좌표를 구성합니다.

1. 코드에서 주어진 것처럼 width, height, focal_length_x, focal_length_y의 관계로 명시적으로 써서 $x_{pixel}$, $y_{pixel}$에서 $x_{hom}$, $y_{hom}$으로의 변환을 수행해도 되고,

1. 아래처럼 width/2, height/2, focal_length_x, focal_length_y로 구성된 Camera Intrinsics의 역행렬 $K^{-1}$을 $x_{pixel}$, $y_{pixel}$에 matrix multiplication하여 $x_{hom}, y_{hom}$을 구해도 됩니다.

• 최종적으로 $[x_{hom}, y_{hom}, 1]$에 $z$를 곱하여 3D points를 구합니다.

Reference

• Depth Anything V2 for Metric Depth Estimation