Visual Programming: Compositional visual reasoning without training

This paper from Allen Institute for AI introduces VISPROG, a neuro-symbolic approach that addresses complex and compositional visual tasks using natural language instructions. Unlike task-specific training methods, VISPROG leverages the contextual learning capabilities of large language models to generate modular programs resembling Python code. These programs are then executed to obtain both the solution to the visual task and a detailed and interpretable rationale.

The generated program consists of multiple lines, with each line potentially invoking various computer vision models, image processing routines, or Python functions to produce intermediate outputs. These outputs can be utilized by subsequent parts of the program, enabling a flexible and modular approach to solving visual tasks.

The authors showcase the versatility of VISPROG by demonstrating its effectiveness on four different tasks: compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. Through these experiments, they highlight the potential of neuro-symbolic approaches like VISPROG to expand the capabilities of AI systems in addressing complex and diverse tasks.

In conclusion, the paper presents VISPROG as a neuro-symbolic approach that effectively tackles complex visual tasks by generating modular programs based on natural language instructions. It emphasizes the benefits of such approaches in broadening the scope of AI systems to cater to a wide range of complex tasks.

Planning-oriented Autonomous Driving

This paper, presented by a research team from Shanghai AI Laboratory, Wuhan University, and SenseTime Research, introduces the Unified Autonomous Driving (UniAD) framework, which aims to improve the performance and coordination of modular tasks in autonomous driving systems. The traditional approach to autonomous driving involves separate models for perception, prediction, and planning, which can lead to errors and inadequate task coordination. UniAD proposes a comprehensive framework that combines these tasks into one network, prioritizing them based on their contribution to the ultimate goal of planning for self-driving cars.

UniAD leverages the strengths of each module and provides complementary feature abstractions to facilitate agent interaction from a global perspective. The framework employs unified query interfaces to enhance communication between tasks and ensure they work together effectively towards planning. The paper demonstrates the effectiveness of the UniAD philosophy by applying it to the nuScenes benchmark, a challenging dataset for autonomous driving. Extensive experiments and ablations show that UniAD outperforms previous state-of-the-art methods in all aspects. The authors have made the code and models publicly available, enabling further research and development in the field.

DynIBaR: Neural Dynamic Image-Based Rendering

This paper, presented by a research team from Google Research and Cornell Tech, introduces a new approach for synthesizing novel views from a monocular video of a complex dynamic scene. Existing methods, such as dynamic Neural Radiance Fields (NeRFs), have shown promising results but struggle with long videos, complex object motions, and uncontrolled camera trajectories, leading to blurry or inaccurate renderings. To overcome these limitations, the authors propose a volumetric image-based rendering framework that aggregates features from nearby views in a scene-motion-aware manner. This approach retains the ability to model complex scenes and view-dependent effects while enabling the synthesis of photo-realistic novel views from long videos with complex dynamics and unconstrained camera trajectories. The authors demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets and successfully apply their approach to challenging real-world videos with difficult camera and object motion, where previous methods fail to produce high-quality renderings.

3D Registration with Maximal Cliques

This paper from Northwestern Polytechnical University, China, introduces a novel method called Maximal Cliques (MAC) for 3D point cloud registration (PCR), which involves aligning a pair of point clouds to find the optimal pose. The authors propose a technique that leverages maximal cliques in a graph to extract local consensus information and generate accurate pose hypotheses. The approach consists of several steps:

1. Construction of a compatibility graph to represent the affinity between initial correspondences.

2. Identification of maximal cliques in the graph, where each clique represents a consensus set. Node-guided clique selection is performed, choosing the clique with the highest graph weight.

3. Computation of transformation hypotheses for the selected cliques using the SVD algorithm, with the best hypothesis used for registration.

The authors conducted extensive experiments on various benchmark datasets, including U3M, 3DMatch, 3DLoMatch, and KITTI. The results demonstrate that MAC significantly improves registration accuracy compared to state-of-the-art methods and enhances the performance of deep-learning-based approaches. When combined with deep-learned methods, MAC achieves a state-of-the-art registration recall of 95.7% on 3DMatch and 78.9% on 3DLoMatch. Overall, MAC proves to be an effective technique for 3D point cloud registration, surpassing existing methods in accuracy and performance.

Featured AI Tools By Marktechpost:

• AdCreative.ai

• Otter AI

• tinyEinstein

• Notion

• Aragon AI

• SaneBox

• StoryBird AI

and many more in our AI Tools Club.