Hey Folks!

This newsletter will discuss some cool AI research papers and AI tools. Happy learning!

➡️ Can One AI Model Master All Audio Tasks? Meet UniAudio: A New Universal Audio Generation System

How can LLMs be employed for universal audio generation across various types? This paper introduces the "UniAudio" system, which capitalizes on LLM's generative prowess to produce diverse audio outputs, from speech and music to sounds and singing. Unlike earlier methods, UniAudio tokenizes the target audio and conditions, merges the source-target data into one sequence, and then predicts the next token via LLM. To address long sequences from the tokenization process, a multi-scale Transformer model is integrated. With a colossal training set of 165K audio hours and 1B parameters, UniAudio captures both the innate characteristics of audio and its correlation with other modalities. This robust training renders UniAudio a pioneering model in audio generation, displaying proficiency in multiple tasks and adaptability for new challenges.

➡️  Meta AI Researchers Introduce a Machine Learning Model that Explores Decoding Speech Perception from Non-Invasive Brain Recordings

Is it possible to decode natural speech from non-invasive brain recordings? This paper from Meta AI introduces a cutting-edge model leveraging contrastive learning to decipher self-supervised representations of perceived speech from non-invasive recordings in a large cohort of healthy individuals. Using integrated data from four public datasets, which included 175 participants undergoing magneto-encephalography or electro-encephalography while listening to stories and sentences, the model achieved remarkable results. It could pinpoint the speech segment from a mere 3-second signal with up to 41% accuracy across participants and even 80% in some. Crucially, this model underscores the potential to decode language from brain activity without necessitating invasive procedures.

➡️  Researchers from Caltech and ETH Zurich Introduce Groundbreaking Diffusion Models: Harnessing Text Captions for State-of-the-Art Visual Tasks and Cross-Domain Adaptations

How can diffusion models, known for their text-to-image synthesis prowess, be optimally utilized for visual tasks? This AI paper delves into enhancing the cross-attention mechanism of diffusion models for better perceptual performance by leveraging automatically generated captions. This innovative method leads to superior results, setting new benchmarks in diffusion-based semantic segmentation on ADE20K, depth estimation on NYUv2, object detection on Pascal VOC, and segmentation on Cityscapes. Moreover, the model demonstrates adaptability to cross-domain scenarios, with exceptional performance on datasets like Watercolor2K, Dark Zurich-val, and Nighttime Driving when aligned with the target domain using model personalization and caption adjustments.

➡️ Microsoft Researchers Introduce SpaceEvo: A Game-Changer for Designing Ultra-Efficient and Quantized Neural Networks for Real-World Devices

How can one design accurate quantized neural networks (QNN) that also ensure low latency on real-world devices using Neural Architecture Search (NAS)? Addressing this problem, this paper introduces "SpaceEvo". This method identifies that existing search spaces lead to quantization inefficiencies, slowing INT8 inference speed due to quantization-unfriendly choices. SpaceEvo crafts a quantization-friendly search space tailored for specific hardware by identifying hardware-preferred configurations and operators, guided by a quantification metric called the Q-T score. The subsequent models, named SEQnet, demonstrate remarkable INT8 accuracy improvements and, when tested on real devices, surpass manually-designed search spaces in speed by up to 2.5x while maintaining accuracy.

➡️ This AI Research Proposes SMPLer-X: A Generalist Foundation Model for 3D/4D Human Motion Capture from Monocular Inputs

How can we achieve more accurate human pose and shape estimation without being confined to specific training datasets? This research presents SMPLer-X, an ambitious approach to expressive human pose and shape estimation (EHPS) that incorporates large-scale data and model architectures. SMPLer-X is developed on the backbone of ViT-Huge and is trained with a massive 4.5M instances from a diverse range of data sources. By systematically exploring 32 EHPS datasets, the researchers ensure their model can handle a wide variety of scenarios, significantly improving EHPS capabilities. Furthermore, they employ vision transformers to optimize model scaling in EHPS. Impressively, SMPLer-X sets new performance records on multiple benchmarks, demonstrating its robustness and transferability.

✅ Featured AI Tools For You