What's included in this newsletter: FalconMamba 7B, Arcee Swarm, XMainframe, Med42-v2 and Sarvam-2B

Small Language Model

FalconMamba 7B Released: The World’s First Attention-Free AI Model with 5500GT Training Data and 7 Billion Parameters

The Technology Innovation Institute (TII) in Abu Dhabi has recently unveiled the FalconMamba 7B, a groundbreaking artificial intelligence model. This model, the first strong attention-free 7B model, is designed to overcome many of the limitations existing AI architectures face, particularly in handling large data sequences. The FalconMamba 7B is released under the TII Falcon License 2.0. It is available as an open-access model within the Hugging Face ecosystem, making it accessible for researchers and developers globally.

FalconMamba 7B distinguishes itself based on the Mamba architecture, originally proposed in the paper “Mamba: Linear-Time Sequence Modeling with Selective State Spaces.” This architecture diverges from the traditional transformer models that dominate the AI landscape today. Transformers, while powerful, have a fundamental limitation in processing large sequences due to their reliance on attention mechanisms, which increase compute and memory costs with sequence length. FalconMamba 7B, however, overcomes these limitations through its architecture, which includes extra RMS normalization layers to ensure stable training at scale. This enables the model to process sequences of arbitrary length without an increase in memory storage, making it capable of fitting on a single A10 24GB GPU.

➡️ Continue reading here!  

Small Language Model

Arcee AI Introduces Arcee Swarm: A Groundbreaking Mixture of Agents MoA Architecture Inspired by the Cooperative Intelligence Found in Nature Itself

Arcee AI, an artificial intelligence AI company focussing specially on small language models, is introducing its first-of-its-kind Arcee Swarm. The release, which is coming soon, is touted to send ripples in the AI community, as it is a pretty new and different solution leveraging specialist models for one framework. What makes Arcee Swarm outstanding is the kind of technological capabilities this would provide, which would probably largely change how AI systems interact with their users, handle complex queries, and produce precise, high-quality outputs across several domains, including high-quality general reasoning, all while also reducing hallucinations.

The Arcee Swarm is purposely intended for a multi-faceted approach, by means of focused and flexible expertise across various domains. Every model in the Swarm is trained to master a specific domain or task, becoming an expert in its area of focus. When implemented with a general model in the Swarm, this system excels not only in specialized tasks but also in general tasks and reasoning, offering a versatile and comprehensive solution. This combination ensures that no matter the query, the Swarm can provide accurate, well-rounded answers that leverage both specialized expertise and general intelligence.

➡️ Continue reading here!

Large Language Model

Researchers at FPT Software AI Center Introduce XMainframe: A State-of-the-Art Large Language Model (LLM) Specialized for Mainframe Modernization to Address the $100B Legacy Code Modernization

Researchers at FPT Software AI Center have developed XMainframe, a state-of-the-art large language model (LLM) specifically designed with expertise in mainframe legacy systems and COBOL codebases. The solution includes the creation of an extensive data collection pipeline to produce high-quality training datasets, significantly enhancing XMainframe’s performance in this specialized domain. Additionally, they introduce MainframeBench, a comprehensive benchmark for evaluating mainframe knowledge through multiple-choice questions, question answering, and COBOL code summarization. Empirical evaluations show that XMainframe consistently outperforms existing state-of-the-art LLMs in these tasks, achieving 30% higher accuracy than DeepSeek-Coder on multiple-choice questions, doubling the BLEU score of Mixtral-Instruct 8x7B on question-answering, and scoring six times higher than GPT-3.5 on COBOL summarization. This work underscores XMainframe’s potential to drive significant advancements in managing and modernizing legacy systems, ultimately enhancing productivity and saving time for software developers....

➡️ Continue reading here!

Recommended AI WEBINAR from Our Partner

Unlock the power of your Snowflake data with LLMs – 29th August (Webinar)
Making sense of data is challenging, but LLM powered applications can simplify it by acting as a natural language interface to your Business Intelligence (BI) tools. In this webinar, Dr. Jasper Schwenzow, Senior NLP Engineer at deepset.ai, will show you how to build an AI system using LLMs to query your database.

You'll learn:
- Why LLMs struggle with tabular data and how to fix it.
- How LLMs bridge gaps between data types.
- Practical examples of companies using this technology for data insights.

Learn more and register for this session on deepset.ai.

Medical AI

Med42-v2 Released: A Groundbreaking Suite of Clinical Large Language Models Built on Llama3 Architecture, Achieving Up to 94.5% Accuracy on Medical Benchmarks

Researchers from M42 Abu Dhabi, UAE, have introduced the Med42-v2, a suite of clinical LLMs built on the advanced Llama3 architecture. Developed by the team at M42 in Abu Dhabi, these models are meticulously fine-tuned using specialized clinical datasets, making them particularly adept at handling medical queries. Unlike generic models, which are often preference-aligned to avoid answering clinical questions, Med42-v2 is specifically trained to engage with such queries, ensuring that it can provide relevant and accurate information to clinicians, patients, & other stakeholders in the healthcare sector….

➡️ Continue reading here!

Indian Small Language Mod

Sarvam AI Releases Samvaad-Hi-v1 Dataset and Sarvam-2B: A 2 Billion Parameter Language Model with 4 Trillion Tokens Focused on 10 Indic Languages for Enhanced NLP

Sarvam AI has recently unveiled its cutting-edge language model, Sarvam-2B. This powerful model, boasting 2 billion parameters, represents a significant stride in Indic language processing. With a focus on inclusivity and cultural representation, Sarvam-2B is pre-trained from scratch on a massive dataset of 4 trillion high-quality tokens, with an impressive 50% dedicated to Indic languages. This development, particularly their ability to understand and generate text in languages, is historically underrepresented in AI research.

They have also introduced the Samvaad-Hi-v1 dataset, a meticulously curated collection of 100,000 high-quality English, Hindi, and Hinglish conversations. This dataset is uniquely designed with an Indic context, making it an invaluable resource for researchers and developers working on multilingual and culturally relevant AI models. Samvaad-Hi-v1 is poised to enhance the training of conversational AI systems that can understand and engage with users more naturally and contextually appropriately across different languages and dialects prevalent in India....

➡️ Continue reading here!