VeloxCon 2024: Innovation in data management

VeloxCon 2024, the premier developer convention that’s devoted to the Velox open-source undertaking, introduced collectively trade leaders, engineers, and lovers to discover the most recent developments and collaborative efforts shaping the way forward for knowledge administration. Hosted by IBM® in partnership with Meta, VeloxCon showcased the most recent innovation in Velox together with undertaking roadmap, Prestissimo (Presto-on-Velox), Gluten (Spark-on-Velox), {hardware} acceleration, and rather more.

An summary of Velox

Velox is a unified execution engine that’s constructed and open-sourced by Meta, geared toward accelerating knowledge administration methods and streamlining their improvement. One of many greatest advantages of Velox is that it consolidates and unifies knowledge administration methods so that you don’t must hold rewriting the engine. As we speak Velox is in varied levels of integration with a number of knowledge methods together with Presto (Prestissimo), Spark (Gluten), PyTorch (TorchArrow), and Apache Arrow. You possibly can learn extra about why Velox was in-built Meta’s engineering blog.

Velox at IBM

Presto is the engine for watsonx.data, IBM’s open knowledge lakehouse platform. During the last 12 months, we’ve been working onerous on advancing Velox for Presto – Prestissimo – at IBM. Presto Java staff are being changed by a C++ course of based mostly on Velox. We now have a number of committers to the Prestissimo undertaking and proceed to associate intently with Meta as we work on constructing Presto 2.0.

A number of the key advantages of Prestissimo embody:

• Hugh efficiency enhance: question processing could be carried out with a lot smaller clusters
• No efficiency cliffs: no Java processes, JVM, or rubbish collections, as reminiscence arbitration improves effectivity
• Simpler to construct and function at scale: Velox provides you reusable and extensible primitives throughout knowledge engines (like Spark)

This 12 months, we plan to do much more with Prestissimo together with:

• The Iceberg reader
• Manufacturing readiness (metrics assortment with Prometheus)
• New Velox system implementation
• TPC-DS benchmark runs

VeloxCon 2024

We labored intently with Meta to arrange VeloxCon 2024, and it was a implausible neighborhood occasion. We heard audio system from Meta, IBM, Pinterest, Intel, Microsoft, and others share what they’re engaged on and their imaginative and prescient for Velox over two dynamic days.

Day 1 highlights

The convention kicked off with classes from Meta together with Amit Purohit reaffirming Meta’s dedication to open supply and neighborhood collaboration. Pedro Pedreira, alongside Manos Karpathiotakis and Deblina Gupta, delved into the idea of composability in knowledge administration, showcasing Velox’s versatility and its alignment with Arrow.

Amit Dutta of Meta explored Prestissimo’s batch effectivity at Meta, shedding gentle on the developments made in optimizing knowledge processing workflows. Remus Lazar, VP Knowledge & AI Software program at IBM offered Velox’s journey inside IBM and imaginative and prescient for its future. Aditi Pandit of IBM adopted with insights into Prestissimo’s integration at IBM, highlighting function enhancements and future plans.

The afternoon classes have been equally insightful, with Jimmy Lu of Meta unveiling the most recent optimizations and options in Velox. Whereas Binwei Yang of Intel mentioned the mixing of Velox with the Apache Gluten undertaking, emphasizing its world influence. Engineers from Pinterest and Microsoft shared their experiences of unlocking knowledge question efficiency by utilizing Velox and Gluten, showcasing tangible efficiency positive factors.

The day concluded with classes from Meta on Velox’s reminiscence administration by Xiaoxuan Meng and a glimpse into the brand new easy aggregation operate interface that was offered by Wei He.

Day 2 highlights

The second day started with a keynote from Orri Erling, co-creator of Velox. He shared insights into Velox Wave and Accelerators, showcasing its potential for acceleration. Krishna Maheshwari from NeuroBlade highlighted their collaboration with the Velox neighborhood, introducing NeuroBlade’s SPU (SQL Processing Unit) and its transformative influence on Velox’s computational pace and effectivity.

Sergei Lewis from Rivos explored the potential of offloading work to accelerators to reinforce Velox’s pipeline efficiency. William Malpica and Amin Aramoon from Voltron Knowledge launched Theseus, a composable, scalable, distributed knowledge analytics engine, utilizing Velox as a CPU backend.

Yoav Helfman from Meta unveiled Nimble, a cutting-edge columnar file format that’s designed to reinforce knowledge storage and retrieval. Pedro Pedreira and Sridhar Anumandla from Meta elaborated on Velox’s new technical governance mannequin, emphasizing its significance in guiding the undertaking’s improvement sustainability.

The day additionally featured classes on Velox’s I/O optimizations by Deepak Majeti from IBM, methods for safeguarding in opposition to Out-Of-Reminiscence (OOM) kills by Vikram Joshi from ComputeAI, and a hands-on demo on debugging Velox functions by Deepak Majeti.

What’s subsequent with Velox

VeloxCon 2024 was a testomony to the colourful ecosystem surrounding the Velox undertaking, showcasing groundbreaking improvements and fostering collaboration amongst trade leaders and builders alike. The convention offered attendees with useful insights, sensible data, and networking alternatives, solidifying Velox’s place as a number one open supply undertaking within the knowledge administration ecosystem.

When you’re excited about studying extra and becoming a member of the Velox neighborhood, listed here are some assets to get began:

Keep tuned for extra updates and developments from the Velox neighborhood, as we proceed to push the boundaries of knowledge administration and speed up innovation collectively.

Try Presto with a free trial of watsonx.data