Data Grid Vector Storage and Search

Extend Data Grid search and storage engine to store vectors and implement k-nearest neighbours algorithm, also known as KNN or k-NN. The embeddings are created outside Data Grid. 

Why is this crucial?

Enterprises and customers have increased the quick adoption of technologies like ChatGPT and the language model capabilities. These technologies come with limitations. An effective vector search can leverage these limitations providing:

• Privacy and security 
• Storing embeddings at scale
• Binary Protocol (hotrod), REST and RESP (Redis protocol) interfaces
• Cost decrease for RH customers

Main Competitors

• Redis Enterprise Vectors https://redis.com/solutions/use-cases/vector-database/
• Elastic Search https://www.elastic.co/blog/may-2023-launch-announcement
• Blog post with Chat GTP https://www.elastic.co/blog/chatgpt-elasticsearch-openai-meets-private-data

Technical evolutions to DG (WIP)

• Upgrade to Lucene 9 to support vectors
• Extend the Search capabilities