The headline needs to capture: Spark 4.2, native vector search, and the question of whether you still need a vector database.

Short summary

Spark 4.2 adds native vector search via SQL primitives like NEAREST BY, letting you run similarity search on embeddings stored in Spark tables without a separate vector database. For batch and analytical workloads this genuinely simplifies your stack. For live, low-latency user-facing retrieval, a dedicated vector DB with ANN indexes like HNSW remains the better choice.

• •Spark 4.2 adds SQL-native vector search (distance, similarity, NEAREST BY top-K join)
• •Batch/analytical retrieval can drop the separate vector DB; real-time serving still needs one
• •No approximate-nearest-neighbor index means exact search at scale is slower than purpose-built vector DBs

Generated with AI, which can make mistakes.