溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
Spark怎樣實現推薦系統中的相似度算法，針對這個問題，這篇文章詳細介紹了相對應的分析和解答，希望可以幫助更多想解決這個問題的小伙伴找到更簡單易行的方法。
在推薦系統中，協同過濾算法是應用較多的，具體又主要劃分為基于用戶和基于物品的協同過濾算法，核心點就是基于"一個人"或"一件物品"，根據這個人或物品所具有的屬性，比如對于人就是性別、年齡、工作、收入、喜好等，找出與這個人或物品相似的人或物，當然實際處理中參考的因子會復雜的多。
def euclidean2(v1: Vector, v2: Vector): Double = {require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +s"=${v2.size}.")val x = v1.toArrayval y = v2.toArrayeuclidean(x, y)}def euclidean(x: Array[Double], y: Array[Double]): Double = {require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +s"=${y.length}.")math.sqrt(x.zip(y).map(p => p._1 - p._2).map(d => d * d).sum)}def euclidean(v1: Vector, v2: Vector): Double = {val sqdist = Vectors.sqdist(v1, v2)math.sqrt(sqdist)}
def pearsonCorrelationSimilarity(arr1: Array[Double], arr2: Array[Double]): Double = {require(arr1.length == arr2.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${arr1.length} and Len(y)" +s"=${arr2.length}.")val sum_vec1 = arr1.sumval sum_vec2 = arr2.sumval square_sum_vec1 = arr1.map(x => x * x).sumval square_sum_vec2 = arr2.map(x => x * x).sumval zipVec = arr1.zip(arr2)val product = zipVec.map(x => x._1 * x._2).sumval numerator = product - (sum_vec1 * sum_vec2 / arr1.length)val dominator = math.pow((square_sum_vec1 - math.pow(sum_vec1, 2) / arr1.length) * (square_sum_vec2 - math.pow(sum_vec2, 2) / arr2.length), 0.5)if (dominator == 0) Double.NaN else numerator / (dominator * 1.0)}
余弦相似度
/** jblas實現余弦相似度 */def cosineSimilarity(v1: DoubleMatrix, v2: DoubleMatrix): Double = {require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(v1)=${x.length} and Len(v2)" +s"=${y.length}.")v1.dot(v2) / (v1.norm2() * v2.norm2())}def cosineSimilarity(v1: Vector, v2: Vector): Double = {require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +s"=${v2.size}.")val x = v1.toArrayval y = v2.toArraycosineSimilarity(x, y)}def cosineSimilarity(x: Array[Double], y: Array[Double]): Double = {require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +s"=${y.length}.")val member = x.zip(y).map(d => d._1 * d._2).sumval temp1 = math.sqrt(x.map(math.pow(_, 2)).sum)val temp2 = math.sqrt(y.map(math.pow(_, 2)).sum)val denominator = temp1 * temp2if (denominator == 0) Double.NaN else member / (denominator * 1.0)}
修正余弦相似度
def adjustedCosineSimJblas(x: DoubleMatrix, y: DoubleMatrix): Double = {require(x.length == y.length, s"SimilarityAlgorithms:DoubleMatrix length do not match: Len(x)=${x.length} and Len(y)" +s"=${y.length}.")val avg = (x.sum() + y.sum()) / (x.length + y.length)val v1 = x.sub(avg)val v2 = y.sub(avg)v1.dot(v2) / (v1.norm2() * v2.norm2())}def adjustedCosineSimJblas(x: Array[Double], y: Array[Double]): Double = {require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +s"=${y.length}.")val v1 = new DoubleMatrix(x)val v2 = new DoubleMatrix(y)adjustedCosineSimJblas(v1, v2)}def adjustedCosineSimilarity(v1: Vector, v2: Vector): Double = {require(v1.size == v2.size, s"SimilarityAlgorithms:Vector dimensions do not match: Dim(v1)=${v1.size} and Dim(v2)" +s"=${v2.size}.")val x = v1.toArrayval y = v2.toArrayadjustedCosineSimilarity(x, y)}def adjustedCosineSimilarity(x: Array[Double], y: Array[Double]): Double = {require(x.length == y.length, s"SimilarityAlgorithms:Array length do not match: Len(x)=${x.length} and Len(y)" +s"=${y.length}.")val avg = (x.sum + y.sum) / (x.length + y.length)val member = x.map(_ - avg).zip(y.map(_ - avg)).map(d => d._1 * d._2).sumval temp1 = math.sqrt(x.map(num => math.pow(num - avg, 2)).sum)val temp2 = math.sqrt(y.map(num => math.pow(num - avg, 2)).sum)val denominator = temp1 * temp2if (denominator == 0) Double.NaN else member / (denominator * 1.0)}
大家如果在實際業務處理中有相關需求，可以根據實際場景對上述代碼進行優化或改造，當然很多算法框架提供的一些算法是對這些相似度算法的封裝，底層還是依賴于這一套，也能幫助大家做更好的了解。比如Spark MLlib在KMeans算法實現中，底層對歐幾里得距離的計算實現。
關于Spark怎樣實現推薦系統中的相似度算法問題的解答就分享到這里了，希望以上內容可以對大家有一定的幫助，如果你還有很多疑惑沒有解開，可以關注億速云行業資訊頻道了解更多相關知識。
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
