摘要：Frequent itemset mining (FIM) is a common approach for discovering hidden frequentpatterns from transactional databases used in prediction, association rules, classification, etc. Aprioriis an FIM elementary algorithm with iterative nature used to find the frequent itemsets. Apriori isused to scan the dataset multiple times to generate big frequent itemsets with different cardinalities.Apriori performance descends when data gets bigger due to the multiple dataset scan to extract thefrequent itemsets. Eclat is a scalable version of the Apriori algorithm that utilizes a vertical layout.The vertical layout has many advantages; it helps to solve the problem of multiple datasets scanningand has information that helps to find each itemset support. In a vertical layout, itemset support canbe achieved by intersecting transaction ids (tidset/tids) and pruning irrelevant itemsets. However,when tids become too big for memory, it affects algorithms efficiency. In this paper, we introduceSHFIM (spark-based hybrid frequent itemset mining), which is a three-phase algorithm that utilizesboth horizontal and vertical layout diffset instead of tidset to keep track of the differences betweentransaction ids rather than the intersections. Moreover, some improvements are developed to decreasethe number of candidate itemsets. SHFIM is implemented and tested over the Spark framework,which utilizes the RDD (resilient distributed datasets) concept and in-memory processing that tacklesMapReduce framework problem. We compared the SHFIM performance with Spark-based Eclatand dEclat algorithms for the four benchmark datasets. Experimental results proved that SHFIMoutperforms Eclat and dEclat Spark-based algorithms in both dense and sparse datasets in terms ofexecution time.
