【SparkAPI Java版】JavaPairRDD——cartesian（三）

Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in `this` and b is in `other`.

该函数返回的是Pair类型的RDD，计算结果是当前RDD和other RDD中每个元素进行笛卡儿计算的结果。最后返回的是CartesianRDD。

//scala def cartesian[U: ClassTag](other: RDD[U]): RDD[(T, U)] //java public static JavaPairRDD cartesian(JavaRDDLike other)

public class Cartesian { public static void main(String[] args) { System.setProperty("hadoop.home.dir","F:\hadoop-2.7.1"); SparkConf conf = new SparkConf().setMaster("local").setAppName("TestSpark"); JavaSparkContext sc = new JavaSparkContext(conf); //JavaPairRDD cartesian示例 JavaPairRDD javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList( new Tuple2(1,11), new Tuple2(2,22), new Tuple2(3,33) )); JavaPairRDD javaPairRDD2 = sc.parallelizePairs(Lists.newArrayList( new Tuple2(7,71), new Tuple2(8,82), new Tuple2(9,93) )); JavaPairRDD, Tuple2> javaPairRDD3 = javaPairRDD1.cartesian(javaPairRDD2); javaPairRDD3.foreach(new VoidFunction, Tuple2>>() { public void call(Tuple2, Tuple2> tuple2Tuple2Tuple2) throws Exception { System.out.println(tuple2Tuple2Tuple2); /*System.out.print("key:"+tuple2Tuple2Tuple2._1+","); System.out.println("value:"+tuple2Tuple2Tuple2._2);*/ } }); //JavaRDD cartesian示例 JavaRDD javaRDD1 = sc.parallelize(Lists.newArrayList(1,2,3)); JavaRDD javaRDD2 = sc.parallelize(Lists.newArrayList(7,8,9)); JavaPairRDD javaPairRDD = javaRDD1.cartesian(javaRDD2); javaPairRDD.foreach(new VoidFunction>() { public void call(Tuple2 integerIntegerTuple2) throws Exception { System.out.println(integerIntegerTuple2); } }); } }

//JavaPairRDD cartesian示例 ((1,11),(7,71)) ((1,11),(8,82)) ((1,11),(9,93)) ((2,22),(7,71)) ((2,22),(8,82)) ((2,22),(9,93)) ((3,33),(7,71)) ((3,33),(8,82)) ((3,33),(9,93)) 19/03/03 23:37:52 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 751 bytes result sent to driver //JavaRDD cartesian示例 (1,7) (1,8) (1,9) (2,7) (2,8) (2,9) (3,7) (3,8) (3,9)

注意：计算笛卡尔积很消耗内存，不要在大量数据的时候随便使用。

EI企业智能 Java spark 可信智能计算服务 TICS 智能数据

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。