output-operations-on-dstreams
2017-08-29 09:28
309 查看
Finally, this can be further optimized by reusing connection objects across multiple RDDs/batches. One can maintain a static pool of connection objects than can be reused as RDDs of multiple batches are pushed to the external system, thus further reducing the overheads. Scala Python dstream.foreachRDD { rdd => rdd.foreachPartition { partitionOfRecords => // ConnectionPool is a static, lazily initialized pool of connections val connection = ConnectionPool.getConnection() partitionOfRecords.foreach(record => connection.send(record)) ConnectionPool.returnConnection(connection) // return to the pool for future reuse } }
http://spark.apache.org/docs/1.6.1/streaming-programming-guide.html#output-operations-on-dstreams
相关文章推荐
- Output Operations on DStreams
- spark-streaming-[7]-Output Operations on DStreams-foreachRDD写Mysql
- 每日一命令(10)cat (concatenate files and print on the standard output)
- System Operations on AWS - Lab 5W - Managing Resources (Windows)
- Assignment | 05-week2 -Part_1-Operations on word vectors
- MATLAB基础学习--Input/Output Operations
- Design a stack with operations on middle element
- 11gR2 Preinstallation Operations On RHEL6.2 64bit Only
- kuangbinOJ 1217 Operations on Grids (暴搜)
- The BulkProcessor class offers a simple interface to flush bulk operations automatically based on th
- 在编译时出现:/usr/bin/ld: final link failed: Nonrepresentable section on output解决方法
- 每次执行过程之前都要set serveroutput on 一下,否则dbms_output.put_line()无显示
- Operations on Surface_mesh in CGAL
- Operations on edges in polyhedron
- System Operations on AWS - Lab 6W - Using Auto Scaling (Windows)
- Coursera 吴恩达DeepLearning.AI 第五课 sequence model 序列模型 第二周 Operations on word vectors - v2
- OpenCV Code: Operations on Mat element
- PE 323 Bitwise-OR operations on random integers (概率dp)
- [SQL Server]:backup, file manipulation operations changes on a database must be serialized.
- set serveroutput on 命令