Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
2017-07-20 21:04
525 查看
装载自http://blog.csdn.net/catoop/article/details/50352334
两个主机建立连接的过程是很复杂的一个过程,涉及到多个数据包的交换,并且也很耗时间。Http连接需要的三次握手开销很大,这一开销对于比较小的http消息来说更大。但是如果我们直接使用已经建立好的http连接,这样花费就比较小,吞吐率更大。
传统的HttpURLConnection并不支持连接池,如果要实现连接池的机制,还需要自己来管理连接对象。对于网络请求这种底层相对复杂的操作,个人以为如果有可用的其他方案,也没有必要自己去管理连接对象。
除了HttpURLConnection,大家肯定还知道HttpClient。一般情况下,普通使用HttpClient已经能满足我们的需求,不过有时候,在我们需要高并发大量的请求网络的时候,还是用“连接池”这样的概念能提升吞吐量。
我们来看下怎么使用 org.apache.httpcomponents.httpclient(版本4.4)提供的连接池来实现我们的高并发网络请求。
使用到的jar包:
org\apache\httpcomponents\httpclient\4.4-beta1\httpclient-4.4-beta1.jar
org\apache\httpcomponents\httpclient-cache\4.4-beta1\httpclient-cache-4.4-beta1.jar
org\apache\httpcomponents\httpcore\4.4-beta1\httpcore-4.4-beta1.jar
下面代码实例中主要使用到 PoolingHttpClientConnectionManager
Post使用方法
两个主机建立连接的过程是很复杂的一个过程,涉及到多个数据包的交换,并且也很耗时间。Http连接需要的三次握手开销很大,这一开销对于比较小的http消息来说更大。但是如果我们直接使用已经建立好的http连接,这样花费就比较小,吞吐率更大。
传统的HttpURLConnection并不支持连接池,如果要实现连接池的机制,还需要自己来管理连接对象。对于网络请求这种底层相对复杂的操作,个人以为如果有可用的其他方案,也没有必要自己去管理连接对象。
除了HttpURLConnection,大家肯定还知道HttpClient。一般情况下,普通使用HttpClient已经能满足我们的需求,不过有时候,在我们需要高并发大量的请求网络的时候,还是用“连接池”这样的概念能提升吞吐量。
我们来看下怎么使用 org.apache.httpcomponents.httpclient(版本4.4)提供的连接池来实现我们的高并发网络请求。
使用到的jar包:
org\apache\httpcomponents\httpclient\4.4-beta1\httpclient-4.4-beta1.jar
org\apache\httpcomponents\httpclient-cache\4.4-beta1\httpclient-cache-4.4-beta1.jar
org\apache\httpcomponents\httpcore\4.4-beta1\httpcore-4.4-beta1.jar
下面代码实例中主要使用到 PoolingHttpClientConnectionManager
package com.hvgroup.zhuhai10086.jms.utils; import java.io.IOException; import java.io.InterruptedIOException; import java.io.UnsupportedEncodingException; import java.net.UnknownHostException; import java.util.ArrayList; import java.util.List; import java.util.Map; import java.util.Set; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import javax.net.ssl.SSLException; import javax.net.ssl.SSLHandshakeException; import org.apache.http.HttpEntity; import org.apache.http.HttpEntityEnclosingRequest; import org.apache.http.HttpHost; import org.apache.http.HttpRequest; import org.apache.http.NameValuePair; import org.apache.http.NoHttpResponseException; import org.apache.http.client.HttpRequestRetryHandler; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.entity.UrlEncodedFormEntity; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.client.methods.HttpPost; import org.apache.http.client.methods.HttpRequestBase; import org.apache.http.client.protocol.HttpClientContext; import org.apache.http.config.Registry; import org.apache.http.config.RegistryBuilder; import org.apache.http.conn.ConnectTimeoutException; import org.apache.http.conn.routing.HttpRoute; import org.apache.http.conn.socket.ConnectionSocketFactory; import org.apache.http.conn.socket.LayeredConnectionSocketFactory; import org.apache.http.conn.socket.PlainConnectionSocketFactory; import org.apache.http.conn.ssl.SSLConnectionSocketFactory; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.impl.conn.PoolingHttpClientConnectionManager; import org.apache.http.message.BasicNameValuePair; import org.apache.http.protocol.HttpContext; import org.apache.http.util.EntityUtils; /** * HttpClient工具类 * * @return * @author SHANHY * @create 2015年12月18日 */ public class HttpClientUtil { static final int timeOut = 10 * 1000; private static CloseableHttpClient httpClient = null; private final static Object syncLock = new Object(); private static void config(HttpRequestBase httpRequestBase) { // 设置Header等 // httpRequestBase.setHeader("User-Agent", "Mozilla/5.0"); // httpRequestBase // .setHeader("Accept", // "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"); // httpRequestBase.setHeader("Accept-Language", // "zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3");// "en-US,en;q=0.5"); // httpRequestBase.setHeader("Accept-Charset", // "ISO-8859-1,utf-8,gbk,gb2312;q=0.7,*;q=0.7"); // 配置请求的超时设置 RequestConfig requestConfig = RequestConfig.custom() .setConnectionRequestTimeout(timeOut) .setConnectTimeout(timeOut).setSocketTimeout(timeOut).build(); httpRequestBase.setConfig(requestConfig); } /** * 获取HttpClient对象 * * @return * @author SHANHY * @create 2015年12月18日 */ public static CloseableHttpClient getHttpClient(String url) { String hostname = url.split("/")[2]; int port = 80; if (hostname.contains(":")) { String[] arr = hostname.split(":"); hostname = arr[0]; port = Integer.parseInt(arr[1]); } if (httpClient == null) { synchronized (syncLock) { if (httpClient == null) { httpClient = createHttpClient(200, 40, 100, hostname, port); } } } return httpClient; } /** * 创建HttpClient对象 * * @return * @author SHANHY * @create 2015年12月18日 */ public static CloseableHttpClient createHttpClient(int maxTotal, int maxPerRoute, int maxRoute, String hostname, int port) { ConnectionSocketFactory plainsf = PlainConnectionSocketFactory .getSocketFactory(); LayeredConnectionSocketFactory sslsf = SSLConnectionSocketFactory .getSocketFactory(); Registry<ConnectionSocketFactory> registry = RegistryBuilder .<ConnectionSocketFactory> create().register("http", plainsf) .register("https", sslsf).build(); PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager( registry); // 将最大连接数增加 cm.setMaxTotal(maxTotal); // 将每个路由基础的连接增加 cm.setDefaultMaxPerRoute(maxPerRoute); HttpHost httpHost = new HttpHost(hostname, port); // 将目标主机的最大连接数增加 cm.setMaxPerRoute(new HttpRoute(httpHost), maxRoute); // 请求重试处理 HttpRequestRetryHandler httpRequestRetryHandler = new HttpRequestRetryHandler() { public boolean retryRequest(IOException exception, int executionCount, HttpContext context) { if (executionCount >= 5) {// 如果已经重试了5次,就放弃 return false; } if (exception instanceof NoHttpResponseException) {// 如果服务器丢掉了连接,那么就重试 return true; } if (exception instanceof SSLHandshakeException) {// 不要重试SSL握手异常 return false; } if (exception instanceof InterruptedIOException) {// 超时 return false; } if (exception instanceof UnknownHostException) {// 目标服务器不可达 return false; } if (exception instanceof ConnectTimeoutException) {// 连接被拒绝 return false; } if (exception instanceof SSLException) {// SSL握手异常 return false; } HttpClientContext clientContext = HttpClientContext .adapt(context); HttpRequest request = clientContext.getRequest(); // 如果请求是幂等的,就再次尝试 if (!(request instanceof HttpEntityEnclosingRequest)) { return true; } return false; } }; CloseableHttpClient httpClient = HttpClients.custom() .setConnectionManager(cm) .setRetryHandler(httpRequestRetryHandler).build(); return httpClient; } private static void setPostParams(HttpPost httpost, Map<String, Object> params) { List<NameValuePair> nvps = new ArrayList<NameValuePair>(); Set<String> keySet = params.keySet(); for (String key : keySet) { nvps.add(new BasicNameValuePair(key, params.get(key).toString())); } try { httpost.setEntity(new UrlEncodedFormEntity(nvps, "UTF-8")); } catch (UnsupportedEncodingException e) { e.printStackTrace(); } } /** * GET请求URL获取内容 * * @param url * @return * @author SHANHY * @throws IOException * @create 2015年12月18日 */ public static String post(String url, Map<String, Object> params) throws IOException { HttpPost httppost = new HttpPost(url); config(httppost); setPostParams(httppost, params); CloseableHttpResponse response = null; try { response = getHttpClient(url).execute(httppost, HttpClientContext.create()); HttpEntity entity = response.getEntity(); String result = EntityUtils.toString(entity, "utf-8"); EntityUtils.consume(entity); return result; } catch (Exception e) { // e.printStackTrace(); throw e; } finally { try { if (response != null) response.close(); } catch (IOException e) { e.printStackTrace(); } } } /** * GET请求URL获取内容 * * @param url * @return * @author SHANHY * @create 2015年12月18日 */ public static String get(String url) { HttpGet httpget = new HttpGet(url); config(httpget); CloseableHttpResponse response = null; try { response = getHttpClient(url).execute(httpget, HttpClientContext.create()); HttpEntity entity = response.getEntity(); String result = EntityUtils.toString(entity, "utf-8"); EntityUtils.consume(entity); return result; } catch (IOException e) { e.printStackTrace(); } finally { try { if (response != null) response.close(); } catch (IOException e) { e.printStackTrace(); } } return null; } public static void main(String[] args) { // URL列表数组 String[] urisToGet = { "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497", "http://blog.csdn.net/catoop/article/details/38849497" }; long start = System.currentTimeMillis(); try { int pagecount = urisToGet.length; ExecutorService executors = Executors.newFixedThreadPool(pagecount); CountDownLatch countDownLatch = new CountDownLatch(pagecount); for (int i = 0; i < pagecount; i++) { HttpGet httpget = new HttpGet(urisToGet[i]); config(httpget); // 启动线程抓取 executors .execute(new GetRunnable(urisToGet[i], countDownLatch)); } countDownLatch.await(); executors.shutdown(); } catch (InterruptedException e) { e.printStackTrace(); } finally { System.out.println("线程" + Thread.currentThread().getName() + "," + System.currentTimeMillis() + ", 所有线程已完成,开始进入下一步!"); } long end = System.currentTimeMillis(); System.out.println("consume -> " + (end - start)); } static class GetRunnable implements Runnable { private CountDownLatch countDownLatch; private String url; public GetRunnable(String url, CountDownLatch countDownLatch) { this.url = url; this.countDownLatch = countDownLatch; } @Override public void run() { try { System.out.println(HttpClientUtil.get(url)); } finally { countDownLatch.countDown(); } } } }>
Post使用方法
// 其中 params 为 Map<String, Object> params String ret = HttpClientUtil.post(url, params); jsonRet = new JSONObject(ret);1
2
3
1
2
3
[/code]
一开始我是使用传统的 HttpURLConnection 来做网络请求的,查了很多资料,有不少说 HttpURLConnection 效率高的。可是经过我修改实现方法后,HttpClient 连接池版本的网络请求相对比较稳定。这也说明,我们并不请尽信他人解说,凡事还是要寻找适合自己的方法,真正的解决自己的问题,才是王道。
===========================================
在使用 HttpURLConnection 的时候,大并发对外做网络请求的时候,前期请求耗时还好,后面耗时越来越高。下面是我之前的实现代码:@Deprecated protected JSONObject callRestfulOld(String url, Map<String, Object> params) { String temp; String ret=""; JSONObject jsonRet=null; String sign = generateSign("POST", url, params);// 对参数进行加密签名 if(sign.isEmpty()) return new JSONObject("{\"ret_code\":-1,\"err_msg\":\"generateSign error\"}"); params.put("sign", sign); try{ URL u = new URL(url); HttpURLConnection conn = (HttpURLConnection)u.openConnection(); conn.setRequestMethod("POST"); conn.setConnectTimeout(10000); conn.setDoOutput(true); conn.setDoInput(true); conn.setUseCaches(false); StringBuffer param = new StringBuffer(); for (String key: params.keySet()) { param.append(key).append("=").append(URLEncoder.encode(params.get(key).toString(), "UTF-8")).append("&"); } conn.getOutputStream().write(param.toString().getBytes("UTF-8"));//System<span class="hljs-preprocessor">.out</span><span class="hljs-preprocessor">.println</span>(param)<span class="hljs-comment">;</span> conn<span class="hljs-preprocessor">.getOutputStream</span>()<span class="hljs-preprocessor">.flush</span>()<span class="hljs-comment">;</span> conn<span class="hljs-preprocessor">.getOutputStream</span>()<span class="hljs-preprocessor">.close</span>()<span class="hljs-comment">;</span> InputStreamReader isr = new InputStreamReader(conn<span class="hljs-preprocessor">.getInputStream</span>())<span class="hljs-comment">; </span> BufferedReader br = new BufferedReader(isr)<span class="hljs-comment">; </span> while((temp = br<span class="hljs-preprocessor">.readLine</span>()) != null){ <span class="hljs-keyword">ret</span> += temp<span class="hljs-comment">; </span> } br<span class="hljs-preprocessor">.close</span>()<span class="hljs-comment">; </span> isr<span class="hljs-preprocessor">.close</span>()<span class="hljs-comment">;</span> conn<span class="hljs-preprocessor">.disconnect</span>()<span class="hljs-comment">;</span> //System<span class="hljs-preprocessor">.out</span><span class="hljs-preprocessor">.println</span>(<span class="hljs-keyword">ret</span>)<span class="hljs-comment">;</span> jsonRet = new JSONObject(<span class="hljs-keyword">ret</span>)<span class="hljs-comment">;</span> } catch(java<span class="hljs-preprocessor">.net</span><span class="hljs-preprocessor">.SocketTimeoutException</span> e) { //e<span class="hljs-preprocessor">.printStackTrace</span>()<span class="hljs-comment">;</span> jsonRet = new JSONObject(<span class="hljs-string">"{\"ret_code\":-1,\"err_msg\":\"call restful timeout\"}"</span>)<span class="hljs-comment">;</span> } catch(Exception e) { //e<span class="hljs-preprocessor">.printStackTrace</span>()<span class="hljs-comment">;</span> jsonRet = new JSONObject(<span class="hljs-string">"{\"ret_code\":-1,\"err_msg\":\"call restful error\"}"</span>)<span class="hljs-comment">;</span> } return jsonRet<span class="hljs-comment">;</span> }</code>[/code]
相关文章推荐
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- Http请求连接池 - HttpClient 的 PoolingHttpClientConnectionManager
- 使用HttpClient的PoolingHttpClientConnectionManager实现Http请求连接池
- PoolingHttpClientConnectionManager连接池探究
- poolingHttpclientConnectionmanager 使用
- httpclient线程池,poolinghttpclientconnectionmanager
- org.apache.http.impl.conn.PoolingClientConnectionManager
- 使用PoolingHttpClientConnectionManager解决友盟(umeng)推送在多线程环境推送失败的问题
- httpclient源码分析之 PoolingHttpClientConnectionManager 获取连接
- HttpClient PoolingClientConnectionManager 参数含义
- 使用HttpClient4.5 PoolingHttpClientConnectionManager 提交Http请求
- HttpClient连接池抛出大量ConnectionPoolTimeoutException: Timeout waiting for connection异常排查
- httpclient 3.x 中 MultiThreadedHttpConnectionManager参数理解
- 判断HttpURLConnection与HttpClient请求
- get和post请求(HttpUrlConnection、HttpClient、AsyncHttpClient)多线程下载,断点续传