linux使用wget
2016-01-19 17:55
716 查看
from the web, via HTTP, HTTPS and FTP protocols. When you are using wget to
download a file at a particular HTTP url, wget sends an appropriate
HTTP request to a destination web server.
To view default HTTP request header being sent by wget, you
can use "-d" option.
$ wget -d http://www.google.com/
---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.12 (linux-gnu) Accept: */* Host: www.google.com Connection: Keep-Alive ---request end---
Sometimes you may want to customize the default HTTP request header used by wget.
For example, you may want to customize "User-Agent" field as some sites rely on "User-Agent" string to block robots like wget to
retrieve their content. You may want to add an additional "Accept-Encoding" field in order to test encoding schemes of your web server. In some other cases, you may need to set "Host" field properly to be able to access a web server running on name-based virtual
hosting.
wget allows you to send an HTTP request with custom HTTP headers.
To supply custom HTTP headers, use "--header" option. You can use "--header" option as many time as you want in a single run.
$ wget -d --header="User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11" --header="Referer: http://xmodulo.com/" --header="Accept-Encoding: compress, gzip" http://www.google.com/
---request begin--- GET / HTTP/1.0 User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 Accept: */* Host: www.google.com Connection: Keep-Alive Referer: http://xmodulo.com/ Accept-Encoding: compress, gzip ---request end---
If you would like to permanently set the default HTTP request header you want to use with wget,
you can use ~/.wgetrc configuration file. You can specify as many header fields as you want in ~/.wgetrc.
$ vi ~/.wgetrc
header = User-Agent: Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 header = Referer: http://xmodulo.com/ header = Accept-Encoding: compress, gzip
Once you have configured ~/.wgetrc, you no longer need to use "--header" option with wget.
curl is another command-line tool with similar functionality
as wget. The curl utility
also allows you to set a custom HTTP header. Refer to this
guideline for detail on curl.
相关文章推荐
- Linux 动画显示
- Linux内核-源码目录结构分析
- Linux下mysql中文乱码问题
- linux grep 正则表达式
- centos6 安装多个mysql5.7.10
- linux加密文件系统
- Linux系统下Hadoop环境的搭建
- centos 6 编译emacs-24.5
- Centos7修改profile错误导致命令行不能用,情况的解救方案
- Linux下" >/dev/null 2>&1 "相关知识说明
- 转: Linux与JVM的内存关系分析
- Linux调试死锁
- Linux学习第4天:linux系统管理
- Linux学习第3天:linux目录管理
- 每天一个linux命令(3):pwd命令
- linux 误删文件后 恢复
- centos下c代码 更改配置文件中配置项的数值
- linux 如何查看防火墙是否开启
- linux查看机器信息:cpu、内存和磁盘
- Centos搭建SVN服务器三步曲