爬虫入门 requests库

异常处理代码框架

%title插图%num

import requests
def getHtmlText(url):
try:
r=requests.get(url,timeout=30)
r.raise_for_status()/不是200 引发httperror
r.encoding=r.apparent_encoding
return r.text
except:
return “产生异常”
if __name__ ==”__main__”:
url=”http://www.ba23idu1.com”
print(getHtmlText(url))
r在这是response类型的对象

Http 协议：

**HTTP**，Hypertext Transfer Protocol，超文本传输协议

HTTP是一个基于“请求与响应”模式的、无状态的应用层协议

HTTP协议采用URL作为定位网络资源的标识，URL格式如下：

http://host[:port][path]

host: 合法的Internet主机域名或IP地址

port: 端口号，缺省端口为80

path: 请求资源的路径

%title插图%num

requests库的七个主要方法:

用法：

head:
import requests
r=requests.head(“http://www.baidu.com”)
print(r.headers)
print(r.text)
/*
{‘Cache-Control’: ‘private, no-cache, no-store, proxy-revalidate, no-transform’, ‘Connection’: ‘keep-alive’, ‘Content-Encoding’: ‘gzip’, ‘Content-Type’: ‘text/html’, ‘Date’: ‘Fri, 07 May 2021 01:25:26 GMT’, ‘Last-Modified’: ‘Mon, 13 Jun 2016 02:50:08 GMT’, ‘Pragma’: ‘no-cache’, ‘Server’: ‘bfe/1.0.8.18’}
*/

post
payload ={‘key1′:’value1′,’key2′:’value2′}
r=requests.post(url,data=payload)
或者
r=requests.post(url,data=’abc’)
// 向url post一个字符串 自动编码为data

put
payload = {‘key1’: ‘value1’, ‘key2’: ‘value2’}
r = requests.put(url, data = payload)
1
2
requests.request(method, url, **kwargs)

method 请求方式，对应get/put/post等7种

url 拟获取页面的url链接

**kwargs 控制访问的参数，共13个

method: get head post put patch delete options

**kwargs:
params : 字典或字节序列，作为参数增加到url中
kv = {‘key1’: ‘value1’, ‘key2’: ‘value2’}
r = requests.request(‘GET’, ‘http://python123.io/ws’, params=kv)
data : 字典、字节序列或文件对象，作为Request的内容
kv = {‘key1’: ‘value1’, ‘key2’: ‘value2’}
r = requests.request(‘POST’, ‘http://python123.io/ws’, data=kv)
json : JSON格式的数据，作为Request的内容
kv = {‘key1’: ‘value1’}
r = requests.request(‘POST’, ‘http://python123.io/ws’, json=kv)
headers : 字典，HTTP定制头
hd = {‘user‐agent’: ‘Chrome/10’}
r = requests.request(‘POST’, ‘http://python123.io/ws’, headers=hd)
cookies : 字典或CookieJar，Request中的cookie
auth : 元组，支持HTTP认证功能
files : 字典类型，传输文件
fs = {‘file’: open(‘data.xls’, ‘rb’)}
r = requests.request(‘POST’, ‘http://python123.io/ws’, files=fs)
timeout : 设定超时时间，秒为单位
r= requests.request(‘GET’, ‘http://www.baidu.com’, timeout=10)
proxies : 字典类型，设定访问代理服务器，可以增加登录认证
pxs = { ‘http’: ‘http://user:pass@10.10.10.1:1234’
‘https’: ‘https://10.10.10.1:4321’ }
r = requests.request(‘GET’, ‘http://www.baidu.com’, proxies=pxs)
allow_redirects : True/False，默认为True，重定向开关
stream : True/False，默认为True，获取内容立即下载开关
verify : True/False，默认为True，认证SSL证书开关
cert : 本地SSL证书路径

requests.get(url, params=None, **kwargs)
requests.head(url, **kwargs)
requests.post(url, data=None, json=None, **kwargs)
requests.put(url, data=None, **kwargs)
requests.patch(url, data=None, **kwargs)
requests.delete(url, **kwargs)
requests.get(url, params=None, **kwargs)

模拟浏览器

import** requests

kv={‘user-agent’:’Mozilla/5.0′}

r=requests.get(“https://www.amazon.com/”,headers=kv)

print(r.status_code)

200

百度接口:
http://www.baidu.com/s?wd=keyword
360:
http://www.so.com/s?q=keyword

import requests
kv={‘wd’:’王泽坤’}
r=requests.get(“http://www.baidu.com/s”,params=kv)
print(r.text)

ip查询接口:

http://m.ip138.com/ip.asp?ip=ipaddress