Use squid to set up the http proxy method

Recently, in the process of building some infrastructure, I encountered the problem of http agency. Mainly, the tools on many hosts only support the http_proxy setting without authentication information, such as: export http_proxy='http://10.10.1.1:8090', but not export http_proxy='http://tonybai: passwd@10.10.1.1:8090 '.

Or it only provides proxy_host and proxy_port in its command-line options, but does not support carrying authentication information. Internal access to external information must also go through the company's authenticated proxy server, which, in short, makes me feel bad. Thus, an idea was generated: whether it is possible to set up an internal http intermediate agent, which can be accessed by the internal host of the department through the agent configuration without identity authentication information, and the intermediate agent will forward all internal http request to the company agent and carry the configured authentication information.

As for the agency of http, I am totally innocent, so Google opened (just as Google has not been very helpful recently, you know why).

First tested 1 tinyproxy, this tool is quite small and simple, through the apt - get can be installed directly under ubuntu, / etc tinyproxy/tinyproxy conf configuration is very simple and clear. However, the configuration line in the profile involved in forwarding to upstream proxy server only supports the form "Upstream host:port" but not "Upstream tonybai:passwd @host :port", and there is no other place to support the configuration of authentication information. On its official bugzilla, many people have reported this situation, but it seems that the latest version does not include this feature, 10 points pity!

So I plan to change a heavyweight agent tool -nginx. The default installation of nginx under Ubuntu 9.04 is version 0.65. Although nginx is powerful, the configuration is not so "complicated", but the problem is that nginx itself seems to focus more on load balancing and reverse generation, and there is little data to meet my problem scenario. There are so many nginx configuration commands and variables that it is difficult to figure out what they mean in a short time. Several attempts to match the cat with the tiger were unsuccessful. I thumbed through the only eleven books of nginx in China, the real nginx, but it was too thick. I turned over three chapters and put them down. Change tools!

The most traditional open source free http proxy tool is squid. It is estimated that its market share is also among the best. The default installation of Ubuntu 9.04 is version 2.7, which is not very old. The official website of squid still provides detailed configuration documents of version 2.7. The default configuration file for squid is super large, with nearly 5k lines, though most of the content is commented out. So you use the command to filter out the uncommented lines, which are the actual configuration in effect.

There is also less information on the Internet about how squid can forward the http request received to http proxy server, the superior with the identification right, but let me find one. Try it out with this configuration recommendation. / etc squid/squid conf configuration in the following:

access_log /var/log/squid/access.log squid
debug_options ALL,1
hosts_file /etc/hosts
coredump_dir /var/spool/squid

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32

acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12 # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network

http_port 10.10.13.17:3128

http_access allow localnet
http_access allow localhost
http_access deny all

cache_peer proxy.yourcompany.com parent port_of_company_httpproxy 0 no-query default login=user:passwd
never_direct allow localnet

After configuration, restart squid (sudo/etc/init d/squid restart). Change the proxy configuration of the Chrome browser to this proxy, try opening "baidu.com", and get stuck in a long wait. So open squid access log/var/log/squid/access log, failed to see the following information:

TCP_DENIED/400 1709 GET error: es1064en-request / -text /html
TCP_DENIED/400 1709 GET error: invalid-request, NONE/ -text /html
10.10.13.235 TCP_DENIED/400 1678 GET error: invalid-request, NONE/ -text /htm

What a mistake! Switch to IE, and you'll see the same error as before. In/var/log/squid/cache log, can also find the following error:

2012/11/21 13:43:56| clientTryParseRequest: FD 12 (10.10.13.235:4247) Invalid Request

Constantly changing the squid.conf configuration, constantly changing the browser proxy configuration, and constantly failing. Always changing the browser's proxy configuration felt like a 10-minute struggle, so I switched to the curl tool to test the proxy. curl recognizes the http_proxy environment variable. Change the http_proxy environment variables to export http_proxy = http: / / 10.10.13.17:3128, at the command line type curl http: / / baidu com, incredibly obtain the following results:

$ curl http://baidu.com
<html>
<meta http-equiv="refresh" content="0;url=http://www.baidu.com/">
</html>

Back to access.log observed the following success log:

1353476863.916 0 10.10.13.235 TCP_HIT / 200 677 GET http: / / baidu com / � NONE / - text/html

So I tried to download external files with wget, access external svn repository with subversion, and install ruby package with rvm, all of which were successful! That's what I want! I was accidentally hit! Although so far I still don't know why http request issued by the browser cannot be recognized ^_^.

Squid is a powerful http agent, which is used by many enterprises as the tool of http agent at corporate level. Its configuration reference is enough to write a thick book (there are already such books on the market). Fortunately, I don't need the weird configuration for my scenario.