useradd nginx<\/code><\/pre>\n\n\n\nAfter that we need to edit the configuration file (check out where it’s located on your installation)<\/p>\n\n\n\n
vi \/var\/nginx\/nginx.conf<\/code><\/pre>\n\n\n\nthen delete all the content using<\/p>\n\n\n\n
:1,$d<\/code><\/pre>\n\n\n\npaste this configuration<\/p>\n\n\n\n
load_module \/usr\/lib\/nginx\/modules\/ngx_stream_module.so;\nworker_rlimit_nofile 65536;\nuser nginx;\nworker_processes 1;\nerror_log \/var\/log\/nginx\/error.log warn;\npid \/var\/run\/nginx.pid;\nevents { worker_connections 65536; }\nhttp { }\ninclude \/etc\/nginx\/tcpconf.d\/*;<\/code><\/pre>\n\n\n\nand finally save and quit.<\/p>\n\n\n\n
This configuration will load the stream module, which is what we need to take the incoming TCP stream on a local port and redirect it to another host on its port. We also specify a high limit for the number of workers since we need to make sure nginx can work on enough files at the same time. It’s also important to notice that the worker limit MUST be a power of 2<\/strong> or it will be disregarded! Yeah, that costed me almost an hour of debugging\u2026<\/p>\n\n\n\nWe should check if the file limits for the nginx user are high enough or if we should increase them. To test it, you can type<\/p>\n\n\n\n
su - nginx\nulimit -Hn\nulimit -Sn<\/code><\/pre>\n\n\n\nIf the limits are too low, you can edit this file<\/p>\n\n\n\n
vi \/etc\/sysctl.conf<\/code><\/pre>\n\n\n\nand append this line<\/p>\n\n\n\n
fs.file-max = 70000<\/code><\/pre>\n\n\n\nAfter saving and closing, edit this other file<\/p>\n\n\n\n
vi \/etc\/security\/limits.conf<\/code><\/pre>\n\n\n\nand add these two lines<\/p>\n\n\n\n
nginx soft nofile 10000\nnginx hard nofile 30000<\/code><\/pre>\n\n\n\nMake sure to use tabs instead of spaces as the latter can cause issues to some people. Finally, edit the file<\/p>\n\n\n\n
vi \/etc\/pam.d\/common-session<\/code><\/pre>\n\n\n\nand append this line<\/p>\n\n\n\n
session required pam_limits.so<\/code><\/pre>\n\n\n\nNow that we edited all the required files, we can reload the sysctl process and nginx (although killing the nginx would be a better way to ensure it reloads entirely)<\/p>\n\n\n\n
sysctl -p\nnginx -s reload<\/code><\/pre>\n\n\n\nWe’re almost good to go! We just need a few more steps. First of all, create a folder called tcpconf.d inside your nginx root folder (where the nginx.conf file is)<\/p>\n\n\n\n
mkdir tcpconf.d\ncd tcpconf.d<\/code><\/pre>\n\n\n\nInside this folder we need to create a file with this syntax (you can give it any name you want)<\/p>\n\n\n\n
stream {\nserver { listen 30000; proxy_pass 1.2.3.4:8080; }\nserver { listen 30001; proxy_pass 5.6.7.8:8080; }\n}<\/code><\/pre>\n\n\n\nYou should be able to generate this file pretty easily by using this python script that I wrote for the occasion<\/p>\n\n\n\n
output = open('output.txt', 'w+')\np = open('proxies.txt', 'r')\nproxies = p.read().splitlines()\np.close()\n\noutput.write('stream{\\n')\n\ni = 30000\nfor proxy in proxies:\n\toutput.write('server { listen %d; proxy_pass %s; }\\n' % (i, proxy))\n\ti += 1\n\noutput.write('}\\n')\noutput.close()<\/code><\/pre>\n\n\n\nThis script will take your proxy list in the format of one proxy per line (e.g. 12.34.56.78:8080) and generate a file called output.txt that is ready to be placed inside the tcpconf.d folder. By default this script will use the ports from 30000 onwards, so if you want to use other ports you can edit that value but do not choose a low value<\/strong> since many of the ports are reserved for other processes (like SSH on port 22) and will give you errors when you start nginx.<\/p>\n\n\n\nFinally we need to generate a proxy list that we can use in our favourite scraping software (like OpenBullet for example). Run this python script<\/p>\n\n\n\n
output = open('output.txt', 'w+')\n\nip = 'YOUR_SERVER_IP'\nfor i in range(30000, 55000):\n\toutput.write('%s:%d\\r\\n' % (ip, i))\n\noutput.close()<\/code><\/pre>\n\n\n\nYou have to replace <\/strong>your own IP and define <\/strong>your port range basing on how many proxies you used (I had 25000 so I needed to output 25000 ports starting from port 30000, which is the beginning of the default port range of the previous script).<\/p>\n\n\n\nFinally, it’s very important that you import these proxies as SOCKS5<\/strong> or they will not work, since we’re redirecting TCP streams and not HTTP requests (they are on two different layers of the protocol stack).<\/p>\n\n\n\nCongratulations, we are officially done and we can finally start nginx<\/p>\n\n\n\n
service nginx start<\/code><\/pre>\n\n\n\nIf you have a large amount of proxies this might take some time, but if you patiently wait you should have your middle proxies visible on all the incoming ports, ready to accept some traffic!<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
This was a great learning experience as I tried to use nginx for something I didn’t know it could do, by using the stream module which is optional and needs to be activated with a directive in the configuration file. I also learned how linux handles the maximum open files allowed for each process and messed around with various configuration files in order to get it to work. Next time I will try one of the open source VPN solutions and report my findings on this blog.<\/p>\n","protected":false},"excerpt":{"rendered":"
There are many services that provide a large quantity of proxies that you can use when scraping websites to avoid getting timed out. But what if you want to buy a single proxy package and use it from multiple computers on different networks? Types of security The main types of security that proxy services use… <\/p>\n
Read More<\/i><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_markdown_editor_remember":false},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/posts\/9"}],"collection":[{"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/openbullet.dev\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9"}],"version-history":[{"count":11,"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/posts\/9\/revisions"}],"predecessor-version":[{"id":21,"href":"https:\/\/openbullet.dev\/index.php?rest_route=\/wp\/v2\/posts\/9\/revisions\/21"}],"wp:attachment":[{"href":"https:\/\/openbullet.dev\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/openbullet.dev\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/openbullet.dev\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}