Python implementation of the bulk download of RFC documents

  • 2020-04-02 14:40:30
  • OfStack

There are a lot of RFC documents, and sometimes you want to browse even if you are not connected to the Internet, so you can only download one copy and keep it locally.
I took a look at the address list, and it went like this:
(link: http://www.networksorcery.com/enp/rfc/rfc1000.txt)
.
(link: http://www.networksorcery.com/enp/rfc/rfc6409.txt)

Ha ha, very suitable for bulk download, the first thought is thunderbolt......
When it was available, I found that it only supported three digit extensions (with thunderbolt 7). I wanted to place exactly four digits...
Depressed under the initiation of their own idea!
This is a very good thing to do in python, the principle is very simple, very little code, read first is fast.
The code is as follows:


#! /usr/bin/python
'''
  File      : getRFC.py
  Author    : Mike
  E-Mail    : Mike_Zhang@live.com
'''
import urllib,os,shutil,time def downloadHtmlPage(url,tmpf = ''):
    i = url.rfind('/')
    fileName = url[i+1:]
    if tmpf : fileName = tmpf
    print url,"->",fileName
    urllib.urlretrieve(url,fileName)
    print 'Downloaded ',fileName   
    time.sleep(0.2)
    return fileName
   
# http://www.networksorcery.com/enp/rfc/rfc1000.txt
# http://www.networksorcery.com/enp/rfc/rfc6409.txt
if __name__ == '__main__':
    addr = 'http://www.networksorcery.com/enp/rfc'   
    dirPath = "RFC"
    #startIndex = 1000
    startIndex = int(raw_input('start : '))
    #endIndex = 6409
    endIndex = int(raw_input('end : '))
    if startIndex > endIndex :
        print 'Input error!'       
    if False == os.path.exists(dirPath):
        os.makedirs(dirPath)   
    fileDownloadList = []
    logFile = open("log.txt","w")
    for i in range(startIndex,endIndex+1):
        try:           
            t_url = '%s/rfc%d.txt' % (addr,i)
            fileName = downloadHtmlPage(t_url)
            oldName = './'+fileName
            newName = './'+dirPath+'/'+fileName
            if True == os.path.exists(oldName):
                shutil.move(oldName,newName)
                print 'Moved ',oldName,' to ',newName
        except:
            msgLog = 'get %s failed!' % (i)
            print msgLog
            logFile.write(msgLog+'n')
            continue
    logFile.close()

In addition to RFC documentation, the program can be modified to do other things, such as bulk downloading of mp3s, e-books, and so on.

Ok, that's all, I hope it helped you.


Related articles: