Of Stack

Implementation method of downloading road network pictures in batches and generating html

  • 2021-06-28 06:13:24
  • OfStack

Using ajax to load content asynchronously, we find the relevant code in its js code 


type : 'POST',
    url : '/index.php/request/new_data2/' + times + '/'+locinfo[domn][0],
    dataType : 'json',

The returned json string is a serialized array containing dictionaries, with dict ['t'] and dict ['i'], dict ['t'] containing the description of the picture, and dict ['i'] containing the url of the picture. Knowing this, you can start the python script

import related modules 


# -*- coding: utf-8 -*-
import urllib2 as url
import json
import sys
import os
from datetime import *

(Fixed unable to get bug of the specified type, the last number in the requested url represents the type)

Get json: index is the download page, type is tws (too obscene) tr (too hot) tgx (too funny) tml (too cute) tht (too nice tyy (too eye-catching) 1

Then you create the html file 


def create_html(alllist,name):
  html_head='<html><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><title>duilu</title><body>'
  html_end="</body></html>"
  f=open("%s.html"%(name),"w")
  f.write(html_head)
  for x in range(len(alllist)):
    f.write('<div><img src="%s/%s.gif"/>'%(name,str(x)))
    f.write('<p>%s</p></div>'%(alllist[x]['t'].encode('utf-8')))   
  f.write(html_end)
  f.close()

Download pictures 


def download(list,dirname,index=0):
  os.chdir(dirname)
  for dict in list:
    imgurl=dict['i']
    text= dict['t']
    print index
    print imgurl
    print text
    res=url.urlopen(imgurl)
    img_type=".gif"
    content_type=res.headers["content-type"]
    if content_type=="image/jpeg":
      type=".jgp"
    filepath="%s"%(str(index)+img_type)
    f=open(filepath,"wb")
    f.write(res.read())
    f.close()
    res.close()
    index+=1
  os.chdir("../")

The main function, which is used to call the above functions 


def start(type,lenght):
  lenght=int(lenght)
  now=datetime.now()
  now=now.strftime("%m-%d %H.%M.%S")
  os.mkdir(type+now)
  
  alllist=[]
  for x in range(0,lenght):
    list=get_json(x,type)
    alllist.extend(list)
  create_html(alllist,type+now)
  download(alllist,type+now)
  print "\r\n\r\n==============OK==============\r\n\r\n"

1 loop body to get user input 


while(True):
  print " Input 
 tws( Too obscene 
 ) tr( Too hot 
 ) tgx( Too funny 
 ) tml( It's so cute 
 ) tht( Too nice to hear 
  tyy( Too eye-catching 
 )  Yeah 
 1\r\nexit: Quit 
 "
  type=raw_input()
  all_type=["tgx","tws","tyy","tr","tml","tht"]
  if type in all_type:
    print " Type the number of pages to download 
 :"
    lenght=raw_input()
    start(type,lenght)
  elif type=="exit":
    break
  else:
    print "\r\n Input error 
 \r\n"

When ok is complete, the script will generate a time-named html file and a folder with the same name in the current directory to store the pictures.

After 1 test, it took a few minutes to download more than 100 pictures, so I don't think I need to download them with multiple threads.

You can also slightly modify the place where html is generated, and turn it into a paged display, and then drag the webpage into an Android phone to see it. It is also good

Using python is that simple!

Related articles: