Using Java to implement HTTP multithreading breakpoints download the file of one
- 2020-04-01 01:25:14
- OfStack
Rationale: use URLConnection to get information about the length of the file to be downloaded, the header, and so on, and set the header for the response. In addition, the input stream is obtained through URLConnection, and the file is divided into specified blocks, and a separate thread is created for each block to complete the data reading and writing. The information of the downloaded file is read by the input stream, and the information is written to the local file randomly by RandomAccessFile. At the same time, the data written by each thread is a file pointer, which is the length of the data written, and it needs to be stored in a temporary file. In this way, when the download is not completed, the next time you download, you will read the length of the last downloaded file from this file, and then continue to download from the last location. And write the length of this download to this file.
One, download file information class, entity
Encapsulates information about the resource to be downloaded
package com.hoo.entity;
/**
* <b>function:</b> Download the file information class
* @author hoojo
* @createDate 2011-9-21 In the afternoon 05:14:58
* @file DownloadInfo.java
* @package com.hoo.entity
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class DownloadInfo {
//Download file url
private String url;
//Download file name
private String fileName;
//Download file path
private String filePath;
//How many sections to download, each section with a thread to complete the download
private int splitter;
//Download file default save path
private final static String FILE_PATH = "C:/temp";
//Default number of blocks and threads
private final static int SPLITTER_NUM = 5;
public DownloadInfo() {
super();
}
public DownloadInfo(String url) {
this(url, null, null, SPLITTER_NUM);
}
public DownloadInfo(String url, int splitter) {
this(url, null, null, splitter);
}
public DownloadInfo(String url, String fileName, String filePath, int splitter) {
super();
if (url == null || "".equals(url)) {
throw new RuntimeException("url is not null!");
}
this.url = url;
this.fileName = (fileName == null || "".equals(fileName)) ? getFileName(url) : fileName;
this.filePath = (filePath == null || "".equals(filePath)) ? FILE_PATH : filePath;
this.splitter = (splitter < 1) ? SPLITTER_NUM : splitter;
}
/**
* <b>function:</b> through url Get the file name
* @author hoojo
* @createDate 2011-9-30 In the afternoon 05:00:00
* @param url
* @return
*/
private String getFileName(String url) {
return url.substring(url.lastIndexOf("/") + 1, url.length());
}
public String getUrl() {
return url;
}
public void setUrl(String url) {
if (url == null || "".equals(url)) {
throw new RuntimeException("url is not null!");
}
this.url = url;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = (fileName == null || "".equals(fileName)) ? getFileName(url) : fileName;
}
public String getFilePath() {
return filePath;
}
public void setFilePath(String filePath) {
this.filePath = (filePath == null || "".equals(filePath)) ? FILE_PATH : filePath;
}
public int getSplitter() {
return splitter;
}
public void setSplitter(int splitter) {
this.splitter = (splitter < 1) ? SPLITTER_NUM : splitter;
}
@Override
public String toString() {
return this.url + "#" + this.fileName + "#" + this.filePath + "#" + this.splitter;
}
}
Write a random file
package com.hoo.download;
import java.io.IOException;
import java.io.RandomAccessFile;
/**
* <b>function:</b> Write to file, save file
* @author hoojo
* @createDate 2011-9-21 In the afternoon 05:44:02
* @file SaveItemFile.java
* @package com.hoo.download
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class SaveItemFile {
//Storing files
private RandomAccessFile itemFile;
public SaveItemFile() throws IOException {
this("", 0);
}
public SaveItemFile(String name, long pos) throws IOException {
itemFile = new RandomAccessFile(name, "rw");
//Starts writing data at the specified pos location
itemFile.seek(pos);
}
/**
* <b>function:</b> The synchronous method is written to the file
* @author hoojo
* @createDate 2011-9-26 In the afternoon 12:21:22
* @param buff The buffer array
* @param start The starting position
* @param length The length of the
* @return
*/
public synchronized int write(byte[] buff, int start, int length) {
int i = -1;
try {
itemFile.write(buff, start, length);
i = length;
} catch (IOException e) {
e.printStackTrace();
}
return i;
}
public void close() throws IOException {
if (itemFile != null) {
itemFile.close();
}
}
}
This class basically completes writing to the local specified file pointer and returns the length of the current written file (the file pointer). This class will be called by the thread, and the file will be called by the thread after it is divided into corresponding blocks. Each thread will call this class to complete a random write to the file.
A single thread downloads a file
package com.hoo.download;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import com.hoo.util.LogUtils;
/**
* <b>function:</b> Single thread downloads files
* @author hoojo
* @createDate 2011-9-22 In the afternoon 02:55:10
* @file DownloadFile.java
* @package com.hoo.download
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class DownloadFile extends Thread {
//Download file url
private String url;
//Download file starting location
private long startPos;
//Download file end location
private long endPos;
//Thread id
private int threadId;
//Download completed
private boolean isDownloadOver = false;
private SaveItemFile itemFile;
private static final int BUFF_LENGTH = 1024 * 8;
public DownloadFile(String url, String name, long startPos, long endPos, int threadId) throws IOException {
super();
this.url = url;
this.startPos = startPos;
this.endPos = endPos;
this.threadId = threadId;
//Download write file contents in chunks
this.itemFile = new SaveItemFile(name, startPos);
}
@Override
public void run() {
while (endPos > startPos && !isDownloadOver) {
try {
URL url = new URL(this.url);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
//Set the connection timeout to 10000ms
conn.setConnectTimeout(10000);
//Set the read data timeout time to 10000ms
conn.setReadTimeout(10000);
setHeader(conn);
String property = "bytes=" + startPos + "-";
conn.setRequestProperty("RANGE", property);
//Output log information
LogUtils.log(" start " + threadId + " : " + property + endPos);
//printHeader(conn);
//Gets the file input stream and reads the file contents
InputStream is = conn.getInputStream();
byte[] buff = new byte[BUFF_LENGTH];
int length = -1;
LogUtils.log("#start#Thread: " + threadId + ", startPos: " + startPos + ", endPos: " + endPos);
while ((length = is.read(buff)) > 0 && startPos < endPos && !isDownloadOver) {
//Writes the contents of the file, returning the length of the last write
startPos += itemFile.write(buff, 0, length);
}
LogUtils.log("#over#Thread: " + threadId + ", startPos: " + startPos + ", endPos: " + endPos);
LogUtils.log("Thread " + threadId + " is execute over!");
this.isDownloadOver = true;
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (itemFile != null) {
itemFile.close();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
if (endPos < startPos && !isDownloadOver) {
LogUtils.log("Thread " + threadId + " startPos > endPos, not need download file !");
this.isDownloadOver = true;
}
if (endPos == startPos && !isDownloadOver) {
LogUtils.log("Thread " + threadId + " startPos = endPos, not need download file !");
this.isDownloadOver = true;
}
}
/**
* <b>function:</b> Print the download file header information
* @author hoojo
* @createDate 2011-9-22 In the afternoon 05:44:35
* @param conn HttpURLConnection
*/
public static void printHeader(URLConnection conn) {
int i = 1;
while (true) {
String header = conn.getHeaderFieldKey(i);
i++;
if (header != null) {
LogUtils.info(header + ":" + conn.getHeaderField(i));
} else {
break;
}
}
}
/**
* <b>function:</b> Set up the URLConnection Masquerading request information
* @author hoojo
* @createDate 2011-9-28 In the afternoon 05:29:43
* @param con
*/
public static void setHeader(URLConnection conn) {
conn.setRequestProperty("User-Agent", "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092510 Ubuntu/8.04 (hardy) Firefox/3.0.3");
conn.setRequestProperty("Accept-Language", "en-us,en;q=0.7,zh-cn;q=0.3");
conn.setRequestProperty("Accept-Encoding", "utf-8");
conn.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
conn.setRequestProperty("Keep-Alive", "300");
conn.setRequestProperty("connnection", "keep-alive");
conn.setRequestProperty("If-Modified-Since", "Fri, 02 Jan 2009 17:00:05 GMT");
conn.setRequestProperty("If-None-Match", ""1261d8-4290-df64d224"");
conn.setRequestProperty("Cache-conntrol", "max-age=0");
conn.setRequestProperty("Referer", "http://www.baidu.com");
}
public boolean isDownloadOver() {
return isDownloadOver;
}
public long getStartPos() {
return startPos;
}
public long getEndPos() {
return endPos;
}
}
This class basically completes the file download for a single thread and reads the resource information for the specified url through the URLConnection. The contents of the file are then read with InputStream, and then the SaveItemFile class is called to write locally the contents of the block being read.
Four, section multi-threaded writing file content
package com.hoo.download;
import java.io.DataInputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;
import com.hoo.entity.DownloadInfo;
import com.hoo.util.LogUtils;
/**
* <b>function:</b> Download files in batches
* @author hoojo
* @createDate 2011-9-22 In the afternoon 05:51:54
* @file BatchDownloadFile.java
* @package com.hoo.download
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class BatchDownloadFile implements Runnable {
//Download file information
private DownloadInfo downloadInfo;
//A group of locations to start downloading
private long[] startPos;
//A group of end download locations
private long[] endPos;
//Sleep time
private static final int SLEEP_SECONDS = 500;
//Subthread download
private DownloadFile[] fileItem;
//The length of the file
private int length;
//Whether the first file
private boolean first = true;
//Whether to stop downloading
private boolean stop = false;
//Temporary file information
private File tempFile;
public BatchDownloadFile(DownloadInfo downloadInfo) {
this.downloadInfo = downloadInfo;
String tempPath = this.downloadInfo.getFilePath() + File.separator + downloadInfo.getFileName() + ".position";
tempFile = new File(tempPath);
//If there is a file at the read point location
if (tempFile.exists()) {
first = false;
//Just read the content
try {
readPosInfo();
} catch (IOException e) {
e.printStackTrace();
}
} else {
//The length of the array is the number of segments
startPos = new long[downloadInfo.getSplitter()];
endPos = new long[downloadInfo.getSplitter()];
}
}
@Override
public void run() {
// First download, get the download The length of the file
if (first) {
length = this.getFileSize();// To obtain The length of the file
if (length == -1) {
LogUtils.log("file length is know!");
stop = true;
} else if (length == -2) {
LogUtils.log("read file length is error!");
stop = true;
} else if (length > 0) {
for (int i = 0, len = startPos.length; i < len; i++) {
int size = i * (length / len);
startPos[i] = size;
//Sets the location of the last end point
if (i == len - 1) {
endPos[i] = length;
} else {
size = (i + 1) * (length / len);
endPos[i] = size;
}
LogUtils.log("start-end Position[" + i + "]: " + startPos[i] + "-" + endPos[i]);
}
} else {
LogUtils.log("get file length is error, download is stop!");
stop = true;
}
}
//The child thread starts the download
if (!stop) {
//Creates an array of single-threaded download objects
fileItem = new DownloadFile[startPos.length];//startPos.length = downloadInfo.getSplitter()
for (int i = 0; i < startPos.length; i++) {
try {
//Creates a specified number of single-threaded download objects, each thread independently completes the download of the specified block content
fileItem[i] = new DownloadFile(
downloadInfo.getUrl(),
this.downloadInfo.getFilePath() + File.separator + downloadInfo.getFileName(),
startPos[i], endPos[i], i
);
fileItem[i].start();//Start the thread and start downloading
LogUtils.log("Thread: " + i + ", startPos: " + startPos[i] + ", endPos: " + endPos[i]);
} catch (IOException e) {
e.printStackTrace();
}
}
//Loop writes download file length information
while (!stop) {
try {
writePosInfo();
LogUtils.log("downloading ... ");
Thread.sleep(SLEEP_SECONDS);
stop = true;
} catch (IOException e) {
e.printStackTrace();
} catch (InterruptedException e) {
e.printStackTrace();
}
for (int i = 0; i < startPos.length; i++) {
if (!fileItem[i].isDownloadOver()) {
stop = false;
break;
}
}
}
LogUtils.info("Download task is finished!");
}
}
private void writePosInfo() throws IOException {
DataOutputStream dos = new DataOutputStream(new FileOutputStream(tempFile));
dos.writeInt(startPos.length);
for (int i = 0; i < startPos.length; i++) {
dos.writeLong(fileItem[i].getStartPos());
dos.writeLong(fileItem[i].getEndPos());
//LogUtils.info("[" + fileItem[i].getStartPos() + "#" + fileItem[i].getEndPos() + "]");
}
dos.close();
}
/**
* <b>function:</b> Reads the location of the write point
* @author hoojo
* @createDate 2011-9-23 In the afternoon 05:30:29
* @throws IOException
*/
private void readPosInfo() throws IOException {
DataInputStream dis = new DataInputStream(new FileInputStream(tempFile));
int startPosLength = dis.readInt();
startPos = new long[startPosLength];
endPos = new long[startPosLength];
for (int i = 0; i < startPosLength; i++) {
startPos[i] = dis.readLong();
endPos[i] = dis.readLong();
}
dis.close();
}
/**
* <b>function:</b> Gets the length of the downloaded file
* @author hoojo
* @createDate 2011-9-26 In the afternoon 12:15:08
* @return
*/
private int getFileSize() {
int fileLength = -1;
try {
URL url = new URL(this.downloadInfo.getUrl());
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
DownloadFile.setHeader(conn);
int stateCode = conn.getResponseCode();
//Determine whether HTTP status is HTTP/1.1 206 Partial Content or 200 OK
if (stateCode != HttpURLConnection.HTTP_OK && stateCode != HttpURLConnection.HTTP_PARTIAL) {
LogUtils.log("Error Code: " + stateCode);
return -2;
} else if (stateCode >= 400) {
LogUtils.log("Error Code: " + stateCode);
return -2;
} else {
//To obtain the length of the
fileLength = conn.getContentLength();
LogUtils.log("FileLength: " + fileLength);
}
// read The length of the file
DownloadFile.printHeader(conn);
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return fileLength;
}
}
This class is primarily finished reading the contents of the specified url resource and getting the length of the resource. Then divide the resource into the specified number of blocks, and store the starting and ending locations of each block in an array. Each block has a separate thread to start the download. Before starting the download, you need to create a temporary file that writes to the start and end download pointer positions of the current download thread.
Five, tool class, test class
Logging utility class
package com.hoo.util;
/**
* <b>function:</b> Logging utility class
* @author hoojo
* @createDate 2011-9-21 In the afternoon 05:21:27
* @file LogUtils.java
* @package com.hoo.util
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public abstract class LogUtils {
public static void log(Object message) {
System.err.println(message);
}
public static void log(String message) {
System.err.println(message);
}
public static void log(int message) {
System.err.println(message);
}
public static void info(Object message) {
System.out.println(message);
}
public static void info(String message) {
System.out.println(message);
}
public static void info(int message) {
System.out.println(message);
}
}
Download tool class
package com.hoo.util;
import com.hoo.download.BatchDownloadFile;
import com.hoo.entity.DownloadInfo;
/**
* <b>function:</b> Block multi-threaded download utility classes
* @author hoojo
* @createDate 2011-9-28 In the afternoon 05:22:18
* @file DownloadUtils.java
* @package com.hoo.util
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public abstract class DownloadUtils {
public static void download(String url) {
DownloadInfo bean = new DownloadInfo(url);
LogUtils.info(bean);
BatchDownloadFile down = new BatchDownloadFile(bean);
new Thread(down).start();
}
public static void download(String url, int threadNum) {
DownloadInfo bean = new DownloadInfo(url, threadNum);
LogUtils.info(bean);
BatchDownloadFile down = new BatchDownloadFile(bean);
new Thread(down).start();
}
public static void download(String url, String fileName, String filePath, int threadNum) {
DownloadInfo bean = new DownloadInfo(url, fileName, filePath, threadNum);
LogUtils.info(bean);
BatchDownloadFile down = new BatchDownloadFile(bean);
new Thread(down).start();
}
}
Download the test class
package com.hoo.test;
import com.hoo.util.DownloadUtils;
/**
* <b>function:</b> Download the test
* @author hoojo
* @createDate 2011-9-23 In the afternoon 05:49:46
* @file TestDownloadMain.java
* @package com.hoo.download
* @project MultiThreadDownLoad
* @blog http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class TestDownloadMain {
public static void main(String[] args) {
/*DownloadInfo bean = new DownloadInfo("http://i7.meishichina.com/Health/UploadFiles/201109/2011092116224363.jpg");
System.out.println(bean);
BatchDownloadFile down = new BatchDownloadFile(bean);
new Thread(down).start();*/
//DownloadUtils.download("http://i7.meishichina.com/Health/UploadFiles/201109/2011092116224363.jpg");
DownloadUtils.download("http://mp3.baidu.com/j?j=2&url=http%3A%2F%2Fzhangmenshiting2.baidu.com%2Fdata%2Fmusic%2F1669425%2F%25E9%2599%25B7%25E5%2585%25A5%25E7%2588%25B1%25E9%2587%258C%25E9%259D%25A2.mp3%3Fxcode%3D2ff36fb70737c816553396c56deab3f1", "aa.mp3", "c:/temp", 5);
}
}
Multithreaded downloads are mostly in parts 3 and 4, but the rest is easy to understand. The source code provides the corresponding comments, easy to understand.