Of Stack

pytorch tutorial notes on the network construction process

  • 2021-11-30 00:25:41
  • OfStack
Directory Construction Network Definition 1 Network loss FunctionBackprop Update Weight

Reference website

Build a network

We can build the network with the torch. nn package, and now that you've seen autograd, nn defines models and differentiates from autograd. An nn. Module includes many layers, and the forward method returns output.

A typical training process includes the following steps:
1. Define a network structure with 1 trainable parameters
2. Develop the input iterata for the dataset
3. Computing Output over the network
4. Calculate loss
5. Back Propagation Calculating Gradients
6. Update weights 


weight = weight - learning_rate * gradient

Define 1 network

Let's define a network 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)

out 


Net(
  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

All we need to do is define the forward and backward functions, which will automatically derive through the functions you define, and you can use all the Tensor operations in the forward functions.
We use the net. parameters () function to return learnable parameters 


params = list(net.parameters())
print(len(params))
print(params[0].size())  # conv1's .weight

out 


10
torch.Size([6, 1, 5, 5])

Let's try the 32*32 input node, because the input to the lenet network should be 32*32, and in order to use lenet on the MNIST dataset, we need to turn the picture reshpe to 32*32 


input = torch.randn(1,1,32,32)
oyt = net(input)
print(out)

out 


tensor([[-0.1346,  0.0581, -0.0396, -0.1136, -0.1128,  0.0180, -0.1226,
         -0.0419, -0.1150,  0.0278]])

Zero derivative buffers All parameters will be randomly derived 


net.zero_grad()
out.backward(torch.randn(1,10))

torch. nn only supports mini-batch, not a single sample
For example, the nn. Conv2d input is a 4-dimensional tensors 


nSamples * nChannels * Height * Width

If you only have a single sample, add a false batch dimension using input. unsqueeze (0)
Before post-processing, let's look at what classes we have learned

Recap:

torch.Tensor - A multi-dimensional array with support for autograd operations like backward(). Also holds the gradient w.r.t. the tensor.
nn.Module - Neural network module. Convenient way of encapsulating parameters, with helpers for moving them to GPU, exporting, loading, etc.
nn.Parameter - A kind of Tensor, that is automatically registered as a parameter when assigned as an attribute to a Module.
autograd.Function - Implements forward and backward definitions of an autograd operation. Every Tensor operation, creates at least a single Function node, that connects to functions that created a Tensor and encodes its history.

At present, we have learned:
1. Define a neural network
2. Handling inputs and using backward propagation
We also need to learn:
1. Calculate loss
2. Update weights

loss Function

Loss function accepts the pair (output traget) as input and calculates a value reflecting the distance to the target.
There are many loss function in the nn package, and the simplest one is nn. MSELoss, which is the mean square error of input and output.

For example 


output = net(input)
target = torch.arrange(1,11)
target = target.view(1m-1)
criterion = nn.MSELoss()
loss = criterion(output,target)
print(loss)

Out: 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)
0

Backprop

All we need to do for back propagation is do loss. backward (), we need to clear the existing gradient

Update weights

The simplest and most common way to update weights is SGD (Stochastic Gradient Descent) 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)
1

We can implement the above formula with simple code: 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)
2

But we can also use different update rules, such as SGD, Nesterov-SGD, Adam, RMSProp, etc.
To use these, we need the torch. optim package, which is also simple to use. 


import torch.optim as optim 
#creat you optimizer
optimizer = optim.SGD(net.parameters(),lr = 0.01)
#in your training loop:
optimizer.zero_grad()
output = net(input)
loss = criterion(output,target)
loss.backward()
optimizer.step()

Note that gradient must be cleared
Now let's call loss. backward () and look at the difference between con1 and bias 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)
4

out 


import torch
import torch as nn
import torch.nn.functional as F
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__(
        #1 input image channel ,6output image channel ,5*5convolytion kernel
        self.conv1 = nn.Conv2d(1,6,5)
        self.conv2 = nn.Conv2d(6,16,5)
        # an affine operation:y = Wx+b
        self.fc1 = nn.Linear(16*5*5,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
    def forward(self,x):
        #max pooling
        x.F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        #2   =     ( 2,2 ) 
        x.F.max_pool2d(F.relu(self.con2(x)),2)
        x = x.view(-1,self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return  x
    def num_flat_features(self,x):
        size = x.size()[1:]
        num_feature = 1
        for s in size:
            num_features *=s
        return num_features

net = Net()
print(net)
5

Now, we see how to use loss function
Important
torch contains many loss function and other packages, the rest of the documentation can be found here
http://pytorch.org/docs/nn

The above is the pytorch tutorial network construction process note details, more information about the pytorch tutorial please pay attention to other related articles on this site!

Related articles: