Menu
Machine learning concepts. Network training and evaluation
1. Building a network model according to the problem being solved
The neural network model consists of two layers – an LSTM layer and an output Dense layer. The reason for choosing an LSTM layer is the need to process sequences of time-related data and to find data correlations. To perform these operations is needed a layer with memory, such as the LSTM layer that is capable of detecting long-term dependencies. The Dense layer limits the number of output parameters to one (corresponding to the closing price), applying an activating function to the outputs of the previous layer. A linear activation function is chosen for the Dense layer so that the neural network can predict higher values than those it was trained with. This can not be achieved with a hyperbolic tangent activation function or a logical sigmoidal activation function.
The input layer of the neural network has 4 neurons – one for each of the input parameters, the hidden LSTM layer – 128 and the output Dense layer – 1, which is the number of possible outputs (remember that we are having a regression task). The number of neurons in the LSTM layer can be changed in order to achieve better results and it is part of the hyperparameters’ tuning (trial and error process).
2. Setting up network hyperparameters
Choosing the right hyperparameters is essential for successful network training. Poorly selected hyperparameters may result in slow or no learning at all.
2.1. Batch size
Batch size is the number of examples propagated through the network. With a selected value of 100 for this hyperparameter, the algorithm divides the training data into groups of 100 records and trains the network with each of the groups. A good starting point when tuning the batch size is 32. Other common choices are 64 and 128.
2.2. Number of epochs and number of iterations
Each epoch represents a complete passage of all training data through the neural network. The small number of epochs is not enough for the network to learn good values of its weight matrixes. Too many epochs lead to a problem known as overfitting, in which the neural network predicts the training data very accurately, but makes bad predictions on data it has never seen. The reason for this is that the network remembers the training data instead of learning the correlations between its items. The “Early stopping” technique can be used for selecting a suitable number of epochs in which the neural network stops learning at the point where the error function stops decreasing. Using this technique solves the overfitting problem.
The number of iterations is equal to the number of all training examples divided by the batch size value. If the data has 1000 examples and the batch size is 250, then the number of iterations is 4, ie. with 4 iterations, 1 epoch can be completed.
2.3. Learning rate
It is one of the most important hyperparamers. Too small or too large values for the learning rate may lead to very poor, very slow or no training. Values typically range from 0.1 to 1.10-6. 1.10-3 is a good starting point to experiment with.
2.4. Activation functions
The LSTM layer has a sigmoidal activating function used to control the LSTM gates because the sigmoidal function has output values ranging from 0 to 1. A linear activation function is selected for the Dense layer activation function.
2.5. Loss function
The mean squared error function is used for solving this regression problem. The produced error by the neural network is measured as the arithmetic mean of the sum of the differences between the predictions and the actual observations on the degree of two. The following formula gives a better explanation of the equation.
3. Network training
After creating and configuring the neural network model, the training process takes place, where the network is trained on the training data. The following graph shows the process of training the neural network. The blue line represents the correct outputs for each example, and the orange one – the predictions made by the network.
4. Network testing
The testing determines how satisfactory the predictions made by the neural network are. The network predicts data that it has not seen, and the predicted values are compared to the correct outputs. The smaller the deviations between the two values, the better predictions it makes. The following graph shows the process of testing the neural network. The blue line represents the correct outputs for each example, and the orange one – the predictions made by the network.
5. Comparing the results
After comparing the two graphs it can be easily seen that the errors on the test data are greater than those on the training. This is the expected result of the comparison. By looking at the test graph, we come to the conclusion that the results of the predictions are satisfactory. The neural network has managed to predict the trend of the cryptocurrency’s market closing price. The next thing we can do is to try other network configurations or fine tune the hyperparameters of the network.
6. Tuning the hyperparameters in order to achieve more satisfactory results
Here is a series of different combinations of hyperparametric values in order to achieve more accurate predictions. The results of the experiments are presented in the following figures. Each of the combinations is trained using the “Early stopping” method.
Training graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 64, loss function error: 0.0040
Testing graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 64, loss function error: 0.0042
Training graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 128, loss function error: 0.0036
Testing graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 128, loss function error: 0.0038
Training graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 256, loss function error: 0.0037
Testing graph. Learning rate: 10-3, batch size: 32 number of neurons in the hidden layer: 256, loss function error: 0.0037
Training graph. Learning rate: 10-3, batch size: 32, number of neurons in the hidden layer: 512, loss function error: 0.0034
Testing graph. Learning rate: 10-3, batch size: 32 number of neurons in the hidden layer: 512, loss function error: 0.0037
Training graph Learning rate: 10-3, batch size: 64, number of neurons in the hidden layer: 512, loss function error: 0.0036
Testing graph. Learning rate: 10-3, batch size: 64, number of neurons in the hidden layer: 512, loss function error: 0.0036
Training graph. Learning rate: 10-3, batch size: 128, number of neurons in the hidden layer: 512, loss function error: 0.0029
Testing graph. Learning rate: 10-3, batch size: 128, number of neurons in the hidden layer: 512, loss function error: 0.0029
After the hyperparameter setting has been made, it is found that the most optimal configuration for the neural network is:
- learning rate: 1.10-3
- batch size: 128
- number of neurons in the hidden layer: 512
7. Persisting the trained model. Exporting the model for further use and loading in other environments
Machine learning models are often trained in a Python environment and are used in production in another environment. The neural network for predicting cryptocurrency prices is modelled in Python and then loaded in Java environment.
To save the trained Python model, use the following expression:
model.save(‘model.h5’)
The model is exported to a file named “model” with extension .h5. Apart from the architecture of the neural network, the weighting coefficients of the network as a result of its training are also recorded. In this way, it can be used directly for forecasting.
Loading the model in Java environment is done using the method importKerasSequentialModelAndWeights()of the KerasModelImport.class which is part of the DL4J machine learning library.
MultiLayerNetwork model = KerasModelImport.importKerasSequentialModelAndWeights(modelPath, true);
Letzte Beiträge
Share :
Share :
Weitere Beiträge
Repository in Android’s MVVM architecture
Repository is a class which purpose is to provide a clean API for accessing data. What that means is that the Repository can gather data from different data sources(different REST APIs, cache, local database storage) and it provides this data to the rest of the app.
Missing Authorization header in Angular 7 HTTP response
Accessing an API back-end from Angular client may result in missing response headers. A common scenario is a missing Authorization header, containing the JSON Web Token (JWT) which is returned from the back-end service when the user logs in successfully.
LiveData
LiveData is an observable data holder class. It is also lifecycle-aware, which means that it respects the lifecycle of the other app components, such as activities, fragments and services. It notifies only active observers, represented by the Observer class.
