Appropriate Number and Allocation of RELUS in Convolutional Neural Networks

Background. Due to that there is no common conception about whether each convolutional layer must be followed with a ReLU, the question on an appropriate number of ReLUs and their allocation is considered. Objective. The goal is to find a law for ascertaining an appropriate number of ReLUs. If this number is less than the number of convolutional layers, then the law shall stand for an appropriate allocation of ReLUs. Methods. A method of evaluating performance on the EEACL26 and CIFAR-10 datasets over various versions of ReLUs' allocation is defined. The performance is evaluated through 4 and 8 epochs for EEACL26 and CIFAR-10, respectively, for each allocation version. The best scores of performance are extracted. Results. In convolutional neural networks with 4 or 5 convolutional layers, the first three convolutional layers shall be followed with ReLUs, and the rest of convolutional layers shall not be ReLUed. It is plausible that appropriateness of ReLUs includes from-the-start compactness of allocating them, i. e. all ReLUs are allocated one by one from the very first convolutional layer. An appropriate number of ReLUs is an integer between a half of the convolutional layers' number and the half increased by 1. Conclusions. In some cases, the gain can grow up to 100 % and more. The gain for CIFAR-10, if any, is of roughly 10 to 20 %. Generally, as the training process goes on, the gain expectedly drops. Nevertheless, the stated appropriateness of number and allocation of ReLUs rationalizes the convolutional neural network architecture. Convolutional neural networks under the appropriate ReLUs' allocation can be progressively optimized further on its other hyperpa?ra?meters.