Dual-Branch Multi-Scale Relation Networks with Tutorial Learning for Few-Shot Learning

Abstract

Few-shot learning refers to training a model with a few labeled data to effectively recognize unseen categories. Recently, numerous approaches have been suggested to improve the extraction of abundant feature information at hierarchical layers or multiple scales for similarity metrics, especially methods based on learnable relation networks, which have demonstrated promising results. However, the roles played by image features in relationship measurement vary at different layers, and effectively integrating features from different layers and multiple scales can improve the measurement capacity of the model. In light of this, we propose a novel method called dual-branch multi-scale relation networks with tutoring learning (DbMRNT) for few-shot learning. Specifically, we first generate deep multiple features using a multi-scale feature generator in Branch 1 while extracting features at hierarchical layers in Branch 2. Then, learnable relation networks are employed in both branches to measure the pairwise similarity of features at each scale or layer. Furthermore, to leverage the dominant role of deep features in the final classification, we introduce a tutorial learning module that enables Branch 1 to tutor the learning process of Branch 2. Ultimately, the relation scores of all scales and layers are integrated to obtain the classification results. Extensive experiments on popular few-shot learning datasets prove that our method outperforms other similar methods.

1. Introduction

Due to the availability of a large amount of labeled data, deep learning yields impressive results in the field of computer vision [1,2,3,4]. However, training deep learning models often relies on a significant amount of labeled data. When faced with limited data and sparse samples, the model is prone to overfitting, resulting in poor generalization ability. This situation does exist in practical problems, such as image data of rare species and medical images. In contrast, humans possess powerful learning and cognitive abilities that allow them to establish knowledge of new concepts through learning only a few examples or even a single concept [5]. Therefore, inspired by human cognitive abilities, determining how to enable machines to establish effective cognition and generalize to new categories through training with only a few samples is one of the significant research issues in the field of deep learning, known as few-shot learning.

Few-shot learning refers to the process of training a model with limited available examples, enabling the model to generalize to new categories and establish robust cognition, simulating the human process of connecting prior knowledge with novel concepts. In the context of few-shot classification problems, a classification model is trained using limited data and then tasked with classifying new categories that were not encountered during training, thus necessitating strong generalization capabilities. To address such challenges, numerous exemplary approaches have been proposed, which can be roughly categorized into three groups: meta-learning-based, data-augmentation-based, and metric-learning-based methods.

Revolving around the meta-learning paradigm, approaches based on meta-learning train a meta-learner across tasks, focusing on training at the task level rather than the sample level. Its objective is to update the meta-learner with task-agnostic knowledge acquired to obtain optimal parameters that can adapt to novel tasks. On the other hand, data-augmentation-based methods attempt to mitigate the problem of limited sample data by generating new data from the existing samples, utilizing additional information, or enhancing sample representations. Meanwhile, metric-learning-based methods focus on constructing an embedding space where robust features are extracted and learning an informative similarity metric in the embedding space. Samples of the same class have a higher similarity metric, while it is lower for samples of different classes. These methods have achieved remarkable success, among which the metric-learning-based methods are the simplest and most effective. In [6,7,8], they first map the images into a feature space and then use predefined distance functions such as Euclidean distance and cosine distance to compute the similarity between query and support image in pairs for classification. Sung et al. [9] proposed a relation network to measure the similarity between samples, which can obtain a suitable distance metric instead of a fixed distance metric through learning. However, the relation network is known sensitive to spatial location, which may result in the misalignment of the compared objects within a certain region. To alleviate this issue, Wu et al. [10] adopted the deformable convolution combined with a dual correlation attention mechanism. Abdelaziz et al. [11] proposed a Kronecker-product module that maps and fuses positionwise correlations between the query and support feature pairs. However, these methods still fail to fully exploit the limited data and ignore a wealth of valuable information, such as the image features at different layers in deep learning models. These features contain distinct image information that is useful for classification. Deeper layers can capture more comprehensive semantic information about the object [12], while shallow layers can learn texture features such as lines and contours [13], which also contribute to enhancing recognition capabilities.

Many methods have been proposed to explore feature information at different layers to better utilize limited samples. Zhang et al. [14] learned a similarity metric of feature pairs at hierarchical layers using a relation network with an attention mechanism and propagated the relations to the next layer. Jiang et al. [15] extracted image features at different layers in the network, fused them with the transposed convolution upsampling method, and then measured the similarity to obtain the final classification results. Wang et al. [16] measured features at multiple layers and introduced a voting mechanism to determine the final classification results. Wu et al. [17] applied relation networks to measure the coarse-grained classification of shallow features and then measured the fine-grained classification of deep features. Similarly, some methods alleviate the problem of data scarcity by capturing multi-scale feature information. Zhang et al. [18] extracted features from original images at different scales and transformed them into second-order features to learn a similarity metric. Chen et al. [19] performed multi-scale transformations on image features to obtain multiple scales of features and learn task-relevant feature representations at each scale, and a similarity-to-class module was employed to achieve classification. Wang et al. [20] devised a multi-scale label propagation network and obtained the final label propagation scores and classification results through a weighted calculation method. Abdelaziz et al. [11] resized the original image into different scales and then trained a shared relation network with a Kronecker-product module for the similarity metric.

Utilizing multi-layer and multi-scale features may relieve the pressure of limited data in few-shot learning [14,15,16,17,18,19,20], while deeper features play a dominant role in the final classification [12,16], but the contribution of shallow features cannot be ignored [13,17]. Therefore, we propose a method named dual-branch multi-scale relation networks with tutoring learning for few-shot learning that effectively utilizes feature information at different layers and fully leverages the dominant role of deep feature information in classification decisions. Specifically, we learn a similarity metric between feature pairs on each deep scale in Branch 1 and on each layer in Branch 2. Recognizing the potential of the metric learning capability of the relation network, in this paper, we adopt relation networks for similarity measurement in both branches [9,11], allowing the network to learn a suitable metric instead of being limited to a predefined distance metric. Furthermore, simply separating the relation networks to learn the similarity of feature pairs at different layers without considering the differences in feature information between layers overlooks the dominant role of deep features. Therefore, we propose using the relation network in Branch 1, which has learned more advanced feature pair similarities, to tutor the relation networks in Branch 2 to learn similarities of shallower feature pairs, resulting in improved classification performance. The contributions of this work can be summarized as follows:

•

We propose a dual-branch multi-scale metric learning method that combines deep multi-scale features and features at different layers and employee relation networks for similarity metrics.

•

A tutoring learning module is proposed, which involves utilizing a deep relation network in Branch 1 that has acquired advanced knowledge to tutor each shallow relation network in Branch 2 to learn similarity metrics, thereby enhancing the metric capability of relation networks.

•

The final classification performance obtained by our dual-branch multi-scale relation metric and tutoring learning outperforms similar methods on few-shot standard datasets.

2. Related Work

2.1. Meta-Learning-Based Methods

This type of method follows the meta-learning paradigm to obtain an excellent meta-learner across tasks by training task-level meta-learners, which in turn trains a meta-learner adequate to adapt quickly to novel tasks and acquire optimal parameters, also known as learning to learn [21]. Finn et al. [22] searched for an appropriate initialization parameter setting of a model and effectively fine-tuned it with a few gradient descent update steps. Munkhdalai et al. [23] employ a recurrent neural network to iterate over samples to gather the needed knowledge in both the base learner and meta-learner for a given problem. Ravi et al. [24] trained an optimizer based on LSTM to achieve more efficient fine-tuning. In contrast, Oreshkin et al. [25] incorporated task-specific knowledge into the feature encoder by scaling and translating the image features and employed auxiliary task cooperation training to acquire a task-dependent metric space while alleviating the complexity of network training. Similarly, Cai et al. [26] utilized memory slots to forecast the parameters of their feature encoder for classifying unlabeled images. Ren et al. [27] used a recurrent neural network based on an attention mechanism to achieve dynamic comparison between samples, whereas Mishra et al. [28] employed a temporal convolutional network with a soft attention mechanism to encode acquired knowledge into a memory module, subsequently utilizing the memory module for targeted information retrieval and classification.

Compared to such methods, the proposed DbMRNT method does not involve designing intricate meta-learners or utilizing external memory modules but achieves superior performance through straightforward end-to-end training.

2.2. Data Augmentation-Based Methods

Methods belonging to this category aim to alleviate the problem of limited samples by obtaining more available samples or information. Antoniou et al. [29] employed generative adversarial networks to generate new samples. Zhang et al. [30] utilized generative adversarial networks to provide additional training signals for classifiers, clarifying the decision boundary. Hariharan et al. [31] employ the concept of generalizing from intraclass sample variations to different categories and utilize generative models for data augmentation. Wang et al. [32] created new samples by incorporating noise into original images and considered the generative and classification models as a whole for simultaneous updates. Zhang et al. [33] introduced the saliency-guided hallucination method to generate new mixed background-foreground samples for additional training. Chen et al. [34] first employed a meta-learner and a deformation network to achieve image deformation and expand the dataset and then proposed utilizing semantic information to expand the data, employing an encoder to map the feature space onto the semantic space for data augmentation, and subsequently leveraging a decoder to reduce the expanded semantic information back into the feature space [35]. Xing et al. [36] proposed an adaptive modal mixing mechanism to integrate image feature information and semantic information, thereby enhancing the performance of the model. Meanwhile, Schwartz et al. [37] introduce a weighted fusion approach to combine multiple semantic information and visual prototypes for generating the ultimate fusion prototype used in similarity measurement and classification.

In contrast to data-augmentation-based methods, our method refrains from expanding the dataset or leveraging additional information such as natural label information or image description details. Instead, we aim to fully exploit the limited samples to the best extent possible.

2.3. Metric-Learning-Based Methods

This type of method typically evaluates the pairwise similarity of features between support and query samples in a feature embedding space, wherein high similarity indicates samples belonging to the same category, while low similarity suggests different categories. The general similarity measures encompass fixed metrics like Euclidean distance and cosine similarity distance or a learnable network. Koch et al. [6] introduce deep neural networks into few-shot learning, where they learn the features of a pair of samples through weight-shared convolutional networks and measure similarity using Euclidean distance. Vinyals et al. [8] encoded the support set and query set using different LSTM networks, measured feature similarity with an attention-weighted metric function and proposed a classic training mechanism for few-shot learning scenarios named the $episode$ mechanism. Snell et al. [7] proposed the existence of a class prototype in each class’s feature space, obtained the prototype representation by calculating the mean of same-class samples, and performed classification by measuring distances between query samples and class prototypes. Sung et al. [9] suggest replacing traditional predefined nonparametric measurement methods with a learnable neural network, allowing the network to autonomously learn an appropriate similarity metric. Differing from previous approaches that rely on image-level features for similarity calculation [6,7,8], Li et al. [38] propose the concept of local descriptors to replace image-level feature metrics; it classifies by measuring the k-nearest neighbors of the local descriptors between the query sample and the categories. Xue et al. [39] propose assigning higher weights to regions with greater similarity between images by calculating feature similarities individually for each region in comparison to the query image and subsequently combining scores across all regions for classification decisions. Zhang et al. [40] transform feature representation into a second-order feature representation to facilitate similarity metrics and then devise a metric strategy for multi-scale relation networks based on this foundation [18]. To address the issue of relation networks being sensitive to the spatial positions of target objects, Wu et al. [10] incorporated deformable convolution into the feature extractor and combined it with a dual attention mechanism. Additionally, Xue et al. [41] introduced relative position and map networks based on an attention mechanism to determine the significance of each spatial location in the image features during comparison and designed a nonlinear metric using a relative map network module. In refs [14,15,16] extract features of different layers to measure similarity, demonstrating that feature maps at hierarchical layers possess distinct classification characteristics and contributions, while [11,18,19,20,42] employ multi-scale feature extraction techniques to capture a broader range of feature information and boost the accuracy of classification. Simultaneously, there exist numerous methods to enhance model performance by integrating attention mechanisms [43,44,45,46,47,48,49] and methods that explore intra-class and inter-class relations [50,51,52,53,54] all falling under the category of metric-learning-based methods.

Our method falls under the category of metric-learning-based methods and shares similarities with multi-layer methods [14,15,16] and multi-scale methods [18,19,20,42]. We evaluate our methodology against them and several methodologies as baselines. The proposed method adopts a dual-branch structure, maximizing the utilization of feature information from different layers and scales. Additionally, learnable relation networks are employed instead of a fixed metric function for similarity measurement. The main difference lies in the branch-guided paradigm proposed in this paper, which enables the metric network to better learn how to measure similarity and effectively improve classification accuracy. The most important aspect is that the proposed tutoring learning approach enables the relation networks to learn a similarity metric better, effectively improving classification accuracy.

3. Methodology

3.1. Problem Definition

Due to the scarcity of labeled data, traditional image classification training methods are not suitable for few-shot learning tasks. To address this issue, a training mechanism known as episode [8] is widely adopted and recognized as an effective approach for solving few-shot learning problems. This training mechanism facilitates learning by constructing numerous meta-tasks. Typically, the dataset is divided into training, testing, and validation sets with a crucial consideration that the categories in the training set are mutually exclusive from those in the testing and validation sets. As shown in Figure 1, training tasks are formulated based on the training set, while test tasks are constructed from the testing set. Each task is further comprised of support set S and query set Q. During the training process, a set of C categories is randomly generated from the training set, and K samples are sampled from every category to create the support set. Subsequently, M samples are sampled from the remaining samples in each category of the selected C categories to create the query set, thereby constructing a C-way K-shot classification task. During the inference stage, we carry out the same sampling as in the training stage, as described above, the only difference is that the query set samples in the inference stage are unlabeled. One meta-task is learned in each training iteration, referred to as an episode, which comprises the support set $S={\left\{\left(x_s,y_s\right)\right\}}_{s=1}^n$ and the query set $Q={\left\{\left(x_q,y_q\right)\right\}}_{q=1}^m$, where $n=C\times K$ and $m=C\times M$.

The few-shot classification task aims to classify the samples in the query set into their respective categories by learning these meta-tasks, and during the testing phase, a C-way K-shot testing task is constructed from the testing set, leveraging the knowledge acquired during the training phase for classification.

3.2. Model Overview

We propose a novel end-to-end metric learning model named dual-branch multi-scale relation networks with tutoring learning to address the challenge of few-shot classification problems. Our method is based on the concept of multiple layers and multi-scale analysis while incorporating relation network metrics, and the model framework is illustrated in Figure 2.

As shown in Figure 2, our model has a dual-branch structure, which is mainly composed of four modules: a feature extractor $F_{\theta }$, a multi-scale feature generator $G_{\varphi }$, a relation network module $R_{\phi }$, and a tutoring learning module T. The samples are fed into the feature extractor to obtain their respective feature maps. Subsequently, these feature maps serve as input for the multi-scale feature generator, which generates feature maps at five different scales. Next, the features of the query image and support images from various categories are concatenated in the channel dimension at each scale to form support–query feature pairs. These pairs are then input into the relation network module. In this branch, multi-scale feature pairs use a shared relation network for similarity metrics, resulting in separate relation scores and losses at each scale. The relation score is a five-dimensional vector indicating the probability that the query image belongs to one of five categories. In Branch 2, we extract two groups of feature maps from the middle and last layers of the feature extractor, as well as an additional group of deeper feature maps from the fifth subbranch of the multi-scale feature generator. Subsequently, these three groups of feature maps are concatenated in the channel dimension to obtain support–query feature pairs, which are then fed into the relation network module in Branch 2. Relation scores and losses at hierarchy layers are obtained by training three weight-unshared relation networks. Additionally, to better utilize the relation networks in measuring shallow feature pair information and contribute to overall performance, we propose a tutoring learning module T. In this module, the knowledge summarization of Branch 1 and Branch 2 is obtained by aggregating their own relation scores. Subsequently, the richer knowledge acquired by the relation network in Branch 1 is transferred to Branch 2 by constructing soft labels, facilitating the tutoring process from Branch 1 to Branch 2.

3.3. Generation of Multi-Scale and Multi-Layer Features

The support image $x_{si}$ and query image $x_{qj}$ are fed into the feature extractor $F_{\theta }$ to obtain two groups of feature representations with different layers: ${\Phi }_s^d={\left\{F_{\theta }^d\left(x_s\right)\right\}}_{d=1}^2$ and ${\Phi }_q^d={\left\{F_{\theta }^d\left(x_q\right)\right\}}_{d=1}^2$, which serve as the feature representations for different depths in Branch 2, while the deeper features ${\Phi }^2$ are utilized as input for the multi-scale feature generator.

Generating multiple scale features helps to obtain multiple unique representations of features from original images to mitigate the problem of limited data in few-shot situations. This paper proposes a multi-scale feature generator inspired by [19]. Additionally, to learn deeper features while reducing the size of the feature maps, a 3 × 3 convolutional layer is added before the 2 × 2 max-pooling layer in the fifth subbranch. As shown in Figure 3, the multi-scale feature generator consists of five subbranches: the first subbranch does nothing; the second subbranch contains a 3 × 3 convolutional layer; the third subbranch contains a 5 × 5 convolutional layer; the fourth subbranch contains a 1 × 7 convolutional layer before a 7 × 1 convolutional layer; and the fifth subbranch contains a 3 × 3 convolutional layer and a 2 × 2 max-pooling layer.

With the help of the multi-scale feature generator, we obtain deep multi-scale features for both support and query images ${\Phi }_s^z={\left\{G_{\varphi }^z\left(F_{\theta }^2\left(x_s\right)\right)\right\}}_{z=1}^5$ and ${\Phi }_q^z={\left\{G_{\varphi }^z\left(F_{\theta }^2\left(x_q\right)\right)\right\}}_{z=1}^5$ and then feed them into the relation network module in Branch 1. Simultaneously, the outputs from the fifth part of the multi-scale feature generator, ${\Phi }_s^5=G_{\varphi }^5\left(F_{\theta }^2\left(x_s\right)\right)$ and ${\Phi }_q^5=G_{\varphi }^5\left(F_{\theta }^2\left(x_q\right)\right)$, will serve as the deepest features ${\Phi }^3$ in Branch 2, alongside the features ${\Phi }_s^d={\left\{F_{\theta }^d\left(x_s\right)\right\}}_{d=1}^2$ and ${\Phi }_q^d={\left\{F_{\theta }^d\left(x_q\right)\right\}}_{d=1}^2$ extracted by the feature extractor at the different layers. This combination enables pairwise similarity measurements at hierarchy layers within Branch 2.

3.4. Relation Networks Module

A relation network [9] is employed to supersede the predefined distance metric and learn a suitable approach to learn the similarity metric between support and query feature pairs. Therefore, the support feature of each class is concatenated with the query feature in the channel dimension after obtaining the output of the previous module, and then the support–query feature pairs are fed into this module to obtain the ultimate relation scores.

The proposed relation networks module is designed differently in the dual-branch structure. In Branch 1, there are 5 different scales. In each scale, the query feature is spliced with the support features of each category, and the feature pairs are fed into the weight-shared relation network. However, the common relation network [9] is known sensitive to the spatial location of the compared objects [10]. We employ an improved relation network [11] in Branch 1, which incorporates a Kronecker-product ($KP$) module [55,56] to acquire positionwise correlation maps between feature pairs for more enhanced metric learning and improved generalization capability. Initially, we are given two original features to form a support–query feature pair from the identical scale., and the $KP$ module is applied to generate spatial correlation maps: ${\widehat{\beta }}_{\left({\Phi }_s^z,{\Phi }_q^z\right)}=KP\left({\Phi }_s^z,{\Phi }_q^z\right)$ and ${\widehat{\beta }}_{\left({\Phi }_q^z,{\Phi }_s^z\right)}=KP\left({\Phi }_q^z,{\Phi }_s^z\right)$ by applying the feature maps. Next, the spatial correlation maps created are concatenated with the original feature maps at each scale, and we have a relation score as follows:

$$r_{s,q}^z=R_{\phi }\left(Concate\left({\Phi }_s^z,{\widehat{\beta }}_{\left({\Phi }_s^z,{\Phi }_q^z\right)},{\widehat{\beta }}_{\left({\Phi }_q^z,{\Phi }_s^z\right)},{\Phi }_q^z\right)\right),z\in \left\{1,...,5\right\}$$

(1)

where z denotes the different scales, and $Concate\left(·\right)$ refers to the operation of splicing the features in the channel dimension. Therefore, for a C-way K-shot task, the total loss of Branch 1 is:

$${\displaystyle {\mathcal{L}}_Z=\sum _{z=1}^5\sum _{i=1}^n\sum _{j=1}^m{\ell }^{CE}\left(r_{i,j}^z,\mathbf{1}\left(y_i=y_j\right)\right)}$$

(2)

where ${\ell }^{CE}$ denotes the cross-entropy loss, $n=C\times K$ refers to the number of support images and $m=C\times M$ refers to the number of query images in an episode. $r_{i,j}^z$ is the relation score belonging to support image $x_{si}$ and query image $x_{qj}$. $\mathbf{1}\left(·\right)$ is equal to 1 when the condition is satisfied; otherwise, it equals 0.

In the relation network module of Branch 2, the inputs consist of two sets of feature representations: ${\Phi }_s^d={\left\{F_{\theta }^d\left(x_s\right)\right\}}_{d=1}^2$ and ${\Phi }_q^d={\left\{F_{\theta }^d\left(x_q\right)\right\}}_{d=1}^2$ extracted by the feature extractor at different depths, as well as a set of feature representations ${\Phi }_s^5$ and ${\Phi }_q^5$ outputted by the fifth part of the multi-scale feature generator. The specific details are illustrated in Figure 4. At each layer, the query image features are concatenated with the support image features of each class and fed into the relation networks for similarity measurement. Due to the relatively large feature size at the low layer, we omit the introduction of the $KP$ module here and solely adopt the relation networks proposed in [9]. Simultaneously, we transfer the relation information from shallow layers to deeper relation networks and concatenate it with deeper feature pairs. Ultimately, we obtain relation scores from three distinct layers:

$$r_{i,j}^d=R_{\phi }^d\left(Concate\left({\Phi }_i^d,{\Phi }_{i,j}^{d-1},{\Phi }_j^d\right)\right),d\in \left\{1,2,3\right\},i\in \left\{1,...,n\right\},j\in \left\{1,...,m\right\}$$

(3)

where d denotes the hierarchy layers, $Concate\left(·\right)$ refers to the operation of splicing the features in the channel dimension, ${\Phi }_{i,j}^{d-1}$ refers to the relation information from the previous layer, and the first layer does not have this item. Therefore, the total loss of Branch 2 can be formulated as follows:

$${\displaystyle {\mathcal{L}}_D=\sum _{d=1}^3\sum _{i=1}^n\sum _{j=1}^m{\ell }^{CE}\left(r_{i,j}^d,\mathbf{1}\left(y_i=y_j\right)\right)}$$

(4)

3.5. Tutoring Learning Module

The relation network replaces the conventional measurement method based on predefined functions, enabling the model to learn an appropriate approach for the similarity metric. However, in few-shot learning problems with limited examples, it becomes crucial to consider every detail. Although employing relation networks separately for measuring the similarity of different hierarchical features can enhance model performance, it overlooks the distinctions and impacts of these different features. As the outcomes of deep features have a significant influence over the final classification, tutoring similarity metrics learning of shallow features from deep feature metrics is essential. To improve the capability of the relation networks model to measure feature pair similarity, we propose a tutoring learning module that utilizes the Branch 1 relation network to instruct the Branch 2 relation networks in learning similarity metrics.

The tutorial learning module is inspired by knowledge distillation [57], which can improve the accuracy of network models. Combining with knowledge distillation techniques can further improve the measurement capabilities of relation networks. Specifically, we consider the relation network in Branch 1 as a teacher network due to the weight-sharing mechanism that enables it to acquire comprehensive deep feature similarity across multiple scales, thereby exhibiting strong measurement capabilities. Conversely, the relation networks in Branch 2 are regarded as student networks since they learned less or basic similarity knowledge, so they can benefit from the tutoring provided by Branch 1. We first sum the relation scores at each scale in Branch 1, which represents the knowledge learned about sample pair similarity. Then, we sum the relation scores in Branch 2, which denotes the knowledge learned by Branch 2:

$${\displaystyle r^Z=\sum _{z=1}^5{r}^z,r^D=\sum _{d=1}^3{r}^d}$$

(5)

Obviously, the weight-shared relation network in Branch 1 learns more comprehensive knowledge at different scales in deep layers, exhibiting robust recognition capabilities, while the relation networks in Branch 2 independently learn and primarily rely on shallow and fundamental features, serving as a supplementary component for final classification. Therefore, this paper proposes a tutoring learning module T to facilitate the transfer of knowledge acquired in Branch 1 to Branch 2, thereby enhancing the overall recognition capability. We leverage the learning summary $r^Z$ from Branch 1 to generate a soft label for tutoring the similarity metric in Branch 2, incorporating hyperparameter t for appropriate regularization. The specific implementation is as follows:

$${\mathcal{L}}_T={\ell }^{KL}\left(softmax\left(r^Z/t\right),softmax\left(r^D/t\right)\right)$$

(6)

where ${\ell }^{KL}$ denotes the Kullback–Leibler divergence, and $softmax\left(·\right)$ indicates normalization. The loss function of the tutoring learning module is: 

$${\mathcal{L}}_{KD}=\alpha {\mathcal{L}}_D+\left(1-\alpha \right){\mathcal{L}}_T$$

(7)

where $\alpha$ denotes the weight hyperparameter, which is set to 0.3 in this paper to control the contribution of the two parts in the distillation loss. Therefore, the total loss of our model is composed of the losses of Branch 1, Branch 2 and the tutoring learning module, and can be formulated as follows:

$$\mathcal{L}=\underset{\theta ,\varphi ,\phi }{min}\left({\mathcal{L}}_Z+{\mathcal{L}}_D+{\mathcal{L}}_{KD}\right)$$

(8)

where the ultimate classification result is determined by the sum of the relation scores of all scales and layers, as depicted below:

$$r^{final}=r^Z+r^D/3$$

(9)

4. Experiments and Results

To assess the efficacy of our approach, we conducted experiments on commonly employed few-shot learning datasets, namely, miniImageNet and tieredImageNet, as well as a fine-grained image classification dataset called Stanford Cars. Subsequently, we compared our approach with other state-of-the-art approaches.

4.1. Datasets

MiniImageNet [8] is a subset of ImageNet [58]. There are 100 classes with 600 pictures in each class, totaling 60,000 color pictures. There are 64 classes prepared for training, while 16 categories and 20 categories are prepared for validating and testing, respectively.

TieredImageNet [27] comprises 779,165 images distributed across 608 categories (34 high-level categories) that are hierarchically sampled from the 1000 categories of ImageNet. In the training set, there are 351 categories (20 high-level categories), while the validation and test sets consist of 97 categories (6 high-level categories) and 160 categories (8 high-level categories), respectively. Due to its distinct classification characteristics compared to miniImageNet, this dataset poses greater challenges in few-shot learning.

Stanford cars [59] is a primarily used fine-grained dataset. It includes 16,185 color images of 196 categories of cars. There are 130 classes prepared for training, while 17 categories and 49 categories are prepared for validating and testing, respectively.

To ensure a fair comparison, all of our experiments resize images to a size of 84 × 84 dimensions, and all the datasets are divided according to the standard protocol. In this paper, miniImageNet was divided according to [8], tieredImageNet was divided according to [27], and Stanford Cars was divided according to [38]. The splits of the datasets are shown in

Table 1. The splits of the datasets.

Dataset	Class-All	Class-Train	Class-Val	Class-Test
miniImageNet [8]	100	64	16	20
tieredImageNet [27]	608	351	97	100
StanfordCars [59]	196	130	17	49

.

4.2. Network Architecture

For a fair comparison, we employ the identical feature extractor $F_{\theta }$ as utilized by the majority of few-shot learning methods. $F_{\theta }$ consists of four convolutional layers, each comprising a convolution block with 64 channels, a 3 × 3 convolution kernel, a BatchNorm layer, and a ReLU layer. Among these layers, only the first two contain a 2 × 2 max-pooling layer at the end. The input image size is set to 84 × 84 × 3. The multi-scale feature generator comprises five parallel parts, as illustrated in Figure 2. The relation networks module adopts the main structure of RN [9], which consists of two convolution blocks and two fully connected layers, each with 64 channels. Each layer utilizes a 3 × 3 convolution kernel, a BatchNorm layer, a ReLU layer, and a 2 × 2 max-pooling layer. The output size of the last convolution block is set to 64. The dimensionality of the first fully connected layer is reduced to 8, followed by a ReLU layer. Finally, the last fully connected layer outputs a one-dimensional classification score. The relation networks module in this paper consists of two branches. The relation network in Branch 1 incorporates an adaptive global average pooling layer to resize the input feature map to 10 × 10, and a Kronecker-product module is used to obtain spatial correlation maps of feature pairs. Meanwhile, the input of Branch 2 does not undergo a global average pooling layer, and because of the different sizes of feature maps, an adaptive global average pooling layer is employed before the fully connected layer.

4.3. Setup

Our experiment followed the C-way K-shot episode training mechanism [8] and was conducted in both 5-way 1-shot and 5-shot settings. In the 5-way 1-shot setting, each training episode consisted of randomly sampled images from five categories, with 1 support sample and 15 query samples per category, resulting in a total of eighty images (5 × 1 + 5 × 15 = 80). For the 5-way 5-shot setting, each training episode included five support images per category and fifteen query images per category, totaling one hundred images (5 × 5 + 5 × 15 = 100). The Adam optimizer was employed with an initial learning rate of 0.001 halved every 150,000 episodes. A total of 500,000 randomly sampled episodes are used for training. For data augmentation, we applied random resize crop, random color jittering, random horizontal flipping, and random erasing during training following the strategy described in [11]. All experiments were trained end-to-end without any fine-tuning during the testing phase. During the inference stage, we selected 15 query images per category in each episode for both 5-way 1-shot and 5-shot classification in the test set, which is the same as the training stage. The final classification accuracy was calculated as an average of over 600 test episodes with a confidence interval of 95%.

4.4. Results

4.4.1. Results of the Main Datasets

We compare our method with current metric-based methods [7,8,9,10,11,19,20,39,40,41,53] as well as some meta-learning based methods [22,23,24,26,27,28]. All the images of our experiments are resized to 84 × 84 dimensions, and a feature extractor with four convolutional layers is employed.

The experimental results of our model on the main datasets are showed in

Table 2. Few-shot classification results on the miniImageNet.

Model	Type	5-Way 1-Shot	5-Way 5-Shot
Meta-learn Nets [24]	Meta	43.44 ± 0.77	60.60 ± 0.71
MetaGAN [30]	Meta	46.13 ± 1.78	60.71 ± 0.89
MAML [22]	Meta	48.70 ± 1.84	63.11 ± 0.92
Meta Nets [23]	Meta	49.21 ± 0.96	-
Meta SSL [27]	Meta	50.41 ± 1.84	64.39 ± 0.92
MM-Net [26]	Meta	53.37 ± 0.48	66.97 ± 0.35
SNAIL [28]	Meta	55.71 ± 0.99	68.88 ± 0.92
Matching Nets [8]	Metric	43.56 ± 0.84	55.31 ± 0.73
Prototypical Nets [7]	Metric	49.42 ± 0.78	68.20 ± 0.66
GNN [53]	Metric	50.33 ± 0.36	66.41 ± 0.63
Relation Nets [9]	Metric	50.44 ± 0.82	65.32 ± 0.70
DN4 [38]	Metric	51.24 ± 0.74	71.02 ± 0.64
MSDN [16]	Metric	52.59 ± 0.81	68.51 ± 0.69
SoSN [40]	Metric	52.96 ± 0.83	68.63 ± 0.68
RPMN [41]	Metric	53.35 ± 0.78	69.35 ± 0.59
RCN [39]	Metric	53.47 ± 0.84	71.63 ± 0.70
DCN [14]	Metric	53.48 ± 0.78	67.63 ± 0.59
MATANet [19]	Metric	53.63 ± 0.83	72.67 ± 0.76
MSFN [42]	Metric	54.48 ± 1.23	69.06 ± 0.69
PARN [10]	Metric	55.22 ± 0.82	71.55 ± 0.66
MSLPN [20]	Metric	56.52 ± 0.92	73.45 ± 0.86
MsKPRN [11]	Metric	57.02 ± 0.88	72.06 ± 0.68
MRN [54]	Metric	57.83 ± 0.69	71.13 ± 0.50
ABNet [49]	Metric	58.12 ± 0.94	72.02 ± 0.75
DbMRNT(Ours)	Metric	58.82 ± 0.88	74.19 ± 0.63

and

Table 3. Few-shot classification results on the tieredImageNet.

Model	Type	5-Way 1-Shot	5-Way 5-Shot
MAML [22]	Meta	51.67 ± 1.81	70.30 ± 1.75
Prototypical Network [7]	Metric	53.31 ± 0.89	72.69 ± 0.74
Relation Nets [9]	Metric	54.48 ± 0.93	71.32 ± 0.78
CGRN [52]	Metric	55.07 ± 0.20	71.34 ± 0.30
HMRN [51]	Metric	57.98 ± 0.26	74.70 ± 0.24
MSLPN [20]	Metric	58.69 ± 0.96	74.12 ± 0.73
DCN [14]	Metric	60.58 ± 0.72	72.42 ± 0.69
ABNet [49]	Metric	62.10 ± 0.96	75.11 ± 0.78
MRN [54]	Metric	62.65 ± 0.84	74.20 ± 0.64
DbMRNT(Ours)	Metric	62.95 ± 0.92	79.00 ± 0.70

, respectively. It can be observed from the table that the proposed dual-branch multi-scale relation networks with tutoring learning outperform other metric-based methods and similar multi-scale approaches in both 5-way 1-shot and 5-shot scenarios.

Our proposed method makes fuller use of image information at hierarchical layers, such as deep multi-scale information in Branch 1 and feature information at different layers in Branch 2. The rich knowledge learned makes the model have robust recognition and classification ability. Compared with MSDN [16] with only different depth features, our method improves by approximately 6% and 5.7% in 5-way 1-shot and 5-way 5-shot scenarios, respectively, on the miniImageNet dataset. At the same time, compared with the method MsKPRN [11], which also adopts multi-scale and relation network metrics, our method improves by 1.8% in 5-way 1-shot and 2.13% in 5-shot. Compared with ABNet [49], which splits the image into significant patches and combines the attention mechanism, our method does not process original images too much, nor does it introduce the attention mechanism, but improves by 0.7% and 2.17% in 1-shot and 5-shot settings, respectively.

On the tieredImageNet dataset, our method achieves a performance improvement of 2.37% and 6.58% for 1-shot and 5-shot scenarios, respectively, compared with DCN [14], which employs multi-scale measurements at different depths. In comparison with MRN [54], which also utilizes relation networks for similarity metrics and incorporates external memory units to store information and class relationships, our model demonstrates only a marginal enhancement of 0.3% but improved by 4.8% in 5-shot scenarios. This is because our method can conduct tutoring learning of the dual-branch relation networks module, which uses the relation network containing more high-dimensional and global information to tutor the relation networks containing more low-dimensional and local information, transfers the learned knowledge between different layers of relation networks, and further improves the task adaptive measurement ability of relation networks, which is more robust and has higher classification accuracy than a single measurement method.

4.4.2. Results of the Fine-Grained Dataset

Fine-grained datasets pose greater challenges than standard datasets due to small inter-class differences and large intra-class variations [11]. Our method outperforms MsKPRN, DN4, GNN, Relation Nets, and other methods in both scenarios. Compared to the second-highest performing method, MsKPRN, our accuracy is higher by 1.66% and 0.82% in the respective scenarios. While MsKPRN processes the original image to a size of 84 × 84 and scales it to 64 × 64 and 48 × 48 as inputs to obtain deep multi-scale feature information and improves relation networks for classification tasks, our approach employs an improved relation network as a metric network for Branch 1 while using Branch 2 to extract features of different depths from the image. Experimental results in

Table 4. Few-shot classification results on the Stanford Cars.

Model	5-Way 1-Shot	5-Way 5-Shot
Matching Nets [8]	34.80 ± 0.98	44.70 ± 1.03
Prototypical Nets [7]	40.90 ± 1.01	52.93 ± 1.03
Relation Nets [9]	47.67 ± 0.47	60.59 ± 0.40
GNN [53]	55.85 ± 0.97	71.25 ± 0.89
DN4 [38]	61.51 ± 0.44	89.60 ± 0.44
MsKPRN [11]	76.64 ± 0.84	89.88 ± 0.46
DbMRNT(Ours)	78.30 ± 0.83	90.70 ± 0.44

show the effectiveness of our dual-branch design and tutoring learning module.

5. Discussion

We perform ablation experiments on the miniImageNet dataset in this section to objectively analyze the effectiveness of our suggested dual-branch multi-scale structure and the tutorial learning module.

5.1. The Effect of the Dual-Branch Structure

The classification effect of each branch in few-shot scenarios is investigated in experiments, and the results are presented in

Table 5. The effect of the dual-branch structure on miniImageNet.

Model	5-Way 1-Shot	5-Way 5-Shot
Branch 1	57.11 ± 0.84	72.85 ± 0.61
Branch 2	56.18 ± 0.85	70.06 ± 0.66
B1 + B2	57.48 ± 0.84	73.46 ± 0.64
B1 + B2 + T	57.88 ± 0.85	73.76 ± 0.62

. From the table, it can be observed that the combined model of both branches exhibits improved performance compared to either individual Branch alone in few-shot classification tasks. In the 5-way 1-shot setting, there is a 0.37% improvement over Branch 1 and a 1.3% improvement over Branch 2. Similarly, in the 5-way 5-shot setting, there is a respective improvement of 0.61% and 3.4% compared to Branch 1 and Branch 2. The experiments validate that deep features (Branch 1) containing richer semantic information contribute significantly to final classification outcomes in multi-scale settings while incorporating shallow information enhances overall performance. Notably, our approach employs a tutoring learning module T where we utilize Branch 1 to tutor the learning process of Branch 2 by capturing the similarity between support and query image pairs, resulting in remarkable improvements in classification accuracy by approximately +0.4% for the 1-shot setting and +0.3% for the 5-shot setting. Our experiments demonstrate the efficacy of our dual-branch architecture combined with a tutoring learning module, which also benefits from the adaptable learning of similarity metrics facilitated by relation networks. Our method harnesses the potential inherent in this type of measurement.

5.2. The Effect of Information Transfer between Layers

In this experiment, we conduct relevant experiments in 5-way 1-shot and 5-shot scenarios to validate the classification effectiveness of selecting and processing intermediate features at different layers in Branch 2. The results are presented in

Table 6. The effect of information transfer between layers on miniImageNet.

Model	5-Way 1-Shot	5-Way 5-Shot
Baseline	57.88 ± 0.85	73.76 ± 0.62
B1 + B2 * $w/o$ ${\Phi }^{d-1}$	58.41 ± 0.84	74.19 ± 0.63
B1 + B2 * + T	58.82 ± 0.88	73.90 ± 0.60

. The baseline model represents a dual-branch structure with a tutoring learning module. In this model, the shallowest layer of Branch 2 is obtained from the output of the first layer of the feature extractor. ‘*’ indicates that the second layer of the feature extractor is chosen to form the shallowest layer of Branch 2 while delaying its max-pooling layer. Compared to our baseline model, when selecting features from the second layer, our model demonstrates an improvement in classification performance by 0.94% and 0.14% for the 5-way 1-shot and 5-shot settings, respectively. These experimental findings indicate that further processing of shallow feature information contributes to enhanced classification accuracy and highlights the crucial impact of selecting shallow features as well. Additionally, Table 6 reveals that knowledge transfer between internal layers within the relation network module in Branch 2 also aids in improving accuracy (by approximately +0.41%) for the case of a 5-way 1-shot scenario; however, such knowledge transfer has a negative effect when dealing with a more data-rich setting such as a 5-shot scenario (i.e., having five labeled examples per class). Further analysis suggests that this occurs because under such settings, where models learn relatively robust prototypes, higher-level features carrying semantic information exert greater influence on final classification decisions. Therefore, the information transferred from shallower layers instead affects overall decision-making.

5.3. The Effect of Hyperparameter t

The primary contribution of our approach lies in the tutoring learning module, which significantly enhances classification accuracy. Leveraging the learnability of relation networks, the tutoring learning module employs the relation network in Branch 1 to tutor relation networks in Branch 2 in learning image pair similarities. Specifically, the knowledge acquired by Branch 1 is utilized to generate soft labels and is subsequently transferred to Branch 2. We design a hyperparameter t to control the degree of softening. The experimental results are presented in

Table 7. The effect of hyperparameter t on miniImageNet.

Model	5-Way 1-Shot	5-Way 5-Shot
$t=1$	58.71 ± 0.88	73.72 ± 0.61
$t=7$	58.73 ± 0.85	73.90 ± 0.60
$t=20$	58.82 ± 0.88	73.66 ± 0.60

. $t=1$ represents no softening processing, and the larger the value of t is, the greater the degree of softening and the smaller the category difference. The results indicate that the optimal classification performance is achieved at $t=20$ in the 5-way 1-shot scenario, whereas it occurs at $t=7$ in the 5-shot scenario. This discrepancy can be attributed to the heightened scarcity of samples in the 1-shot case, leading the model to exploit additional inter-class information, which is mitigated in the 5-shot case.

6. Conclusions

In this paper, we propose a novel few-shot learning method for effectively integrating deep multi-scale feature information and feature information from multiple hierarchical layers. The proposed method, DbMRNT, maximizes the utilization of limited data in the few-shot learning problem and enhances the accuracy of classification. The introduced tutoring learning approach capitalizes on the learnability of the relation network, leveraging the deep and comprehensive knowledge acquired by the relation network in Branch 1 to tutor the shallow relation networks in Branch 2 in learning simpler information, thereby further enhancing classification performance in the few-shot learning problem. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed method.

For future work, we intend to explore other backbone networks and utilize larger images, design ways to capture effective target regions in images to better leverage relation networks, and explore intra-class and inter-class information at multi-scale and hierarchical feature information.