📝 主旨内容

1. Recurrent Neural Networks (RNNs):  RNNs are a class of artificial neural networks designed to handle sequential data such as time series, speech, or text. They maintain a hidden state across time steps, allowing them to capture temporal dependencies between inputs. Variants of RNNs include Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks, which address issues related to vanishing gradients when dealing with longer sequences. Despite their successes, RNNs can struggle with capturing long-range dependencies due to challenges associated with gradient propagation through time.

2. Graph Neural Networks (GNNs):  GNNs are specialized neural network architectures developed for handling graph structured data, where nodes represent entities and edges denote relationships between those entities. By designing appropriate convolutional layers over graphs, GNNs learn node representations that encode structural and semantic features from neighboring nodes and edges. Applications range from social networks, recommendation systems, chemistry, and physics simulations to traffic prediction and more. Popular GNN variants include Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), Message Passing Neural Networks (MPNNs), and many others.

3. Other Similar Models: There exist several other deep learning architectures suitable for diverse data structures and problem domains. Some examples are:

• Convolutional Neural Networks (CNNs): CNNs are primarily applied to grid-like data, such as images, videos, and signals. Through convolution filters and pooling operations, they efficiently extract local patterns and hierarchical features.

• Autoencoders (AEs): AEs are unsupervised generative models that consist of two main parts - encoder and decoder. Encoders map input data into latent space representations, whereas decoders reconstruct the original input from these encoded vectors. Autoencoders serve applications including dimensionality reduction, anomaly detection, feature learning, and representation learning.

• Generative Adversarial Networks (GANs): GANs comprise two competing subnetworks - generator and discriminator. Generators synthesize new samples resembling real data, while discriminators distinguish generated samples from actual ones. Training both networks simultaneously results in improved sample quality and diversity. GANs find uses in image generation, style transfer, video predictions, and more.

1. IID (Independent and Identically Distributed): IID data consists of independent, identically distributed samples, meaning each datum point is sampled independently from the same underlying distribution. Importantly, knowing the value of one data point does not affect the probability of observing another data point. Many classical machine learning algorithms assume IID data, making them easier to analyze mathematically and guaranteeing satisfactory performance under ideal conditions.

2. OOD (Out-of-Distribution): OOD data refers to data that comes from a different distribution than the one the model was trained on. Specifically, OOD data violates the assumption of identically distributed data. Running a model on OOD data can lead to unpredictable or incorrect outputs because the model wasn't exposed to similar data during training. Therefore, detecting and appropriately handling OOD data is increasingly recognized as an important aspect of building robust and reliable machine learning systems.

1. Robustness: refers to a desirable property of machine learning models, denoting insensitivity or resilience to disturbances, fluctuations, or perturbations affecting inputs, training data, environment, or operational conditions. Broadly speaking, robustness manifests in four principal flavors

2. What is FFN?
Description:
FFN stands for Feed-Forward Network, which is a basic type of artificial neural network architecture. It belongs to the family of Multi-Layer Perceptron (MLP) models, consisting of neurons arranged in an acyclic graph manner, meaning there are no loops connecting the nodes.
Feed-Forward Networks operate by passing information linearly from input nodes through hidden layers to the output nodes. At each stage, a set of weights and biases determines how much importance is assigned to the input features. Then, an activation function (such as sigmoid, tanh, or ReLU) is applied element-wise to introduce nonlinearity, allowing the model to learn complex representations and make nonlinear predictions.
Here are some key features of FFNs:
1. Layered architecture: Input layer -> Hidden layers -> Output layer.
2. Information flows strictly in one direction (feed-forward); hence, no feedback loops.
3. Weights and biases are learned through backpropagation and optimization techniques.
4. Activation functions are applied element-wise to add nonlinearity.
5. Capacity to learn complex relationships grows with the addition of hidden layers.
FFNs can be used for various tasks, such as function approximation, regression, and classification. More complicated architectures, like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), build upon the FFN principle, adding specific structures and constraints to handle spatial or temporal dependencies.

3. What is activation function？
Description:
An activation function is a mathematical transformation applied element-wise to the weighted sum of inputs in a neural network. Introducing nonlinearity in this fashion enables artificial neural networks to learn complex and nonlinear relationships between inputs and outputs. The choice of activation function influences the learning capability, convergence speed, and overall performance of a neural network. Some common activation functions include:
1. Sigmoid: It produces an S-shaped curve, compressing values between 0 and 1. Mathematically defined as:
2. Tanh (Hyperbolic Tangent): Another S-shaped function, but symmetric around the origin, mapping values between -1 and 1. Computed as:
3. Rectified Linear Unit (ReLU): A piecewise linear function that returns z if z > 0 and 0 otherwise. It's widely preferred due to simplicity and fast computation. Mathematically: Parametric ReLU (PReLU): A variant of ReLU, introducing a parametric slope (α) for negative input regions, controlling the level of nonlinearity. Calculated as:
4. Softplus: Smooth version of ReLU, eliminating abrupt transitions, computed as:
5. Swish: A newer activation function introduced by Google Brain team, showing promising results. It computes:
Choosing the right activation function for a given problem depends on various factors, including the nature of the data, model architecture, and specific task requirements. Designing custom activation functions can be beneficial in certain scenarios, keeping in mind the fundamental principles of nonlinearity, continuity, monotonocity, and boundedness.
 

4. What is CLIP?
Description:
CLIP stands for Contrastive Language-Image Pretraining, a framework introduced by OpenAI in January 2021. The CLIP model combines a large-scale vision transformer and a text-based transformer language model to bridge the gap between computer vision and natural language processing. By doing so, CLIP can associate textual descriptions with corresponding images, making it possible to train the model on vast internet-scale data without manually annotated pairs of images and their corresponding text.
CLIP trains an image encoder and a text encoder in a contrastive learning setup, comparing millions of image-caption pairs. The idea is to maximize the cosine similarity between matched pairs while pushing apart mismatched ones. Once trained, the model performs zero-shot classification, suggesting its ability to recognize novel classes without finetuning.
Key Features of CLIP:
1. Scalability: Trained on a dataset containing 400 million (image, text) pairs scraped from the Internet.
2. Versatility: Supports a variety of vision tasks, such as image retrieval, classification, and generation.
3. Text-based Control: Users can instruct the model using natural language commands, opening doors to intuitive user interfaces.
4. Strong Performance: Demonstrates strong performance on ImageNet zero-shot classification, rivaling fully supervised state-of-the-art models.
While CLIP shows great promise, it faces limitations too. Being pre-trained on internet data, the model may pick up harmful biases and associations present online. Additionally, the computational demand involved in training such models remains high, making CLIP less accessible for individuals or organizations with limited resources.
Overall, CLIP symbolizes a step towards bringing together text and images in a seamless manner, revolutionizing the way computers reason about multimodal data. Future developments in this area will bring us closer to more robust AI systems that can effortlessly navigate through complex data environments.

5. What is Ablation Study?(消融实验)
Description:
An ablation study is a systematic method for evaluating the necessity and performance of individual components or features in a machine learning model or system. Researchers remove or modify a specific module, parameter, or feature and observe the effect on the overall performance to determine its importance and utility.
By conducting ablation experiments, researchers gain insights into:
1. Contribution of individual components: Determine the impact of removing or tweaking a specific component on the overall performance metric, establishing the significance of the examined element.
2. Interactions between components: Study the combined influence of multiple components and their effect on the whole system's performance.
3. Model robustness: Verify the model's sensitivity to changes in specific components, gauging its stability and resilience.
Common Uses of Ablation Studies:
1. Model Architecture: Investigate the significance of individual layers, units, or architectural choices in neural networks.
2. Hyperparameters: Observe the impact of varying specific hyperparameters, such as learning rates, batch sizes, and regularization coefficients.
3. Loss Functions: Compare the performance of different loss functions, measuring their contribution to the model's overall success.
4. Regularization techniques: Test the influence of various regularizers, such as L1, L2, Dropout, or batch normalization.
5. Embedding Spaces: Examine the impact of different embedding spaces, comparing their qualitative and quantitative contributions to the model.
Procedure for Ablation Studies:
1. Choose the Component: Define the component or feature to be analyzed and removed or modified.
2. Train Baseline Model: Train the baseline model with all components intact. Record its performance on the validation and test datasets.
3. Remove or Modify Component: Make the planned changes to the selected component.
4. Retrain Model: Train the adjusted model with changed components. Again, record its performance on the validation and test datasets.
5. Compare Results: Document and analyze the variation in performance between the baseline and modified models. Draw conclusions regarding the importance and utility of the studied component.
In summary, ablation studies provide valuable insights into the importance and utility of individual components and features in machine learning models or systems. Systematically removing or altering components reveals their contribution to the overall performance and highlights the robustness of the model. Ultimately, ablation studies help researchers design and engineer more efficient and effective AI systems.

6. PPO (Proximal Policy Optimization)(算法)
Description:
Proximal Policy Optimization (PPO) is a gradient-based policy optimization algorithm for reinforcement learning problems. Some key things to know about PPO:
• PPO is an on-policy algorithm, meaning it uses data collected by the latest policy to update the policy parameters. This provides more stable performance compared to off-policy methods like DDQN.
• The goal of PPO is to maximize the probability of actions taken by the policy for each state, while keeping the new policy close to the old one. This helps stabilize training and prevent jumping to poor local optima.
• It achieves this using a clipped objective function that explicitly limits how much one update can change the policy parameters. This clipping term prevents unconstrained policy updates.
• PPO uses several tricks like generalized advantage estimation (GAE) and sampling minibatches to stabilize training and reduce sample complexity compared to vanilla policy gradient methods.
• It has been shown to achieve state-of-the-art performance on many continuous control and 3D locomotion tasks from OpenAI Gym and Roboschool.
• PPO is relatively easy to implement and is considered one of the most widely used and effective policy gradient algorithms in practice.
• Hyperparameters like clipping range, learning rate, number of epochs, minibatch size require tuning for best results on different tasks.
So in summary, PPO achieves stable policy improvement through clipped objective training and has become a standard algorithm for continuous control problems.

7. RLHF
Description:
RLHF stands for Reinforcement Learning from Human Feedback. It is a paradigm of reinforcement learning in which a human provides feedback to help the agent learn an optimal policy. This is in contrast to traditional reinforcement learning, where the agent must learn from the environment without any human intervention.
The goal of RLHF is to enable an agent to learn from human feedback quickly, so that it can perform well on a variety of tasks. This can be achieved in a number of ways, such as:
• The human can provide positive or negative feedback on the agent's actions.
• The human can provide suggestions for specific actions that the agent should take.
• The human can provide advice on how the agent should trade-off different objectives.
RLHF has been used to solve a variety of tasks, including:
• Robot control
• Natural language processing
• Games
RLHF is an active area of research, and new methods are constantly being developed to improve the agent's ability to learn from human feedback.
Here are some additional details about RLHF:
• RLHF is often more efficient than traditional reinforcement learning, because the human can provide insights into the agent's behavior that would not be available to the agent learning from the environment alone.
• RLHF can be used to solve a wider range of tasks, including those that are difficult to solve using traditional reinforcement learning.
• RLHF is an active area of research, and new methods are constantly being developed to improve the agent's ability to learn from human feedback.
I hope this helps you to better understand RLHF.

8. MAE-MAE(Masked Autoencoders) - 知乎 (zhihu.com)

9. a

10. d

11. a

12. d

🤗 总结归纳

📎 参考文章

• 一些引用

• 引用文章