AI, Vision lover
This article aims to introduce the pioneering work (InstDisc) that leads to contrastive learning.
Paper: Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018. Updated version accessed at: https://arxiv.org/abs/1805.01978v1.
关于数据挖掘的一些知识，内容包括分类，聚类，异常值检测，关联性分析等等。
大量截图和思路来自 the University of Queensland 的Dr Hongzhi Yin所教课程INFS7203 Data Mining。
What are common drawbacks for the present contrastive learning methods?
When Does Contrastive Visual Representation Learning Work?
[2105.05837v1] When Does Contrastive Visual Representation Learning Work? (arxiv.org)
Pie charts can show numbers, but they always show percentages.
Line graphs show numbers changing over a period of time
A line graph always has 2,3,4,5 lines on a graph. And your job is to compare the lines, not decribe them sepratately.
Why Contrastive Learning can work? It is also an interesting field and has many excellent papers.
At present, mutual information is a key point in my cognition. Of course, this chapter will continue to be updated as my research progresses.
In addition to the lack of high-level semantic representations, the sampling of positive and negative samples for instance discrimination tasks also has obvious problems. Positive samples are all from its own data augmentation, while negative samples are directly sampled from a large memory (within the same batch, memory bank or a momentum encoder). The problem with this is that negative samples may have positive sample, which they are the same class, but are different instances. Another problem is that hard negative samples and easy negative samples have the same weights in InfoNCE loss , which make model too simple if it focus on too many easy negative samples. In this chapter, we will see many interesting works with different methods on solving sampling problem in CL.
So far, we have discussed Contrastive Learning(CL) frameworks such as InstDisc, SimCLR, MoCo, SimSiam, BYOL, which still the mainstream framework in the lastest CL works. Instead of frameworks, CL still has many also problems or weakness. This chapter will discribe one of it weakness: lacking semantic presententation due to instance discrimination pretext task, i.e. a model good at discriminating instances will lose higher level information like their clusters. Therefore, let’s see how researchers integrate clustering methods into Contrastive Learning. Moreover, this chapter and the next will be like a short survey of the lastest CL papers, which means I will not write every paper clearly and discuss every techniques. We will focus on papers’ motivation, main method and try to give my own insights.