To "let go of everything" refers to an understanding of experience beyond concepts.
Usually we label everything either consciously or unconsciously and experience very little that is fresh to our mind without previous conditioning. 'To let go of everything' refers to a level of experiencing which does not rely on previously formed patterns of response. In this state there is less judgement and more potential for creative response. It is an act of surrender.
If we can train our mind to be in this state of openness we will be able to relate more to what is happening as it happens. This aspect of t'ai chi is very important for the martial application. If an adversary is going to attack you it is vital that you do not prejudge or anticipate anything about the nature, direction or intention of the attack. The perception of the adversary must be pure and completely objective. Then the response can be well timed and completely appropriate.
If you are not so interested in the martial application of t'ai chi there is still value to this approach applied to daily life. When you have learned to open your mind you will appreciate the uniqueness of your experiences and will be more connected to your responses. In some way it is a little like returning the state of being a child when there are few previously formed attitudes and responses and everything is intensely experienced as new. Yet of course this perception goes with all of the adult qualities of our matured intelligence.
To understand this process of labelling which conditions our experience we must spend some time contemplating how it occurs.
When we recognise something we quickly categorise it in relation to our previously formed concepts. So when we see something we recognise as a table we see the basic elements of say a flat surface and legs and then our mind has an experience of recognition. At the moment of recognition we have an image in our mind. We don't encounter just the colours, shapes, textures and sensory experiences any longer, we have an image of 'table' which coexists with our experience of the table. The familiarity with the image of the table will prevent us from appreciating the uniqueness of the experience in front of us.