difference between let and allow

There is a difference in the infinitive form following let and allow and permit:

You allow them to leave early.

You permit them to leave early.

But

You let them leave early.