Past and future are based on our perception. Directionality is based on thermodynamic interactions. It is always happening that way, whether we are observing it or not.