Patrick is enjoying his vacay with family