With the Syrian crisis entering its 8th year, refugees have become the focus of research across multiple disciplines, including design and HCI research. While some researchers have reflected upon designing with refugees, these accounts have been limited to conducting design workshops in formal spaces. Through reflecting on our experiences of conducting design research in informal refugee settlements in Lebanon we unpack lessons learnt, design practices and research approaches that facilitate design engagements with refugees. We highlight the value in participants configuring the design space, using a dialogical approach as well as creating a safe space for both participants and the researcher. We also reflect on the roles that researchers may take on when conducting similar research. By doing so we contribute specific design practices that may be transferrable to other similar contexts.