Biodiesel is the most commonly used liquid biofuel for several reasons; it is less toxic, biodegradable, and renewable, reduces greenhouse gases emissions. In the present study mixture of edible and non-edible oils (linseed, palm, karanja and thumba oil) was converted in to biodiesel in a reverse flow helical coil reactor (RFHR) through transesterification. Both homogeneous and heterogeneous catalyst i.e. KOH and   KF impregnated snail shell (KF/SS) were used as catalysts. The heterogeneous catalyst was characterized using FT-IR, XRD, SEM-EDS, XPS, and TGA. The best calcination conditions were observed at 850ºC for 4 hours based on biodiesel yield. Response surface Methodology based Box Behnken Design was used to optimize the reaction variables such as residence time, reaction temperature and catalyst concentration for biodiesel synthesis in RFHR using mixture of oils for both KOH and KF/SS catalysts.  A quadratic model was created for the prediction of the biodiesel yield. The R² value of the model for RFHR was 0.98 for homogeneous catalyst which indicates the satisfactory accuracy of the model. Residence time due to reverse flow mixing shows the positive effect on biodiesel yield. The optimum combinations of the findings include residence time of 5 min, the reaction temperature of 63°C, and catalyst concentration of 2.1 gm for homogeneous catalyst. At these reaction conditions, the predicted and observed biodiesel yield was 99.9% and 99.8%. In the case of heterogeous catalyst the optimal conditions were as follows: catalyst concentration of 3.7 wt%, time 22.8min, reactor temperature 61.72oC with biodiesel yield of 97%. Based on the optimum condition, the predicted biodiesel conversion was 98% while the actual experimental value was 97.03%. The comparison of result with homogeneous and heterogeous catalyst shows that heterogeous catalyst requires more residence time compare to homogeneous one.