Analyzing big data sets by using different panelized regression methods with application: surveys of multidimensional poverty in Iraq

Abstract

Poverty phenomenon is very substantial topic that determines the future of societies and governments and the way that they deal with education, health and economy. Sometimes poverty takes multidimensional trends through education and health. The research aims at studying multidimensional poverty in Iraq by using panelized regression methods, to analyze Big Data sets from demographical surveys collected by the Central Statistical Organization in Iraq. We choose classical penalized regression method represented by the Ridge Regression. Moreover, we choose another penalized method, which is the Smooth Integration of Counting and Absolute Deviation (SICA) to analyze Big Data sets related to the different poverty forms in Iraq. Euclidian Distance (ED) was used to compare the two methods and the research conclude that the SICA method is better than Ridge estimator with Big Data conditions.