On regularisation methods for analysis of high dimensional data
Sirimongkolkasem, Tanin; Drikvandi, Reza
High dimensional data are rapidly growing in many domains due to the development of technological advances which helps collect data with a large number of variables to better understand a given phenomenon of interest. Particular examples appear in genomics, fMRI data analysis, large-scale healthcare analytics, text/image analysis and astronomy. In the last two decades regularisation approaches have become the methods of choice for analysing such high dimensional data. This paper aims to study the performance of regularisation methods, including the recently proposed method called de-biased lasso, for the analysis of high dimensional data under different sparse and non-sparse situations. Our investigation concerns prediction, parameter estimation and variable selection. We particularly study the effects of correlated variables, covariate location and effect size which have not been well investigated. We find that correlated data when associated with important variables improve those common regularisation methods in all aspects, and that the level of sparsity can be reflected not only from the number of important variables but also from their overall effect size and locations. The latter may be seen under a non-sparse data structure. We demonstrate that the de-biased lasso performs well especially in low dimensional data, however it still suffers from issues, such as multicollinearity and multiple hypothesis testing, similar to the classical regression methods.
Sirimongkolkasem, T., & Drikvandi, R. (2019). On regularisation methods for analysis of high dimensional data. Annals of Data Science, 6(4), 737-763. https://doi.org/10.1007/s40745-019-00209-4
|Journal Article Type||Article|
|Acceptance Date||Apr 7, 2019|
|Online Publication Date||Apr 13, 2019|
|Deposit Date||Oct 6, 2020|
|Publicly Available Date||Nov 2, 2020|
|Journal||Annals of Data Science|
|Peer Reviewed||Peer Reviewed|
Published Journal Article
Publisher Licence URL
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
You might also like
MEGH: A parametric class of general hazard models for clustered survival data
Diagnostic tools for random effects in general mixed models
Invited session "Recent advances in biostatistics"
Sparse principal component analysis for natural language processing
Joint modelling of longitudinal data involving time-varying covariates