In this paper, we pro pose a logistic kernel machine regression model for binary outcomes, where covariate effects are modeled parametri cally and the genetic pathway effect is modeled paramet rically or nonparametrically using the kernel machine method. A main contribution of this paper is to establish a connection between logistic kernel machine regression and the logistic mixed model. We show that the kernel machine estimator of the genetic pathway effect can be obtained from the estimator of the random effects in the corresponding logistic mixed model. This connection pro vides a convenient vehicle to connect the powerful kernel machine method with the popular mixed model method in the statistical literature.

This mixed model connection also provides an unified framework for statistical infer ence for model parameters, including the regression coef ficients, the nonparametric genetic pathway function, and the regularization and kernel parameters. Based on the proposed logistic kernel machine regression model, we develop a new test for the nonlinear pathway effect on dis ease risk. An appealing feature of the proposed test is that it performs well without the need to correctly specify the functional form of the effects of each gene or their interac tions. This feature has a significant practical implication when analyzing genetic pathway data, as the true relation ship between the pathway and the disease outcome is often unknown. We extend the results to generalized ker nel machine regression for a class of continuous and dis crete outcomes and discuss its connection with generalized linear mixed models.

Recently, Wei and Li proposed a nonparametric path way based regression to model pathway data. NPR is a pathway based gradient boosting procedure, where the base learner is usually a regression or classification tree. It provides a flexible approach in modeling pathways and interactions among genes within a pathway. Michalowski et al. proposed a Bayesian Belief Net work approach for pathway data. Neither method is like lihood based. Thus parameter estimation and inference cannot be casted within a unified likelihood framework. It is hence Carfilzomib difficult to estimate and quantify the overall pathway effect on disease risk and assess its statistical uncertainty. Secondly, a primary interest in this paper is to test for the statistical significance of the overall pathway effect on the risk of a disease.

Both NPR and Bayesian belief network do not provide such a statistical test for the pathway effect. For example, NPR uses an importance score to rank the relative importance of each pathway. It lacks formal inferential procedure for assessing the statis tical significance of a pathway. Further, when considering a single pathway, the importance score loses its meaning in assessing the importance of a pathway.