GridSearchCV - XGBoost - Early Stopping

Question

I am trying to do a hyperparameter search using scikit-learn's GridSearchCV on XGBoost. During grid search I'd like it to early stop since it reduces search time drastically and (expecting to) have better results on my prediction/regression task. I am using XGBoost via its Scikit-Learn API. model = xgb.XGBRegressor() GridSearchCV(model, paramGrid, verbose=verbose ,fit_params={'early_stopping_rounds':42}, cv=TimeSeriesSplit(n_splits=cv).get_n_splits([trainX, trainY]), n_jobs=n_jobs, iid=iid).fit(trainX,trainY) I tried to give early stopping parameters with using fit_params, but then it throws this error which is basically because of the lack of validation set which is required for early stopping: /opt/anaconda/anaconda3/lib/python3.5/site-packages/xgboost/callback.py in callback(env=XGBoostCallbackEnv(model= 192 score = env.evaluation_result_list[-1][1] score = undefined env.evaluation_result_list = [] 193 if len(state) == 0: 194 init(env) 195 best_score = state['best_score'] 196 best_iteration = state['best_iteration'] How can I apply GridSearch on XGBoost with using early_stopping_rounds? note: model is working without gridsearch, also GridSearch works without 'fit_params={'early_stopping_rounds':42} Select the correct answer from above options

Answer 1

We use early stopping to stop the model training and evaluation when a pre-specified threshold achieved. To perform early stopping, you have to use an evaluation metric as a parameter in the fit function. XGboost: XGBoost is an open-source software library that provides a gradient boosting to optimize loss during training. You can learn more about XGBOOST here. import xgboost as xgb from sklearn.model_selection import GridSearchCV from sklearn.model_selection import TimeSeriesSplit cv = 2 trainX= [[1], [2], [3], [4], [5]] trainY = [1, 2, 3, 4, 5] # these are the evaluation sets testX = trainX testY = trainY paramGrid = {"subsample" : [0.5, 0.8]} fit_params={"early_stopping_rounds":42, "eval_metric" : "mae", "eval_set" : [[testX, testY]]} model = xgb.XGBRegressor() gridsearch = GridSearchCV(model, paramGrid, verbose=1 , fit_params=fit_params, cv=TimeSeriesSplit(n_splits=cv).get_n_splits([trainX,trainY])) gridsearch.fit(trainX,trainY) For more details, studying Gradient Boosting will provide some amazing insights. Hope this answer helps.