机器学习实战系列[一]:工业蒸汽量预测(最新版本上篇)含数据探索特征工程等

机器学习实战系列[一]：工业蒸汽量预测

背景介绍

相关描述

经脱敏后的锅炉传感器采集的数据（采集频率是分钟级别），根据锅炉的工况，预测产生的蒸汽量。

数据说明

结果评估

预测结果以mean square error作为评判标准。

1.数据探索性分析

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from scipy import stats

import warnings
warnings.filterwarnings("ignore"
 
%matplotlib inline

# 下载需要用到的数据集
!wget http://tianchi-media.oss-cn-beijing.aliyuncs.com/DSW/Industrial_Steam_Forecast/zhengqi_test.txt
!wget http://tianchi-media.oss-cn-beijing.aliyuncs.com/DSW/Industrial_Steam_Forecast/zhengqi_train.txt

--2023-03-23 18:10:23--  http://tianchi-media.oss-cn-beijing.aliyuncs.com/DSW/Industrial_Steam_Forecast/zhengqi_test.txt
正在解析主机 tianchi-media.oss-cn-beijing.aliyuncs.com (tianchi-media.oss-cn-beijing.aliyuncs.com... 49.7.22.39
正在连接 tianchi-media.oss-cn-beijing.aliyuncs.com (tianchi-media.oss-cn-beijing.aliyuncs.com|49.7.22.39|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 466959 (456K [text/plain]
正在保存至: “zhengqi_test.txt.1”

zhengqi_test.txt.1  100%[===================>] 456.01K  --.-KB/s    in 0.04s   

2023-03-23 18:10:23 (10.0 MB/s - 已保存 “zhengqi_test.txt.1” [466959/466959]

--2023-03-23 18:10:23--  http://tianchi-media.oss-cn-beijing.aliyuncs.com/DSW/Industrial_Steam_Forecast/zhengqi_train.txt
正在解析主机 tianchi-media.oss-cn-beijing.aliyuncs.com (tianchi-media.oss-cn-beijing.aliyuncs.com... 49.7.22.39
正在连接 tianchi-media.oss-cn-beijing.aliyuncs.com (tianchi-media.oss-cn-beijing.aliyuncs.com|49.7.22.39|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 714370 (698K [text/plain]
正在保存至: “zhengqi_train.txt.1”

zhengqi_train.txt.1 100%[===================>] 697.63K  --.-KB/s    in 0.04s   

2023-03-23 18:10:24 (17.9 MB/s - 已保存 “zhengqi_train.txt.1” [714370/714370]

# **读取数据文件**
# 使用Pandas库`read_csv(`函数进行数据读取，分割符为‘\t’
train_data_file = "./zhengqi_train.txt"
test_data_file =  "./zhengqi_test.txt"

train_data = pd.read_csv(train_data_file, sep='\t', encoding='utf-8'
test_data = pd.read_csv(test_data_file, sep='\t', encoding='utf-8'

1.1 查看数据信息

#查看特征变量信息
train_data.info(

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2888 entries, 0 to 2887
Data columns (total 39 columns:
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V0      2888 non-null   float64
 1   V1      2888 non-null   float64
 2   V2      2888 non-null   float64
 3   V3      2888 non-null   float64
 4   V4      2888 non-null   float64
 5   V5      2888 non-null   float64
 6   V6      2888 non-null   float64
 7   V7      2888 non-null   float64
 8   V8      2888 non-null   float64
 9   V9      2888 non-null   float64
 10  V10     2888 non-null   float64
 11  V11     2888 non-null   float64
 12  V12     2888 non-null   float64
 13  V13     2888 non-null   float64
 14  V14     2888 non-null   float64
 15  V15     2888 non-null   float64
 16  V16     2888 non-null   float64
 17  V17     2888 non-null   float64
 18  V18     2888 non-null   float64
 19  V19     2888 non-null   float64
 20  V20     2888 non-null   float64
 21  V21     2888 non-null   float64
 22  V22     2888 non-null   float64
 23  V23     2888 non-null   float64
 24  V24     2888 non-null   float64
 25  V25     2888 non-null   float64
 26  V26     2888 non-null   float64
 27  V27     2888 non-null   float64
 28  V28     2888 non-null   float64
 29  V29     2888 non-null   float64
 30  V30     2888 non-null   float64
 31  V31     2888 non-null   float64
 32  V32     2888 non-null   float64
 33  V33     2888 non-null   float64
 34  V34     2888 non-null   float64
 35  V35     2888 non-null   float64
 36  V36     2888 non-null   float64
 37  V37     2888 non-null   float64
 38  target  2888 non-null   float64
dtypes: float64(39
memory usage: 880.1 KB

此训练集数据共有2888个样本，数据中有V0-V37共计38个特征变量，变量类型都为数值类型，所有数据特征没有缺失值数据；
数据字段由于采用了脱敏处理，删除了特征数据的具体含义；target字段为标签变量

test_data.info(

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1925 entries, 0 to 1924
Data columns (total 38 columns:
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   V0      1925 non-null   float64
 1   V1      1925 non-null   float64
 2   V2      1925 non-null   float64
 3   V3      1925 non-null   float64
 4   V4      1925 non-null   float64
 5   V5      1925 non-null   float64
 6   V6      1925 non-null   float64
 7   V7      1925 non-null   float64
 8   V8      1925 non-null   float64
 9   V9      1925 non-null   float64
 10  V10     1925 non-null   float64
 11  V11     1925 non-null   float64
 12  V12     1925 non-null   float64
 13  V13     1925 non-null   float64
 14  V14     1925 non-null   float64
 15  V15     1925 non-null   float64
 16  V16     1925 non-null   float64
 17  V17     1925 non-null   float64
 18  V18     1925 non-null   float64
 19  V19     1925 non-null   float64
 20  V20     1925 non-null   float64
 21  V21     1925 non-null   float64
 22  V22     1925 non-null   float64
 23  V23     1925 non-null   float64
 24  V24     1925 non-null   float64
 25  V25     1925 non-null   float64
 26  V26     1925 non-null   float64
 27  V27     1925 non-null   float64
 28  V28     1925 non-null   float64
 29  V29     1925 non-null   float64
 30  V30     1925 non-null   float64
 31  V31     1925 non-null   float64
 32  V32     1925 non-null   float64
 33  V33     1925 non-null   float64
 34  V34     1925 non-null   float64
 35  V35     1925 non-null   float64
 36  V36     1925 non-null   float64
 37  V37     1925 non-null   float64
dtypes: float64(38
memory usage: 571.6 KB

测试集数据共有1925个样本，数据中有V0-V37共计38个特征变量，变量类型都为数值类型

# 查看数据统计信息
train_data.describe(

test_data.describe(

# 查看数据字段信息
train_data.head(

test_data.head(

1.2 可视化探索数据

fig = plt.figure(figsize=(4, 6  # 指定绘图对象宽度和高度
sns.boxplot(train_data['V0'],orient="v", width=0.5

<matplotlib.axes._subplots.AxesSubplot at 0x7faf89f46950>

# 画箱式图
# column = train_data.columns.tolist([:39]  # 列表头
# fig = plt.figure(figsize=(20, 40  # 指定绘图对象宽度和高度
# for i in range(38:
#     plt.subplot(13, 3, i + 1  # 13行3列子图
#     sns.boxplot(train_data[column[i]], orient="v", width=0.5  # 箱式图
#     plt.ylabel(column[i], fontsize=8
# plt.show(
#箱图自行打开

查看数据分布图

查看特征变量‘V0’的数据分布直方图，并绘制Q-Q图查看数据是否近似于正态分布

plt.figure(figsize=(10,5

ax=plt.subplot(1,2,1
sns.distplot(train_data['V0'],fit=stats.norm
ax=plt.subplot(1,2,2
res = stats.probplot(train_data['V0'], plot=plt

# train_cols = 6
# train_rows = len(train_data.columns
# plt.figure(figsize=(4*train_cols,4*train_rows

# i=0
# for col in train_data.columns:
#     i+=1
#     ax=plt.subplot(train_rows,train_cols,i
#     sns.distplot(train_data[col],fit=stats.norm
    
#     i+=1
#     ax=plt.subplot(train_rows,train_cols,i
#     res = stats.probplot(train_data[col], plot=plt
# plt.show(
#QQ图自行打开

由上面的数据分布图信息可以看出，很多特征变量（如'V1','V9','V24','V28'等）的数据分布不是正态的，数据并不跟随对角线，后续可以使用数据变换对数据进行转换。

ax = sns.kdeplot(train_data['V0'], color="Red", shade=True
ax = sns.kdeplot(test_data['V0'], color="Blue", shade=True
ax.set_xlabel('V0'
ax.set_ylabel("Frequency"
ax = ax.legend(["train","test"]

# dist_cols = 6
# dist_rows = len(test_data.columns
# plt.figure(figsize=(4*dist_cols,4*dist_rows

# i=1
# for col in test_data.columns:
#     ax=plt.subplot(dist_rows,dist_cols,i
#     ax = sns.kdeplot(train_data[col], color="Red", shade=True
#     ax = sns.kdeplot(test_data[col], color="Blue", shade=True
#     ax.set_xlabel(col
#     ax.set_ylabel("Frequency"
#     ax = ax.legend(["train","test"]
    
#     i+=1
# plt.show(
#自行打开

查看特征'V5', 'V17', 'V28', 'V22', 'V11', 'V9'数据的数据分布

drop_col = 6
drop_row = 1

plt.figure(figsize=(5*drop_col,5*drop_row

i=1
for col in ["V5","V9","V11","V17","V22","V28"]:
    ax =plt.subplot(drop_row,drop_col,i
    ax = sns.kdeplot(train_data[col], color="Red", shade=True
    ax = sns.kdeplot(test_data[col], color="Blue", shade=True
    ax.set_xlabel(col
    ax.set_ylabel("Frequency"
    ax = ax.legend(["train","test"]
    
    i+=1
plt.show(

drop_columns = ['V5','V9','V11','V17','V22','V28']
# 合并训练集和测试集数据，并可视化训练集和测试集数据特征分布图

可视化线性回归关系

查看特征变量‘V0’与'target'变量的线性回归关系

fcols = 2
frows = 1

plt.figure(figsize=(8,4

ax=plt.subplot(1,2,1
sns.regplot(x='V0', y='target', data=train_data, ax=ax, 
            scatter_kws={'marker':'.','s':3,'alpha':0.3},
            line_kws={'color':'k'};
plt.xlabel('V0'
plt.ylabel('target'

ax=plt.subplot(1,2,2
sns.distplot(train_data['V0'].dropna(
plt.xlabel('V0'

plt.show(

1.2.2 查看变量间线性回归关系

# fcols = 6
# frows = len(test_data.columns
# plt.figure(figsize=(5*fcols,4*frows

# i=0
# for col in test_data.columns:
#     i+=1
#     ax=plt.subplot(frows,fcols,i
#     sns.regplot(x=col, y='target', data=train_data, ax=ax, 
#                 scatter_kws={'marker':'.','s':3,'alpha':0.3},
#                 line_kws={'color':'k'};
#     plt.xlabel(col
#     plt.ylabel('target'
    
#     i+=1
#     ax=plt.subplot(frows,fcols,i
#     sns.distplot(train_data[col].dropna(
    # plt.xlabel(col
    #已注释图片生成，自行打开

1.2.2 查看特征变量的相关性

data_train1 = train_data.drop(['V5','V9','V11','V17','V22','V28'],axis=1
train_corr = data_train1.corr(
train_corr

声明：本文由用户投稿上传，本站不保证内容的真实与正确性，并且不承担相关法律责任，如有侵权请提供版权资料并联系删除！

编程笔记 » 机器学习实战系列[一]:工业蒸汽量预测(最新版本上篇)含数据探索特征工程等