Python Discussion Forums

There's no such thing as a stupid question. Ask. Discuss. Learn.

Please anyone create UDF for this programm (Pyspark) Python

Could you please create User defined function for this program

#______________________________________________________________________
from pyspark.sql import SparkSession
from pyspark.ml.classification import LogisticRegression

spark = SparkSession.builder.appName('titanic_logreg').getOrCreate()
df = spark.read.csv('titanic.csv', inferSchema = True, header = True)
df.show(3)

df.columns

my_col = df.select([ 'Survived',
'Pclass',
'Sex',
'Age',
'SibSp',
'Parch',
'Fare',
'Embarked'])

final_data = my_col.na.drop()
final_data.show(3)

from pyspark.ml.feature import (VectorAssembler, StringIndexer, VectorIndexer, OneHotEncoder)

gender_indexer = StringIndexer(inputCol = 'Sex', outputCol = 'SexIndex')
gender_encoder = OneHotEncoder(inputCol = 'SexIndex', outputCol = 'SexVec')

embark_indexer = StringIndexer(inputCol = 'Embarked', outputCol = 'EmbarkIndex')
embark_encoder = OneHotEncoder(inputCol = 'EmbarkIndex', outputCol = 'EmbarkVec')

assembler = VectorAssembler(inputCols = ['Pclass', 'SexVec', 'Age', 'SibSp', 'Fare', 'EmbarkVec'], outputCol = 'features')

from pyspark.ml import Pipeline
log_reg = LogisticRegression(featuresCol='features', labelCol = 'Survived')

pipeline= Pipeline(stages= [gender_indexer, embark_indexer,
gender_encoder, embark_encoder,
assembler, log_reg])

train, test = final_data.randomSplit([0.7, 0.3])

fit_model = pipeline.fit(train)
results = fit_model.transform(test)

results.select('prediction', 'Survived').show(3)

from pyspark.ml.evaluation import BinaryClassificationEvaluator

eval = BinaryClassificationEvaluator(rawPredictionCol = 'rawPrediction', labelCol = 'Survived')
AUC = eval.evaluate(results)
AUC

#____________________________________

Suresh Kumar 5 months ago
  • Sawantbhai

    Got a RBP4 for my bday want some project ideas that I can make from scratch using python
    Title says it all! I've been coding in python on and off for about a year now. I wouldn't say that I'm amazing at coding but I do want to advance! I feel like using this device and having some type of project will get me motivated to better my skills. So please if anyone has any neat ideas on what I can build from scratch that would be great! :) thanks again!


  • Jerrytubs Jerrytubs

    Title says it all! I've been coding in python on and off for about a year now. I wouldn't say that I'm amazing at coding but I do want to advance! I feel like using this device and having some type of project will get me motivated to better my skills. So please if anyone has any neat ideas on what I can build from scratch that would be great! :) thanks again! https://openweb.onl/whatsapp-web/ https://modplus.pro/whatsapp-plus/ https://routerlogin.dev/

    i agree!!


  • Terry Joe

    Yea even im not able to solve the issue.


  • Whitelisted Links