Skip to content

Instantly share code, notes, and snippets.

@angusdev

angusdev/spark_vertical.py

Created September 4, 2024 08:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save angusdev/e87fd3b7cc2bbabae7e6fd457831b9d5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save angusdev/e87fd3b7cc2bbabae7e6fd457831b9d5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_vertical.py
from pyspark.sql import SparkSession
from pyspark.sql import Row
# Initialize Spark Session
spark = SparkSession.builder.appName("Vertical Display").getOrCreate()
# Example DataFrame
data = [("Alice", 25), ("Bob", 30)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)
# Display DataFrame in vertical format
def vertical_format(row):
return [(col, str(row[col])) for col in row.asDict()]
# Collect each row, reshape into vertical format, and print
for row in df.collect():
vertical_rows = vertical_format(row)
for col, value in vertical_rows:
print(f"{col}: {value}")
print("-" * 20) # To separate rows for better readability
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment