Show HN: SQLFrame – I ran PySpark without Spark on a SQL database Recently I open-sourced SQLFrame, a DataFrame library that implements the PySpark DataFrame API but removes Spark as a dependency. It does this by generating the corresponding SQL for the DataFrame operations using SQLGlot. Since the output is SQL this also means that the PySpark DataFrame API can now be used directly against other databases without the Spark middleman. I built this because of two common problems I have faced in my career: 1. I prefer to write complex pipelines in PySpark but they can be hard to read for SQL-proficient co-workers. Therefore I find myself in a tradeoff between maintainability and accessibility. 2. I really enjoy using the PySpark DataFrame API but not every project requires Spark and therefore I'm not able to use the DataFrame library I am most proficient in. The library currently focuses on transformation pipelines (reading from and writing to tables) and data analysis as key use cases. It does offer some ability to read from files directly but they must be small although this can be improved over time if there is demand for it. SQLFrame currently supports BigQuery, DuckDB, and Postgres with Clickhouse, Redshift, Snowflake, Spark, and Trino in development or planned. You can use the "Standalone" session to test running against any engine supported by SQLGlot but there could be issues with more advanced functions that will be resolved once officially supported by SQLFrame. Blog post for more details: https://ift.tt/U0pTQN4... Would love to answer any questions or hear any feedback you may have! https://ift.tt/gEPLHpo May 21, 2024 at 06:39AM
Show HN: SQLFrame – I ran PySpark without Spark on a SQL database https://ift.tt/4liObUq
Related Articles
Show HN: I built a SaaS thanks to my wife https://ift.tt/LbKSNqoShow HN: I built a SaaS thanks to my wife I’m Michał, and I’d like to … Read More
Show HN: Iterm-Mcp – AI Terminal/REPL Control for iTerm2 https://ift.tt/iyRzJrjShow HN: Iterm-Mcp – AI Terminal/REPL Control for iTerm2 Hi HN! Ever w… Read More
Show HN: Cdlog: nicer directory navigation for Bash https://ift.tt/Tzcxa41Show HN: Cdlog: nicer directory navigation for Bash https://ift.tt/2Gv… Read More
Show HN: I made a price aggregator to find the best PC parts deals across eBay https://ift.tt/iJaVKs6Show HN: I made a price aggregator to find the best PC parts deals acr… Read More
Show HN: Workflow86 - An AI business analyst and automation engineer https://ift.tt/UysVDmoShow HN: Workflow86 - An AI business analyst and automation engineer H… Read More
Show HN: Audiocube – A 3D DAW for Spatial Audio https://ift.tt/Mc2UzyJShow HN: Audiocube – A 3D DAW for Spatial Audio I’ve recently released… Read More
Show HN: Mcp-Agent – Build effective agents with Model Context Protocol https://ift.tt/kxRhUASShow HN: Mcp-Agent – Build effective agents with Model Context Protoco… Read More
Show HN: Share your path to resolve issues with Savvy's Chrome Extension https://ift.tt/KswYMrHShow HN: Share your path to resolve issues with Savvy's Chrome Extensi… Read More
0 Comments: